大神帮忙看下20行的python代码,文件io和数据库插入太慢,pythonio,新学python,在做一
大神帮忙看下20行的python代码,文件io和数据库插入太慢,pythonio,新学python,在做一
新学python,在做一个把爬虫爬下来的股票数据从文件插入到mysql数据库,速度很慢,不知道瓶颈是文件处理还是sql插入问题,望大神不吝赐教,感激不尽!
# encoding: utf-8#!/usr/bin/pythonimport MySQLdbimport osdb = MySQLdb.connect(host="localhost",user="root",passwd="4242",\ db="STOCKS", charset="utf8", use_unicode=True)cursor = db.cursor()files = os.listdir("shanghaiHistory")for fileName in files: print "start insert " + fileName isTheFirstLine = True for line in open("shanghaiHistory/" + fileName): if isTheFirstLine: isTheFirstLine = False continue lineList = line.split(',') lineList.insert(0,fileName[:6]) sql = "insert into shanghaiHistory(Code,Date,Open,High,Low,Close,Volume,AdjClose)\ values ('%s','%s','%s','%s','%s','%s','%s','%s')" % tuple(lineList) try: cursor.execute(sql) db.commit() except: db.rollback()db.close()
文件格式是形如:
Date,Open,High,Low,Close,Volume,Adj Close2012-12-21,0.9151,0.9151,0.9151,0.9151,000,0.91512012-12-20,0.896,0.898,0.896,0.898,22924300,0.8982012-12-19,0.895,0.898,0.895,0.897,37858100,0.8972012-12-18,0.896,0.896,0.895,0.896,43542500,0.8962012-12-17,0.895,0.897,0.895,0.896,41877700,0.8962012-12-14,0.895,0.897,0.894,0.897,35734700,0.8972012-12-13,0.894,0.896,0.894,0.895,11613800,0.8952012-12-12,0.89,0.895,0.89,0.894,26586500,0.894
两个多小时才插入45W条数据,怎么才能提速一点呢.
编橙之家文章,
相关内容
- 用django做购物车,为什么购物车中相同的物品不能合并
- Python 协程的无限递归的问题,python协程递归,hi, 最近看
- Python 二维数组赋值,python二维数组赋值,grid = [[0,]
- 自动化重构方案与实践,自动化重构方案,如何进行代码
- python+webdriver想登录淘宝,输入用户名密码点登录后,提
- python把HTML存入Mysql,python存入mysql,用pyspider爬虫爬
- 知乎的qq登录怎么用python和webdriver定位到“账号密码登
- python爬虫模拟登陆,post后如何返回retUrl,pythonreturl,想
- python manage.py runserver 127.0.0.1:80 时出错,manage.pyrunserve
- 如何利用pycharm调试Django?能打印出变量的数据结构吗,
评论关闭