Python scrapy模拟登陆知乎重定向问题,pythonscrapy,当我使用scrapy想进
Python scrapy模拟登陆知乎重定向问题,pythonscrapy,当我使用scrapy想进
当我使用scrapy想进行模拟登陆, 然后抓取首页的问题和答案时, 一直显示重定向问题pythonfrom scrapy.contrib.spiders import CrawlSpider, Rulefrom scrapy.selector import Selectorfrom scrapy.contrib.linkextractors.sgml import SgmlLinkExtractorfrom scrapy.http import Request, FormRequestfrom zhihu.items import ZhihuItemclass ZhihuSipder(CrawlSpider) : name = "zhihu" allowed_domains = ["www.zhihu.com"] start_urls = [ "http://www.zhihu.com" ] rules = ( Rule(SgmlLinkExtractor(allow = r'http://www\.zhihu\.com/question/\d+'), callback = 'parse_page'), ) def start_requests(self): return [Request("https://www.zhihu.com/login", callback = self.post_login)] #FormRequeset出问题了 def post_login(self, response): print 'Preparing login' xsrf = Selector(response).xpath('//input[@name="_xsrf"]/@value').extract()[0] print xsrf ############## return [FormRequest.from_response(response, #"http://www.zhihu.com/login", formdata = { '_xsrf': xsrf, 'email': '@qq.com', 'password': '123456', 'rememberme': 'y', }, callback = self.parse_page )] def parse_page(self, response): problem = Selector(response) item = ZhihuItem() item['url'] = response.url item['title'] = problem.xpath('//h2[@class="zm-item-title zm-editable-content"]/text()').extract() item['description'] = problem.xpath('//div[@class="zm-editable-content"]/text()').extract() item['answer']= problem.xpath('//div[@class=" zm-editable-content clearfix"]/text()').extract() return item
使用命令运行爬虫, 可以正确打印xsrf, 但无法成功登陆
$ scrapy crawl zhihu错误结果如下2014-12-18 14:45:11+0800 [zhihu] INFO: Spider opened2014-12-18 14:45:11+0800 [zhihu] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)2014-12-18 14:45:11+0800 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:60232014-12-18 14:45:11+0800 [scrapy] DEBUG: Web service listening on 127.0.0.1:60802014-12-18 14:45:11+0800 [zhihu] DEBUG: Redirecting (301) to <GET https://www.zhihu.com/> from <GET https://www.zhihu.com/login>2014-12-18 14:45:11+0800 [zhihu] DEBUG: Redirecting (302) to <GET http://www.zhihu.com/> from <GET https://www.zhihu.com/>2014-12-18 14:45:12+0800 [zhihu] DEBUG: Crawled (200) <GET http://www.zhihu.com/> (referer: None)Preparing logind117e46de0dcc5e8ee2f0c7031fcafe92014-12-18 14:45:12+0800 [zhihu] DEBUG: Redirecting (302) to <GET http://www.zhihu.com/> from <POST http://www.zhihu.com/login>2014-12-18 14:45:12+0800 [zhihu] DEBUG: Filtered duplicate request: <GET http://www.zhihu.com/> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)2014-12-18 14:45:12+0800 [zhihu] INFO: Closing spider (finished)2014-12-18 14:45:12+0800 [zhihu] INFO: Dumping Scrapy stats:
希望得到解答, 为什么不能成功登陆, 非常疑惑, 非常感谢
实际上已经登录了, 在 parse_page 方法里面添加一句
print response.body
把返回的 body 复制到一个 html 文件中, 用浏览器打开,就知道了。
上面其实已经成功登录, 后来自己测试过了, 只不过没有调用一个抓取页面url的函数
编橙之家文章,
相关内容
- Python3.4 import base64报AttributeRrror错误如何解决,,如图i
- Python 下载对应文件服务器的response该怎么写,pythonres
- xml文件很大python方法提取节点中多行文本怎么办,xml
- 说Python多线程是伪命题正确吗?,python多线程伪命题,我
- Python用mysql.connector select报错如何解决,,python中使用m
- Python什么方法可以将多行unicode字符串前u去除,pythonu
- 正则匹配出每一条记录后面的python字符串,怎么做?,
- Ruby语言对于研究网络安全方面会有提升吗?,ruby网络安
- Django写博客model关于这块该怎么写,djangomodel,初学编程
- Python第三方模块中有限制频率的模块吗,python第三方
评论关闭