python利用beautifulSoup实现爬虫,python爬虫
python利用beautifulSoup实现爬虫,python爬虫
以前讲过利用phantomjs做爬虫抓网页 http://www.bkjia.com/article/55789.htm 是配合选择器做的
利用 beautifulSoup(文档 :http://www.crummy.com/software/BeautifulSoup/bs4/doc/)这个python模块,可以很轻松的抓取网页内容
# coding=utf-8 import urllib from bs4 import BeautifulSoup url ='http://www.baidu.com/s' values ={'wd':'网球'} encoded_param = urllib.urlencode(values) full_url = url +'?'+ encoded_param response = urllib.urlopen(full_url) soup =BeautifulSoup(response) alinks = soup.find_all('a')
上面可以抓取百度搜出来结果是网球的记录。
beautifulSoup内置了很多非常有用的方法。
几个比较好用的特性:
构造一个node元素
复制代码 代码如下:
soup = BeautifulSoup('<b class="boldest">Extremely bold</b>')
tag = soup.b
type(tag)
# <class 'bs4.element.Tag'>
属性可以使用attr拿到,结果是字典
复制代码 代码如下:
tag.attrs
# {u'class': u'boldest'}
或者直接tag.class取属性也可。
也可以自由操作属性
tag['class'] = 'verybold' tag['id'] = 1 tag # <blockquote class="verybold" id="1">Extremely bold</blockquote> del tag['class'] del tag['id'] tag # <blockquote>Extremely bold</blockquote> tag['class'] # KeyError: 'class' print(tag.get('class')) # None
还可以随便操作,查找dom元素,比如下面的例子
1.构建一份文档
html_doc = """ <html><head><title>The Dormouse's story</title></head> <p><b>The Dormouse's story</b></p> <p>Once upon a time there were three little sisters; and their names were <a href="http://example.com/elsie" id="link1">Elsie</a>, <a href="http://example.com/lacie" id="link2">Lacie</a> and <a href="http://example.com/tillie" id="link3">Tillie</a>; and they lived at the bottom of a well.</p> <p>...</p> """ from bs4 import BeautifulSoup soup = BeautifulSoup(html_doc)
2.各种搞
soup.head # <head><title>The Dormouse's story</title></head> soup.title # <title>The Dormouse's story</title> soup.body.b # <b>The Dormouse's story</b> soup.a # <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a> soup.find_all('a') # [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>, # <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, # <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>] head_tag = soup.head head_tag # <head><title>The Dormouse's story</title></head> head_tag.contents [<title>The Dormouse's story</title>] title_tag = head_tag.contents[0] title_tag # <title>The Dormouse's story</title> title_tag.contents # [u'The Dormouse's story'] len(soup.contents) # 1 soup.contents[0].name # u'html' text = title_tag.contents[0] text.contents for child in title_tag.children: print(child) head_tag.contents # [<title>The Dormouse's story</title>] for child in head_tag.descendants: print(child) # <title>The Dormouse's story</title> # The Dormouse's story len(list(soup.children)) # 1 len(list(soup.descendants)) # 25 title_tag.string # u'The Dormouse's story'
用这两个参数:findAll('div',{'class':'content'})
可以的
soup = BeautifulSoup(获取的数据, fromEncoding='gbk')
items = soup.findAll(attrs = {'class':'small'})
for item in items:
print item
相关内容
- python服务器端收发请求的实现代码,python服务器端
- python提示No module named images的解决方法,pythonnamed
- Python中asyncore的用法实例,pythonasyncore
- Python学习之asyncore模块用法实例教程,pythonasyncore
- Python基于twisted实现简单的web服务器,pythontwistedweb
- Python中logging模块的用法实例,pythonlogging
- python中ConfigParse模块的用法,pythonconfigparse
- python实现通过shelve修改对象实例,pythonshelve
- python使用Berkeley DB数据库实例,pythonberkeley
- python实现udp数据报传输的方法,pythonudp数据报
评论关闭