Python3之urllib模块,python3urllib,简介 urllib
Python3之urllib模块,python3urllib,简介 urllib
简介
urllib是python的一个获取url(Uniform Resource Locators,统一资源定位符),可以用来抓取远程的数据。
常用方法
(1)urlopen
urllib.request.urlopen(url, data=None,[timeout,]*,cafile=None,capath=None,cadefault=False,context=None)
urllib.request.urlopen() 可以获取页面,获取页面内容的数据格式为bytes类型,需要进行decode()解码,转换成str类型。
参数说明:
url : 需要打开的网址data : 字典形式,默认为None时是GET方法,data不为空时,urlopen()的提交方式为POST,注意POST提交时,data需要转换为字节;timeout : 设置网站访问的超时时间1234567 | from |
1234 | # 使用with语句 |
注意:urllib.request 使用相同的接口来处理所有类型的url,比如:
1 | req |
urlopen返回对象提供的方法:
read(),readline(),readlines(),fileno(),close() : 对HTTPResponse类型数据进行操作info() : 返回HTTPMessage对象,表示远程服务器返回的头信息getcode() : 返回HTTP状态码,如果是http请求,200请求成功完成,404网页未找到geturl(): 返回请求的url(2)Request
urllib.request.Request(url,data=None,headers={},method=None)
123456789101112 | from |
(3)parse.urlencode
urllib.parse.urlencode(query, doseq=False,safe=‘‘,encoding=None,errors=None)
urlencode()的主要作用就是将url附上要提交的数据. 对data数据进行编码。
1234567891011121314151617181920212223 | from |
(4)代理request.ProxyHandler(proxies=None)
当需要抓取的网站设置了访问限制,这时就需要用到代理来抓取数据。
12345678910111213 | from |
(5)异常处理
urlopen在不能处理某个响应的时候会抛出URLError, HTTPError是URLError的子类,在遇到HTTP URL的特殊情况时被抛出。异常类来自于 urllib.error 模块。
URLError:
一般来说,URLError被抛出是因为没有网络连接(没有到指定服务器的路径),或者是指定服务器不存在。在这种情况下,抛出的异常将会包含一个‘reason’属性,
这是包含一个错误码和一段错误信息的元组.
1234567 | req |
HTTPError :
每一个来自服务器的HTTP响应都包含一个数字的“状态码”。有时状态码表明服务器不能执行请求。默认的处理程序会为你处理其中的部分响应(比如,如果响应是“重定向”,
要求客户端从一个不同的URL中获取资料,那么urllib将会为你处理这个)。对于那些不能处理的响应,urlopen将会抛出一个HTTPError。
典型的错误包括‘404’(页面未找到),‘403’(请求禁止),和‘401’(请求认证)。
# Table mapping response codes to messages; entries have the
# form {code: (shortmessage, longmessage)}.
responses = {
100: (‘Continue‘, ‘Request received, please continue‘),
101: (‘Switching Protocols‘,
‘Switching to new protocol; obey Upgrade header‘),
200: (‘OK‘, ‘Request fulfilled, document follows‘),
201: (‘Created‘, ‘Document created, URL follows‘),
202: (‘Accepted‘,
‘Request accepted, processing continues off-line‘),
203: (‘Non-Authoritative Information‘, ‘Request fulfilled from cache‘),
204: (‘No Content‘, ‘Request fulfilled, nothing follows‘),
205: (‘Reset Content‘, ‘Clear input form for further input.‘),
206: (‘Partial Content‘, ‘Partial content follows.‘),
300: (‘Multiple Choices‘,
‘Object has several resources -- see URI list‘),
301: (‘Moved Permanently‘, ‘Object moved permanently -- see URI list‘),
302: (‘Found‘, ‘Object moved temporarily -- see URI list‘),
303: (‘See Other‘, ‘Object moved -- see Method and URL list‘),
304: (‘Not Modified‘,
‘Document has not changed since given time‘),
305: (‘Use Proxy‘,
‘You must use proxy specified in Location to access this ‘
‘resource.‘),
307: (‘Temporary Redirect‘,
‘Object moved temporarily -- see URI list‘),
400: (‘Bad Request‘,
‘Bad request syntax or unsupported method‘),
401: (‘Unauthorized‘,
‘No permission -- see authorization schemes‘),
402: (‘Payment Required‘,
‘No payment -- see charging schemes‘),
403: (‘Forbidden‘,
‘Request forbidden -- authorization will not help‘),
404: (‘Not Found‘, ‘Nothing matches the given URI‘),
405: (‘Method Not Allowed‘,
‘Specified method is invalid for this server.‘),
406: (‘Not Acceptable‘, ‘URI not available in preferred format.‘),
407: (‘Proxy Authentication Required‘, ‘You must authenticate with ‘
‘this proxy before proceeding.‘),
408: (‘Request Timeout‘, ‘Request timed out; try again later.‘),
409: (‘Conflict‘, ‘Request conflict.‘),
410: (‘Gone‘,
‘URI no longer exists and has been permanently removed.‘),
411: (‘Length Required‘, ‘Client must specify Content-Length.‘),
412: (‘Precondition Failed‘, ‘Precondition in headers is false.‘),
413: (‘Request Entity Too Large‘, ‘Entity is too large.‘),
414: (‘Request-URI Too Long‘, ‘URI is too long.‘),
415: (‘Unsupported Media Type‘, ‘Entity body in unsupported format.‘),
416: (‘Requested Range Not Satisfiable‘,
‘Cannot satisfy request range.‘),
417: (‘Expectation Failed‘,
‘Expect condition could not be satisfied.‘),
500: (‘Internal Server Error‘, ‘Server got itself in trouble‘),
501: (‘Not Implemented‘,
‘Server does not support this operation‘),
502: (‘Bad Gateway‘, ‘Invalid responses from another server/proxy.‘),
503: (‘Service Unavailable‘,
‘The server cannot process the request due to a high load‘),
504: (‘Gateway Timeout‘,
‘The gateway server did not receive a timely response‘),
505: (‘HTTP Version Not Supported‘, ‘Cannot fulfill request.‘),
}
错误码
异常处理方式:
12345678 | req |
或者:
123456789101112 | from |
Python3之urllib模块
相关内容
- Python——决策树实战:california房价预测,pythoncaliforn
- python 实现选课系统,python实现选课,角色:学校、学员、
- 使用python编写微信跳一跳的自动脚本,python编写脚本
- Python学习笔记整理总结【ORM(SQLAlchemy)】,ormsqlalche
- python-day7--%s与%d的使用,python2中的input及raw_input,,#co
- 深度学习之前期准备工作--python,pip,numpy,tensorflow安装,
- Ubuntu Desktop 16.04 LTS 下成功配置Jupyter的两个python内核版
- 使用牛顿-拉弗森法定义平方根函数(Newton-Raphson metho
- 基于python+selenium的框架思路,pythonselenium,设想:1、使用
- Sublime Text3 配置Python3编译环境,sublimepython3,Sublime Te
评论关闭