问题描述:
在middleware中使用ip代理的时候,报错如下
2019-01-05 21:16:15 [scrapy.core.scraper] ERROR: Error downloading <GET http://httpbin.org/ip>
Traceback (most recent call last):
File "e:\anaconda3\lib\site-packages\twisted\internet\defer.py", line 1416, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "e:\anaconda3\lib\site-packages\twisted\python\failure.py", line 491, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "e:\anaconda3\lib\site-packages\scrapy\core\downloader\middleware.py", line 43, in process_request
defer.returnValue((yield download_func(request=request,spider=spider)))
File "e:\anaconda3\lib\site-packages\scrapy\utils\defer.py", line 45, in mustbe_deferred
result = f(*args, **kw)
File "e:\anaconda3\lib\site-packages\scrapy\core\downloader\handlers\__init__.py", line 65, in download_request
return handler.download_request(request, spider)
File "e:\anaconda3\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 67, in download_request
return agent.download_request(request)
File "e:\anaconda3\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 331, in download_request
method, to_bytes(url, encoding='ascii'), headers, bodyproducer)
File "e:\anaconda3\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 252, in request
proxyEndpoint = self._getEndpoint(self._proxyURI)
File "e:\anaconda3\lib\site-packages\twisted\web\client.py", line 1635, in _getEndpoint
return self._endpointFactory.endpointForURI(uri)
File "e:\anaconda3\lib\site-packages\twisted\web\client.py", line 1513, in endpointForURI
raise SchemeNotSupported("Unsupported scheme: %r" % (uri.scheme,))
twisted.web.error.SchemeNotSupported: Unsupported scheme: b''
middleware代码如下:
class IpProxyDownloaderMiddleware(object):
def __init__(self):
self.proxy_list = [
"39.137.107.98:80",
"111.177.190.124:9999",
"175.25.26.184:10800"
]
def process_request(self, request, spider):
proxy = random.choice(self.proxy_list)
# print("proxy:", proxy)
request.meta["proxy"] = proxy
settings文件:
# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
'Useragent.middlewares.UseragentDownloaderMiddleware': 543,
'Useragent.middlewares.IpProxyDownloaderMiddleware': 300,
}
原因分析:
在middleware中代理ip并没有加"https://" 或者"http://",具体是哪一个就要看你使用的是哪种类型的了
因为我用的都是高匿代理所以都需要加https,修改后代码:
class IpProxyDownloaderMiddleware(object):
def __init__(self):
self.proxy_list = [
"39.137.107.98:80",
"111.177.190.124:9999",
"175.25.26.184:10800"
]
def process_request(self, request, spider):
proxy = random.choice(self.proxy_list)
# print("proxy:", proxy)
request.meta["proxy"] = "https://" + proxy
重新执行,成功!!