-
开放代理
class IpProxyDownloadMiddleware(object):
PROXIES = ['110.52.235.131:9999','110.52.235.249:9999','112.17.38.141:3128']
def process_request(self,request,spider):
proxy = random.choice(self.PROXIES)
request.meta['proxy'] = 'http://'+proxy
注:上面在设置代理时,要添加协议名称,http://
或https://
,否则会报错。
twisted.web.error.SchemeNotSupported: Unsupported scheme: b''
-
独享代理
class IpProxyDownloadMiddleware(object): def process_request(self,request,spider): proxy = "121.199.6.124:16816" user_password = "970138074:rcdj35ur" request.meta['proxy'] = 'http://'+proxy #bytes bs64_user_password = base64.b64encode(user_password.encode('utf-8')) request.headers['Proxy-Authorization'] = 'Basic' + bs64_user_password.decode('utf-8')