1.在middlewares.py添加如下代码
class SeleniumMiddleware(object):
‘’’
selenium
‘’’
def init(self):
self.browser = webdriver.Chrome(executable_path="/cdrom/chromedriver")
def process_request(self,request,spider):
self.browser.get(request.url)
time.sleep(10)
#在此拦截引擎发到scrapy下载器的request,使用模拟器进行下载,将下载结果封装
HtmlResponse,发送给spider
return HtmlResponse(url=self.browser.current_url,body=self.browser.page_source,encoding='utf-8',request=request)
2.打开settings.py的middlewares注释
‘’‘
#543是中间件
DOWNLOADER_MIDDLEWARES = {
‘simulatorspider.middlewares.SeleniumMiddleware’: 543,
}
’‘’