使用Scrapy 提取器
设置好规则
rules = ( Rule(LinkExtractor(allow=r'type=4&page=\d+'), callback="parse_pages", follow=True) )
。。。
其他代码省略,只说关键代码;
满心期待运行 scrapy crawl myspider_name
CRITICAL: Unhandled error in Deferred:
2018-07-24 18:52:17 [twisted] CRITICAL:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 79, in crawl
self.spider = self._create_spider(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/scrapy/crawler.py", line 102, in _create_spider
return self.spidercls.from_crawler(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/scrapy/spiders/crawl.py", line 100, in from_crawler
spider = super(CrawlSpider, cls).from_crawler(crawler, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/scrapy/spiders/__init__.py", line 51, in from_crawler
spider = cls(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/scrapy/spiders/crawl.py", line 40, in __init__
self._compile_rules()
File "/usr/local/lib/python3.7/site-packages/scrapy/spiders/crawl.py", line 92, in _compile_rules
self._rules = [copy.copy(r) for r in self.rules]
TypeError: 'Rule' object is not iterable
尼玛,竟然报出一个严重的错误。
更为奇葩的是因为一个逗号的问题。。。
rules = ( Rule(LinkExtractor(allow=r'type=4&page=\d+'), callback="parse_pages", follow=True), )
在Rule规则 之后加个逗号,重新运行scrapy 爬虫,错误解决。。。