DUPEFILTER_CLASS='scrapy_redis.dupefilter.RFPDupeFilter'
SCHEDULER = 'scrapy_redis.scheduler.Scheduler'
SCHEDULER_PERSIST = True
REDIS_URL = 'redis://127.0.0.1:6379'
scrapy在settings中添加redis,可以实现断点续传
猜你喜欢
转载自www.cnblogs.com/duanlinxiao/p/10826702.html
今日推荐
周排行