custom_setting:是对setting中的文件内容进行覆盖。
这里我们以爬取知乎的网站为例。
知识点介绍: custom_settings :对框架中的内容进行覆盖,比如我想覆盖setting中的headers的内容,那么只要将header的内容写入custom_settings中,然后改变headers的值即可, 当程序再次运行时会覆盖以前setting的headers值,而运行你修改之后的内容。
知乎最开始带有heaser的setting文件内容如下:、
DEFAULT_REQUEST_HEADERS = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en',
}
运行zhihu文件出现状态码·200
2018-08-03 10:53:57 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.zhihu.com/robots.txt> (refere
r: None)
编写zhihu文件的内容:
# -*- coding: utf-8 -*-
import scrapy
class ZhihuSpider(scrapy.Spider):
name = 'zhihu'
allowed_domains = ['www.zhihu.com']
start_urls = ['http://www.zhihu.com/']
custom_settings = {
'DEFAULT_REQUEST_HEADERS' : {
'User-Agent': None,
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en',
}
}
def parse(self, response):
pass
运行结果为:
2018-08-03 10:56:14 [scrapy.core.engine] DEBUG: Crawled (400) <GET http://www.zhihu.com/robots.txt> (referer
: None)
2018-08-03 10:56:19 [scrapy.core.engine] DEBUG: Crawled (400) <GET http://www.zhihu.com/> (referer: None)