需要在setting.py里找到ROBOTSTXT_OBEY并设为false
来源:https://blog.csdn.net/yimingsilence/article/details/52119720