最近用scrapy开始写爬虫来做个代理池,程序写好了,开始使用pyinstall打包,遇到了各种坑
使用的库
scrapy,apscheduler,等等 1. 在随机头的时候本来使用的是faker库,打包实在是搞不定,改用fake_useragent了,至少效果一样,faker库报错 Invalid configuration for faker locale en-us实在是搞不定。放弃
2.使用的打包方式是 pyinstaller -D main.spec,修改main.spec的spec文件的hiddenimport数据,添加了一堆库,这样就会默认加载进来,不需要修改代码,其实也是一个报错一个加载的过程,比较繁琐
hiddenimports=
[
'scrapy.spiderloader',
'scrapy.statscollectors',
'scrapy.logformatter',
'scrapy.dupefilters',
'scrapy.squeues',
'scrapy.extensions.spiderstate',
'scrapy.extensions.corestats',
'scrapy.extensions.telnet',
'scrapy.extensions.logstats',
'scrapy.extensions.memusage',
'scrapy.extensions.memdebug',
'scrapy.extensions.feedexport',
'scrapy.extensions.closespider',
'scrapy.extensions.debug',
'scrapy.extensions.httpcache',
'scrapy.extensions.statsmailer',
'scrapy.extensions.throttle',
'scrapy.core.scheduler',
'scrapy.core.engine',
'scrapy.core.scraper',
'scrapy.core.spidermw',
'scrapy.core.downloader',
'scrapy.downloadermiddlewares.stats',
'scrapy.downloadermiddlewares.httpcache',
'scrapy.downloadermiddlewares.cookies',
'scrapy.downloadermiddlewares.useragent',
'scrapy.downloadermiddlewares.httpproxy',
'scrapy.downloadermiddlewares.ajaxcrawl',
'scrapy.downloadermiddlewares.chunked',
'scrapy.downloadermiddlewares.decompression',
'scrapy.downloadermiddlewares.defaultheaders',
'scrapy.downloadermiddlewares.downloadtimeout',
'scrapy.downloadermiddlewares.httpauth',
'scrapy.downloadermiddlewares.httpcompression',
'scrapy.downloadermiddlewares.redirect',
'scrapy.downloadermiddlewares.retry',
'scrapy.downloadermiddlewares.robotstxt',
'scrapy.spidermiddlewares.depth',
'scrapy.spidermiddlewares.httperror',
'scrapy.spidermiddlewares.offsite',
'scrapy.spidermiddlewares.referer',
'scrapy.spidermiddlewares.urllength',
'scrapy.pipelines',
'scrapy.core.downloader.handlers.http',
'scrapy.core.downloader.contextfactory',
'sqlite3',
'fake_useragent'
],
3.会出现
FileNotFoundError: [Errno 2] No such file or directory: 'D:\\code\\python\\python-scrapy\\dist\\main\\scrapy\\VERSION'
[10816] Failed to execute script main
错误
把E:\soft\develop\Python\Python37\Lib\site-packages\scrapy目录下的mime.types,VERSION拷贝到exe目录下的scrapy文件夹中
4.
会报lookupError错误
解决办法是修改代码
添加
修改
改成下图这样
最后参考了以下链接
https://blog.csdn.net/u011962883/article/details/53436824
https://blog.csdn.net/La_vie_est_belle/article/details/79017358
https://blog.csdn.net/weixin_42052836/article/details/82315118
不得不说,python打包就是个坑,还是c++好