Scrapy框架安装配置
1.创建一个虚拟环境,在虚拟环境中进行操作
安装:
pip install virtualenv
创建:
到对应目录下:
virtualenv 虚拟环境名称
进入虚拟环境:
D:\virtu\spiderenv\Scripts>activate
2.安装twiste依赖,
https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
到该网站下载对应python版本对应位数的twisted,下载后
pip install 路径\twisted包名
3.安装pywin32
https://www.lfd.uci.edu/~gohlke/pythonlibs/#pywin32
下载对应python版本的安装包后安装:
pip install 路径\pywin32包名
或者直接安装
安装后在虚拟环境中运行python,输入import win32com
,如果没有报错,则说明安装成功
D:\virtu\spiderenv\Scripts>python
Python 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 13:35:33) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import win32com
4.安装lxml
pip install lxml
5.安装Scrapy
pip install scrapy
6.验证安装是否成功,如下情况则为安装成功
(spiderenv) D:\virtu\spiderenv\Scripts>scrapy
Scrapy 1.5.1 - no active project
Usage:
scrapy <command> [options] [args]
Available commands:
bench Run quick benchmark test
fetch Fetch a URL using the Scrapy downloader
genspider Generate new spider using pre-defined templates
runspider Run a self-contained spider (without creating a project)
settings Get settings values
shell Interactive scraping console
startproject Create new project
version Print Scrapy version
view Open URL in browser, as seen by Scrapy
[ more ] More commands available when run from project directory
Use "scrapy <command> -h" to see more info about a command