方法和第一篇影评的爬取方法一模一样
1、加载必要的请求库
# -*- coding: utf-8 -*-
__author__ = 'ma'
import requests
from bs4 import BeautifulSoup
2、进入豆瓣分析网站
豆瓣电影传送门
随便选一首歌曲,速度与激情主题曲。See You Again传送门
第一页
第二页
这就简单了朋友们
comments/hot?p=1
comments/hot?p=2
那么第三页肯定就是
comments/hot?p=3
def get_page(url):
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36"}
response = requests.get(url, headers=headers)
if response.status_code == 200:
return response.text
return None
因为这个url随页面变化而变化,并且信息也在源代码中,所以为静态爬虫
例如第一个
进行检查我们发现评论都在列表里
在这一标签下
此时我们加载Beautifulsoup解析HTML
for i in range(0,35):#此页数为1-10页
url = "https://music.douban.com/subject/26362557/comments/hot?p=" + str(i+1)
print("爬取第"+str(i+1)+"页")
html = get_page(url)
soup = BeautifulSoup(html, 'html5lib')
comments = soup.find(attrs={'class': "comment-list hot"})
for item in comments.find_all(attrs={'class': "short"}):
print(item.string)
此时可以获取评论信息
3、完整代码(音乐没有反爬虫,可全部爬取)
# -*- coding: utf-8 -*-
__author__ = 'ma'
import requests
from bs4 import BeautifulSoup
def get_page(url):
headers = {"User-Agent": "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.106 Mobile Safari/537.36"}
response = requests.get(url, headers=headers)
if response.status_code == 200:
return response.text
return None
for i in range(0,35):#此页数为1-35页
url = "https://music.douban.com/subject/26362557/comments/hot?p=" + str(i+1)
print("爬取第"+str(i+1)+"页")
html = get_page(url)
soup = BeautifulSoup(html, 'html5lib')
comments = soup.find(attrs={'class': "comment-list hot"})
for item in comments.find_all(attrs={'class': "short"}):
print(item.string)