获取一篇新浪文章信息的简单爬虫

具体代码如下，有详细的注释：

# python3
# 主要采用的 requests库，BeautifulSoup库
import requests
from bs4 import BeautifulSoup

#将要爬取的新浪文章的网址
url = "http://news.sina.com.cn/c/xl/2018-07-18/doc-ihfnsvyz7503763.shtml"

#获得文章的内容
def getNewsContent(newsurl):
    #访问网页，res == 200，则访问成功
    res = requests.get(newsurl) 
    #制定编码格式 'utf-8'
    res.encoding = 'utf-8'  

    # res.text 为该网页的html文件
    # 使用BeauitfulSoup 对 html(res.text)进行内容提取
    # BeauitfulSoup 中文官方文档：https://www.crummy.com/software/BeautifulSoup/bs4/doc.zh/#id37
    soup = BeautifulSoup(res.text, 'html.parser')
    
    # 使用BeauitfulSoup的CSS选择器
    # BeauitfulSoup的CSS选择器的基本用法 见附录1 
    context = ''
    for Pargraph in soup.select('#article_content p')[:-2]:
        context += (Pargraph.text)
        context += '\n'
    return context

def getNewsDetail(newsurl):
    result = {}
    res = requests.get(newsurl)  
    res.encoding = 'utf-8'  
    soup = BeautifulSoup(res.text, 'html.parser')
    
    # 获得文章的标题
    result['title'] = soup.select('.main-title')[0].text  
    # 获得文章的时间
    result['date'] = soup.select('.date')[0].text
    result['context'] = getNewsContent(newsurl)
    print(result['title'])  
    print(result['date'])
    print(result['context'])

getNewsDetail(url)

附录一：

### 附录一 ###
CSS选择器
# BeauitfulSoup 中文官方文档：https://www.crummy.com/software/BeautifulSoup/bs4/doc.zh/#id37

Beautiful Soup支持大部分的CSS选择器 [6] ,在 Tag 或 BeautifulSoup 对象的 .select() 方法中传入字符串参数,即可使用CSS选择器的语法找到tag:

soup.select("title")
# [<title>The Dormouse's story</title>]

soup.select("p nth-of-type(3)")
# [<p class="story">...</p>]

通过tag标签逐层查找:

soup.select("body a")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie"  id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select("html head title")
# [<title>The Dormouse's story</title>]

找到某个tag标签下的直接子标签 [6] :

soup.select("head > title")
# [<title>The Dormouse's story</title>]

soup.select("p > a")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie"  id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select("p > a:nth-of-type(2)")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]

soup.select("p > #link1")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]

soup.select("body > a")
# []

找到兄弟节点标签:

soup.select("#link1 ~ .sister")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie"  id="link3">Tillie</a>]

soup.select("#link1 + .sister")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]

通过CSS的类名查找:

soup.select(".sister")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select("[class~=sister]")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

通过tag的id查找:

soup.select("#link1")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]

soup.select("a#link2")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]

通过是否存在某个属性来查找:

soup.select('a[href]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

通过属性的值来查找:

soup.select('a[href="http://example.com/elsie"]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]

soup.select('a[href^="http://example.com/"]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select('a[href$="tillie"]')
# [<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select('a[href*=".com/el"]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]

通过语言设置来查找:

multilingual_markup = """
 <p lang="en">Hello</p>
 <p lang="en-us">Howdy, y'all</p>
 <p lang="en-gb">Pip-pip, old fruit</p>
 <p lang="fr">Bonjour mes amis</p>
"""
multilingual_soup = BeautifulSoup(multilingual_markup)
multilingual_soup.select('p[lang|=en]')
# [<p lang="en">Hello</p>,
#  <p lang="en-us">Howdy, y'all</p>,
#  <p lang="en-gb">Pip-pip, old fruit</p>]

对于熟悉CSS选择器语法的人来说这是个非常方便的方法.Beautiful Soup也支持CSS选择器API,
如果你仅仅需要CSS选择器的功能,那么直接使用 lxml 也可以,而且速度更快,支持更多的CSS选择器语法,
但Beautiful Soup整合了CSS选择器的语法和自身方便使用API.

获取一篇新浪文章信息的简单爬虫

猜你喜欢