PhantomJS这个headless浏览器还是很好用的，配合着v8与qt化的webkit引擎，JavaScript可以很好得去处理得到的页面数据。
去年在整合各个平台搭建搜索引擎的时候，写了一个爬虫脚本，去掉一些认证和特殊功能，一个简单的js就放到github上了：
https://github.com/dna2github/dna2oldmemory/blob/master/dna2poem/phantomjs/spider.js
使用的时候直接：
phantomjs spider.js <url>
这个小爬虫包含了：
－ 资源加载监听
－ 英语分词器和词频统计
－ DOM里<a>的href抓取
这样，当phantomjs加载一个页面的时候，就可以知道访问的所有api，图像，css，js等资源url是什么，加载了多长时间。
分词器和词频统计是为了后面给文章进行大概的分类，提高搜索结果的质量。
对于<a>的href抓取，这里没有判断href是不是javascript调用；抓取后有两个用途，一个是建立page rank模型，另一个是可以继续爬取下一个网页。
例如：
phantomjs spider.js https://en.wikipedia.org/wiki/Special:Random
我们可以得到如下结果：- Sun Mar 27 2016 11:40:50 GMT+0800 (CST) - Sun Mar 27 2016 11:40:51 GMT+0800 (CST) <GET: 302> /wiki/Special:Random
- Sun Mar 27 2016 11:40:51 GMT+0800 (CST) - Sun Mar 27 2016 11:40:51 GMT+0800 (CST) <GET: 200> /wiki/Adele_Wong
- Sun Mar 27 2016 11:40:51 GMT+0800 (CST) - Sun Mar 27 2016 11:40:52 GMT+0800 (CST) <GET: 200> /w/load.php?debug=false&lang=en&modules=ext.cite.styles%7Cext.gadget.DRN-wizard%2CReferenceTooltips%2Ccharinsert%2Cfeatured-articles-links%2CrefToolbar%2Cswitcher%2Cteahouse%7Cext.tmh.thumbnail.styles%7Cext.uls.nojs%7Cext.visualEditor.desktopArticleTarget.noscript%7Cext.wikimediaBadges%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.raggett%2CsectionAnchor%7Cmediawiki.skinning.interface%7Cskins.vector.styles%7Cwikibase.client.init&only=styles&skin=vector
- Sun Mar 27 2016 11:40:51 GMT+0800 (CST) - Sun Mar 27 2016 11:40:52 GMT+0800 (CST) <GET: 200> /w/load.php?debug=false&lang=en&modules=site&only=styles&skin=vector
- Sun Mar 27 2016 11:40:51 GMT+0800 (CST) - Sun Mar 27 2016 11:40:52 GMT+0800 (CST) <GET: 200> /w/load.php?debug=false&lang=en&modules=startup&only=scripts&skin=vector
- Sun Mar 27 2016 11:40:52 GMT+0800 (CST) - Sun Mar 27 2016 11:40:52 GMT+0800 (CST) <GET: 200> /w/load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=vector&version=tbOYKAbm
url: https://en.wikipedia.org/wiki/Adele_Wong
= frequencies ==========================================
total: 546
in 18
wong 16
film 13
she 8
singapore 8
at 7
of 7
festival 6
s 6
this 6
page 5
for 5
also 5
which 5
lead 5
as 5
links 5
chinese 5
wikipedia 5
2009 5
= word stream ==========================================
adele wong from wikipedia free encyclopedia this chinese name family name wong this article may require cleanup to meet wikipedia s quality standards specific problem some of links not working anymore more citation needed as well please help improve this article if you july 2013 adele wong wong in 2008 music video knowing 懂 chinese name 王 惠 珊 pinyin wáng huìsh ā n mandarin ancestry shanghai china born 8 march 1983 singapore occupation actress singer writer screenwriter years active 2006-present adele wong born 8 march 1983 singaporean actress singer writer screenwriter contents 1 biography 2 filmography 3 references 4 external links biography edit born in singapore wong traces her ancestry to shanghai china she graduated from national university of singapore nus with honors degree in sociology at nus wong active in dance theatre upon graduation wong began her professional career in acting hosting writing she played lead role in damned 1 which part of omnibus feature film which premiered at osaka asian film festival in 2009 wong also played lead role in garden girls short film which premiered at singapore film festival in 2009 wong also active in television in singapore where she has been seen in several programs she recently starred as female lead annie in jack neo s first english-language drama series for okto titled happily ever after she also seen in 2009 mediacorp chinese new year television film kuchinta family reunion wong currently filming chinese-language television drama he ain t pesky he s my brother for mediacorp s channel 8 where she plays lead female character opposite christopher lee premiering in july 2009 wong s feature film debut as female lead role in days which screened at tokyo international film festival singapore international film festival short listed for new talent award at shanghai international film festival wong also sang theme song for film knowing 懂 2 which written produced by jim lim 林 倛 玉 besides acting wong writes both fiction non-fiction she hosts column on sinema s website 3 she also recently awarded grant for short film by singapore film commission 4 hanging gardens starring wong go into production in august 2009 filmography edit short films damned garden girls feature films days music videos 懂 knowing serials perfect deception as evelyn references edit damned in osaka asian film festival 懂 knowing music video sinema column sfc external links edit adele wong at internet movie database days official movie site watch my ass official movie site watch my ass myspace site categories 1983 birthsnational university of singapore alumnisingaporean actressesliving peoplesingaporean people of chinese descent navigation menu not logged intalkcontributionscreate accountlog inarticletalkreadeditview history main page contents featured content current events random article donate to wikipedia wikipedia store interaction help about wikipedia community portal recent changes contact page tools what links here related changes upload file special pages permanent link page information wikidata item cite this page print export create book download as pdf printable version languages add links this page last modified on 26 november 2013 at 21 33 text available under creative commons attribution-sharealike license additional terms may apply by using this site you agree to terms of use privacy policy wikipedia® registered trademark of wikimedia foundation inc non-profit organization privacy policyabout wikipediadisclaimerscontact wikipediadeveloperscookie statementmobile view
= links ================================================
https://en.wikipedia.org/wiki/Adele_Wong#mw-head
https://en.wikipedia.org/wiki/Adele_Wong#p-search
https://en.wikipedia.org/wiki/Chinese_name
https://en.wikipedia.org/wiki/Chinese_surname
https://en.wikipedia.org/wiki/Wang_(surname)
https://en.wikipedia.org/wiki/Wikipedia:Cleanup
https://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style
https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit
https://en.wikipedia.org/wiki/File:Adele_Wong.jpg
https://en.wikipedia.org/wiki/Standard_Mandarin
https://en.wikipedia.org/wiki/Ancestral_home_(Chinese)
https://en.wikipedia.org/wiki/Shanghai
https://en.wikipedia.org/wiki/China
https://en.wikipedia.org/wiki/Singapore
https://en.wikipedia.org/wiki/Adele_Wong#Biography
https://en.wikipedia.org/wiki/Adele_Wong#Filmography
https://en.wikipedia.org/wiki/Adele_Wong#References
https://en.wikipedia.org/wiki/Adele_Wong#External_links
https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit§ion=1
https://en.wikipedia.org/wiki/National_University_of_Singapore
https://en.wikipedia.org/wiki/Adele_Wong#cite_note-1
https://en.wikipedia.org/wiki/Singapore_Film_Festival
https://en.wikipedia.org/wiki/Jack_Neo
https://en.wikipedia.org/wiki/Okto
https://en.wikipedia.org/wiki/Happily_Ever_After_(TV_series)
https://en.wikipedia.org/wiki/MediaCorp
https://en.wikipedia.org/wiki/MediaCorp
https://en.wikipedia.org/wiki/Christopher_Lee
https://en.wikipedia.org/w/index.php?title=The_Days_(Singaporean_Film)&action=edit&redlink=1
https://en.wikipedia.org/wiki/Tokyo_International_Film_Festival
https://en.wikipedia.org/wiki/Singapore_International_Film_Festival
https://en.wikipedia.org/wiki/Shanghai_International_Film_Festival
https://en.wikipedia.org/wiki/Adele_Wong#cite_note-2
https://en.wikipedia.org/w/index.php?title=Jim_Lim&action=edit&redlink=1
https://en.wikipedia.org/wiki/Adele_Wong#cite_note-3
https://en.wikipedia.org/wiki/Adele_Wong#cite_note-4
https://en.wikipedia.org/wiki/Hanging_Gardens
https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit§ion=2
https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit§ion=3
https://en.wikipedia.org/wiki/Adele_Wong#cite_ref-1
http://www.oaff.jp/lineup/lineup02.html#meet_02
https://en.wikipedia.org/wiki/Adele_Wong#cite_ref-2
https://www.youtube.com/watch?v=grf-FZqkPao&feature=relate
https://en.wikipedia.org/wiki/Adele_Wong#cite_ref-3
http://www.oaff.jp/lineup/lineup02.html#meet_02
https://en.wikipedia.org/wiki/Adele_Wong#cite_ref-4
http://www.sfc.org.sg/main.html
https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit§ion=4
http://www.imdb.com/name/nm3044125/
https://en.wikipedia.org/wiki/Internet_Movie_Database
http://www.thedaysmovie.com/
http://watchmyass.wordpress.com/
https://www.myspace.com/watchmyassmovie
https://en.wikipedia.org/wiki/Wikipedia:Persondata
https://en.wikipedia.org/wiki/Singapore
https://en.wikipedia.org/w/index.php?title=Adele_Wong&oldid=583442184
https://en.wikipedia.org/wiki/Help:Category
https://en.wikipedia.org/wiki/Category:1983_births
https://en.wikipedia.org/wiki/Category:National_University_of_Singapore_alumni
https://en.wikipedia.org/wiki/Category:Singaporean_actresses
https://en.wikipedia.org/wiki/Category:Living_people
https://en.wikipedia.org/wiki/Category:Singaporean_people_of_Chinese_descent
https://en.wikipedia.org/wiki/Category:Articles_needing_cleanup_from_July_2013
https://en.wikipedia.org/wiki/Category:All_articles_needing_cleanup
https://en.wikipedia.org/wiki/Category:Cleanup_tagged_articles_with_a_reason_field_from_July_2013
https://en.wikipedia.org/wiki/Category:Wikipedia_pages_needing_cleanup_from_July_2013
https://en.wikipedia.org/wiki/Category:Use_dmy_dates_from_November_2010
https://en.wikipedia.org/wiki/Special:MyTalk
https://en.wikipedia.org/wiki/Special:MyContributions
https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Adele+Wong&type=signup
https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Adele+Wong
https://en.wikipedia.org/wiki/Adele_Wong
https://en.wikipedia.org/wiki/Talk:Adele_Wong
https://en.wikipedia.org/wiki/Adele_Wong#
https://en.wikipedia.org/wiki/Adele_Wong
https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=edit
https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=history
https://en.wikipedia.org/wiki/Adele_Wong#
https://en.wikipedia.org/wiki/Main_Page
https://en.wikipedia.org/wiki/Main_Page
https://en.wikipedia.org/wiki/Portal:Contents
https://en.wikipedia.org/wiki/Portal:Featured_content
https://en.wikipedia.org/wiki/Portal:Current_events
https://en.wikipedia.org/wiki/Special:Random
https://donate.wikimedia.org/wiki/Special:FundraiserRedirector?utm_source=donate&utm_medium=sidebar&utm_campaign=C13_en.wikipedia.org&uselang=en
https://shop.wikimedia.org/
https://en.wikipedia.org/wiki/Help:Contents
https://en.wikipedia.org/wiki/Wikipedia:About
https://en.wikipedia.org/wiki/Wikipedia:Community_portal
https://en.wikipedia.org/wiki/Special:RecentChanges
https://en.wikipedia.org/wiki/Wikipedia:Contact_us
https://en.wikipedia.org/wiki/Special:WhatLinksHere/Adele_Wong
https://en.wikipedia.org/wiki/Special:RecentChangesLinked/Adele_Wong
https://en.wikipedia.org/wiki/Wikipedia:File_Upload_Wizard
https://en.wikipedia.org/wiki/Special:SpecialPages
https://en.wikipedia.org/w/index.php?title=Adele_Wong&oldid=583442184
https://en.wikipedia.org/w/index.php?title=Adele_Wong&action=info
https://www.wikidata.org/wiki/Q4681907
https://en.wikipedia.org/w/index.php?title=Special:CiteThisPage&page=Adele_Wong&id=583442184
https://en.wikipedia.org/w/index.php?title=Special:Book&bookcmd=book_creator&referer=Adele+Wong
https://en.wikipedia.org/w/index.php?title=Special:Book&bookcmd=render_article&arttitle=Adele+Wong&returnto=Adele+Wong&oldid=583442184&writer=rdf2latex
https://en.wikipedia.org/w/index.php?title=Adele_Wong&printable=yes
https://en.wikipedia.org/wiki/Adele_Wong#
https://www.wikidata.org/wiki/Q4681907#sitelinks-wikipedia
https://en.wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribution-ShareAlike_3.0_Unported_License
https://creativecommons.org/licenses/by-sa/3.0/
https://wikimediafoundation.org/wiki/Terms_of_Use
https://wikimediafoundation.org/wiki/Privacy_policy
https://www.wikimediafoundation.org/
https://wikimediafoundation.org/wiki/Privacy_policy
https://en.wikipedia.org/wiki/Wikipedia:About
https://en.wikipedia.org/wiki/Wikipedia:General_disclaimer
https://en.wikipedia.org/wiki/Wikipedia:Contact_us
https://www.mediawiki.org/wiki/Special:MyLanguage/How_to_contribute
https://wikimediafoundation.org/wiki/Cookie_statement
https://en.m.wikipedia.org/w/index.php?title=Adele_Wong&mobileaction=toggle_view_mobile
https://wikimediafoundation.org/
https://www.mediawiki.org/
