HtmlCleaner是使用Java编写的HTML解析开源组件。网络上的HTML一般是比较杂乱的,难以进一步处理,需要大量的时间去剔除无效的内容,按照顺序整理tag,attribute等。HtmlCleaner能够将任意的HTML格式化为XML。HtmlCleaner会同浏览器一样默认的创建DOM(Document Object Model)。不论怎样,都需要自定义tag筛选策略和调整方式。
HtmlCleaner可以配置相应的参数(http://htmlcleaner.sourceforge.net/parameters.php),可以在在Java代码中定义,如:
HtmlCleaner cleaner = new HtmlCleaner( new ConfigFileTagProvider(myConfigFile) );
也可以在命令行中定义,如:
java -jar htmlcleanerXX.jar [src = <url | file>] [incharset = <charset>] [dest = <file>] [outcharset = <charset>] [taginfofile = <file>] [options...]
可以通过http://htmlcleaner.sourceforge.net/javause.php,查看基本的用法。
既然HtmlCleaner在内部将HTML组织成XML,那么使用XPath也是理所当然的。作为例子,抓取“豆瓣电影TOP250”
package org.htmlcleaner; import java.net.URL; import java.util.ArrayList; import java.util.List; /** * @author Reymont.Li * @DateTime 2015年8月17日 下午6:36:26 */ public class DoubanTest { // http://movie.douban.com/top250 // http://movie.douban.com/top250?start=200&filter=&type= // http://movie.douban.com/top250?start=225&filter=&type= public static void main(String[] args) throws Exception { HtmlCleaner cleaner = new HtmlCleaner(); String urlAddr = "http://movie.douban.com/top250"; List<String> names = new ArrayList<String>(); List<String> stars = new ArrayList<String>(); List<String> remarks = new ArrayList<String>(); for (int i = 0; i <= 10; i++) { TagNode root = cleaner.clean(new URL(urlAddr + "?start=" + i * 25)); Object[] nameArray = root.evaluateXPath("//div[@class='info']//a/span[1]"); for (Object nameNode : nameArray) { String name = nameNode instanceof TagNode ? ((TagNode) nameNode).getText().toString() : nameNode.toString(); names.add(name.trim()); } Object[] starArray = root.evaluateXPath("//div[@class='info']//em"); for (Object starNode : starArray) { String star = starNode instanceof TagNode ? ((TagNode) starNode).getText().toString() : starNode.toString(); stars.add(star.trim()); } Object[] remarkArray = root.evaluateXPath("//div[@class='info']//div[@class='star']/span[2]"); for (Object remarkNode : remarkArray) { String remark = remarkNode instanceof TagNode ? ((TagNode) remarkNode).getText().toString() : remarkNode.toString(); remarks.add(remark.trim()); } } for (int i = 0; i < names.size(); i++) { System.out.println( (i + 1) + "\t name:" + names.get(i) + "\t\t star:" + stars.get(i) + "\t remark:" + remarks.get(i)); } } }
参考:
HtmlCleaner Project Home Page
http://htmlcleaner.sourceforge.net/javause.php
htmlcleaner使用及xpath语法初探 - reesun的专栏 - 博客频道 - CSDN.NET
http://blog.csdn.net/reesun/article/details/8021201
Java - XPath解析爬取内容 - 推酷
http://www.tuicool.com/articles/Uzmmue
Java 语言的 XPath API
http://www.ibm.com/developerworks/cn/xml/x-javaxpathapi.html