1、controller.getFetchChain().process(curi,this);
1.1、org.archive.crawler.prefetch.Preselector,
1.2、org.archive.crawler.prefetch.PreconditionEnforcer,
1.3、org.archive.modules.fetcher.FetchDNS,
//httpclient
1.4、org.archive.modules.fetcher.FetchHTTP,
1.5、org.archive.modules.extractor.ExtractorHTTP,
1.6、org.archive.modules.extractor.ExtractorHTML,
1.7、org.archive.modules.extractor.ExtractorCSS,
1.8、org.archive.modules.extractor.ExtractorJS,
1.9、org.archive.modules.extractor.ExtractorSWF
2、controller.getFrontier().beginDisposition(curi);
BdbFrontier.beginDisposition(curi)
3、controller.getDispositionChain().process(curi,this);
3.1、org.archive.modules.writer.MirrorWriterProcessor
//getCandidateChain().process(prereq, null);//add seeds
3.2、org.archive.crawler.postprocessor.CandidatesProcessor
//run isInScope(curi)
3.2.1、org.archive.crawler.prefetch.CandidateScoper
3.2.2、org.archive.crawler.prefetch.FrontierPreparer
// set schedulingDirective
3.2.2.1、curi.setSchedulingDirective(getSchedulingDirective(curi));
// set canonicalized version
3.2.2.2、curi.setCanonicalString(canonicalize(curi));
// set queue key
3.2.2.3、curi.setClassKey(getClassKey(curi));
// set cost
3.2.2.4、curi.setHolderCost(getCost(curi));
// set URI precedence
3.2.2.5、getUriPrecedencePolicy().uriScheduled(curi);
3.3、org.archive.crawler.postprocessor.DispositionProcessor
org.archive.crawler.framework.ToeThread
猜你喜欢
转载自sharehua.iteye.com/blog/1749324
今日推荐
周排行