Examples of org.apache.nutch.parse.ParseResult

org.apache.nutch.parse.ParseResult
A utility class that stores result of a parse. Internally a ParseResult stores < {@link Text}, {@link Parse}> pairs.
Parsers may return multiple results, which correspond to parts or other associated documents related to the original URL.

There will be usually one parse result that corresponds directly to the original URL, and possibly many (or none) results that correspond to derived URLs (or sub-URLs).

      status.setArgs(new String[] { metaTags.getRefreshHref().toString(),
          Integer.toString(metaTags.getRefreshTime()) });
    }
    ParseData parseData = new ParseData(status, title, outlinks,
        content.getMetadata(), nutchMetadata);
    ParseResult parseResult = ParseResult.createParseResult(
        content.getUrl(), new ParseImpl(text, parseData));


    // run filters on parse
    ParseResult filteredParse = this.htmlParseFilters.filter(content,
        parseResult, metaTags, root);
    if (metaTags.getNoCache()) { // not okay to cache
      for (Map.Entry<org.apache.hadoop.io.Text, Parse> entry : filteredParse)
        entry.getValue().getData().getParseMeta()
            .set(Nutch.CACHING_FORBIDDEN_KEY, cachingPolicy);

View Full Code Here

    if (LOG.isInfoEnabled()) {
      LOG.info("parsing: " + url);
      LOG.info("contentType: " + contentType);
    }


    ParseResult parseResult = new ParseUtil(conf).parse(content);


    NutchDocument doc = new NutchDocument();
    doc.add("id", url);
    Text urlText = new Text(url);


    Inlinks inlinks = null;
    Parse parse = parseResult.get(urlText);
    if (parse == null) {
      LOG.error("Failed to get parse from parse result");
      LOG.error("Available parses in parse result (by URL key):");
      for (Map.Entry<Text, Parse> entry : parseResult) {
        LOG.error("  " + entry.getKey());

View Full Code Here

0 1 2 3

TOP

Related Classes of org.apache.nutch.parse.ParseResult

com.atlantbh.nutch.filter.xpath.XPathHtmlParserFilterTest

org.apache.hadoop.io.Text

org.apache.nutch.indexer.IndexingFiltersChecker

org.apache.nutch.parse.feed.FeedParser

org.apache.nutch.parse.feed.TestFeedParser

org.apache.nutch.parse.mp3.MP3Parser

org.apache.nutch.parse.tika.TikaParser

org.apache.nutch.tools.arc.ArcSegmentCreator

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.