Examples of org.apache.nutch.net.RegexUrlNormalizer

org.apache.nutch.net.RegexUrlNormalizer

Allows users to do regex substitutions on all/any URLs that are encountered, which is useful for stripping session IDs from URLs.

This class must be specified as the URL normalizer to be used in nutch-site.xml or nutch-default.xml. To do this specify the urlnormalizer.class property to have the value: org.apache.nutch.net.RegexUrlNormalizer. The urlnormalizer.regex.file property should also be set to the file name of an xml file which should contain the patterns and substitutions to be done on encountered URLs.

@author Luke Baker

  }


  private void normalizeTest(String weird, String normal) throws Exception {
    String testSrcDir = System.getProperty("test.src.dir");
    String path = testSrcDir + "/org/apache/nutch/net/test-regex-normalize.xml";
    RegexUrlNormalizer normalizer = new RegexUrlNormalizer(path);
    assertEquals(normal, normalizer.normalize(weird));
  }

  }


  private void normalizeTest(String weird, String normal) throws Exception {
    String testSrcDir = System.getProperty("test.src.dir");
    String path = testSrcDir + "/org/apache/nutch/net/test-regex-normalize.xml";
    RegexUrlNormalizer normalizer = new RegexUrlNormalizer(path);
    assertEquals(normal, normalizer.normalize(weird));
  }

Related Classes of org.apache.nutch.net.RegexUrlNormalizer

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.