Package org.apache.nutch.protocol.http.RobotRulesParser

Examples of org.apache.nutch.protocol.http.RobotRulesParser.RobotRuleSet


       boolean[] allowed) {
    String agentsString= agents[0];
    for (int i= 1; i < agents.length; i++)
      agentsString= agentsString + "," + agents[i];
    RobotRulesParser p= new RobotRulesParser(agents);
    RobotRuleSet rules= p.parseRules(ROBOTS_STRINGS[robotsString].getBytes());
    for (int i= 0; i < paths.length; i++) {
      assertTrue("testing robots file "+robotsString+", on agents ("
     + agentsString + "), and path " + TEST_PATHS[i] + "; got "
     + rules.isAllowed(TEST_PATHS[i]) + ", rules are: " + LF
           + rules,
     rules.isAllowed(TEST_PATHS[i]) == allowed[i]);
    }
  }
View Full Code Here

TOP

Related Classes of org.apache.nutch.protocol.http.RobotRulesParser.RobotRuleSet

Copyright © 2018 www.massapicom. All rights reserved.
All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.