Examples of com.flaptor.hounder.crawler.UrlPatterns$OnlyTokens

com.flaptor.hounder.crawler.UrlPatterns
The UrlPatterns class keeps a large number of expressions against which a string can be matched. The expressions are read from a file and must comply with the folowing format: <prefix> [ | <pattern> ] [ || <anti-pattern> ] [ ||| <tokens> ] The following forms are valid: 1. prefix 2. prefix ||| tokens 3. prefix | pattern 4. prefix | pattern ||| tokens 5. prefix | pattern || anti-pattern 6. prefix | pattern || anti-pattern ||| tokens 7. "tokens" is a comma separated list of keywords that are associated with the prefix/regex of that line. The "*" prefix matches everything. If such a line is included, everying else in the file is ignored. A string matches the UrlPattern if it matches any given prefix that was not followed by a regular expression, or if some inicial part of it matches any given prefix, and the rest matches the pattern that followed and does not contain anything that matches the anti-pattern, if specified. Note that while the pattern needs to match the url after the prefix in its entirety, the anti-pattern can match any substring of it. For example, if the file contains the following lines: abc jkl | [0-9]+ xyz | [a-z0-9/]+ || pag[0-9] then the strings "abc", "jkl123" and "xyz/foo/bar" would match, but "ab", "abc3", "jkl", "jklx", "xyz/foo/pag3/bar" and "mno" would not. @author Flaptor Development Team

        super(name, globalConfig);
        maxTitleLength = getModuleConfig().getInt("max.title.length");
        titleSpamActive = (maxTitleLength > 0);
        String urlPatternFile = getModuleConfig().getString("url.pattern.file");
        urlMatchSpamActive = !("".equals(urlPatternFile));
        patterns = new UrlPatterns(urlPatternFile);
    }

View Full Code Here

   */
    public MatchUrlModule (String name, Config globalConfig) throws IOException{
        super(name, globalConfig);
        String filename = getModuleConfig().getString("url.pattern.file");
        String filepath = FileUtil.getFilePathFromClasspath(filename);
        patterns = new UrlPatterns(filepath); // TODO: hotspots should be singleton, otherwise we have two copies in ram.
    }

View Full Code Here

            logger.error(error);
            // TODO check what kind of exception to throw
            throw new RuntimeException(error);
        }
        try {
            patterns = new UrlPatterns(filename);
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

View Full Code Here

  /**
   * @todo UrlPatterns are not singleton, but should be. @link UrlPatterns
   */
    public WhiteListModule (String name, Config globalConfig) throws IOException{
        super(name, globalConfig);
        patterns = new UrlPatterns(getModuleConfig().getString("whitelist.file"));
    }

View Full Code Here

                patterns.add(list); 
            } 
        }


        try {
            this.urlPatterns = new UrlPatterns(patternFile.getAbsolutePath());
        } catch (Exception e) {
            this.urlPatterns = null;
        }
        
        return patterns;

View Full Code Here

            writer.newLine();
        }
        writer.close();


        patterns = list;
        urlPatterns = new UrlPatterns(orig.getAbsolutePath());
    }

View Full Code Here

        super(config);
        field = config.getString("method.keyword.applyto.field");
        String filename = config.getString("method.keyword.patterns.file");
        String filepath = FileUtil.getFilePathFromClasspath(filename);
        try {
            patterns = new UrlPatterns(filepath);
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

View Full Code Here

    public UrlPatternBoostCondition (Config config) {
        super(config);
        String filename = config.getString("condition.url.patterns.file");
        String filepath = FileUtil.getFilePathFromClasspath(filename);
        try {
            patterns = new UrlPatterns(filepath);
        } catch (IOException e) {   
            throw new RuntimeException(e);
        }
    }

View Full Code Here

TOP

Related Classes of com.flaptor.hounder.crawler.UrlPatterns$OnlyTokens

com.flaptor.hounder.crawler.bean.PatternFileBean

com.flaptor.hounder.crawler.modules.boost.KeywordBoostMethod

com.flaptor.hounder.crawler.modules.boost.UrlPatternBoostCondition

com.flaptor.hounder.crawler.modules.MatchUrlModule

com.flaptor.hounder.crawler.modules.PatternClassifierModule

com.flaptor.hounder.crawler.modules.SpamDetectorModule

com.flaptor.hounder.crawler.modules.WhiteListModule

java.io.FileReader

java.io.BufferedReader

java.io.IOException

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.