Examples of SURTTokenizer

org.archive.wayback.surt.SURTTokenizer
provides iterative Url reduction for prefix matching to find ever coarser grained URL-specific configuration. Assumes that a prefix binary search is being attempted for each returned value. First value is the entire SURT url String, with TAB appended. Second removes CGI ARGs. Then each subsequent path segment ('/' separated) is removed. Then the login:password, if present is removed. Then the port, if not :80 or omitted on the initial URL. Then each subsequent authority segment(. separated) is removed. the nextSearch() method will return null, finally, when no broader searches can be attempted on the URL. @author brad @version $Date: 2007-05-29 22:15:55 +0100 (Tue, 29 May 2007) $, $Revision: 1748 $

Examples of org.archive.wayback.surt.SURTTokenizer

    exclusionMap = map;
  }
  
  protected boolean isExcluded(String url) {
    try {
      SURTTokenizer st = new SURTTokenizer(url);
      while(true) {
        String nextSearch = st.nextSearch();
        if(nextSearch == null) {
          break;
        }
        LOGGER.fine("EXCLUSION-MAP:Checking " + nextSearch);
        if(exclusionMap.containsKey(nextSearch)) {

View Full Code Here

Examples of org.archive.wayback.surt.SURTTokenizer

  /* (non-Javadoc)
   * @see org.archive.wayback.accesscontrol.ExclusionAuthority#checkExclusion(java.lang.String, java.lang.String, java.lang.String)
   */
  public ExclusionResponse checkExclusion(String userAgent, String urlString, 
      String captureDate) throws Exception {
    SURTTokenizer tokenizer;
    try {
      tokenizer = new SURTTokenizer(urlString);
    } catch (URIException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
      throw new Exception(e);
    }


    while(true) {
      String surtPrefix = tokenizer.nextSearch();
      if(surtPrefix == null) {
        break;
      }
      AdministrativeExclusionRule rule = getRuleFor(surtPrefix,captureDate);
      if(rule != null) {

View Full Code Here

Examples of org.archive.wayback.surt.SURTTokenizer

    }
  }
  
  protected boolean isExcluded(String url) {
    try {
      SURTTokenizer st = new SURTTokenizer(url, canonicalizer.isSurtForm());
      while(true) {
        String nextSearch = st.nextSearch();
        if(nextSearch == null) {
          break;
        }
        LOGGER.fine("EXCLUSION-MAP:Checking " + nextSearch);
        if(exclusionMap.containsKey(nextSearch)) {

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.