Examples of SrxTextIterator

net.sourceforge.segment.srx.SrxTextIterator
Represents text iterator splitting text according to rules in SRX file. The algorithm idea is as follows: 1. Rule matcher list is created based on SRX file and language. Each rule matcher is responsible for matching before break and after break regular expressions of one break rule. 2. Each rule matcher is matched to the text. If the rule was not found the rule matcher is removed from the list. 3. First rule matcher in terms of its break position in text is selected. 4. List of exception rules corresponding to break rule is retrieved. 5. If none of exception rules is matching in break position then the text is marked as split and new segment is created. In addition all rule matchers are moved so they start after the end of new segment (which is the same as break position of the matched rule). 6. All the rules that have break position behind last matched rule break position are moved until they pass it. 7. If segment was not found the whole process is repeated. In streaming version of this algorithm character buffer is searched. When the end of it is reached or break position is in the margin (break position > buffer size - margin) and there is more text, the buffer is moved in the text until it starts after last found segment. If this happens rule matchers are reinitialized and the text is searched again. Streaming version has a limitation that read buffer must be at least as long as any segment in the text. As this algorithm uses lookbehind extensively but Java does not permit infinite regular expressions in lookbehind, so some patterns are finitized. For example a* pattern will be changed to something like a{0,100}. @author loomchild

Examples of net.sourceforge.segment.srx.SrxTextIterator

  }


  @Override
  public final List<String> tokenize(final String text) {
    final List<String> segments = new ArrayList<>();
    final TextIterator textIterator = new SrxTextIterator(document, languageCode + parCode, text);
    while (textIterator.hasNext()) {
      segments.add(textIterator.next());
    }
    return segments;
  }

View Full Code Here

Examples of net.sourceforge.segment.srx.SrxTextIterator

    this.languageCode = languageCode;
  }
  
  public List<String> split(String string) {
    TextIterator textIterator = 
      new SrxTextIterator(document, languageCode, string);
    List<String> segmentList = new ArrayList<String>();
    while(textIterator.hasNext()) {
      String segment = textIterator.next();
      segmentList.add(segment);
    }
    return segmentList;
    }

View Full Code Here

Examples of net.sourceforge.segment.srx.SrxTextIterator

      } else {
        throw new IllegalArgumentException("For accurate algorithm preload option (-r) is mandatory.");
      }
    } else if (algorithm == Algorithm.ultimate) {
      if (text != null) {
        textIterator = new SrxTextIterator(document, languageCode, text, parameterMap);
      } else {
        textIterator = new SrxTextIterator(document, languageCode, reader, parameterMap);
      }
    } else if (algorithm == Algorithm.fast) {
      if (text != null) {
        textIterator = new FastTextIterator(document, languageCode, text, parameterMap);
      } else {

View Full Code Here

Examples of net.sourceforge.segment.srx.SrxTextIterator

    }
  }


  static List<String> tokenize(String text, SrxDocument srxDocument, String code) {
    List<String> segments = new ArrayList<>();
    TextIterator textIterator = new SrxTextIterator(srxDocument, code, text);
    while (textIterator.hasNext()) {
      segments.add(textIterator.next());
    }
    return segments;
  }

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.