List of tokenizePos() Examples

Examples of tokenizePos()

opennlp.tools.tokenize.SimpleTokenizer.tokenizePos()
opennlp.tools.tokenize.Tokenizer.tokenizePos()
Finds the boundaries of atomic parts in a string. @param s The string to be tokenized. @return The Span[] with the spans (offsets into s) for eachtoken as the individuals array elements.
opennlp.tools.tokenize.TokenizerME.tokenizePos()
Tokenizes the string. @param d The string to be tokenized. @return A span array containing individual tokens as elements.

Examples of opennlp.tools.tokenize.Tokenizer.tokenizePos()


    private List<Token> tokenize(Section section,String langauge) {
        Tokenizer tokenizer = getTokenizer(langauge);
        String text = section.getSpan();
        List<Token> tokens = new ArrayList<Token>(text.length()/5); //assume avr. token length is 5
        opennlp.tools.util.Span[] tokenSpans = tokenizer.tokenizePos(section.getSpan());
        for(int i=0;i<tokenSpans.length;i++){
            Token token = section.addToken(tokenSpans[i].getStart(), tokenSpans[i].getEnd());
            log.trace(" > add {}",token);
            tokens.add(token);
        }

View Full Code Here

Examples of opennlp.tools.tokenize.TokenizerME.tokenizePos()


    Span[] sentSpans = sentenceDetector.sentPosDetect(intext);
    for (Span sentSpan : sentSpans) {
      String sentence = sentSpan.getCoveredText(intext).toString();
      int start = sentSpan.getStart();
      Span[] tokSpans = tokenizer.tokenizePos(sentence);
      String[] tokens = new String[tokSpans.length];
      // System.out.println("\n\nTokens:");
      for (int i = 0; i < tokens.length; i++) {
        tokens[i] = tokSpans[i].getCoveredText(sentence).toString();
        // System.out.println(tokens[i]);

View Full Code Here

0 1

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.