Examples of org.exist.storage.analysis.SimpleTokenizer

org.exist.storage.analysis.SimpleTokenizer
This is the default class used by the fulltext indexer for tokenizing a string into words. Known token types are defined by class Token. @author Wolfgang Meier

        LOG.debug(e);
      }
    }
    if (tokenizer == null) {
      LOG.debug("using simple tokenizer");
      tokenizer = new SimpleTokenizer();
    }


    if (stem)
      {stemmer = new PorterStemmer();}
    tokenizer.setStemming(stem);

View Full Code Here

    throws XPathException {
    if(args[0].isEmpty())
      {return Sequence.EMPTY_SEQUENCE;}
    
    final ValueSequence result = new ValueSequence();
    final SimpleTokenizer tokenizer = new SimpleTokenizer();
    tokenizer.setText(args[0].getStringValue());
    TextToken token = tokenizer.nextToken(false);
    while(token != null && token.getType() != TextToken.EOF) {
      result.add(new StringValue(token.getText()));
      token = tokenizer.nextToken(false);
    }
    return result;
  }

View Full Code Here

TOP

Related Classes of org.exist.storage.analysis.SimpleTokenizer

org.exist.storage.TextSearchEngine

org.exist.xquery.functions.text.Tokenize

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.