Examples of org.apache.lucene.analysis.pattern.PatternTokenizer

Package org.apache.lucene.analysis.pattern

Examples of org.apache.lucene.analysis.pattern.PatternTokenizer

org.apache.lucene.analysis.pattern.PatternTokenizer
This tokenizer uses regex pattern matching to construct distinct tokens for the input stream. It takes two arguments: "pattern" and "group".
- "pattern" is the regular expression.
- "group" says which group to extract into tokens.
group=-1 (the default) is equivalent to "split". In this case, the tokens will be equivalent to the output from (without empty tokens): {@link String#split(java.lang.String)}

Using group >= 0 selects the matching group as the token. For example, if you have:
```
 pattern = \'([^\']+)\' group = 0 input = aaa 'bbb' 'ccc' 
```
the output will be two tokens: 'bbb' and 'ccc' (including the ' marks). With the same input but using group=1, the output would be: bbb and ccc (no ' marks)

NOTE: This Tokenizer does not output tokens that are of zero length.
@see Pattern

                return "pattern";
            }


            @Override public Tokenizer create(Reader reader) {
                try {
                    return new PatternTokenizer(reader, Regex.compile("\\W+", null), -1);
                } catch (IOException e) {
                    throw new ElasticSearchIllegalStateException("failed to parse default pattern");
                }
            }
        }));

View Full Code Here

        this.group = settings.getAsInt("group", -1);
    }


    @Override public Tokenizer create(Reader reader) {
        try {
            return new PatternTokenizer(reader, pattern, group);
        } catch (IOException e) {
            throw new ElasticSearchIllegalStateException("failed to create pattern tokenizer", e);
        }
    }

View Full Code Here

   * Split the input using configured pattern
   */
  @Override
  public Tokenizer create(final Reader in) {
    try {
      return new PatternTokenizer(in, pattern, group);
    } catch( IOException ex ) {
      throw new RuntimeException("IOException thrown creating PatternTokenizer instance", ex);
    }
  }

View Full Code Here

        this.stopWords = stopWords;
    }


    @Override
    protected TokenStreamComponents createComponents(String s) {
        final Tokenizer tokenizer = new PatternTokenizer(pattern, -1);
        TokenStream stream = tokenizer;
        if (lowercase) {
            stream = new LowerCaseFilter(stream);
        }
        if (stopWords != null) {

View Full Code Here

        this.group = settings.getAsInt("group", -1);
    }


    @Override
    public Tokenizer create() {
        return new PatternTokenizer(pattern, group);
    }

View Full Code Here

TOP

Related Classes of org.apache.lucene.analysis.pattern.PatternTokenizer

org.apache.lucene.analysis.pattern.PatternTokenizerFactory

org.elasticsearch.index.analysis.PatternAnalyzer

org.elasticsearch.index.analysis.PatternTokenizerFactory

org.elasticsearch.indices.analysis.IndicesAnalysisService

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.