Examples of org.apache.lucene.analysis.LowerCaseTokenizer

org.apache.lucene.analysis.LowerCaseTokenizer
LowerCaseTokenizer performs the function of LetterTokenizer and LowerCaseFilter together. It divides text at non-letters and converts them to lower case. While it is functionally equivalent to the combination of LetterTokenizer and LowerCaseFilter, there is a performance advantage to doing the two tasks at once, hence this (redundant) implementation.
Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces.

  /* (non-Javadoc)
   * @see org.apache.lucene.analysis.Analyzer#tokenStream(java.lang.String, java.io.Reader)
   */
  public TokenStream tokenStream(String arg0, Reader arg1)
  {
    return new SynonymTokenizer(new LowerCaseTokenizer(arg1), synonyms);
  }

View Full Code Here

  /* (non-Javadoc)
   * @see org.apache.lucene.analysis.Analyzer#tokenStream(java.lang.String, java.io.Reader)
   */
  public TokenStream tokenStream(String arg0, Reader arg1)
  {
    return new SynonymTokenizer(new LowerCaseTokenizer(arg1), synonyms);
  }

View Full Code Here

     *               provides access to the input text
     * @return an instance of TokenStream
     */
    @Override
    public TokenStream tokenStream(String fieldName, Reader reader) {
       return new PorterStemFilter(new StopFilter(enablePositionIncements, new LowerCaseTokenizer(reader), stopWords));
    }

View Full Code Here

    assertEquals("Wrong payload for the target " + target + ": " + b[0], (byte) target, b[0]);
  }


  private static class PayloadAnalyzer extends Analyzer {
    public TokenStream tokenStream(String fieldName, Reader reader) {
      return new PayloadFilter(new LowerCaseTokenizer(reader));
    }

View Full Code Here

/**
 * @version $Id: LowerCaseTokenizerFactory.java 591158 2007-11-01 22:37:42Z hossman $
 */
public class LowerCaseTokenizerFactory extends BaseTokenizerFactory {
  public LowerCaseTokenizer create(Reader input) {
    return new LowerCaseTokenizer( input );
  }

View Full Code Here

    assertEquals("Wrong payload for the target " + target + ": " + b[0], (byte) target, b[0]);
  }


  private static class PayloadAnalyzer extends Analyzer {
    public TokenStream tokenStream(String fieldName, Reader reader) {
      return new PayloadFilter(new LowerCaseTokenizer(reader));
    }

View Full Code Here

  
  public static class QPTestAnalyzer extends Analyzer {


    /** Filters LowerCaseTokenizer with StopFilter. */
    public final TokenStream tokenStream(String fieldName, Reader reader) {
      return new QPTestFilter(new LowerCaseTokenizer(reader));
    }

View Full Code Here


  public static class QPTestAnalyzer extends Analyzer {


    /** Filters LowerCaseTokenizer with StopFilter. */
    public final TokenStream tokenStream(String fieldName, Reader reader) {
      return new QPTestFilter(new LowerCaseTokenizer(reader));
    }

View Full Code Here

  }


  class PayloadAnalyzer extends Analyzer {


    public TokenStream tokenStream(String fieldName, Reader reader) {
      TokenStream result = new LowerCaseTokenizer(reader);
      result = new PayloadFilter(result, fieldName);
      return result;
    }

View Full Code Here


  private class PayloadAnalyzer extends Analyzer {




    public TokenStream tokenStream(String fieldName, Reader reader) {
      TokenStream result = new LowerCaseTokenizer(reader);
      result = new PayloadFilter(result, fieldName);
      return result;
    }

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.lucene.analysis.LowerCaseTokenizer

BugAnalyzer.PorterStemAnalyzer

org.apache.lucene.analysis.br.TestBrazilianStemmer

org.apache.lucene.analysis.de.TestGermanAnalyzer

org.apache.lucene.index.TestIndexWriterExceptions

org.apache.lucene.index.TestMultiLevelSkipList$PayloadAnalyzer

org.apache.lucene.queryParser.precedence.TestPrecedenceQueryParser$QPTestAnalyzer

org.apache.lucene.queryParser.QueryParserTestBase$QPTestAnalyzer

org.apache.lucene.queryParser.standard.TestQPHelper$QPTestAnalyzer

org.apache.lucene.queryParser.standard.TestQueryParserWrapper$QPTestAnalyzer

org.apache.lucene.queryParser.TestQueryParser$QPTestAnalyzer

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.