Examples of HunspellStemFilter

org.apache.lucene.analysis.hunspell.HunspellStemFilter
TokenFilter that uses hunspell affix rules and words to stem tokens. Since hunspell supports a word having multiple stems, this filter can emit multiple tokens for each consumed token
Note: This filter is aware of the {@link KeywordAttribute}. To prevent certain terms from being passed to the stemmer {@link KeywordAttribute#isKeyword()} should be set to truein a previous {@link TokenStream}. Note: For including the original term as well as the stemmed version, see {@link org.apache.lucene.analysis.miscellaneous.KeywordRepeatFilterFactory}

Examples of org.apache.lucene.analysis.hunspell.HunspellStemFilter

        }
    }


    @Override
    public TokenStream create(TokenStream tokenStream) {
        return new HunspellStemFilter(tokenStream, dictionary, dedup);
    }

View Full Code Here

Examples of org.apache.lucene.analysis.hunspell.HunspellStemFilter

   *
   * @param tokenStream TokenStream that will be filtered
   * @return HunspellStemFilter that filters the TokenStream 
   */
  public TokenStream create(TokenStream tokenStream) {
    return new HunspellStemFilter(tokenStream, dictionary);
  }

View Full Code Here

Examples of org.apache.lucene.analysis.hunspell.HunspellStemFilter

   * @param tokenStream TokenStream that will be filtered
   * @return HunspellStemFilter that filters the TokenStream 
   */
  @Override
  public TokenStream create(TokenStream tokenStream) {
    return new HunspellStemFilter(tokenStream, dictionary);
  }

View Full Code Here

Examples of org.apache.lucene.analysis.hunspell.HunspellStemFilter

  
  /** Simple test for KeywordAttribute */
  public void testKeywordAttribute() throws IOException {
    MockTokenizer tokenizer = new MockTokenizer(new StringReader("lucene is awesome"));
    tokenizer.setEnableChecks(true);
    HunspellStemFilter filter = new HunspellStemFilter(tokenizer, dictionary);
    assertTokenStreamContents(filter, new String[]{"lucene", "lucen", "is", "awesome"}, new int[] {1, 0, 1, 1});
    
    // assert with keyword marker
    tokenizer = new MockTokenizer(new StringReader("lucene is awesome"));
    CharArraySet set = new CharArraySet(TEST_VERSION_CURRENT, Arrays.asList("Lucene"), true);
    filter = new HunspellStemFilter(new SetKeywordMarkerFilter(tokenizer, set), dictionary);
    assertTokenStreamContents(filter, new String[]{"lucene", "is", "awesome"}, new int[] {1, 1, 1});
  }

View Full Code Here

Examples of org.apache.lucene.analysis.hunspell.HunspellStemFilter

  
  /** simple test for longestOnly option */
  public void testLongestOnly() throws IOException {
    MockTokenizer tokenizer = new MockTokenizer(new StringReader("lucene is awesome"));
    tokenizer.setEnableChecks(true);
    HunspellStemFilter filter = new HunspellStemFilter(tokenizer, dictionary, true, true);
    assertTokenStreamContents(filter, new String[]{"lucene", "is", "awesome"}, new int[] {1, 1, 1});
  }

View Full Code Here

Examples of org.apache.lucene.analysis.hunspell.HunspellStemFilter

  public void testRandomStrings() throws Exception {
    Analyzer analyzer = new Analyzer() {
      @Override
      protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
        Tokenizer tokenizer = new MockTokenizer(reader, MockTokenizer.WHITESPACE, false);
        return new TokenStreamComponents(tokenizer, new HunspellStemFilter(tokenizer, dictionary));
      }
    };
    checkRandomData(random(), analyzer, 1000*RANDOM_MULTIPLIER);
  }

View Full Code Here

Examples of org.apache.lucene.analysis.hunspell.HunspellStemFilter

  public void testEmptyTerm() throws IOException {
    Analyzer a = new Analyzer() {
      @Override
      protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
        Tokenizer tokenizer = new KeywordTokenizer(reader);
        return new TokenStreamComponents(tokenizer, new HunspellStemFilter(tokenizer, dictionary));
      }
    };
    checkOneTerm(a, "", "");
  }

View Full Code Here

Examples of org.apache.lucene.analysis.hunspell.HunspellStemFilter

    }
    Analyzer a = new Analyzer() {
      @Override
      protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
        Tokenizer tokenizer = new KeywordTokenizer(reader);
        return new TokenStreamComponents(tokenizer, new HunspellStemFilter(tokenizer, d));
      }
    };
    checkOneTerm(a, "NoChAnGy", "NoChAnGy");
  }

View Full Code Here

Examples of org.apache.lucene.analysis.hunspell.HunspellStemFilter

        longestOnly = settings.getAsBoolean("longest_only", false);
    }


    @Override
    public TokenStream create(TokenStream tokenStream) {
        return new HunspellStemFilter(tokenStream, dictionary, dedup, longestOnly);
    }

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.