Examples of TokenFilter

edu.ucla.sspace.text.TokenFilter
A utility for asserting what tokens are valid and invalid within a stream of tokens. A filter may be either inclusive or exclusive.
An inclusive filter will accept only those tokens with which it was initialized. For an example, an inclusive filter initialized with all of the words in the english dictionary would exclude all misspellings or foreign words in a token stream.
An exclusive filter will aceept only those tokens that are not in set with which it was initialized. An exclusive filter is often used with a list of common words that should be excluded, which is also known as a "stop list."
{@code TokenFilter} instances may be combined into a linear chain of filters.This allows for a highly configurable filter to be made from mulitple rules. Chained filters are created in a linear order and each filter must accept the token for the last filter to return {@code}. If the any of the earlier filters return {@code false}, then the token is not accepted.
This class also provides a static utility function {@link #loadFromSpecification(String) loadFromSpecification} for initializing achain of filters from a text configuration. This is intended to facility command-line tools that want to provide easily configurable filters. An example configuration might look like: include=top-tokens.txt:test-words.txt,exclude=stop-words.txt @see FilteredIterator
org.apache.lucene.analysis.TokenFilter
A TokenFilter is a TokenStream whose input is another token stream.
This is an abstract class. NOTE: subclasses must override {@link #incrementToken()} if the new TokenStream API is usedand {@link #next(Token)} or {@link #next()} if the oldTokenStream API is used.
See {@link TokenStream}
org.apache.tools.ant.filters.TokenFilter
This splits up input into tokens and passes the tokens to a sequence of filters. @author Peter Reilly @since Ant 1.6 @see BaseFilterReader @see ChainableReader @see DynamicConfigurator
org.apache.uima.conceptMapper.support.tokens.TokenFilter

Examples of org.apache.lucene.analysis.TokenFilter

    check("कठिन", "कठिन");
  }
  
  private void check(String input, String output) throws IOException {
    Tokenizer tokenizer = new MockTokenizer(new StringReader(input), MockTokenizer.WHITESPACE, false);
    TokenFilter tf = new HindiStemFilter(tokenizer);
    assertTokenStreamContents(tf, new String[] { output });
  }

View Full Code Here

Examples of org.apache.lucene.analysis.TokenFilter

    check("ত্‍", "ৎ");
  }
  
  private void check(String input, String output) throws IOException {
    Tokenizer tokenizer = new MockTokenizer(new StringReader(input), MockTokenizer.WHITESPACE, false);;
    TokenFilter tf = new IndicNormalizationFilter(tokenizer);
    assertTokenStreamContents(tf, new String[] { output });
  }

View Full Code Here

Examples of org.apache.lucene.analysis.TokenFilter


      @Override
      public TokenStream tokenStream(String fieldName, Reader reader) {
        MockTokenizer tokenizer = new MockTokenizer(reader, MockTokenizer.SIMPLE, true);
        tokenizer.setEnableChecks(false); // disable workflow checking as we forcefully close() in exceptional cases.
        return new TokenFilter(tokenizer) {
          private int count = 0;


          @Override
          public boolean incrementToken() throws IOException {
            if (count++ == 5) {

View Full Code Here

Examples of org.apache.lucene.analysis.TokenFilter

    Analyzer analyzer = new Analyzer(new Analyzer.PerFieldReuseStrategy()) {
      @Override
      protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
        Tokenizer tokenizer = new MockTokenizer(reader);
        if (fieldName.contains("payloadsFixed")) {
          TokenFilter filter = new MockFixedLengthPayloadFilter(new Random(0), tokenizer, 1);
          return new TokenStreamComponents(tokenizer, filter);
        } else if (fieldName.contains("payloadsVariable")) {
          TokenFilter filter = new MockVariableLengthPayloadFilter(new Random(0), tokenizer);
          return new TokenStreamComponents(tokenizer, filter);
        } else {
          return new TokenStreamComponents(tokenizer);
        }
      }

View Full Code Here

Examples of org.apache.lucene.analysis.TokenFilter

  public void testInvalidOffsets() throws Exception {
    Analyzer analyzer = new Analyzer() {
      @Override
      protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
        Tokenizer tokenizer = new MockTokenizer(reader, MockTokenizer.WHITESPACE, false);
        TokenFilter filters = new ASCIIFoldingFilter(tokenizer);
        filters = new EdgeNGramTokenFilter(filters, EdgeNGramTokenFilter.Side.FRONT, 2, 15);
        return new TokenStreamComponents(tokenizer, filters);
      }
    };
    assertAnalyzesTo(analyzer, "mosfellsbær",

View Full Code Here

Examples of org.apache.lucene.analysis.TokenFilter

          
          final Object args[] = newFilterArgs(random, spec.stream, ctor.getParameterTypes());
          if (broken(ctor, args)) {
            continue;
          }
          final TokenFilter flt = createComponent(ctor, args, descr);
          if (flt != null) {
            spec.offsetsAreCorrect &= !brokenOffsets(ctor, args);
            spec.stream = flt;
            break;
          }

View Full Code Here

Examples of org.apache.lucene.analysis.TokenFilter

    Analyzer analyzer = new Analyzer(Analyzer.PER_FIELD_REUSE_STRATEGY) {
      @Override
      protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
        Tokenizer tokenizer = new MockTokenizer(reader);
        if (fieldName.contains("payloadsFixed")) {
          TokenFilter filter = new MockFixedLengthPayloadFilter(new Random(0), tokenizer, 1);
          return new TokenStreamComponents(tokenizer, filter);
        } else if (fieldName.contains("payloadsVariable")) {
          TokenFilter filter = new MockVariableLengthPayloadFilter(new Random(0), tokenizer);
          return new TokenStreamComponents(tokenizer, filter);
        } else {
          return new TokenStreamComponents(tokenizer);
        }
      }

View Full Code Here

Examples of org.apache.lucene.analysis.TokenFilter


      @Override
      public TokenStreamComponents createComponents(String fieldName, Reader reader) {
        MockTokenizer tokenizer = new MockTokenizer(reader, MockTokenizer.SIMPLE, true);
        tokenizer.setEnableChecks(false); // disable workflow checking as we forcefully close() in exceptional cases.
        return new TokenStreamComponents(tokenizer, new TokenFilter(tokenizer) {
          private int count = 0;


          @Override
          public boolean incrementToken() throws IOException {
            if (count++ == 5) {

View Full Code Here

Examples of org.apache.lucene.analysis.TokenFilter


  public void testElision() throws Exception {
    String test = "Plop, juste pour voir l'embrouille avec O'brian. M'enfin.";
    Tokenizer tokenizer = new StandardTokenizer(TEST_VERSION_CURRENT, newAttributeFactory(), new StringReader(test));
    CharArraySet articles = new CharArraySet(TEST_VERSION_CURRENT, asSet("l", "M"), false);
    TokenFilter filter = new ElisionFilter(tokenizer, articles);
    List<String> tas = filter(filter);
    assertEquals("embrouille", tas.get(4));
    assertEquals("O'brian", tas.get(6));
    assertEquals("enfin", tas.get(7));
  }

View Full Code Here

Examples of org.apache.tools.ant.filters.TokenFilter

        LineContainsRegExp lcre = new LineContainsRegExp();
        RegularExpression regexp = new RegularExpression();
        regexp.setPattern("^import .+;");
        lcre.addConfiguredRegexp(regexp);
        filterChain.add(lcre);
        TokenFilter tf = new TokenFilter();
        TokenFilter.ReplaceRegex rre = new TokenFilter.ReplaceRegex();
        rre.setPattern("import (.+);.*");
        rre.setReplace("\\1");
        tf.add(rre);
        filterChain.add(tf);
        concat.addFilterChain(filterChain);
    }

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.