Examples of org.apache.lucene.analysis.miscellaneous.LimitTokenCountFilter

org.apache.lucene.analysis.miscellaneous.LimitTokenCountFilter

This TokenFilter limits the number of tokens while indexing. It is a replacement for the maximum field length setting inside {@link org.apache.lucene.index.IndexWriter}.

By default, this filter ignores any tokens in the wrapped {@code TokenStream}once the limit has been reached, which can result in {@code reset()} being called prior to {@code incrementToken()} returning {@code false}. For most {@code TokenStream} implementations this should be acceptable, and faster then consuming the full stream. If you are wrapping a {@code TokenStream} which requires that the full stream of tokens be exhausted in order to function properly, use the {@link #LimitTokenCountFilter(TokenStream,int,boolean) consumeAllTokens} option.

    consumeAllTokens = getBoolean(CONSUME_ALL_TOKENS_KEY, false);
  }


  @Override
  public TokenStream create(TokenStream input) {
    return new LimitTokenCountFilter(input, maxTokenCount, consumeAllTokens);
  }

        this.consumeAllTokens = settings.getAsBoolean("consume_all_tokens", DEFAULT_CONSUME_ALL_TOKENS);
    }


    @Override
    public TokenStream create(TokenStream tokenStream) {
        return new LimitTokenCountFilter(tokenStream, maxTokenCount, consumeAllTokens);
    }

Examples of org.apache.lucene.analysis.miscellaneous.LimitTokenCountFilter

Related Classes of org.apache.lucene.analysis.miscellaneous.LimitTokenCountFilter