List of reusableTokenStream() Examples

Examples of reusableTokenStream()

org.apache.lucene.analysis.Analyzer.reusableTokenStream()
Creates a TokenStream that is allowed to be re-used from the previous time that the same thread called this method. Callers that do not need to use more than one TokenStream at the same time from this analyzer should use this method for better performance.
org.apache.lucene.analysis.ReusableAnalyzerBase.reusableTokenStream()
This method uses {@link #createComponents(String,Reader)} to obtain aninstance of {@link TokenStreamComponents}. It returns the sink of the components and stores the components internally. Subsequent calls to this method will reuse the previously stored components if and only if the {@link TokenStreamComponents#reset(Reader)} method returnedtrue. Otherwise a new instance of {@link TokenStreamComponents} is created. @param fieldName the name of the field the created TokenStream is used for @param reader the reader the streams source reads from

Examples of org.apache.lucene.analysis.Analyzer.reusableTokenStream()

          stringReader.init(stringValue);
          reader = stringReader;
        }
        
        // Tokenize field
        stream = analyzer.reusableTokenStream(field.name(), reader);
      }


      // reset the TokenStream to the first token
      stream.reset();

View Full Code Here

Examples of org.apache.lucene.analysis.Analyzer.reusableTokenStream()


    if (!TokenizerChain.class.isInstance(analyzer)) {


      TokenStream tokenStream = null;
      try {
        tokenStream = analyzer.reusableTokenStream(context.getFieldName(), new StringReader(value));
      } catch (IOException e) {
        throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e);
      }
      NamedList<List<NamedList>> namedList = new NamedList<List<NamedList>>();
      namedList.add(tokenStream.getClass().getName(), convertTokensToNamedLists(analyzeTokenStream(tokenStream), context));

View Full Code Here

Examples of org.apache.lucene.analysis.Analyzer.reusableTokenStream()

    public TokenStream reusableTokenStream(String fieldName, Reader reader)
            throws IOException {
        if (indexingConfig != null) {
            Analyzer propertyAnalyzer = indexingConfig.getPropertyAnalyzer(fieldName);
            if (propertyAnalyzer != null) {
                return propertyAnalyzer.reusableTokenStream(fieldName, reader);
            }
        }
        return defaultAnalyzer.reusableTokenStream(fieldName, reader);
    }
}

View Full Code Here

Examples of org.apache.lucene.analysis.Analyzer.reusableTokenStream()

    public TokenStream reusableTokenStream(String fieldName, Reader reader)
            throws IOException {
        if (indexingConfig != null) {
            Analyzer propertyAnalyzer = indexingConfig.getPropertyAnalyzer(fieldName);
            if (propertyAnalyzer != null) {
                return propertyAnalyzer.reusableTokenStream(fieldName, reader);
            }
        }
        return defaultAnalyzer.reusableTokenStream(fieldName, reader);
    }
}

View Full Code Here

Examples of org.apache.lucene.analysis.Analyzer.reusableTokenStream()


    if (!TokenizerChain.class.isInstance(analyzer)) {


      TokenStream tokenStream = null;
      try {
        tokenStream = analyzer.reusableTokenStream(context.getFieldName(), new StringReader(value));
        tokenStream.reset();
      } catch (IOException e) {
        throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e);
      }
      NamedList<List<NamedList>> namedList = new NamedList<List<NamedList>>();

View Full Code Here

Examples of org.apache.lucene.analysis.Analyzer.reusableTokenStream()

                Tokenizer t = new WhitespaceTokenizer(Lucene.VERSION, reader);
                return new TokenStreamComponents(t, new UniqueTokenFilter(t));
            }
        };


        TokenStream test = analyzer.reusableTokenStream("test", new StringReader("this test with test"));
        CharTermAttribute termAttribute = test.addAttribute(CharTermAttribute.class);
        assertThat(test.incrementToken(), equalTo(true));
        assertThat(termAttribute.toString(), equalTo("this"));


        assertThat(test.incrementToken(), equalTo(true));

View Full Code Here

Examples of org.apache.lucene.analysis.Analyzer.reusableTokenStream()

                Tokenizer t = new WhitespaceTokenizer(Lucene.VERSION, reader);
                return new TokenStreamComponents(t, new TruncateTokenFilter(t, 3));
            }
        };


        TokenStream test = analyzer.reusableTokenStream("test", new StringReader("a bb ccc dddd eeeee"));
        CharTermAttribute termAttribute = test.addAttribute(CharTermAttribute.class);
        assertThat(test.incrementToken(), equalTo(true));
        assertThat(termAttribute.toString(), equalTo("a"));


        assertThat(test.incrementToken(), equalTo(true));

View Full Code Here

Examples of org.apache.lucene.analysis.Analyzer.reusableTokenStream()


        // Logic similar to QueryParser#getFieldQuery


        TokenStream source;
        try {
            source = analyzer.reusableTokenStream(field, new FastStringReader(text));
            source.reset();
        } catch (IOException e) {
            source = analyzer.tokenStream(field, new FastStringReader(text));
        }
        CachingTokenFilter buffer = new CachingTokenFilter(source);

View Full Code Here

Examples of org.apache.lucene.analysis.Analyzer.reusableTokenStream()

                    ArrayList<TextFragment> fragsList = new ArrayList<TextFragment>();
                    try {
                        for (Object textToHighlight : textsToHighlight) {
                            String text = textToHighlight.toString();
                            Analyzer analyzer = context.mapperService().documentMapper(hitContext.hit().type()).mappers().indexAnalyzer();
                            TokenStream tokenStream = analyzer.reusableTokenStream(mapper.names().indexName(), new FastStringReader(text));
                            TextFragment[] bestTextFragments = highlighter.getBestTextFragments(tokenStream, text, false, numberOfFragments);
                            for (TextFragment bestTextFragment : bestTextFragments) {
                                if (bestTextFragment != null && bestTextFragment.getScore() > 0) {
                                    fragsList.add(bestTextFragment);
                                }

View Full Code Here

Examples of org.apache.lucene.analysis.Analyzer.reusableTokenStream()

        }


        List<AnalyzeResponse.AnalyzeToken> tokens = Lists.newArrayList();
        TokenStream stream = null;
        try {
            stream = analyzer.reusableTokenStream(field, new FastStringReader(request.text()));
            stream.reset();
            CharTermAttribute term = stream.addAttribute(CharTermAttribute.class);
            PositionIncrementAttribute posIncr = stream.addAttribute(PositionIncrementAttribute.class);
            OffsetAttribute offset = stream.addAttribute(OffsetAttribute.class);
            TypeAttribute type = stream.addAttribute(TypeAttribute.class);

View Full Code Here

0 1 2

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.