Examples of org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer

Package org.apache.lucene.analysis.cn.smart

Examples of org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer

org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer
kipedia.org/wiki/Hidden_Markov_Model">Hidden Markov Model. A large training corpus was used to calculate Chinese word frequency probability.

This analyzer requires a dictionary to provide statistical data. SmartChineseAnalyzer has an included dictionary out-of-box.

The included dictionary data is from ICTCLAS1.0. Thanks to ICTCLAS for their hard work, and for contributing the data under the Apache 2 License!

WARNING: The status of the analyzers/smartcn analysis.cn.smart package is experimental. The APIs and file formats introduced here might change in the future and will not be supported anymore in such a case.

    }
    System.out.println("Length = " + testString.length() + "  | " +testString.getBytes().length);
    
    StringReader reader = new StringReader(testString);
    
    SmartChineseAnalyzer ss = new SmartChineseAnalyzer(Version.LUCENE_30);  
    
    
    try {
      Thread.sleep(5000);
    } catch (InterruptedException e1) {
      // TODO Auto-generated catch block
      e1.printStackTrace();
    }
    
    long begin = System.currentTimeMillis();   
    
    TokenStream tokenStream = ss.tokenStream("", reader);
    try {
      while(tokenStream.incrementToken()){
//        TermAttribute termAttribute = tokenStream.getAttribute(TermAttribute.class);
//        System.out.println(termAttribute.toString());

View Full Code Here


    @Inject
    public SmartChineseAnalyzerProvider(Index index, @IndexSettings Settings indexSettings, Environment env, @Assisted String name, @Assisted Settings settings) {
        super(index, indexSettings, name, settings);


        analyzer = new SmartChineseAnalyzer(SmartChineseAnalyzer.getDefaultStopSet());
    }

View Full Code Here

    @Inject
    public SmartChineseIndicesAnalysis(Settings settings, IndicesAnalysisService indicesAnalysisService) {
        super(settings);


        // Register smartcn analyzer
        indicesAnalysisService.analyzerProviderFactories().put("smartcn", new PreBuiltAnalyzerProviderFactory("smartcn", AnalyzerScope.INDICES, new SmartChineseAnalyzer()));


        // Register smartcn_tokenizer tokenizer
        indicesAnalysisService.tokenizerFactories().put("smartcn_tokenizer", new PreBuiltTokenizerFactoryFactory(new TokenizerFactory() {
            @Override
            public String name() {

View Full Code Here

TOP

Related Classes of org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer

org.apache.lucene.analysis.cn.smart.SentenceTokenizer

org.apache.lucene.analysis.cn.smart.WordTokenFilter

org.apache.lucene.analysis.core.StopFilter

org.apache.lucene.analysis.en.PorterStemFilter

org.apache.lucene.analysis.PorterStemFilter

org.apache.lucene.analysis.StopFilter

org.apache.lucene.analysis.Tokenizer

org.apache.lucene.analysis.TokenStream

org.elasticsearch.index.analysis.SmartChineseAnalyzerProvider

org.elasticsearch.indices.analysis.smartcn.SmartChineseIndicesAnalysis

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.