Examples of org.apache.lucene.analysis.cn.ChineseTokenizer

org.apache.lucene.analysis.cn.ChineseTokenizer

Tokenize Chinese text as individual chinese characters.

The difference between ChineseTokenizer and CJKTokenizer is that they have different token parsing logic.

For example, if the Chinese text "C1C2C3C4" is to be indexed:

The tokens returned from ChineseTokenizer are C1, C2, C3, C4.
The tokens returned from the CJKTokenizer are C1C2, C2C3, C3C4.

Therefore the index created by CJKTokenizer is much larger.

The problem is that when searching for C1, C1C2, C1C3, C4C2, C1C2C3 ... the ChineseTokenizer works, but the CJKTokenizer will not work.

@version 1.0

    super.init(args);
    warnDeprecated("Use StandardTokenizerFactory instead.");
  }
  
  public ChineseTokenizer create(Reader in) {
    return new ChineseTokenizer(in);
  }

 */
@Deprecated
public class ChineseTokenizerFactory extends TokenizerFactory {
  
  public ChineseTokenizer create(Reader in) {
    return new ChineseTokenizer(in);
  }


import org.apache.lucene.analysis.cn.ChineseTokenizer;


public class ChineseTokenizerFactory extends BaseTokenizerFactory {
  public ChineseTokenizer create(Reader in) {
    return new ChineseTokenizer( in );
  }

Related Classes of org.apache.lucene.analysis.cn.ChineseTokenizer

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.