Package com.atilika.kuromoji.viterbi

Examples of com.atilika.kuromoji.viterbi.ViterbiLattice

50
51
52
53
54
55
56
57
58
        this.viterbiSearcher = new ViterbiSearcher(this.viterbiBuilder, mode, dictionaries.getCosts(), dictionaries.getUnknownDictionary());
    this.formatter = new ViterbiFormatter(dictionaries.getCosts());
  }

  public String debugTokenize(String text) {
    ViterbiLattice lattice = this.viterbiBuilder.build(text);
    List<ViterbiNode> bestPath = this.viterbiSearcher.search(lattice);
    return this.formatter.format(lattice, bestPath);
  }
View Full Code Here
168
169
170
171
172
173
174
175
176
177
178
     * @return list of Token
     */
    private List<Token> doTokenize(int offset, String sentence) {
        ArrayList<Token> result = new ArrayList<Token>();

        ViterbiLattice lattice = viterbiBuilder.build(sentence);
        List<ViterbiNode> bestPath = viterbiSearcher.search(lattice);
        for (ViterbiNode node : bestPath) {
            int wordId = node.getWordId();
            if (node.getType() == ViterbiNode.Type.KNOWN && wordId == -1) { // Do not include BOS/EOS
                continue;
View Full Code Here
TOP

Related Classes of com.atilika.kuromoji.viterbi.ViterbiLattice

  • com.atilika.kuromoji.AbstractTokenizer
  • com.atilika.kuromoji.DebugTokenizer
    • Copyright © 2018 www.massapicom. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.