Examples of joshua.decoder.ff.lm.buildin_lm.LMGrammarJAVA

joshua.decoder.ff.lm.buildin_lm.LMGrammarJAVA
this class implement (1) read the LM file into a Trie data structure (2) get LM probablity for a given n-grm (3) get equivilent state @author Zhifei Li, @version $LastChangedDate:2008-07-28 18:44:45 -0400 (Mon, 28 Jul 2008) $

    
    SymbolTable vocab = new Vocabulary();
    
    Corpus corpus = suffixArray.getCorpus();
    
    NGramLanguageModel largeLM = new LMGrammarJAVA(
        vocab,
        lmOrder,
        largeArpaLM,
        JoshuaConfiguration.use_left_equivalent_state,
        JoshuaConfiguration.use_right_equivalent_state);
    
    NGramLanguageModel testLM = new LMGrammarJAVA(
        vocab,
        lmOrder,
        testArpaLM,
        JoshuaConfiguration.use_left_equivalent_state,
        JoshuaConfiguration.use_right_equivalent_state);
    
    this.weights = new float[corpus.getNumSentences()];
    
    for (int i=0, n=corpus.getNumSentences(); i<n; i++) {
      Phrase sentence = corpus.getSentence(i);
      int[] words = sentence.getWordIDs();
      double largeProbLM = largeLM.ngramLogProbability(words);
      double testProbLM = testLM.ngramLogProbability(words);
      double ratio = testProbLM - largeProbLM;
      this.weights[i] = (float) ratio;
    }
  }

View Full Code Here

//              JoshuaConfiguration.lm_file,
//              this.symbolTable
//          ));
      
      // using the built-in JAVA implementation of LM, may not be as scalable as SRILM
      this.languageModel = new LMGrammarJAVA(
        this.symbolTable,
        JoshuaConfiguration.lmOrder,
        JoshuaConfiguration.lm_file,
        JoshuaConfiguration.use_left_equivalent_state,
        JoshuaConfiguration.use_right_equivalent_state);

View Full Code Here

      
    } else {
      //p_lm = new LMGrammar_JAVA(g_lm_order, lm_file, use_left_euqivalent_state);
      //big bug: should load the consistent symbol files
      p_symbolTable = new BuildinSymbol(remote_symbol_tbl);
      p_lm = new LMGrammarJAVA((BuildinSymbol)p_symbolTable, g_lm_order, lm_file, use_left_euqivalent_state, use_right_euqivalent_state);
    }
  }

View Full Code Here

TOP

Related Classes of joshua.decoder.ff.lm.buildin_lm.LMGrammarJAVA

joshua.decoder.ff.lm.distributed_lm.LMServer

joshua.decoder.JoshuaDecoder

joshua.prefix_tree.LMAdaptingRuleExtractor

joshua.util.io.LineReader

joshua.util.Regex

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.