Examples of opennlp.ccg.lexicon.Tokenizer.format()

opennlp.ccg.lexicon.Tokenizer.format()
Returns a string for the given list of words, in the format expected by the SRILM tool for factored language models.

          item.setAttribute("info", actualID);
          outRoot.addContent(item);
          signMap.put(actualID, thisParse);
          // Add parsed words as a separate LF element
          Element fullWordsElt = new Element("full-words");
          fullWordsElt.addContent(tokenizer.format(thisParse.getWords()));
          item.addContent(fullWordsElt);
          if (predInfo != null) {
        Element predInfoElt = new Element("pred-info");
        predInfoElt.setAttribute("data", predInfo);
        item.addContent(predInfoElt);

View Full Code Here

                  words.add(Word.createWord("</s>"));
              }
              // write str, add to unique set
              String str = (!withFactors)
                  ? tokenizer.getOrthography(words, semClassReplacement)
                  : tokenizer.format(words, semClassReplacement);
              tOut.println(str);
              unique.add(str);
              System.out.print("."); // indicate progress
          }
          System.out.println();

View Full Code Here

              RegressionInfo.TestItem testItem = tbInfo.getItem(i); 
            if (testItem.numOfParses == 0) continue; // check grammatical
            Sign sign = testItem.sign;
            List<Word> factors = GenerativeSyntacticModel.getFactors(sign);
            for (Word w : factors) {
              tOut.print(tokenizer.format(w));
              tOut.print(" ");
            }
            tOut.println();
              System.out.print("."); // indicate progress
          }

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.