Examples of edu.stanford.nlp.process.Tokenizer

edu.stanford.nlp.process.Tokenizer
Tokenizers break up text into individual Objects. These objects may be Strings, Words, or other Objects. A Tokenizer extends the Iterator interface, but provides a lookahead operation peek(). An implementation of this interface is expected to have a constructor that takes a single argument, a Reader. @author Teg Grenager (grenager@stanford.edu)




  public static void main(String[] args) throws IOException {


    Reader in = new FileReader(args[0]);
    Tokenizer st = new NegraPennTokenizer(in);


    while (st.hasNext()) {
      String s = (String) st.next();
      System.out.println(s);
    }
  }

View Full Code Here

TOP

Related Classes of edu.stanford.nlp.process.Tokenizer

edu.stanford.nlp.trees.international.negra.NegraPennTokenizer

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.