Examples of cc.mallet.pipe.TokenSequenceMatchDataAndTarget

cc.mallet.pipe.TokenSequenceMatchDataAndTarget

Run a regular expression over the text of each token; replace the text with the substring matching one regex group; create a target TokenSequence from the text matching another regex group.

For example, if you have a data file containing one line per token, and the label also appears on that line, you can first get a TokenSequence in which the text of each line is the Token.getText() of each token, then run this pipe, and separate the target information from the data information. For example to process the following,

 BACKGROUND Then PERSON Mr. PERSON Smith BACKGROUND said ...

use new TokenSequenceMatchDataAndTarget (Pattern.compile ("([A-Z]+) (.*)"), 2, 1). @author Andrew McCallum mccallum@cs.umass.edu

  static String toy = "A a\nB b\nC c\nD d\nB b\nC c\n";


  public void testStartState() {
    Pipe p = new SerialPipes(new Pipe[] {
        new LineGroupString2TokenSequence(),
        new TokenSequenceMatchDataAndTarget(Pattern
            .compile("^(\\S+) (.*)"), 2, 1),
        new TokenSequenceParseFeatureString(false), new TokenText(),
        new TokenSequence2FeatureVectorSequence(true, false),
        new Target2LabelSequence(), new PrintInputAndTarget(), });

Related Classes of cc.mallet.pipe.TokenSequenceMatchDataAndTarget

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.