Examples of CoreLabelTokenFactory

edu.stanford.nlp.process.CoreLabelTokenFactory
Constructs {@link CoreLabel}s from Strings optionally with beginning and ending (character after the end) offset positions in an original text. The makeToken method will put the token in the OriginalTextAnnotation AND TextAnnotation keys (2 places!), and optionally records begin and position after offsets in BeginPositionAnnotation and EndPositionAnnotation. If the tokens are built in PTBTokenizer with an "invertible" tokenizer, you will also get a BeforeAnnotation and for the last token an AfterAnnotation.You can also get an empty CoreLabel token @author Anna Rafferty @author Sonal Gupta (now implements CoreTokenFactory, you can make tokens using many options)

Examples of edu.stanford.nlp.process.CoreLabelTokenFactory

      atbOptions.put(option, "true");
    }
  }


  public static ArabicTokenizer<CoreLabel> newArabicTokenizer(Reader r, Properties lexerProperties) {
    return new ArabicTokenizer<CoreLabel>(r, new CoreLabelTokenFactory(), lexerProperties);
  }

View Full Code Here

Examples of edu.stanford.nlp.process.CoreLabelTokenFactory

    protected final LexedTokenFactory<T> factory;


    protected Properties lexerProperties = new Properties();


    public static TokenizerFactory<CoreLabel> newTokenizerFactory() {
      return new ArabicTokenizerFactory<CoreLabel>(new CoreLabelTokenFactory());
    }

View Full Code Here

Examples of edu.stanford.nlp.process.CoreLabelTokenFactory

  public MUCMentionExtractor(Dictionaries dict, Properties props, Semantics semantics) throws Exception {
    super(dict, semantics);
    String fileName = props.getProperty(Constants.MUC_PROP);
    fileContents = IOUtils.slurpFile(fileName);
    currentOffset = 0;
    tokenizerFactory = PTBTokenizer.factory(new CoreLabelTokenFactory(false), "");
    stanfordProcessor = loadStanfordProcessor(props);
  }

View Full Code Here

Examples of edu.stanford.nlp.process.CoreLabelTokenFactory

    protected final LexedTokenFactory<T> factory;
    protected Properties lexerProperties = new Properties();
    protected boolean splitCompoundOption = false;


    public static TokenizerFactory<CoreLabel> newTokenizerFactory() {
      return new FrenchTokenizerFactory<CoreLabel>(new CoreLabelTokenFactory());
    }

View Full Code Here

Examples of edu.stanford.nlp.process.CoreLabelTokenFactory

    this.flags = flags;


    try {
      this.featureFactory = (FeatureFactory) Class.forName(flags.featureFactory).newInstance();
      if (flags.tokenFactory == null)
        tokenFactory = (CoreTokenFactory<IN>) new CoreLabelTokenFactory();
      else
        this.tokenFactory = (CoreTokenFactory<IN>) Class.forName(flags.tokenFactory).newInstance();
    } catch (Exception e) {
      throw new RuntimeException(e);
    }

View Full Code Here

Examples of edu.stanford.nlp.process.CoreLabelTokenFactory


        // Output bag
        DataBag bagOfTokens = bagFactory.newDefaultBag();
                
        StringReader textInput = new StringReader(input.get(0).toString());
        PTBTokenizer ptbt = new PTBTokenizer(textInput, new CoreLabelTokenFactory(), "");


        for (CoreLabel label; ptbt.hasNext(); ) {
          label = (CoreLabel)ptbt.next();
          Tuple termText = tupleFactory.newTuple(label.toString());
          bagOfTokens.add(termText);

View Full Code Here

0 1

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.