Examples of opennlp.model.TwoPassDataIndexer

opennlp.model.TwoPassDataIndexer
Collecting event and context counts by making two passes over the events. The first pass determines which contexts will be used by the model, and the second pass creates the events in memory containing only the contexts which will be used. This greatly reduces the amount of memory required for storing the events. During the first pass a temporary event file is created which is read during the second pass.

        (opennlp.tools.parser.lang.en.HeadRules) rules, ParserType.TREEINSERT);
  }
  
  @Deprecated
  public static AbstractModel train(opennlp.model.EventStream es, int iterations, int cut) throws java.io.IOException {
    return opennlp.maxent.GIS.trainModel(iterations, new TwoPassDataIndexer(es, cut));
  }

View Full Code Here

    
    EventStream es = new ChunkerEventStream(in, contextGenerator);
    HashSumEventStream hses = new HashSumEventStream(es);
    
    AbstractModel maxentModel = opennlp.maxent.GIS.trainModel(iterations, 
        new TwoPassDataIndexer(hses, cutoff));
    
    manifestInfoEntries.put(BaseModel.TRAINING_EVENTHASH_PROPERTY, 
        hses.calculateHashSum().toString(16));
    
    return new ChunkerModel(lang, maxentModel, manifestInfoEntries);

View Full Code Here

       featureGenerator = createFeatureGenerator();
     
     EventStream eventStream = new NameFinderEventStream(samples, type,
         new DefaultNameContextGenerator(featureGenerator));
     HashSumEventStream hses = new HashSumEventStream(eventStream);
     AbstractModel nameFinderModel = GIS.trainModel(iterations, new TwoPassDataIndexer(hses, cutoff));
     
     manifestInfoEntries.put(BaseModel.TRAINING_EVENTHASH_PROPERTY, 
         hses.calculateHashSum().toString(16));
     
     return new TokenNameFinderModel(languageCode, nameFinderModel,

View Full Code Here

     return NameFinderME.train(languageCode, type, samples, resources, 100, 5);
   }
   
  @Deprecated
  public static GISModel train(EventStream es, int iterations, int cut) throws IOException {
    return GIS.trainModel(iterations, new TwoPassDataIndexer(es, cut));
  }

View Full Code Here

   *
   * @return the new model
   */
   @Deprecated
  public static AbstractModel train(DocumentCategorizerEventStream eventStream) throws IOException {
    return GIS.trainModel(100, new TwoPassDataIndexer(eventStream, 5));
  }

View Full Code Here

      throws IOException {
    
    Map<String, String> manifestInfoEntries = new HashMap<String, String>();
    ModelUtil.addCutoffAndIterations(manifestInfoEntries, cutoff, iterations);
    
    AbstractModel model = GIS.trainModel(iterations, new TwoPassDataIndexer(
        new DocumentCategorizerEventStream(samples, featureGenerators), cutoff));
    
    return new DoccatModel(languageCode, model, manifestInfoEntries);
  }

View Full Code Here

0 1 2

TOP

Related Classes of opennlp.model.TwoPassDataIndexer

com.tamingtext.qa.AnswerTypeClassifier

opennlp.maxent.MaxentPrepAttachTest

opennlp.maxent.quasinewton.QNTrainerTest

opennlp.perceptron.PerceptronPrepAttachTest

opennlp.tools.chunker.ChunkerME

opennlp.tools.doccat.DocumentCategorizerME

opennlp.tools.namefind.NameFinderME

opennlp.tools.parser.chunking.Parser

opennlp.tools.parser.treeinsert.Parser

opennlp.tools.postag.POSTaggerTrainer

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.