Examples of org.terrier.structures.indexing.singlepass.hadoop.HadoopRunWriter

org.terrier.structures.indexing.singlepass.hadoop.HadoopRunWriter

RunWriter for the MapReduce indexer. Provides functionality to write term posting lists out to the map task outputcollector during a MapReduce indexing job. Map and flush numbers are also passed with the posting list to allow for docids to be corrected later from side-effect files. @author Richard McCreadie and Craig Macdonald

  protected void forceFlush() throws IOException
  {
    //logger.info("Map "+mapTaskID+", flush requested, containing "+numberOfDocsSinceFlush+" documents, flush "+flushNo);
    if (mp == null)
      throw new IOException("Map flushed before any documents were indexed");
    mp.finish(new HadoopRunWriter(outputPostingListCollector, mapTaskID, splitnum, flushNo));
    RunData.writeInt(currentId);
    if (currentReporter != null)
      currentReporter.incrCounter(Counters.INDEXER_FLUSHES, 1);
    System.gc();
    createMemoryPostings();

Related Classes of org.terrier.structures.indexing.singlepass.hadoop.HadoopRunWriter

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.