Package org.terrier.structures.indexing.singlepass.hadoop

Examples of org.terrier.structures.indexing.singlepass.hadoop.BitPostingIndexInputFormat


   
   
    void checkDirectIndexHSplits(Index index, int maxTermId, int numberOfTerms, int documentLengths[], int[] documentPointers)
      throws Exception
    {
      BitPostingIndexInputFormat informat = new BitPostingIndexInputFormat();
      JobConf jc = HadoopPlugin.getJobFactory("testSplits").newJob();
      HadoopUtility.toHConfiguration(index, jc);
      BitPostingIndexInputFormat.setStructures(jc, "direct", "document");
      InputSplit[] splits = informat.getSplits(jc, 2);
     
      TIntHashSet termIds = new TIntHashSet();
     
      long tokens = 0;
      long pointers = 0;
      int docid = 0;
     
      for(InputSplit split : splits)
      {
        RecordReader<IntWritable, IntObjectWrapper<IterablePosting>> rr = informat.getRecordReader(split, jc, null);
        IntWritable key = rr.createKey();
        IntObjectWrapper<IterablePosting> value = rr.createValue();
        while(rr.next(key, value))
        {
          docid = key.get();
View Full Code Here

TOP

Related Classes of org.terrier.structures.indexing.singlepass.hadoop.BitPostingIndexInputFormat

Copyright © 2018 www.massapicom. All rights reserved.
All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.