Examples of it.unimi.dsi.mg4j.index.BitStreamHPIndexWriter

it.unimi.dsi.mg4j.index.BitStreamHPIndexWriter
Writes a bitstream-based high-performance index. The comments about offsets in the documentation of {@link BitStreamIndexWriter} apply here, too.
The difference between indices generated by this class and those generated by {@link BitStreamIndexWriter} lie in the levelof interleaving. Indices generated by this class have positions in a separate stream (similarly to Lucene), and a compulsory skip structure (an extension of that used by a {@link BitStreamIndexWriter}) that indexes both the main index file and the positions file. This can result in major performance improvement in the resolution of position-based operators (e.g., phrases) and in the evaluation of {@linkplain VignaScorer proximity-based scorers}. Since the overhead due to the additional skip structure and to the separate positions stream is negligible, indices generated by this class are the default in MG4J.
Presently, indices generated by this class cannot carry payloads: you must use a {@link BitStreamIndexWriter}in that case. Moreover, only nonparametric indices can be used for positions (this limitation rules out {@link Coding#GOLOMB} and {@link Coding#INTERPOLATIVE}). @author Sebastiano Vigna @since 1.2

    if ( ! metadataOnly ) {
      if ( interleaved ) {
        if ( ! skips ) indexWriter = new BitStreamIndexWriter( outputBasename, numberOfDocuments, true, writerFlags );
        else indexWriter = new SkipBitStreamIndexWriter( outputBasename, numberOfDocuments, true, skipBufferSize, writerFlags, skips ? ( quantum < 0 ? 0 : quantum ) : -1, skips ? height : -1 );
      }
      else indexWriter = new BitStreamHPIndexWriter( outputBasename, numberOfDocuments, true, skipBufferSize, writerFlags, quantum < 0 ? 0 : quantum, height );
      if ( skips && quantum < 0 ) variableQuantumIndexWriter = (VariableQuantumIndexWriter)indexWriter;
      //else indexWriter = new SqrtSkipIndexWriter( outputBasename, numberOfDocuments, true, writerFlags );
    }
  }

View Full Code Here

    skips |= ! interleaved;
    if ( skips && ( quantum <= 0 || height < 0 ) ) throw new IllegalArgumentException( "You must specify a positive quantum and a nonnegative height (variable quanta are not available when partitioning documentally)." );
    
    for ( int i = 0; i < numIndices; i++ ) {
      String name = localBasename[ i ]; 
      if ( ! interleaved ) indexWriter[ i ] = new BitStreamHPIndexWriter( localBasename[ i ], strategy.numberOfDocuments( i ), true, skipBufferSize, writerFlags, quantum, height );
      else if ( ! skips ) indexWriter[ i ] = new BitStreamIndexWriter( localBasename[ i ], strategy.numberOfDocuments( i ), true, writerFlags );
      else indexWriter[ i ] = new SkipBitStreamIndexWriter( localBasename[ i ], strategy.numberOfDocuments( i ), true, skipBufferSize, writerFlags, quantum, height );
      
      if ( haveCounts ) localGlobCounts[ i ] = new OutputBitStream( name + DiskBasedIndex.GLOBCOUNTS_EXTENSION );
      localFrequencies[ i ] = new OutputBitStream( name + DiskBasedIndex.FREQUENCIES_EXTENSION );

View Full Code Here

TOP

Related Classes of it.unimi.dsi.mg4j.index.BitStreamHPIndexWriter

it.unimi.dsi.fastutil.ints.Int2IntRBTreeMap

it.unimi.dsi.fastutil.io.FastBufferedInputStream

it.unimi.dsi.fastutil.io.FastByteArrayOutputStream

it.unimi.dsi.io.OutputBitStream

it.unimi.dsi.lang.MutableString

it.unimi.dsi.mg4j.tool.Combine

it.unimi.dsi.mg4j.tool.PartitionDocumentally

it.unimi.dsi.util.Properties

java.io.FileInputStream

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.