Examples of it.unimi.dsi.mg4j.util.SemiExternalOffsetList

it.unimi.dsi.mg4j.util.SemiExternalOffsetList
Provides semi-external random access to offsets of an {@link it.unimi.dsi.mg4j.index.Index index}.
This class is a semi-external {@link it.unimi.dsi.fastutil.longs.LongList} thatMG4J uses as default for accessing term offsets.
When the number of terms in the index grows, storing each offset as a long in an array can consume hundred of megabytes of memory, and most of this memory is wasted, as it is occupied by offsets of hapax legomena (terms occurring just once in the collection). Instead, this class accesses offsets in their compressed forms, and provides entry points for random access to each offset. At construction time, entry points are computed with a certain step, which is the number of offsets accessible from each entry point, or, equivalently, the maximum number of offsets that will be necessary to read to access a given offset.
Warning: This class is not thread safe, and needs to be synchronised to be used in a multithreaded environment. @author Fabien Campagne @author Sebastiano Vigna

    if ( payload == null && randomAccess ) {
      int offsetStep = queryProperties != null && queryProperties.get( UriKeys.OFFSETSTEP ) != null ? Integer.parseInt( queryProperties.get( UriKeys.OFFSETSTEP ) ) : DEFAULT_OFFSET_STEP;
      
      if ( offsetStep < 0 ) { // Memory-mapped
        offsetStep  = -offsetStep;
        offsets = LongLists.synchronize( new SemiExternalOffsetList( 
            new InputBitStream( ByteBufferInputStream.map( new FileInputStream( basename + DiskBasedIndex.OFFSETS_EXTENSION ).getChannel(), MapMode.READ_ONLY ) ),
            offsetStep, numberOfTerms + 1 ) );
      }
      else {
        offsets = offsetStep == 0? 
            DiskBasedIndex.readOffsets( basename + DiskBasedIndex.OFFSETS_EXTENSION, numberOfTerms ) :
              LongLists.synchronize( new SemiExternalOffsetList( new InputBitStream( basename + DiskBasedIndex.OFFSETS_EXTENSION, 1024 ), offsetStep, numberOfTerms + 1 ) );
      }
      if ( offsets.size() != numberOfTerms + 1 ) throw new IllegalStateException( "The length of the offset list (" + offsets.size() + ") is not equal to the number of terms plus one (" + numberOfTerms + " + 1)" );
    }
    else offsets = null;

View Full Code Here

    public void testSemiExternalOffsetListGammaCoding() throws IOException {


    long[] offsets = { 10, 300, 450, 650, 1000, 1290, 1699 };
    LongList listOffsets = new LongArrayList( offsets );


    SemiExternalOffsetList list = new SemiExternalOffsetList( buildInputStream( listOffsets ), 1, listOffsets.size() );
    for ( int i = 0; i < offsets.length; ++i ) {
      assertEquals( ( "test failed for index: " + i ), offsets[ i ], list.getLong( i ) );
    }


    list = new SemiExternalOffsetList( buildInputStream( listOffsets ), 2, listOffsets.size() );
    for ( int i = 0; i < offsets.length; ++i ) {
      assertEquals( ( "test failed for index: " + i ), offsets[ i ], list.getLong( i ) );
    }


    list = new SemiExternalOffsetList( buildInputStream( listOffsets ), 4, listOffsets.size() );
    for ( int i = 0; i < offsets.length; ++i ) {
      assertEquals( ( "test failed for index: " + i ), offsets[ i ], list.getLong( i ) );
    }


    list = new SemiExternalOffsetList( buildInputStream( listOffsets ), 7, listOffsets.size() );
    for ( int i = 0; i < offsets.length; ++i ) {
      assertEquals( ( "test failed for index: " + i ), offsets[ i ], list.getLong( i ) );
    }
    
    list = new SemiExternalOffsetList( buildInputStream( listOffsets ), 8, listOffsets.size() );
    for ( int i = 0; i < offsets.length; ++i ) {
      assertEquals( ( "test failed for index: " + i ), offsets[ i ], list.getLong( i ) );
    }
    }

View Full Code Here

    public void testEmptySemiExternalOffsetListGammaCoding() throws IOException {


    long[] offsets = {  };
    LongList listOffsets = new LongArrayList( offsets );


    new SemiExternalOffsetList( buildInputStream( listOffsets ), 1, listOffsets.size() );
    assertTrue( true );
    }

View Full Code Here

TOP

Related Classes of it.unimi.dsi.mg4j.util.SemiExternalOffsetList

it.unimi.dsi.mg4j.index.DiskBasedIndex

it.unimi.dsi.mg4j.util.SemiExternalOffsetListTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.