Examples of it.unimi.dsi.fastutil.io.FastBufferedInputStream

it.unimi.dsi.fastutil.io.FastBufferedInputStream
sun.com/bugdatabase/view_bug.do?bug_id=6222822">bug 6222822; don't be fooled by the “closed, fixed” label), this class peeks at the underlying stream and if it is {@link System#in} it usesrepeated reads instead of calling {@link InputStream#skip(long)} on the underlying stream; moreover,skips and reads are tried alternately, so to guarantee that skipping less bytes than requested can be caused only by reaching the end of file.
This class keeps also track of the number of bytes read so far, so to be able to implemented {@link MeasurableStream#position()}independently of underlying input stream.
This class has limited support for {@linkplain #readLine(byte[],int,int,EnumSet) “reading a line”}(whatever that means) from the underlying input stream. You can choose the set of {@linkplain FastBufferedInputStream.LineTerminator line terminators} thatdelimit lines.

Warning: Since fastutil 6.0.0, this class detects a implementations of {@link MeasurableStream} instead of subclasses MeasurableInputStream (which is deprecated). @since 4.4


    final boolean openStream = fbis == null;
    if ( openStream ){
      f = Arrays.binarySearch( firstDocument, index );
      if ( f < 0 ) f = -f - 2;
      fbis = new FastBufferedInputStream( new FileInputStream( file[ f ] ) );
    }


    long start = pointers.get( f ).getLong( index - firstDocument[ f ] );
    fbis.position( start );
    final long end = pointers.get( f ).getLong( index - firstDocument[ f ] + 1 );

View Full Code Here

  }




  public Document getDocument( InputStream rawContent, Reference2ObjectMap<Enum<?>,Object> metadata ) throws IOException {
    
    final FastBufferedInputStream fbis = new FastBufferedInputStream( rawContent );
    int startedHeader = 0; // 0 == false, 1 == true, 2 === header started and uri collected
    boolean foundDocNo = false;
    
    int l = fbis.readLine( buffer );
    if ( l < 0 ) throw new EOFException();
    if ( ! TRECDocumentCollection.equals( buffer, l, DOC_OPEN  ) ) throw new IllegalStateException ( "Document does not start with <DOC>: " + new String( buffer, 0, l ) );
    
    while ( ( l = fbis.readLine( buffer ) ) != -1 ) {
      if ( !foundDocNo && startsWith( buffer, l, DOCNO_OPEN ) ) {
        foundDocNo = true;
        metadata.put( MetadataKeys.TITLE, new String( buffer, DOCNO_OPEN.length, l - ( DOCNO_OPEN.length + DOCNO_CLOSE.length ) ) );
      }

View Full Code Here

     */
    @Override
    protected void parseContent(final int fileIndex, InputStream is, final DataOutputStream metadataStream) throws IOException {
        LOGGER.debug("Processing files %d (%s)", fileIndex, files[fileIndex]);
        checkBuffers();
        FastBufferedInputStream fbis = new FastBufferedInputStream(is,
                bufferSize);
        final boolean debugEnabled = LOGGER.isDebugEnabled();


        EventHandler handler = new EventHandler() {
            @Override

View Full Code Here

public class CatTRECDocument extends AbstractTask {
    static final Logger logger = Logger.getLogger(CatTRECDocument.class);


    @Override
    public int execute() throws Throwable {
        FastBufferedInputStream in = new FastBufferedInputStream(System.in);


        byte[] buffer = new byte[8192];
        byte[] docnoBuffer = new byte[512];


        Charset charset = Charset.forName("ISO-8859-1");

View Full Code Here

      int max = min + step;


      System.out.println (start);
      l.output("START " + inFile);


      final FastBufferedInputStream in =
    new FastBufferedInputStream(new FileInputStream (new File(inFile)));


      GZWarcRecord record = new GZWarcRecord();
      Filter<WarcRecord> filter = Filters.adaptFilterBURL2WarcRecord (new TrueFilter());
      WarcFilteredIterator it = new WarcFilteredIterator(in, record, filter);

View Full Code Here

0 1

TOP

Related Classes of it.unimi.dsi.fastutil.io.FastBufferedInputStream

it.unimi.dsi.mg4j.document.SimpleCompressedDocumentCollection

it.unimi.dsi.mg4j.document.TRECDocumentCollection

it.unimi.dsi.mg4j.document.TRECHeaderDocumentFactory

it.unimi.dsi.mg4j.document.WikipediaDocumentCollection

it.unimi.dsi.mg4j.index.BitStreamHPIndexWriter

it.unimi.dsi.mg4j.index.remote.RemoteIndexServerConnection

it.unimi.dsi.mg4j.index.remote.RemoteInputStream$ServerThread

it.unimi.dsi.mg4j.index.SkipBitStreamIndexWriter

net.bpiwowar.mg4j.extensions.tasks.CatTRECDocument

net.bpiwowar.mg4j.extensions.trec.TRECDocumentCollection

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.