Examples of org.apache.hadoop.io.compress.bzip2.CBZip2InputStream$Data

Package org.apache.hadoop.io.compress.bzip2

Examples of org.apache.hadoop.io.compress.bzip2.CBZip2InputStream$Data

org.apache.tika.parser.pkg.bzip2.CBZip2InputStream
An input stream that decompresses from the BZip2 format (without the file header chars) to be read as any other stream.
The decompression requires large amounts of memory. Thus you should call the {@link #close() close()} method as soon as possible, to forceCBZip2InputStream to release the allocated memory. See {@link CBZip2OutputStream CBZip2OutputStream} for information about memoryusage.

CBZip2InputStream reads bytes from the compressed source stream via the single byte {@link java.io.InputStream#read() read()} method exclusively.Thus you should consider to use a buffered source stream.

This Ant code was enhanced so that it can de-compress blocks of bzip2 data. Current position in the stream is an important statistic for Hadoop. For example in LineRecordReader, we solely depend on the current position in the stream to know about the progess. The notion of position becomes complicated for compressed files. The Hadoop splitting is done in terms of compressed file. But a compressed file deflates to a large amount of data. So we have handled this problem in the following way. On object creation time, we find the next block start delimiter. Once such a marker is found, the stream stops there (we discard any read compressed data in this process) and the position is updated (i.e. the caller of this class will find out the stream location). At this point we are ready for actual reading (i.e. decompression) of data. The subsequent read calls give out data. The position is updated when the caller of this class has read off the current block + 1 bytes. In between the block reading, position is not updated. (We can only update the postion on block boundaries).

Instances of this class are not threadsafe.

        else if(lowerName.endsWith(".tar.bz") || lowerName.endsWith(".tar.bz2") || lowerName.endsWith(".tbz"))
        {
          FileInputStream is = new FileInputStream(archive);
          is.read(); // read 'B'
          is.read(); // read 'Z'
          archiveInputStream = new ArchiveStreamFactory().createArchiveInputStream("tar", new CBZip2InputStream(is));
        }
        else if(lowerName.endsWith(".tar"))
        {
          archiveInputStream = new ArchiveStreamFactory().createArchiveInputStream("tar", new FileInputStream(archive));
        }

View Full Code Here


    public BZip2CompressionOutputStream(OutputStream out)
        throws IOException {
      super(out);
      writeStreamHeader();
      this.output = new CBZip2OutputStream(out);
    }

View Full Code Here


    private void internalReset() throws IOException {
      if (needsReset) {
        needsReset = false;
        writeStreamHeader();
        this.output = new CBZip2OutputStream(out);
      }
    }

View Full Code Here


    private void internalReset() throws IOException {
      if (needsReset) {
        needsReset = false;
        writeStreamHeader();
        this.output = new CBZip2OutputStream(out);
      }
    }

View Full Code Here


    private void internalReset() throws IOException {
      if (needsReset) {
        needsReset = false;
        writeStreamHeader();
        this.output = new CBZip2OutputStream(out);
      }
    }

View Full Code Here


    private void internalReset() throws IOException {
      if (needsReset) {
        needsReset = false;
        writeStreamHeader();
        this.output = new CBZip2OutputStream(out);
      }
    }

View Full Code Here


    private void internalReset() throws IOException {
      if (needsReset) {
        needsReset = false;
        writeStreamHeader();
        this.output = new CBZip2OutputStream(out);
      }
    }

View Full Code Here


    private void internalReset() throws IOException {
      if (needsReset) {
        needsReset = false;
        writeStreamHeader();
        this.output = new CBZip2OutputStream(out);
      }
    }

View Full Code Here


    private void internalReset() throws IOException {
      if (needsReset) {
        needsReset = false;
        writeStreamHeader();
        this.output = new CBZip2OutputStream(out);
      }
    }

View Full Code Here

        xhtml.startDocument();


        // At the end we want to close the bzip2 stream to release any associated
        // resources, but the underlying document stream should not be closed
        InputStream gzip =
            new CBZip2InputStream(new CloseShieldInputStream(stream));
        try {
            Metadata entrydata = new Metadata();
            String name = metadata.get(Metadata.RESOURCE_NAME_KEY);
            if (name != null) {
                if (name.endsWith(".tbz")) {
                    name = name.substring(0, name.length() - 4) + ".tar";
                } else if (name.endsWith(".tbz2")) {
                    name = name.substring(0, name.length() - 5) + ".tar";
                } else if (name.endsWith(".bz")) {
                    name = name.substring(0, name.length() - 3);
                } else if (name.endsWith(".bz2")) {
                    name = name.substring(0, name.length() - 4);
                }
                entrydata.set(Metadata.RESOURCE_NAME_KEY, name);
            }
            parseEntry(gzip, xhtml, entrydata);
        } finally {
            gzip.close();
        }


        xhtml.endDocument();
    }

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.hadoop.io.compress.bzip2.CBZip2InputStream$Data

at.molindo.utils.io.StreamUtils

cc.twittertools.index.IndexStatuses

codec.asn1.ASN1ObjectIdentifier

codec.pkcs12.AuthenticatedSafe

com.endgame.binarypig.util.BuildSequenceFileFromArchive

com.google.gwt.ant.taskdefs.TarCat$UntarCompressionMethod

com.google.gwt.user.client.ui.RadioButton

com.google.refine.importing.ImportingUtilities

com.inmethod.grid.common.AbstractGridRow

com.inmethod.grid.common.AttachPrelightBehavior

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.