Examples of org.apache.hadoop.io.compress.bzip2.CBZip2InputStream

Package org.apache.hadoop.io.compress.bzip2

Examples of org.apache.hadoop.io.compress.bzip2.CBZip2InputStream

org.apache.tools.bzip2.CBZip2InputStream
An input stream that decompresses from the BZip2 format (without the file header chars) to be read as any other stream.
The decompression requires large amounts of memory. Thus you should call the {@link #close() close()} method as soon as possible, to forceCBZip2InputStream to release the allocated memory. See {@link CBZip2OutputStream CBZip2OutputStream} for information about memoryusage.

CBZip2InputStream reads bytes from the compressed source stream via the single byte {@link java.io.InputStream#read() read()} method exclusively.Thus you should consider to use a buffered source stream.

This Ant code was enhanced so that it can de-compress blocks of bzip2 data. Current position in the stream is an important statistic for Hadoop. For example in LineRecordReader, we solely depend on the current position in the stream to know about the progess. The notion of position becomes complicated for compressed files. The Hadoop splitting is done in terms of compressed file. But a compressed file deflates to a large amount of data. So we have handled this problem in the following way. On object creation time, we find the next block start delimiter. Once such a marker is found, the stream stops there (we discard any read compressed data in this process) and the position is updated (i.e. the caller of this class will find out the stream location). At this point we are ready for actual reading (i.e. decompression) of data. The subsequent read calls give out data. The position is updated when the caller of this class has read off the current block + 1 bytes. In between the block reading, position is not updated. (We can only update the postion on block boundaries).

Instances of this class are not threadsafe.


    private void internalReset() throws IOException {
      if (needsReset) {
        needsReset = false;
        writeStreamHeader();
        this.output = new CBZip2OutputStream(out);
      }
    }

View Full Code Here


    private void internalReset() throws IOException {
      if (needsReset) {
        needsReset = false;
        writeStreamHeader();
        this.output = new CBZip2OutputStream(out);
      }
    }

View Full Code Here


    private void internalReset() throws IOException {
      if (needsReset) {
        needsReset = false;
        writeStreamHeader();
        this.output = new CBZip2OutputStream(out);
      }
    }

View Full Code Here


    private void internalReset() throws IOException {
      if (needsReset) {
        needsReset = false;
        writeStreamHeader();
        this.output = new CBZip2OutputStream(out);
      }
    }

View Full Code Here


    private void internalReset() throws IOException {
      if (needsReset) {
        needsReset = false;
        writeStreamHeader();
        this.output = new CBZip2OutputStream(out);
      }
    }

View Full Code Here


    private void internalReset() throws IOException {
      if (needsReset) {
        needsReset = false;
        writeStreamHeader();
        this.output = new CBZip2OutputStream(out);
      }
    }

View Full Code Here

        xhtml.startDocument();


        // At the end we want to close the bzip2 stream to release any associated
        // resources, but the underlying document stream should not be closed
        InputStream gzip =
            new CBZip2InputStream(new CloseShieldInputStream(stream));
        try {
            Metadata entrydata = new Metadata();
            String name = metadata.get(Metadata.RESOURCE_NAME_KEY);
            if (name != null) {
                if (name.endsWith(".tbz")) {
                    name = name.substring(0, name.length() - 4) + ".tar";
                } else if (name.endsWith(".tbz2")) {
                    name = name.substring(0, name.length() - 5) + ".tar";
                } else if (name.endsWith(".bz")) {
                    name = name.substring(0, name.length() - 3);
                } else if (name.endsWith(".bz2")) {
                    name = name.substring(0, name.length() - 4);
                }
                entrydata.set(Metadata.RESOURCE_NAME_KEY, name);
            }
            parseEntry(gzip, xhtml, entrydata);
        } finally {
            gzip.close();
        }


        xhtml.endDocument();
    }

View Full Code Here

        final int b2 = is.read();
        if (b1 != 'B' || b2 != 'Z')
        {
            throw new FileSystemException("vfs.provider.compressedFile/not-a-compressedFile-file.error", name);
        }
        return new CBZip2InputStream(is);
    }

View Full Code Here

            if (sourcefile.getName().endsWith(".bz2")) {
                int b = is.read();
                if (b != 'B') throw new IOException("Invalid bz2 content.");
                b = is.read();
                if (b != 'Z') throw new IOException("Invalid bz2 content.");
                is = new CBZip2InputStream(is);
            } else if (sourcefile.getName().endsWith(".gz")) {
                is = new GZIPInputStream(is);
            }
            BufferedReader r = new BufferedReader(new java.io.InputStreamReader(is, "UTF-8"), 4 * 1024 * 1024);
            String t;

View Full Code Here

                throw new Exception("Invalid bz2 content.");
            }           
            
            int read = 0;
            final byte[] data = new byte[1024];
            final CBZip2InputStream zippedContent = new CBZip2InputStream(source);        
            
            tempFile = File.createTempFile("bunzip","tmp");
            tempFile.deleteOnExit();
            
            // creating a temp file to store the uncompressed data
            final FileOutputStream out = new FileOutputStream(tempFile);
            
            // reading gzip file and store it uncompressed
            while((read = zippedContent.read(data, 0, 1024)) != -1) {
                out.write(data, 0, read);
            }
            zippedContent.close();
            out.close();
            
            // creating a new parser class to parse the unzipped content
            docs = TextParser.parseSource(location, null, null, tempFile);
        } catch (final Exception e) {

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.hadoop.io.compress.bzip2.CBZip2InputStream

at.molindo.utils.io.StreamUtils

cc.twittertools.index.IndexStatuses

com.endgame.binarypig.util.BuildSequenceFileFromArchive

com.google.gwt.ant.taskdefs.TarCat$UntarCompressionMethod

com.google.refine.importing.ImportingUtilities

com.packetloop.packetpig.loaders.pcap.PcapRecordReader

com.packetloop.packetpig.loaders.pcap.StreamingPcapRecordReader

edu.umd.cloud9.collection.wikipedia.WikipediaPagesBz2InputStream

er.woinstaller.archiver.XarFile

mwt.wow.mpq.BZip2Reader

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.