Examples of org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigStatsOutputSizeReader

Package org.apache.pig.backend.hadoop.executionengine.mapReduceLayer

Examples of org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigStatsOutputSizeReader

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigStatsOutputSizeReader
Interface to implement when you want to customize the way of computing the size of output in PigStats. Since not every storage is file-based (e.g. HBaseStorage), the output size cannot always be computed as the total size of output files. @see FileBasedOutputSizeReader

     * defaults to FileBasedOutputSizeReader.
     * @param sto POStore
     * @param conf configuration
     */
    public static long getOutputSize(POStore sto, Configuration conf) {
        PigStatsOutputSizeReader reader = null;
        String readerNames = conf.get(
                PigStatsOutputSizeReader.OUTPUT_SIZE_READER_KEY,
                FileBasedOutputSizeReader.class.getCanonicalName());


        for (String className : readerNames.split(",")) {
            reader = (PigStatsOutputSizeReader) PigContext.instantiateFuncFromSpec(className);
            if (reader.supports(sto, conf)) {
                LOG.info("using output size reader: " + className);
                try {
                    return reader.getOutputSize(sto, conf);
                } catch (FileNotFoundException e) {
                    LOG.warn("unable to find the output file", e);
                    return -1;
                } catch (IOException e) {
                    LOG.warn("unable to get byte written of the job", e);

View Full Code Here

     * defaults to FileBasedOutputSizeReader.
     * @param sto POStore
     * @param conf configuration
     */
    public static long getOutputSize(POStore sto, Configuration conf) {
        PigStatsOutputSizeReader reader = null;
        String readerNames = conf.get(
                PigStatsOutputSizeReader.OUTPUT_SIZE_READER_KEY,
                FileBasedOutputSizeReader.class.getCanonicalName());


        for (String className : readerNames.split(",")) {
            reader = (PigStatsOutputSizeReader) PigContext.instantiateFuncFromSpec(className);
            if (reader.supports(sto, conf)) {
                LOG.info("using output size reader: " + className);
                try {
                    return reader.getOutputSize(sto, conf);
                } catch (FileNotFoundException e) {
                    LOG.warn("unable to find the output file", e);
                    return -1;
                } catch (IOException e) {
                    LOG.warn("unable to get byte written of the job", e);

View Full Code Here

     * defaults to FileBasedOutputSizeReader.
     * @param sto POStore
     * @param conf configuration
     */
    static long getOutputSize(POStore sto, Configuration conf) {
        PigStatsOutputSizeReader reader = null;
        String readerNames = conf.get(
                PigStatsOutputSizeReader.OUTPUT_SIZE_READER_KEY,
                FileBasedOutputSizeReader.class.getCanonicalName());


        for (String className : readerNames.split(",")) {
            reader = (PigStatsOutputSizeReader) PigContext.instantiateFuncFromSpec(className);
            if (reader.supports(sto)) {
                LOG.info("using output size reader: " + className);
                try {
                    return reader.getOutputSize(sto, conf);
                } catch (FileNotFoundException e) {
                    LOG.warn("unable to find the output file", e);
                    return -1;
                } catch (IOException e) {
                    LOG.warn("unable to get byte written of the job", e);

View Full Code Here

TOP

Related Classes of org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigStatsOutputSizeReader

org.apache.pig.tools.pigstats.JobStats

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.