List of aggregateStats() Examples

Examples of aggregateStats()

org.apache.hadoop.hive.ql.stats.StatsAggregator.aggregateStats()
This method aggregates a given statistic from all tasks (partial stats). After aggregation, this method also automatically removes all records that have been aggregated. @param keyPrefix a prefix of the keys used in StatsPublisher to publish stats. Any rows that starts with the same prefix will be aggregated. For example, if the StatsPublisher uses the following compound key to publish stats: the output directory name (unique per FileSinkOperator) + the partition specs (only for dynamic partitions) + taskID (last component of task file) The keyPrefix for aggregation could be first 2 components. This will aggregates stats across all tasks for each partition. @param statType a string noting the key to be published. Ex: "numRows". @return a string representation of a long value, null if there are any error/exception.

Examples of org.apache.hadoop.hive.ql.stats.StatsAggregator.aggregateStats()

          tableSize += fileStatus[i].getLen();
        }
        tblStats.setSize(tableSize);


        // In case of a non-partitioned table, the key for stats temporary store is "rootDir"
        String rows = statsAggregator.aggregateStats(work.getAggKey(), StatsSetupConst.ROW_COUNT);
        if (rows != null) {
          tblStats.setNumRows(Long.parseLong(rows));
        }
      } else {
        // Partitioned table:

View Full Code Here

Examples of org.apache.hadoop.hive.ql.stats.StatsAggregator.aggregateStats()

          PartitionStatistics newPartStats = new PartitionStatistics();


          // In that case of a partition, the key for stats temporary store is "rootDir/[dynamic_partition_specs/]%"
          String partitionID = work.getAggKey() + Warehouse.makePartPath(partn.getSpec());


          String rows = statsAggregator.aggregateStats(partitionID, StatsSetupConst.ROW_COUNT);
          if (rows != null) {
            newPartStats.setNumRows(Long.parseLong(rows));
          }


          fileSys = partn.getPartitionPath().getFileSystem(conf);

View Full Code Here

Examples of org.apache.hadoop.hive.ql.stats.StatsAggregator.aggregateStats()

      fillStatMap("400", "3000");
      assertTrue(statsPublisher.publishStat("file_00001", stats));




      // aggregate existing stats
      String rows0 = statsAggregator.aggregateStats("file_00000", StatsSetupConst.ROW_COUNT);
      assertEquals("200", rows0);
      String usize0 = statsAggregator.aggregateStats("file_00000",
          StatsSetupConst.RAW_DATA_SIZE);
      assertEquals("1000", usize0);

View Full Code Here

Examples of org.apache.hadoop.hive.ql.stats.StatsAggregator.aggregateStats()




      // aggregate existing stats
      String rows0 = statsAggregator.aggregateStats("file_00000", StatsSetupConst.ROW_COUNT);
      assertEquals("200", rows0);
      String usize0 = statsAggregator.aggregateStats("file_00000",
          StatsSetupConst.RAW_DATA_SIZE);
      assertEquals("1000", usize0);


      String rows1 = statsAggregator.aggregateStats("file_00001", StatsSetupConst.ROW_COUNT);
      assertEquals("400", rows1);

View Full Code Here

Examples of org.apache.hadoop.hive.ql.stats.StatsAggregator.aggregateStats()

      assertEquals("200", rows0);
      String usize0 = statsAggregator.aggregateStats("file_00000",
          StatsSetupConst.RAW_DATA_SIZE);
      assertEquals("1000", usize0);


      String rows1 = statsAggregator.aggregateStats("file_00001", StatsSetupConst.ROW_COUNT);
      assertEquals("400", rows1);
      String usize1 = statsAggregator.aggregateStats("file_00001",
          StatsSetupConst.RAW_DATA_SIZE);
      assertEquals("3000", usize1);

View Full Code Here

Examples of org.apache.hadoop.hive.ql.stats.StatsAggregator.aggregateStats()

          StatsSetupConst.RAW_DATA_SIZE);
      assertEquals("1000", usize0);


      String rows1 = statsAggregator.aggregateStats("file_00001", StatsSetupConst.ROW_COUNT);
      assertEquals("400", rows1);
      String usize1 = statsAggregator.aggregateStats("file_00001",
          StatsSetupConst.RAW_DATA_SIZE);
      assertEquals("3000", usize1);


      // close connections
      assertTrue(statsPublisher.closeConnection());

View Full Code Here

Examples of org.apache.hadoop.hive.ql.stats.StatsAggregator.aggregateStats()

      assertTrue(statsPublisher.publishStat("file_00001_a", stats));
      fillStatMap("500", "4000");
      assertTrue(statsPublisher.publishStat("file_00001_b", stats));


      // aggregate existing stats
      String rows0 = statsAggregator.aggregateStats("file_00000", StatsSetupConst.ROW_COUNT);
      assertEquals("500", rows0);
      String usize0 = statsAggregator.aggregateStats("file_00000",
          StatsSetupConst.RAW_DATA_SIZE);
      assertEquals("3000", usize0);

View Full Code Here

Examples of org.apache.hadoop.hive.ql.stats.StatsAggregator.aggregateStats()

      assertTrue(statsPublisher.publishStat("file_00001_b", stats));


      // aggregate existing stats
      String rows0 = statsAggregator.aggregateStats("file_00000", StatsSetupConst.ROW_COUNT);
      assertEquals("500", rows0);
      String usize0 = statsAggregator.aggregateStats("file_00000",
          StatsSetupConst.RAW_DATA_SIZE);
      assertEquals("3000", usize0);


      String rows1 = statsAggregator.aggregateStats("file_00001", StatsSetupConst.ROW_COUNT);
      assertEquals("900", rows1);

View Full Code Here

Examples of org.apache.hadoop.hive.ql.stats.StatsAggregator.aggregateStats()

      assertEquals("500", rows0);
      String usize0 = statsAggregator.aggregateStats("file_00000",
          StatsSetupConst.RAW_DATA_SIZE);
      assertEquals("3000", usize0);


      String rows1 = statsAggregator.aggregateStats("file_00001", StatsSetupConst.ROW_COUNT);
      assertEquals("900", rows1);
      String usize1 = statsAggregator.aggregateStats("file_00001",
          StatsSetupConst.RAW_DATA_SIZE);
      assertEquals("7000", usize1);

View Full Code Here

Examples of org.apache.hadoop.hive.ql.stats.StatsAggregator.aggregateStats()

          StatsSetupConst.RAW_DATA_SIZE);
      assertEquals("3000", usize0);


      String rows1 = statsAggregator.aggregateStats("file_00001", StatsSetupConst.ROW_COUNT);
      assertEquals("900", rows1);
      String usize1 = statsAggregator.aggregateStats("file_00001",
          StatsSetupConst.RAW_DATA_SIZE);
      assertEquals("7000", usize1);


      // aggregate non-existent stats
      String rowsX = statsAggregator.aggregateStats("file_00002", StatsSetupConst.ROW_COUNT);

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.