Examples of org.apache.hadoop.metrics.MetricsRecord

org.apache.hadoop.metrics.MetricsRecord
A named and optionally tagged set of records to be sent to the metrics system.
A record name identifies the kind of data to be reported. For example, a program reporting statistics relating to the disks on a computer might use a record name "diskStats".
A record has zero or more tags. A tag has a name and a value. To continue the example, the "diskStats" record might use a tag named "diskName" to identify a particular disk. Sometimes it is useful to have more than one tag, so there might also be a "diskType" with value "ide" or "scsi" or whatever.
A record also has zero or more metrics. These are the named values that are to be reported to the metrics system. In the "diskStats" example, possible metric names would be "diskPercentFull", "diskPercentBusy", "kbReadPerSecond", etc.
The general procedure for using a MetricsRecord is to fill in its tag and metric values, and then call update() to pass the record to the client library. Metric data is not immediately sent to the metrics system each time that update() is called. An internal table is maintained, identified by the record name. This table has columns corresponding to the tag and the metric names, and rows corresponding to each unique set of tag values. An update either modifies an existing row in the table, or adds a new row with a set of tag values that are different from all the other rows. Note that if there are no tags, then there can be at most one row in the table.
Once a row is added to the table, its data will be sent to the metrics system on every timer period, whether or not it has been updated since the previous timer period. If this is inappropriate, for example if metrics were being reported by some transient object in an application, the remove() method can be used to remove the row and thus stop the data from being sent.
Note that the update() method is atomic. This means that it is safe for different threads to be updating the same metric. More precisely, it is OK for different threads to call update() on MetricsRecord instances with the same set of tag names and tag values. Different threads should not use the same MetricsRecord instance at the same time.

        poolToJobCounters.put(pool, poolCounters);
      }
      accumulateCounters(poolCounters, counters);


      if (!poolToMetricsRecord.containsKey(pool)) {
        MetricsRecord poolRecord = context.createRecord("pool-" + pool);
        poolToMetricsRecord.put(pool, poolRecord);
      }
    }
  }

View Full Code Here


    // The gets + puts below are OK because only one thread is doing it.
    for (PoolGroupSchedulable poolGroup : poolGroupManager.getPoolGroups()) {
      int poolGroupSessions = 0;
      for (PoolSchedulable pool : poolGroup.getPools()) {
        MetricsRecord poolRecord =
            poolInfoToMetricsRecord.get(pool.getPoolInfo());
        if (poolRecord == null) {
          poolRecord = metrics.getContext().createRecord(
              "pool-" + pool.getName());
          poolInfoToMetricsRecord.put(pool.getPoolInfo(), poolRecord);
        }


        PoolInfoMetrics poolMetrics = new PoolInfoMetrics(pool.getPoolInfo(),
            type, poolRecord);
        poolMetrics.setCounter(
            MetricName.GRANTED, pool.getGranted());
        poolMetrics.setCounter(
            MetricName.REQUESTED, pool.getRequested());
        poolMetrics.setCounter(
            MetricName.SHARE, (long) pool.getShare());
        poolMetrics.setCounter(
            MetricName.MIN, pool.getMinimum());
        poolMetrics.setCounter(
            MetricName.MAX, pool.getMaximum());
        poolMetrics.setCounter(
            MetricName.WEIGHT, (long) pool.getWeight());
        poolMetrics.setCounter(
            MetricName.SESSIONS, pool.getScheduleQueue().size());
        poolMetrics.setCounter(
            MetricName.STARVING, pool.getStarvingTime(now) / 1000);
        Long averageFirstTypeMs =
            poolInfoAverageFirstWaitMs.get(pool.getPoolInfo());
        poolMetrics.setCounter(MetricName.AVE_FIRST_WAIT_MS,
            (averageFirstTypeMs == null) ?
                0 : averageFirstTypeMs.longValue());


        newPoolNameToMetrics.put(pool.getPoolInfo(), poolMetrics);
        poolGroupSessions += pool.getScheduleQueue().size();
      }


      MetricsRecord poolGroupRecord =
          poolInfoToMetricsRecord.get(poolGroup.getName());
      if (poolGroupRecord == null) {
        poolGroupRecord = metrics.getContext().createRecord(
            "poolgroup-" + poolGroup.getName());
        poolInfoToMetricsRecord.put(poolGroup.getPoolInfo(), poolGroupRecord);

View Full Code Here

  @Test
  public void testPushMetric() {
    final MetricsHistogram h = new MetricsHistogram("testHistogram", null);
    genRandomData(h);


    MetricsRecord mr = mock(MetricsRecord.class);
    h.pushMetric(mr);
    
    verify(mr).setMetric("testHistogram_num_ops", 10000L);
    verify(mr).setMetric(eq("testHistogram_min"), anyLong());
    verify(mr).setMetric(eq("testHistogram_max"), anyLong());

View Full Code Here

    metricsRecord.setMetric("adhoc_avg_first_map_wait_ms", nonConfiguredAvgFirstMapWaitTime);
    metricsRecord.setMetric("adhoc_avg_first_reduce_wait_ms", nonConfiguredAvgFirstReduceWaitTime);
  }


  private void submitPoolMetrics(PoolInfo info) {
    MetricsRecord record = poolToMetricsRecord.get(info.poolName);
    if (record == null) {
      record = MetricsUtil.createRecord(context, "pool-" + info.poolName);
      FairScheduler.LOG.info("Create metrics record for pool:" + info.poolName);
      poolToMetricsRecord.put(info.poolName, record);
    }
    record.setMetric("min_map", info.minMaps);
    record.setMetric("min_reduce", info.minReduces);
    record.setMetric("max_map", info.maxMaps);
    record.setMetric("max_reduce", info.maxReduces);
    record.setMetric("running_map", info.runningMaps);
    record.setMetric("running_reduce", info.runningReduces);
    record.setMetric("runnable_map", info.runnableMaps);
    record.setMetric("runnable_reduce", info.runnableReduces);
    record.setMetric("inited_tasks", info.initedTasks);
    record.setMetric("max_inited_tasks", info.maxInitedTasks);
    int runningJobs = info.runningJobs;
    record.setMetric("avg_first_map_wait_ms",
        (runningJobs == 0) ? 0 : info.totalFirstMapWaitTime / runningJobs);
    record.setMetric("avg_first_reduce_wait_ms",
        (runningJobs == 0) ? 0 : info.totalFirstReduceWaitTime / runningJobs);
  }

View Full Code Here

  }


  
  @Override
  protected MetricsRecord newRecord(String recordName) {
    MetricsRecord record = super.newRecord(recordName);
    if (records.isEmpty() || records.contains(recordName)) {
      // Create MBean to expose this record
      // Only if this record is to be exposed through JMX
      getOrCreateMBean(recordName);
    }

View Full Code Here

        poolToJobCounters.put(pool, poolCounters);
      }
      accumulateCounters(poolCounters, counters);


      if (!poolToMetricsRecord.containsKey(pool)) {
        MetricsRecord poolRecord = context.createRecord("pool-" + pool);
        poolToMetricsRecord.put(pool, poolRecord);
      }
    }
  }

View Full Code Here


    // The gets + puts below are OK because only one thread is doing it.
    for (PoolGroupSchedulable poolGroup : poolGroupManager.getPoolGroups()) {
      int poolGroupSessions = 0;
      for (PoolSchedulable pool : poolGroup.getPools()) {
        MetricsRecord poolRecord =
            poolInfoToMetricsRecord.get(pool.getPoolInfo());
        if (poolRecord == null) {
          poolRecord = metrics.getContext().createRecord(
              "pool-" + pool.getName());
          poolInfoToMetricsRecord.put(pool.getPoolInfo(), poolRecord);
        }


        PoolInfoMetrics poolMetrics = new PoolInfoMetrics(pool.getPoolInfo(),
            type, poolRecord);
        poolMetrics.setCounter(
            MetricName.GRANTED, pool.getGranted());
        poolMetrics.setCounter(
            MetricName.REQUESTED, pool.getRequested());
        poolMetrics.setCounter(
            MetricName.SHARE, (long) pool.getShare());
        poolMetrics.setCounter(
            MetricName.MIN, pool.getMinimum());
        poolMetrics.setCounter(
            MetricName.MAX, pool.getMaximum());
        poolMetrics.setCounter(
            MetricName.WEIGHT, (long) pool.getWeight());
        poolMetrics.setCounter(
            MetricName.SESSIONS, pool.getScheduleQueue().size());
        poolMetrics.setCounter(
            MetricName.STARVING, pool.getStarvingTime(now) / 1000);
        Long averageFirstTypeMs =
            poolInfoAverageFirstWaitMs.get(pool.getPoolInfo());
        poolMetrics.setCounter(MetricName.AVE_FIRST_WAIT_MS,
            (averageFirstTypeMs == null) ?
                0 : averageFirstTypeMs.longValue());


        newPoolNameToMetrics.put(pool.getPoolInfo(), poolMetrics);
        poolGroupSessions += pool.getScheduleQueue().size();
      }


      MetricsRecord poolGroupRecord =
          poolInfoToMetricsRecord.get(poolGroup.getName());
      if (poolGroupRecord == null) {
        poolGroupRecord = metrics.getContext().createRecord(
            "poolgroup-" + poolGroup.getName());
        poolInfoToMetricsRecord.put(poolGroup.getPoolInfo(), poolGroupRecord);

View Full Code Here

      IPCLoggerChannelMetrics m = REGISTRY.get(name);
      if (m != null) {
        m.setChannel(ch);
      } else {
        MetricsContext metricsContext = MetricsUtil.getContext("dfs");
        MetricsRecord metricsRecord = MetricsUtil.createRecord(metricsContext,
            "loggerchannel");
        metricsRecord.setTag("loggerchannel", name);
        m = new IPCLoggerChannelMetrics(ch, metricsRecord, name);
        metricsContext.registerUpdater(m);
        REGISTRY.put(name, m);
      }
      return m;

View Full Code Here

TOP

Related Classes of org.apache.hadoop.metrics.MetricsRecord

org.apache.hadoop.corona.SchedulerForType

org.apache.hadoop.hbase.metrics.TestMetricsHistogram

org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics

org.apache.hadoop.mapred.FairSchedulerMetricsInst

org.apache.hadoop.mapred.ProxyJobTracker

org.apache.hadoop.metrics.jmx.JMXContext

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.