Examples of org.apache.hadoop.conf.Configuration$Resource

org.apache.hadoop.conf.Configuration
Provides access to configuration parameters.
Resources

Configurations are specified by resources. A resource contains a set of name/value pairs as XML data. Each resource is named by either a String or by a {@link Path}. If named by a String, then the classpath is examined for a file with that name. If named by a Path, then the local filesystem is examined directly, without referring to the classpath.
Unless explicitly turned off, Hadoop by default specifies two resources, loaded in-order from the classpath:
1. core-default.xml : Read-only defaults for hadoop.
2. core-site.xml: Site-specific configuration for a given hadoop installation.
Applications may add additional resources, which are loaded subsequent to these resources in the order they are added.
Final Parameters

Configuration parameters may be declared final. Once a resource declares a value final, no subsequently-loaded resource can alter that value. For example, one might define a final parameter with:
```
 <property> <name>dfs.hosts.include</name> <value>/etc/hadoop/conf/hosts.include</value> <final>true</final> </property>
```
Administrators typically define parameters as final in core-site.xml for values that user applications may not alter.
Variable Expansion

Value strings are first processed for variable expansion. The available properties are:
1. Other properties defined in this Configuration; and, if a name is undefined here,
2. Properties in {@link System#getProperties()}.
For example, if a configuration resource contains the following property definitions:
```
 <property> <name>basedir</name> <value>/user/${user.name}</value> </property> <property> <name>tempdir</name> <value>${basedir}/tmp</value> </property>
```
When conf.get("tempdir") is called, then ${basedir} will be resolved to another property in this Configuration, while ${user.name} would then ordinarily be resolved to the value of the System property with that name.

    if (outputRoot == null) {
      System.err.println("Destination file-system not provided.");
      printUsageAndExit();
    }


    Configuration conf = getConf();
    Path inputRoot = FSUtils.getRootDir(conf);
    FileSystem inputFs = FileSystem.get(conf);
    FileSystem outputFs = FileSystem.get(outputRoot.toUri(), conf);


    Path snapshotDir = SnapshotDescriptionUtils.getCompletedSnapshotDir(snapshotName, inputRoot);

View Full Code Here

    }
    // either dump using the HLogPrettyPrinter or split, depending on args
    if (args[0].compareTo("--dump") == 0) {
      HLogPrettyPrinter.run(Arrays.copyOfRange(args, 1, args.length));
    } else if (args[0].compareTo("--split") == 0) {
      Configuration conf = HBaseConfiguration.create();
      for (int i = 1; i < args.length; i++) {
        try {
          conf.set("fs.default.name", args[i]);
          conf.set("fs.defaultFS", args[i]);
          Path logPath = new Path(args[i]);
          split(conf, logPath);
        } catch (Throwable t) {
          t.printStackTrace(System.err);
          System.exit(-1);

View Full Code Here

    String nodeName = rss.getServerName().toString();
    this.memberRpcs = new ZKProcedureMemberRpcs(zkw,
        SnapshotManager.ONLINE_SNAPSHOT_CONTROLLER_DESCRIPTION, nodeName);


    // read in the snapshot request configuration properties
    Configuration conf = rss.getConfiguration();
    long wakeMillis = conf.getLong(SNAPSHOT_REQUEST_WAKE_MILLIS_KEY, SNAPSHOT_REQUEST_WAKE_MILLIS_DEFAULT);
    long keepAlive = conf.getLong(SNAPSHOT_TIMEOUT_MILLIS_KEY, SNAPSHOT_TIMEOUT_MILLIS_DEFAULT);
    int opThreads = conf.getInt(SNAPSHOT_REQUEST_THREADS_KEY, SNAPSHOT_REQUEST_THREADS_DEFAULT);


    // create the actual snapshot procedure member
    ThreadPoolExecutor pool = ProcedureMember.defaultPool(wakeMillis, keepAlive, opThreads, nodeName);
    this.member = new ProcedureMember(memberRpcs, pool, new SnapshotSubprocedureBuilder());
  }

View Full Code Here

    // will hang and fail.


    LOG.debug("Launching subprocedure for snapshot " + snapshot.getName() + " from table "
        + snapshot.getTable());
    ForeignExceptionDispatcher exnDispatcher = new ForeignExceptionDispatcher();
    Configuration conf = rss.getConfiguration();
    long timeoutMillis = conf.getLong(SNAPSHOT_TIMEOUT_MILLIS_KEY,
        SNAPSHOT_TIMEOUT_MILLIS_DEFAULT);
    long wakeMillis = conf.getLong(SNAPSHOT_REQUEST_WAKE_MILLIS_KEY,
        SNAPSHOT_REQUEST_WAKE_MILLIS_DEFAULT);


    switch (snapshot.getType()) {
    case FLUSH:
      SnapshotSubprocedurePool taskManager =

View Full Code Here


  ServerManager(final Server master, final MasterServices services,
      final boolean connect) throws ZooKeeperConnectionException {
    this.master = master;
    this.services = services;
    Configuration c = master.getConfiguration();
    maxSkew = c.getLong("hbase.master.maxclockskew", 30000);
    warningSkew = c.getLong("hbase.master.warningclockskew", 10000);
    this.deadservers = new DeadServer();
    this.connection = connect ? HConnectionManager.getConnection(c) : null;
  }

View Full Code Here

    private Path inputArchive;
    private Path inputRoot;


    @Override
    public void setup(Context context) {
      Configuration conf = context.getConfiguration();
      verifyChecksum = conf.getBoolean(CONF_CHECKSUM_VERIFY, true);


      filesGroup = conf.get(CONF_FILES_GROUP);
      filesUser = conf.get(CONF_FILES_USER);
      filesMode = (short)conf.getInt(CONF_FILES_MODE, 0);
      outputRoot = new Path(conf.get(CONF_OUTPUT_ROOT));
      inputRoot = new Path(conf.get(CONF_INPUT_ROOT));


      inputArchive = new Path(inputRoot, HConstants.HFILE_ARCHIVE_DIRECTORY);
      outputArchive = new Path(outputRoot, HConstants.HFILE_ARCHIVE_DIRECTORY);


      try {

View Full Code Here

  throws IOException {
    job.setInputFormatClass(inputFormatClass);
    if (outputValueClass != null) job.setMapOutputValueClass(outputValueClass);
    if (outputKeyClass != null) job.setMapOutputKeyClass(outputKeyClass);
    job.setMapperClass(mapper);
    Configuration conf = job.getConfiguration();
    HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf));
    conf.set(TableInputFormat.INPUT_TABLE, table);
    conf.set(TableInputFormat.SCAN, convertScanToString(scan));
    if (addDependencyJars) {
      addDependencyJars(job);
    }
    initCredentials(job);
  }

View Full Code Here

        // init credentials for remote cluster
        String quorumAddress = job.getConfiguration().get(
            TableOutputFormat.QUORUM_ADDRESS);
        if (quorumAddress != null) {
          String[] parts = ZKUtil.transformClusterKey(quorumAddress);
          Configuration peerConf = HBaseConfiguration.create(job
              .getConfiguration());
          peerConf.set(HConstants.ZOOKEEPER_QUORUM, parts[0]);
          peerConf.set("hbase.zookeeper.client.port", parts[1]);
          peerConf.set(HConstants.ZOOKEEPER_ZNODE_PARENT, parts[2]);
          User.getCurrent().obtainAuthTokenForJob(peerConf, job);
        }
        
        User.getCurrent().obtainAuthTokenForJob(job.getConfiguration(), job);
      } catch (InterruptedException ie) {

View Full Code Here

  public static void initTableReducerJob(String table,
    Class<? extends TableReducer> reducer, Job job,
    Class partitioner, String quorumAddress, String serverClass,
    String serverImpl, boolean addDependencyJars) throws IOException {


    Configuration conf = job.getConfiguration();    
    HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf));
    job.setOutputFormatClass(TableOutputFormat.class);
    if (reducer != null) job.setReducerClass(reducer);
    conf.set(TableOutputFormat.OUTPUT_TABLE, table);
    // If passed a quorum/ensemble address, pass it on to TableOutputFormat.
    if (quorumAddress != null) {
      // Calling this will validate the format
      ZKUtil.transformClusterKey(quorumAddress);
      conf.set(TableOutputFormat.QUORUM_ADDRESS,quorumAddress);
    }
    if (serverClass != null && serverImpl != null) {
      conf.set(TableOutputFormat.REGION_SERVER_CLASS, serverClass);
      conf.set(TableOutputFormat.REGION_SERVER_IMPL, serverImpl);
    }
    job.setOutputKeyClass(ImmutableBytesWritable.class);
    job.setOutputValueClass(Writable.class);
    if (partitioner == HRegionPartitioner.class) {
      job.setPartitionerClass(HRegionPartitioner.class);

View Full Code Here

      System.err.println("Missing snapshot name!");
      printUsageAndExit();
      return 1;
    }


    Configuration conf = getConf();
    fs = FileSystem.get(conf);
    rootDir = FSUtils.getRootDir(conf);


    // Load snapshot information
    if (!loadSnapshotInfo(snapshotName)) {

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.hadoop.conf.Configuration$Resource

azkaban.common.web.HdfsSequenceFileViewer

bbejeck.mapred.joins.map.MapSideJoinDriver

co.nubetech.hiho.dedup.DedupJob

co.nubetech.hiho.job.DBQueryInputJob

co.nubetech.hiho.job.ExportToDB

co.nubetech.hiho.job.ExportToFTPServer

co.nubetech.hiho.job.ExportToMySQLDB

co.nubetech.hiho.job.ExportToOracleDb

co.nubetech.hiho.job.sf.TestExportSalesForceJob

co.nubetech.hiho.job.TestDBQueryInputJob

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of org.apache.hadoop.conf.Configuration$Resource

Resources

Final Parameters

Variable Expansion

Related Classes of org.apache.hadoop.conf.Configuration$Resource