Examples of org.apache.hadoop.conf.Configuration$Resource

org.oasis.wsrp.v2.Resource
Provides access to configuration parameters.
Resources

Configurations are specified by resources. A resource contains a set of name/value pairs as XML data. Each resource is named by either a String or by a {@link Path}. If named by a String, then the classpath is examined for a file with that name. If named by a Path, then the local filesystem is examined directly, without referring to the classpath.
Unless explicitly turned off, Hadoop by default specifies two resources, loaded in-order from the classpath:
1. core-default.xml : Read-only defaults for hadoop.
2. core-site.xml: Site-specific configuration for a given hadoop installation.
Applications may add additional resources, which are loaded subsequent to these resources in the order they are added.
Final Parameters

Configuration parameters may be declared final. Once a resource declares a value final, no subsequently-loaded resource can alter that value. For example, one might define a final parameter with:
```
 <property> <name>dfs.hosts.include</name> <value>/etc/hadoop/conf/hosts.include</value> <final>true</final> </property>
```
Administrators typically define parameters as final in core-site.xml for values that user applications may not alter.
Variable Expansion

Value strings are first processed for variable expansion. The available properties are:
1. Other properties defined in this Configuration; and, if a name is undefined here,
2. Properties in {@link System#getProperties()}.
For example, if a configuration resource contains the following property definitions:
```
 <property> <name>basedir</name> <value>/user/${user.name}</value> </property> <property> <name>tempdir</name> <value>${basedir}/tmp</value> </property>
```
When conf.get("tempdir") is called, then ${basedir} will be resolved to another property in this Configuration, while ${user.name} would then ordinarily be resolved to the value of the System property with that name.

   * have write permissions to this directory, and this should not be a subdirectory of rootdir.
   * After the job is finished, restoreDir can be deleted.
   * @throws IOException if an error occurs
   */
  public static void setInput(Job job, String snapshotName, Path restoreDir) throws IOException {
    Configuration conf = job.getConfiguration();
    conf.set(SNAPSHOT_NAME_KEY, snapshotName);


    Path rootDir = new Path(conf.get(HConstants.HBASE_DIR));
    FileSystem fs = rootDir.getFileSystem(conf);


    restoreDir = new Path(restoreDir, UUID.randomUUID().toString());


    // TODO: restore from record readers to parallelize.
    RestoreSnapshotHelper.copySnapshotForScanner(conf, fs, rootDir, restoreDir, snapshotName);


    conf.set(TABLE_DIR_KEY, restoreDir.toString());
  }

View Full Code Here


    @Override
    public void initialize(InputSplit split, TaskAttemptContext context) throws IOException,
        InterruptedException {


      Configuration conf = context.getConfiguration();
      this.split = (TableSnapshotRegionSplit) split;
      String regionName = this.split.regionName;
      String snapshotName = getSnapshotName(conf);
      Path rootDir = new Path(conf.get(HConstants.HBASE_DIR));
      FileSystem fs = rootDir.getFileSystem(conf);


      Path tmpRootDir = new Path(conf.get(TABLE_DIR_KEY)); // This is the user specified root
      // directory where snapshot was restored


      Path snapshotDir = SnapshotDescriptionUtils.getCompletedSnapshotDir(snapshotName, rootDir);


      //load table descriptor
      HTableDescriptor htd = FSTableDescriptors.getTableDescriptorFromFs(fs, snapshotDir);


      //load region descriptor
      Path regionDir = new Path(snapshotDir, regionName);
      HRegionInfo hri = HRegionFileSystem.loadRegionInfoFileContent(fs, regionDir);


      // create scan
      String scanStr = conf.get(TableInputFormat.SCAN);
      if (scanStr == null) {
        throw new IllegalArgumentException("A Scan is not configured for this job");
      }
      scan = TableMapReduceUtil.convertStringToScan(scanStr);
      scan.setIsolationLevel(IsolationLevel.READ_UNCOMMITTED); // region is immutable, this should be fine,

View Full Code Here

    @Override
    public void map(ImmutableBytesWritable row, final Result value,
                    Context context)
        throws IOException {
      if (replicatedScanner == null) {
        Configuration conf = context.getConfiguration();
        final Scan scan = new Scan();
        scan.setCaching(conf.getInt(TableInputFormat.SCAN_CACHEDROWS, 1));
        long startTime = conf.getLong(NAME + ".startTime", 0);
        long endTime = conf.getLong(NAME + ".endTime", Long.MAX_VALUE);
        String families = conf.get(NAME + ".families", null);
        if(families != null) {
          String[] fams = families.split(",");
          for(String fam : fams) {
            scan.addFamily(Bytes.toBytes(fam));
          }
        }
        scan.setTimeRange(startTime, endTime);
        HConnectionManager.execute(new HConnectable<Void>(conf) {
          @Override
          public Void connect(HConnection conn) throws IOException {
            String zkClusterKey = conf.get(NAME + ".peerQuorumAddress");
            Configuration peerConf = HBaseConfiguration.create(conf);
            ZKUtil.applyClusterKeyToConf(peerConf, zkClusterKey);


            HTable replicatedTable = new HTable(peerConf, conf.get(NAME + ".tableName"));
            scan.setStartRow(value.getRow());
            replicatedScanner = replicatedTable.getScanner(scan);

View Full Code Here

    if (outputKeyClass != null) job.setMapOutputKeyClass(outputKeyClass);
    job.setMapperClass(mapper);
    if (Put.class.equals(outputValueClass)) {
      job.setCombinerClass(PutCombiner.class);
    }
    Configuration conf = job.getConfiguration();
    HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf));
    conf.set(TableInputFormat.INPUT_TABLE, table);
    conf.set(TableInputFormat.SCAN, convertScanToString(scan));
    conf.setStrings("io.serializations", conf.get("io.serializations"),
        MutationSerialization.class.getName(), ResultSerialization.class.getName(),
        KeyValueSerialization.class.getName());
    if (addDependencyJars) {
      addDependencyJars(job);
    }

View Full Code Here

            if (result == null)
                result = defaultCase(theEObject);
            return result;
        }
        case Bpmn2Package.RESOURCE: {
            Resource resource = (Resource) theEObject;
            T result = caseResource(resource);
            if (result == null)
                result = caseRootElement(resource);
            if (result == null)
                result = caseBaseElement(resource);

View Full Code Here

     * <!-- begin-user-doc -->
     * <!-- end-user-doc -->
     * @generated
     */
    public void setResourceRef(Resource newResourceRef) {
        Resource oldResourceRef = resourceRef;
        resourceRef = newResourceRef;
        if (eNotificationRequired())
            eNotify(new ENotificationImpl(this, Notification.SET,
                    Bpmn2Package.RESOURCE_ROLE__RESOURCE_REF, oldResourceRef, resourceRef));
    }

View Full Code Here

   {
      public Resource apply(V1Resource from)
      {
         if (from != null)
         {
            Resource result = WSRPTypeFactory.createResource(from.getResourceName(), WSRPUtils.transform(from.getValues(), RESOURCEVALUE));
            return result;
         }
         else
         {
            return null;

View Full Code Here

   }


   public static Resource createResource(String resourceName, List<ResourceValue> resourceValue)
   {
      ParameterValidation.throwIllegalArgExceptionIfNull(resourceName, "ResourceName");
      Resource resource = new Resource();
      resource.setResourceName(resourceName);


      if (resourceValue != null && !resourceValue.isEmpty())
      {
         resource.getValues().addAll(resourceValue);
      }


      return resource;
   }

View Full Code Here

   }


   public static Resource createResource(String resourceName, List<ResourceValue> resourceValue)
   {
      ParameterValidation.throwIllegalArgExceptionIfNull(resourceName, "ResourceName");
      Resource resource = new Resource();
      resource.setResourceName(resourceName);


      if (resourceValue != null && !resourceValue.isEmpty())
      {
         resource.getValues().addAll(resourceValue);
      }


      return resource;
   }

View Full Code Here

   }


   public static Resource createResource(String resourceName, List<ResourceValue> resourceValue)
   {
      ParameterValidation.throwIllegalArgExceptionIfNull(resourceName, "ResourceName");
      Resource resource = new Resource();
      resource.setResourceName(resourceName);


      if (resourceValue != null && !resourceValue.isEmpty())
      {
         resource.getValues().addAll(resourceValue);
      }


      return resource;
   }

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.hadoop.conf.Configuration$Resource

azkaban.common.web.HdfsSequenceFileViewer

bbejeck.mapred.joins.map.MapSideJoinDriver

co.nubetech.hiho.dedup.DedupJob

co.nubetech.hiho.job.DBQueryInputJob

co.nubetech.hiho.job.ExportToDB

co.nubetech.hiho.job.ExportToFTPServer

co.nubetech.hiho.job.ExportToMySQLDB

co.nubetech.hiho.job.ExportToOracleDb

co.nubetech.hiho.job.sf.TestExportSalesForceJob

co.nubetech.hiho.job.TestDBQueryInputJob

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of org.apache.hadoop.conf.Configuration$Resource

Resources

Final Parameters

Variable Expansion

Related Classes of org.apache.hadoop.conf.Configuration$Resource