Examples of org.apache.hadoop.hbase.migration.nineteen.regionserver.HStoreFile

Package org.apache.hadoop.hbase.migration.nineteen.regionserver

Examples of org.apache.hadoop.hbase.migration.nineteen.regionserver.HStoreFile

org.apache.hadoop.hbase.migration.nineteen.regionserver.HStoreFile
A HStore data file. HStores usually have one or more of these files. They are produced by flushing the memcache to disk.
This one has been doctored to be used in migrations. Private and protecteds have been made public, etc.
Each HStore maintains a bunch of different data files. The filename is a mix of the parent dir, the region name, the column name, and a file identifier. The name may also be a reference to a store file located elsewhere. This class handles all that path-building stuff for you.
An HStoreFile usually tracks 4 things: its parent dir, the region identifier, the column family, and the file identifier. If you know those four things, you know how to obtain the right HStoreFile. HStoreFiles may also reference store files in another region serving either from the top-half of the remote file or from the bottom-half. Such references are made fast splitting regions.
Plain HStoreFiles are named for a randomly generated id as in: 1278437856009925445 A file by this name is made in both the mapfiles and info subdirectories of a HStore columnfamily directoy: E.g. If the column family is 'anchor:', then under the region directory there is a subdirectory named 'anchor' within which is a 'mapfiles' and 'info' subdirectory. In each will be found a file named something like 1278437856009925445, one to hold the data in 'mapfiles' and one under 'info' that holds the sequence id for this store file.
References to store files located over in some other region look like this: 1278437856009925445.hbaserepository,qAReLZD-OyQORZWq_vqR1k==,959247014679548184: i.e. an id followed by the name of the referenced region. The data ('mapfiles') of HStoreFile references are empty. The accompanying info file contains the midkey, the id of the remote store we're referencing and whether we're to serve the top or bottom region of the remote store file. Note, a region is not splitable if it has instances of store file references (References are cleaned up by compactions).
When merging or splitting HRegions, we might want to modify one of the params for an HStoreFile (effectively moving it elsewhere).

    Path regiondir = familydir.getParent();
    Path basedir = regiondir.getParent();
    if (HStoreFile.isReference(mf)) {
      throw new IOException(mf.toString() + " is Reference");
    }
    HStoreFile hsf = new HStoreFile(conf, fs, basedir,
      Integer.parseInt(regiondir.getName()),
      Bytes.toBytes(familydir.getName()), Long.parseLong(mf.getName()), null);
    BloomFilterMapFile.Reader src = hsf.getReader(fs, false, false);
    HFile.Writer tgt = StoreFile.getWriter(fs, familydir,
      conf.getInt("hfile.min.blocksize.size", 64*1024),
      Compression.Algorithm.NONE, getComparator(basedir));
    // From old 0.19 HLogEdit.
    ImmutableBytesWritable deleteBytes =
      new ImmutableBytesWritable("HBASE::DELETEVAL".getBytes("UTF-8"));
    try {
      while (true) {
        HStoreKey key = new HStoreKey();
        ImmutableBytesWritable value = new ImmutableBytesWritable();
        if (!src.next(key, value)) {
          break;
        }
        byte [][] parts = KeyValue.parseColumn(key.getColumn());
        KeyValue kv = deleteBytes.equals(value)?
            new KeyValue(key.getRow(), parts[0], parts[1], 
                key.getTimestamp(), KeyValue.Type.Delete):
              new KeyValue(key.getRow(), parts[0], parts[1], 
                key.getTimestamp(), value.get());
         tgt.append(kv);
      }
      long seqid = hsf.loadInfo(fs);
      StoreFile.appendMetadata(tgt, seqid, 
          hsf.isMajorCompaction());
      // Success, delete src.
      src.close();
      tgt.close();
      hsf.delete();
      // If we rewrote src, delete mapfiles and info dir.
      fs.delete(mf.getParent(), true);
      fs.delete(new Path(familydir, "info"), true);
      LOG.info("Rewrote " + mf.toString() + " as " + tgt.toString());
    } catch (IOException e) {

View Full Code Here

    Path regiondir = familydir.getParent();
    Path basedir = regiondir.getParent();
    if (HStoreFile.isReference(mf)) {
      throw new IOException(mf.toString() + " is Reference");
    }
    HStoreFile hsf = new HStoreFile(conf, fs, basedir,
      Integer.parseInt(regiondir.getName()),
      Bytes.toBytes(familydir.getName()), Long.parseLong(mf.getName()), null);
    BloomFilterMapFile.Reader src = hsf.getReader(fs, false, false);
    String compression = conf.get("migrate.compression", "NONE").trim();
    Compression.Algorithm compressAlgorithm = Compression.Algorithm.valueOf(compression);
    HFile.Writer tgt = StoreFile.getWriter(fs, familydir,
      conf.getInt("hfile.min.blocksize.size", 64*1024),
      compressAlgorithm, getComparator(basedir));
    // From old 0.19 HLogEdit.
    ImmutableBytesWritable deleteBytes =
      new ImmutableBytesWritable("HBASE::DELETEVAL".getBytes("UTF-8"));
    try {
      while (true) {
        HStoreKey key = new HStoreKey();
        ImmutableBytesWritable value = new ImmutableBytesWritable();
        if (!src.next(key, value)) {
          break;
        }
        byte [][] parts = KeyValue.parseColumn(key.getColumn());
        KeyValue kv = deleteBytes.equals(value)?
            new KeyValue(key.getRow(), parts[0], parts[1], 
                key.getTimestamp(), KeyValue.Type.Delete):
              new KeyValue(key.getRow(), parts[0], parts[1], 
                key.getTimestamp(), value.get());
         tgt.append(kv);
      }
      long seqid = hsf.loadInfo(fs);
      StoreFile.appendMetadata(tgt, seqid, 
          hsf.isMajorCompaction());
      // Success, delete src.
      src.close();
      tgt.close();
      hsf.delete();
      // If we rewrote src, delete mapfiles and info dir.
      fs.delete(mf.getParent(), true);
      fs.delete(new Path(familydir, "info"), true);
      LOG.info("Rewrote " + mf.toString() + " as " + tgt.toString());
    } catch (IOException e) {

View Full Code Here

TOP

Related Classes of org.apache.hadoop.hbase.migration.nineteen.regionserver.HStoreFile

org.apache.hadoop.fs.FSDataInputStream

org.apache.hadoop.fs.FSDataOutputStream

org.apache.hadoop.fs.Path

org.apache.hadoop.hbase.migration.nineteen.io.HalfMapFileReader

org.apache.hadoop.hbase.migration.nineteen.io.Reference

org.apache.hadoop.hbase.util.Migrate

java.io.FileNotFoundException

java.io.DataInputStream

java.io.IOException

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.