Examples of com.facebook.LinkBench.generators.MotifDataGenerator

Package com.facebook.LinkBench.generators

Examples of com.facebook.LinkBench.generators.MotifDataGenerator

com.facebook.LinkBench.generators.MotifDataGenerator
A simple data generator where the same sequences of bytes, or "motifs" occur multiple times. This is designed to emulate one particular property of real data that is exploited by compression algorithms. Typically a short sequence of data generated by this generator will not be very compressible on its own, as no motifs will recur, but if multiple output strings are concatenated together then the same motifs will recur repeatedly and the data will be compressible. The motif data generator has a buffer of "shared" motifs, which reoccur frequently in the output of the generator The data generator generates bytes from within the range of values [min, max). There is an additional parameter, which is called uniqueness for lack of a better name. The generator fills a buffer with data in chunks. A chunk is either generated as random new bytes, or is drawn from the "motifs", The uniqueness parameter controls the proportion of new chunks versus duplicated motifs. It is a probability between 0.0 and 1.0. It can also be seen as the expected percentage of bytes are generated from scratch. Control how often motifs appear in data uniqueness = 0.0: all data drawn from motifs uniqueness 1.0: completely independent bytes

  public void testTimingMotif() {
    System.err.println("Testing motif generator");
    System.err.println("=========================");
    DataGenFactory fact = new DataGenFactory() {
      public DataGenerator make(double param) {
        MotifDataGenerator gen = new MotifDataGenerator();
        gen.init(0,  8, param);
        return gen;
      }
    };
    testTiming(fact, 128);
    testTiming(fact, 1024);

View Full Code Here

   *
   * Currently difficult to automatically verify output.
   */
  @Test
  public void testMotif() {
    MotifDataGenerator gen = new MotifDataGenerator();


    System.err.println("uniqueness 0.25");
    gen.init(0, 8, 0.25);
    byte data[] = gen.fill(new Random(), new byte[64]);
    printByteGrid(data);


    System.err.println("uniqueness 0.0");
    gen.init(0, 8, 0.0);
    data = gen.fill(new Random(), new byte[64]);
    printByteGrid(data);


    System.err.println("uniqueness 0.05");
    gen.init(0, 8, 0.05);
    data = gen.fill(new Random(), new byte[64]);
    printByteGrid(data);


    System.err.println("uniqueness 1.0");
    gen.init(0, 8, 1.0);
    data = gen.fill(new Random(), new byte[64]);
    printByteGrid(data);
  }

View Full Code Here

   * compressing a long stream of the data
   * @throws IOException
   */
  @Test
  public void testCompressibility() throws IOException {
    MotifDataGenerator gen = new MotifDataGenerator();
    gen.init(0, 255, 0.5);
    System.err.println("\nUniqueness=0.5 Range=255\n===============");
    testCompressibility(gen, 1024, 10000);
    testCompressibility(gen, 64, 1);


    gen.init(0, 127, 0.5);
    System.err.println("\nUniqueness=0.5 Range=127\n===============");
    testCompressibility(gen, 1024, 10000);
    testCompressibility(gen, 64, 1);


    gen.init(0, 255, 0.0);
    System.err.println("\nUniqueness=0.0 Range=255\n===============");
    testCompressibility(gen, 1024, 10000);
    testCompressibility(gen, 64, 1);


    gen.init(0, 255, 1.0);
    System.err.println("\nUniqueness=1.0 Range=255\n===============");
    testCompressibility(gen, 1024, 10000);
    testCompressibility(gen, 64, 1);
    gen.init(0, 255, 1.0);


    gen.init(0, 127, 1.0);
    System.err.println("\nUniqueness=1.0 Range=127\n===============");
    testCompressibility(gen, 1024, 10000);
    testCompressibility(gen, 64, 1);


    gen.init(0, 1, 1.0);
    System.err.println("\nUniqueness=1.0 Range=1\n===============");
    testCompressibility(gen, 1024, 10000);
    testCompressibility(gen, 64, 1);
  }

View Full Code Here

    gen.init(50, 75);
    return gen;
  }


  private static MotifDataGenerator makeMotifObj() {
    MotifDataGenerator gen = new MotifDataGenerator();
    int start = 50;
    int end = 220;
    double uniqueness = 0.63;
    gen.init(start, end, uniqueness);
    return gen;
  }

View Full Code Here

    gen.init(50, 75);
    return gen;
  }


  private static MotifDataGenerator makeMotifAssoc() {
    MotifDataGenerator gen = new MotifDataGenerator();
    int start = 32;
    int end = 100;
    double uniqueness = 0.225;
    int motifSize = 128;
    gen.init(start, end, uniqueness, motifSize);
    return gen;
  }

View Full Code Here

TOP

Related Classes of com.facebook.LinkBench.generators.MotifDataGenerator

com.facebook.LinkBench.GeneratedDataDump

com.facebook.LinkBench.LinkBenchConfigError

com.facebook.LinkBench.TestDataGen

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.