Examples of TsProcessor

org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor
TsProcessor is a generic processor that can be configured to find the timestamp in the text of a record. By default, this class expects that a record starts with a date in this format: yyyy-MM-dd HH:mm:ss,SSS
This format can be changed with the following configurations.
- TsProcessor.default.time.format - Changes the default time format used by all data types.
- TsProcessor.time.format.[some_data_type] - Overrides the default format for a specific data type.
If the time string is not at the beginning of the record you can configure a regular expression to locate the timestamp text with either of the following configurations. The text found in group 1 of the regular expression match will be used with the configured date format.
- TsProcessor.default.time.regex - Changes the default time location regex of the time text for all data types.
- TsProcessor.time.regex.[some_data_type] - Overrides the default time location regex for a specific data type.

Examples of org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor

    File inputFile = new File( datadir+ File.separator + "ClientTrace.log");
    Path outputFile = new Path(outputDir + "/" + this.getClass().getName() + "/ClientTrace.evt");
    String clusterName = "testClusterName";
    String dataType = "testDataType";
    String streamName = "testStreamName";
    MapProcessor processor = new TsProcessor();


    //create the sequence file
    CreateRecordFile.makeTestSequenceFile(inputFile, outputFile, clusterName,
                                          dataType, streamName, processor);
    //read the output file

View Full Code Here

Examples of org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor

                                           String streamName,
                                           MapProcessor processor) throws IOException {


     //initialize the output collector and the default processor
     MockOutputCollector collector = new MockOutputCollector();
     if (processor == null) processor = new TsProcessor();


     //initialize the sequence file writer
     Configuration conf = new Configuration();
     FileSystem fs = outputFile.getFileSystem(conf);
     FSDataOutputStream out = fs.create(outputFile);

View Full Code Here

Examples of org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor

     File inputFile = new File(args[0]);
     Path outputFile = new Path(args[1]);
     String clusterName = "testClusterName";
     String dataType = "testDataType";
     String streamName = "testStreamName";               
     MapProcessor processor = new TsProcessor();
     Path confFile = null;


     if (args.length > 2) clusterName = args[2];
     if (args.length > 3) dataType = args[3];
     if (args.length > 4) streamName = args[4];


     if (args.length > 5) {
       Class clazz = null;
       try {
         clazz = Class.forName(args[5]);
       }
       catch (ClassNotFoundException e) {
         try {
           clazz = Class.forName(
                 "org.apache.hadoop.chukwa.extraction.demux.processor.mapper." + args[5]);
         }
         catch (Exception e2) {
           throw e;
         }
       }
       processor = (MapProcessor)clazz.newInstance();
     }


     if (args.length > 6) {
       confFile = new Path(args[6]);
       Demux.jobConf = new JobConf(confFile);
     }


     System.out.println("Creating sequence file using the following input:");
     System.out.println("inputFile  : " + inputFile);
     System.out.println("outputFile : " + outputFile);
     System.out.println("clusterName: " + clusterName);
     System.out.println("dataType   : " + dataType);
     System.out.println("streamName : " + streamName);
     System.out.println("processor  : " + processor.getClass().getName());
     System.out.println("confFile   : " + confFile);


     makeTestSequenceFile(inputFile, outputFile, clusterName, dataType, streamName, processor);


     System.out.println("Done");

View Full Code Here

Examples of org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor

                                           String streamName,
                                           MapProcessor processor) throws IOException {


     //initialize the output collector and the default processor
     MockOutputCollector collector = new MockOutputCollector();
     if (processor == null) processor = new TsProcessor();


     //initialize the sequence file writer
     Configuration conf = new Configuration();
     FileSystem fs = outputFile.getFileSystem(conf);
     FSDataOutputStream out = fs.create(outputFile);

View Full Code Here

Examples of org.apache.hadoop.chukwa.extraction.demux.processor.mapper.TsProcessor

     File inputFile = new File(args[0]);
     Path outputFile = new Path(args[1]);
     String clusterName = "testClusterName";
     String dataType = "testDataType";
     String streamName = "testStreamName";               
     MapProcessor processor = new TsProcessor();


     if (args.length > 2) clusterName = args[2];
     if (args.length > 3) dataType = args[3];
     if (args.length > 4) streamName = args[4];


     if (args.length > 5) {
       Class clazz = Class.forName(args[5]);
       processor = (MapProcessor)clazz.newInstance();
     }


     System.out.println("Creating sequence file using the following input:");
     System.out.println("inputFile  : " + inputFile);
     System.out.println("outputFile : " + outputFile);
     System.out.println("clusterName: " + clusterName);
     System.out.println("dataType   : " + dataType);
     System.out.println("streamName : " + streamName);
     System.out.println("processor  : " + processor.getClass().getName());


     makeTestSequenceFile(inputFile, outputFile, clusterName, dataType, streamName, processor);


     System.out.println("Done");
   }

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.