Examples of com.cloudera.flume.handlers.text.output.OutputFormat

Package com.cloudera.flume.handlers.text.output

Examples of com.cloudera.flume.handlers.text.output.OutputFormat

com.cloudera.flume.handlers.text.output.OutputFormat
Output formats take an Event and a OutputStream and then formats the event before writing to the OutputStream. There are many potential formats. Ideally text formats would revert an event back to exactly the version found in a text log file. (an apache one would dump in apache format, syslog would dump in the same format as syslog.log, etc) Oftentimes this is not possible and a best effort version is output. Requiring the outputStream as an arg allows for streaming writes and removes the potential requirement for copies of large memory buffers with large events. It also allow sophisticated (avro, seqfile) output formats to fancy things such as markers for bundling/splitability, adding error correction codes, checksums, etc.


      @Override
      public EventSink build(Context context, String... argv) {
        Preconditions.checkArgument(argv.length <= 1,
            "usage: console[(format)]");
        OutputFormat fmt = DebugOutputFormat.builder().build();
        if (argv.length >= 1) {
          // TODO (jon) handle formats with arguments. Requires language update.
          try {
            fmt = FormatFactory.get().getOutputFormat(argv[0]);
          } catch (FlumeSpecException e) {

View Full Code Here

          throw new IllegalArgumentException(
          "usage: customdfs(\"[(hdfs|file|s3n|...)://namenode[:port]]/path\", \"format\")");
        }


        String format = (args.length == 1) ? null : args[1];
        OutputFormat fmt;
        try {
          fmt = FormatFactory.get().getOutputFormat(format);
        } catch (FlumeSpecException e) {
          LOG.error("failed to load format " + format, e);
          throw new IllegalArgumentException("failed to load format " + format);

View Full Code Here

    return new SinkBuilder() {
      @Override
      public EventSink build(Context context, String... args) {
        Preconditions.checkArgument(args.length >= 1 && args.length <= 2,
            "usage: text(filename[,format])");
        OutputFormat fmt = DebugOutputFormat.builder().build();
        if (args.length >= 2) {
          try {
            fmt = FormatFactory.get().getOutputFormat(args[1]);
          } catch (FlumeSpecException e) {
            LOG.error("Illegal output format " + args[1], e);

View Full Code Here

0 1 2

TOP

Related Classes of com.cloudera.flume.handlers.text.output.OutputFormat

com.cloudera.flume.handlers.avro.AvroDataFileOutputFormat

com.cloudera.flume.handlers.avro.AvroJsonOutputFormat

com.cloudera.flume.handlers.avro.AvroNativeFileOutputFormat

com.cloudera.flume.handlers.avro.TestAvroDataFile

com.cloudera.flume.handlers.debug.ConsoleEventSink

com.cloudera.flume.handlers.debug.TextFileSink

com.cloudera.flume.handlers.hdfs.CustomDfsSink

com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink

com.cloudera.flume.handlers.hive.HiveNotifyingDfsSink

com.cloudera.flume.handlers.seqfile.TestSequenceFileOutputFormat

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.