Examples of org.apache.avro.io.BinaryDecoder$InputStreamByteSource

org.apache.avro.io.BinaryDecoder
ByteSource abstracts the source of data from the core workings of BinaryDecoder. This is very important for performance reasons because InputStream's API is a barrier to performance due to several quirks: InputStream does not in general require that as many bytes as possible have been read when filling a buffer.
InputStream's terminating conditions for a read are two-fold: EOFException and '-1' on the return from read(). Implementations are supposed to return '-1' on EOF but often do not. The extra terminating conditions cause extra conditionals on both sides of the API, and slow performance significantly.
ByteSource implementations provide read() and skip() variants that have stronger guarantees than InputStream, freeing client code to be simplified and faster.
{@link skipSourceBytes} and {@link readRaw} are guaranteed to have read orskipped as many bytes as possible, or throw EOFException. {@link trySkipBytes} and {@link tryRead} are guaranteed to attempt to reador skip as many bytes as possible and never throw EOFException, while returning the exact number of bytes skipped or read. {@link isEof} returnstrue if all the source bytes have been read or skipped. This condition can also be detected by a client if an EOFException is thrown from {@link skipSourceBytes} or {@link readRaw}, or if {@link trySkipBytes} or{@link tryRead} return 0;
A ByteSource also implements the InputStream contract for use by APIs that require it. The InputStream interface must take into account buffering in any decoder that this ByteSource is attached to. The other methods do not account for buffering.

    writer.write(a, new BinaryEncoder(out));
    AnotherSampleRecord b = new AnotherSampleRecord(10);
    writer.write(b, new BinaryEncoder(out));
    ReflectDatumReader reader = new ReflectDatumReader(schm, prefix);
    ByteArrayInputStream in = new ByteArrayInputStream(out.toByteArray());
    Object decoded = reader.read(null, new BinaryDecoder(in));
    assertEquals(a, decoded);
    decoded = reader.read(null, new BinaryDecoder(in));
    assertEquals(b, decoded);
  }

View Full Code Here

    byte[] data = out.toByteArray();


    reader.setSchema(schema);
        
    Object decoded =
      reader.read(null, new BinaryDecoder(new ByteArrayInputStream(data)));
      
    assertEquals("Decoded data does not match.", datum, decoded);
  }

View Full Code Here

    +"\"type\":"+schemaJson+", "
    +"\"default\":"+defaultJson+"}]}";
    Schema expected = Schema.parse(recordJson);
    DatumReader in = new GenericDatumReader(ACTUAL, expected);
    GenericData.Record record = (GenericData.Record)
      in.read(null, new BinaryDecoder(new ByteArrayInputStream(new byte[0])));
    assertEquals("Wrong default.", defaultValue, record.get("f"));
    assertEquals("Wrong toString", expected, Schema.parse(expected.toString()));
  }

View Full Code Here


    long length = in.length();
    in.seek(length-4);
    int footerSize=(in.read()<<24)+(in.read()<<16)+(in.read()<<8)+in.read();
    in.seek(length-footerSize);
    this.vin = new BinaryDecoder(in);
    long l = vin.readMapStart();
    if (l > 0) {
      do {
        for (long i = 0; i < l; i++) {
          String key = vin.readString(null).toString();

View Full Code Here

   * a response or error. */
  public List<ByteBuffer> respond(Transceiver transceiver) throws IOException {
    ByteBufferInputStream bbi =
      new ByteBufferInputStream(transceiver.readBuffers());
    
    Decoder in = new BinaryDecoder(bbi);
    ByteBufferOutputStream bbo =
      new ByteBufferOutputStream();
    Encoder out = new BinaryEncoder(bbo);
    AvroRemoteException error = null;
    RPCContext context = new RPCContext();
    try {
      Protocol remote = handshake(transceiver, in, out);
      if (remote == null)                        // handshake failed
        return bbo.getBufferList();


      // read request using remote protocol specification
      context.setRequestCallMeta(META_READER.read(null, in));
      String messageName = in.readString(null).toString();
      Message m = remote.getMessages().get(messageName);
      if (m == null)
        throw new AvroRuntimeException("No such remote message: "+messageName);
      
      Object request = readRequest(m.getRequest(), in);

View Full Code Here

      
      List<ByteBuffer> response =                 // transceive
        getTransceiver().transceive(bbo.getBufferList());
      
      ByteBufferInputStream bbi = new ByteBufferInputStream(response);
      in = new BinaryDecoder(bbi);
      if (!established)                           // if not established
        readHandshake(in);                        // process handshake
    } while (!established);


    // use remote protocol to read response

View Full Code Here


  @SuppressWarnings("deprecation")
  protected Decoder createDecoder() throws IOException {
    switch(codecType) {
      case BINARY:
        return new BinaryDecoder(getOrCreateInputStream());
      case JSON:
        return new JsonDecoder(schema, getOrCreateInputStream());
    }
    return null;
  }

View Full Code Here

    case BOOLEAN: return val[0] != 0;
    case RECORD:
      // TODO: This is TOO SLOW... OPTIMIZE
      reader.setSchema(schema);
      reader.setExpected(schema);
      BinaryDecoder decoder = new BinaryDecoder(new ByteArrayInputStream(val));
      return reader.read(null, decoder);
    default: throw new RuntimeException("Unknown type: "+type);
    }
  }

View Full Code Here

      try {
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(baos, null);
        ResolvingGrammarGenerator.encode(encoder, field.schema(), json);
        encoder.flush();
        BinaryDecoder decoder =
          DecoderFactory.get().binaryDecoder(baos.toByteArray(), null);
        defaultValue =
          createDatumReader(field.schema()).read(null, decoder);


        defaultValueCache.put(field, defaultValue);

View Full Code Here

      try {
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(baos, null);
        ResolvingGrammarGenerator.encode(encoder, field.schema(), json);
        encoder.flush();
        BinaryDecoder decoder =
          DecoderFactory.get().binaryDecoder(baos.toByteArray(), null);
        defaultValue =
          createDatumReader(field.schema()).read(null, decoder);


        defaultValueCache.put(field, defaultValue);

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.avro.io.BinaryDecoder$InputStreamByteSource

co.cask.cdap.logging.appender.kafka.LoggingEventSerializer

com.adgear.anoa.avro.decode.GenericDatumTextReader

com.cloudera.flume.handlers.avro.TestAvroSerialize

com.fasterxml.jackson.dataformat.avro.AvroSchema

com.senseidb.gateway.kafka.AvroSerializerHelper

havrobase.UserTest

io.s4.serialize.AvroSerDeser

io.s4.serialize.AvroUtils

org.apache.avro.file.DataFileReader

org.apache.avro.generic.GenericData

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.