Examples of co.cask.cdap.data.file.ReadFilter

co.cask.cdap.data.file.ReadFilter
Filter for reading from {@link FileReader}. This class is experimental is still expanding.
See {@link co.cask.cdap.data.stream.StreamDataFileWriter} for the file format.
The methods on this class will be called in the following sequence:
```
 reset() - Called before filtering a new data block acceptTimestamp() - Called when the timestamp of a data block is read getNextTimestampHint() - Called if acceptTimestamp() return false acceptOffset() - Called at each stream event begin offset 
```

      .add("consumer", consumerConfig)
      .toString();
  }


  private ReadFilter createReadFilter(ConsumerConfig consumerConfig, @Nullable ReadFilter extraFilter) {
    ReadFilter baseFilter = createBaseReadFilter(consumerConfig);


    if (extraFilter != null) {
      return ReadFilters.and(extraFilter, baseFilter);
    } else {
      return baseFilter;

View Full Code Here

    // For RoundRobin, the idea is to scatter the events across consumers evenly. Since there is no way to known
    // about the absolute starting point to do true round robin, we employ a good enough hash function on the
    // file offset as a way to spread events across consumers
    final int instanceId = consumerConfig.getInstanceId();


    return new ReadFilter() {
      @Override
      public boolean acceptOffset(long offset) {
        int hashValue = Math.abs(strategy == DequeueStrategy.HASH ? 0 : ROUND_ROBIN_HASHER.hashLong(offset).hashCode());
        return instanceId == (hashValue % groupSize);
      }

View Full Code Here

    startTime = Math.max(startTime, System.currentTimeMillis() - streamConfig.getTTL());


    // Create the stream event reader
    FileReader<StreamEventOffset, Iterable<StreamFileOffset>> reader = createReader(streamConfig, startTime);
    try {
      ReadFilter readFilter = createReadFilter(startTime, endTime);
      List<StreamEvent> events = Lists.newArrayListWithCapacity(100);
      int eventsRead = reader.read(events, getReadLimit(limit), 0, TimeUnit.SECONDS, readFilter);


      // If empty already, return 204 no content
      if (eventsRead <= 0) {

View Full Code Here

   * @param startTime Start timestamp for event to be valid (inclusive).
   * @param endTime End timestamp fo event to be valid (exclusive).
   * @return A {@link ReadFilter} with the specific filtering property.
   */
  private ReadFilter createReadFilter(final long startTime, final long endTime) {
    return new ReadFilter() {


      private long hint;


      @Override
      public void reset() {

View Full Code Here

TOP

Related Classes of co.cask.cdap.data.file.ReadFilter

co.cask.cdap.data.stream.service.StreamFetchHandler

co.cask.cdap.data2.transaction.stream.AbstractStreamFileConsumer

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.