Examples of cascading.tap.hadoop.Lfs

cascading.tap.hadoop.Lfs
Class Lfs is a {@link cascading.tap.Tap} class that provides access to the Local File System via Hadoop.
Note that using a Lfs {@link cascading.tap.Tap} instance in a {@link cascading.flow.Flow} will force a portion of not the whole Flow to be executedin "local" mode forcing the Flow to execute in the current JVM. Mixing with {@link cascading.tap.hadoop.Dfs} and other Taptypes is possible, providing a means to implement complex file/data management functions.
Use {@link cascading.tap.hadoop.Hfs} if you need a Tap instance that inherits the default {@link FileSystem} used by Hadoop.

    similarityFlow.complete();
    }


  public static Tap makeTap( String path, Scheme scheme )
    {
    return path.matches( "^[^:]+://.*" ) ? new Hfs( scheme, path ) : new Lfs( scheme, path );
    }

View Full Code Here

    if( stepStatePath == null )
      throw new FlowException( "unable to find step state from distributed cache" );


    LOG.info( "reading step state from local path: {}", stepStatePath );


    Hfs temp = new Lfs( new TextLine( new Fields( "line" ) ), stepStatePath.toString() );


    TupleEntryIterator reader = null;


    try
      {
      reader = temp.openForRead( new HadoopFlowProcess( jobConf ) );


      if( !reader.hasNext() )
        throw new FlowException( "step state path is empty: " + temp.getIdentifier() );


      return reader.next().getString( 0 );
      }
    catch( IOException exception )
      {
      throw new FlowException( "unable to find state path: " + temp.getIdentifier(), exception );
      }
    finally
      {
      try
        {

View Full Code Here

  // test is not executed, just guarantees flow is run locally


  @Test
  public void testLocalModeSource() throws Exception
    {
    Tap source = new Lfs( new TextLine(), "input/path" );
    Tap sink = new Hfs( new TextLine(), "output/path", SinkMode.REPLACE );


    Pipe pipe = new Pipe( "test" );


    Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );

View Full Code Here


  @Test
  public void testLocalModeSink() throws Exception
    {
    Tap source = new Hfs( new TextLine(), "input/path" );
    Tap sink = new Lfs( new TextLine(), "output/path", SinkMode.REPLACE );


    Pipe pipe = new Pipe( "test" );


    Flow flow = getPlatform().getFlowConnector().connect( source, sink, pipe );

View Full Code Here

    }


  @Test
  public void testFlowID() throws Exception
    {
    Tap source = new Lfs( new TextLine(), "input/path" );
    Tap sink = new Hfs( new TextLine(), "output/path", true );


    Pipe pipe = new Pipe( "test" );


    Map<Object, Object> props = getProperties();

View Full Code Here

    }


  @Test
  public void testCopyConfig() throws Exception
    {
    Tap source = new Lfs( new TextLine(), "input/path" );
    Tap sink = new Hfs( new TextLine(), "output/path", true );


    Pipe pipe = new Pipe( "test" );


    Configuration conf = ( (BaseHadoopPlatform) getPlatform() ).getConfiguration();

View Full Code Here

        defaults.set("parquet.benchmark.bytes.total", "false");
        defaults.set("parquet.benchmark.time.read", "false");


        ((LocalScheme<SourceCtx, SinkCtx>) this.getScheme()).setDefaults(defaults);


        lfs = new Lfs(scheme, path);
        ((LocalScheme<SourceCtx, SinkCtx>) this.getScheme()).setLfs(lfs);
    }

View Full Code Here

      new Fields("in"));


    Pipe pipe = new Pipe("batch-pipe");
    pipe = new Each(pipe, new AnnotateWithSizeOfCurrentBatch(4));


    Tap<JobConf, RecordReader, OutputCollector> dst = new Lfs(new SequenceFile(OUT_FIELD), getTestRoot()+"/out");


    Flow f = CascadingUtil.get().getFlowConnector().connect(src, dst, pipe);
    f.complete();


    TupleEntryIterator tupleEntryIterator = dst.openForRead(CascadingUtil.get().getFlowProcess());
    List<BytesWritable> outStrings = new ArrayList<BytesWritable>(6);
    while (tupleEntryIterator.hasNext()) {
      TupleEntry next = tupleEntryIterator.next();
      BytesWritable bytes = (BytesWritable) next.getObject(OUT_FIELD);
      outStrings.add(bytes);

View Full Code Here

TOP

Related Classes of cascading.tap.hadoop.Lfs

cascading.flow.hadoop.FlowPlatformTest

cascading.flow.hadoop.util.HadoopMRUtil

com.etsy.cascading.tap.local.LocalTap

com.liveramp.cascading_ext.operation.TestBatchFunction

sample.recommender.Main

cascading.tap.TapException

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.