Examples of cascading.scheme.Scheme

cascading.scheme.Scheme
A Scheme defines what is stored in a {@link Tap} instance by declaring the {@link Tuple}field names, and alternately parsing or rendering the incoming or outgoing {@link Tuple}stream, respectively.
A Scheme defines the type of resource data will be sourced from or sinked to.
The default sourceFields are {@link Fields#UNKNOWN} and the default sinkFields are {@link Fields#ALL}.
Any given sourceFields only label the values in the {@link Tuple}s as they are sourced. It does not necessarily filter the output since a given implementation may choose to collapse values and ignore keys depending on the format.
If the sinkFields are {@link Fields#ALL}, the Cascading planner will attempt to resolve the actual field names and make them available via the {@link cascading.scheme.SinkCall#getOutgoingEntry()} method. Sometimes this maynot be possible (in the case the {@link Tap#openForWrite(cascading.flow.FlowProcess)} method is called from usercode directly (without planner intervention).
If the sinkFields are a valid selector, the {@link #sink(cascading.flow.FlowProcess,SinkCall)} method willonly see the fields expected.
Setting the {@code numSinkParts} value to 1 (one) attempts to ensure the output resource has only one part.In the case of MapReduce, this is only a suggestion for the Map side, on the Reduce side it does this by setting the number of reducers to the given value. This may affect performance, so be cautioned.
Note that setting numSinkParts does not force the planner to insert a final Reduce operation in the job, so numSinkParts may be ignored entirely if the final job is Map only. To force the Flow to have a final Reduce, add a {@link cascading.pipe.GroupBy} to the assembly before sinking.

  @Test
  public void testTrapTapSourceSink() throws Exception
    {
    getPlatform().copyFromLocal( inputFileApache );


    Scheme scheme = getPlatform().getTestFailScheme();


    Tap source = getPlatform().getTap( scheme, inputFileApache, SinkMode.KEEP );


    Pipe pipe = new Pipe( "map" );

View Full Code Here

  @Test
  public void testSourceConfInit() throws IOException
    {
    getPlatform().copyFromLocal( inputFileNums20 );


    Scheme scheme = new SchemeWithProperties( new Fields( "line" ) );
    Tap source = getPlatform().getTap( scheme, inputFileNums20, SinkMode.KEEP );


    Pipe pipe = new Pipe( "test" );


    Tap sink = getPlatform().getTextFile( getOutputPath( "sourceconfinit" ), SinkMode.REPLACE );

View Full Code Here


    Tap source = getPlatform().getTextFile( new Fields( "line" ), inputFileNums20, SinkMode.KEEP );


    Pipe pipe = new Pipe( "test" );


    Scheme scheme = new SchemeWithProperties( new Fields( "line" ) );
    Tap sink = getPlatform().getTap( scheme, getOutputPath( "sinkconfinit" ), SinkMode.REPLACE );


    Properties properties = new Properties();
    properties.setProperty( "default", "connector-default" );
    properties.setProperty( "replace", "connector-replace" );

View Full Code Here

    return tap;
    }


  public Scheme createScheme( Stereotype<Protocol, Format> stereotype, Protocol protocol, Format format, Properties properties )
    {
    Scheme scheme = null;
    try
      {
      LOG.info( "using " + getDescription() + " to create scheme for stereotype {} with properties: {}", stereotype.getName(), properties );
      scheme = factoryObject.createScheme( protocol.toString(), format.toString(), stereotype.getFields(), properties );

View Full Code Here


    if( formatHandler == null )
      throw new IllegalArgumentException( "no format handler for format: " + format );


    // do not make loadable, tap loadable will handle dynamic classloader issues
    Scheme scheme = formatHandler.createScheme( tableDef.getStereotype(), protocol, format );


    Resource<Protocol, Format, SinkMode> resource = tableDef.getResourceWith( sinkMode );


    return ( (LingualProtocolHandler) protocolHandler ).createLoadableTap( scheme, resource );
    }

View Full Code Here


    if( protocolHandler == null || formatHandler == null )
      return null;


    // do not make loadable, tap loadable will handle dynamic classloader issues
    Scheme scheme = formatHandler.createScheme( stereotype, resource.getProtocol(), resource.getFormat() );


    return ( (LingualProtocolHandler) protocolHandler ).createLoadableTap( scheme, resource );
    }

View Full Code Here

0 1

TOP

Related Classes of cascading.scheme.Scheme

cascading.flow.hadoop.BuildJobsHadoopPlatformTest

cascading.flow.planner.FlowPlanner

cascading.lingual.catalog.provider.ProviderProxy

cascading.lingual.catalog.SchemaCatalogManager

cascading.property.ConfigDefPlatformTest

cascading.tap.local.LocalTapPlatformTest

cascading.tap.MultiSourceTap

cascading.TraceTest

cascading.TrapPlatformTest

com.scaleunlimited.cascading.scheme.hadoop.SolrSchemeHadoopTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.