Examples of thredds.crawlabledataset.CrawlableDataset

thredds.crawlabledataset.CrawlableDataset
CrawlableDataset represents an abstract dataset that is part of a hierarchical dataset collection. Parent and child datasets can be accessed allowing the collection to be crawled.

The CrawlableDataset interface is a generalization (and simplification) of the java.io.File class. A CrawlableDataset path is made up of ONE or more path segments each seperated by a slash ("/"). The path may start or end with a slash ("/").

Implementation Notes:
1. The thredds.crawlabledataset.CrawlableDatasetFactory requires each CrawlableDataset implementation to define a public constructor with one String argument and one Object argument. The String argument is the path for the CrawlableDataset being constructed, the Object argument is a configuration object.
2. The thredds.cataloggen.CollectionLevelScanner framework does not support building a catalog for the collection based at the CrawlableDataset path "/". So, do not implement your CrawlableDataset so that the path "/" is allowed, or at least so that it is not likely to be used as the root of a dataset collection (e.g., CrawlableDatasetFile is probably safe because the root directory ("/") should never be used as the base of a data collection. If the backend data source on top of which you are implementing CrawlabeDataset uses "/" as its root, you can simply prepend a string (e.g., "myDataCollection" or "root") to the backend path for the CrawlableDataset view of the path.
@author edavis @see thredds.cataloggen.CollectionLevelScanner CollectionLevelScanner uses CrawlableDatasets to scan a dataset collection and create a THREDDS catalog. @since May 3, 2005 20:18:59 -0600

    TestFileDirUtils.addFile( gfs211Dir, "2004050312_gfs_211.nc" );
    TestFileDirUtils.addFile( gfs211Dir, "2004050318_gfs_211.nc" );




    File catWriteDir = new File( tmpDir, "catWriteDir");
    CrawlableDataset collectionCrDs = new CrawlableDatasetFile( tmpDir );
    InvService service = new InvService( "myServer", "File", collectionCrDs.getPath() + "/", null, null );
    CrawlableDatasetFilter filter = null;
    CrawlableDataset topCatCrDs = collectionCrDs.getDescendant( startPath );


    CatGenAndWrite cgaw = null;
    try
    {
      cgaw = new CatGenAndWrite( "DATA", "My data", "", service,
                                 collectionCrDs, topCatCrDs, filter, null, catWriteDir );
    }
    catch ( IllegalArgumentException e )
    {
      fail( "Bad argument: " + e.getMessage() );
      return;
    }


    try
    {
      cgaw.genAndWriteCatalogTree();
    }
    catch ( IOException e )
    {
      fail( "I/O error generating and writing catalogs at and under \"" + topCatCrDs.getPath() + "\": " + e.getMessage() );
      return;
    }


    crawlCatalogs( new File( new File( catWriteDir, startPath), "catalog.xml") );
  }

View Full Code Here

    TestFileDirUtils.addFile( gfs211Dir, "2004050312_gfs_211.nc" );
    TestFileDirUtils.addFile( gfs211Dir, "2004050318_gfs_211.nc" );




    File catWriteDir = new File( tmpDir, "catWriteDir" );
    CrawlableDataset collectionCrDs = new CrawlableDatasetFile( tmpDir );


    InvService service = new InvService( "myServer", "OPeNDAP", "/thredds/dodsC/", null, null );
    CrawlableDatasetFilter filter = null;
    CrawlableDataset topCatCrDs = collectionCrDs.getDescendant( startPath );


    CatGenAndWrite cgaw = null;
    try
    {
      cgaw = new CatGenAndWrite( "DATA", "My data", "tdr", service,
                                 collectionCrDs, topCatCrDs, filter, null, catWriteDir );
    }
    catch ( IllegalArgumentException e )
    {
      fail( "Bad argument: " + e.getMessage() );
      return;
    }


    try
    {
      cgaw.genAndWriteCatalogTree();
    }
    catch ( IOException e )
    {
      fail( "I/O error generating and writing catalogs at and under \"" + topCatCrDs.getPath() + "\": " + e.getMessage() );
      return;
    }


    crawlCatalogs( new File( new File( catWriteDir, startPath), "catalog.xml") );
  }

View Full Code Here

                    .getInstanceToMatchOnDatasetName(
                            matchPattern, substitutionPattern, duration );
    assertTrue( timeCoverageEnhancer != null );


    InvDatasetImpl ds = new InvDatasetImpl( null, dsName );
    CrawlableDataset crDs = new MockCrawlableDataset( dsName, false );


    assertTrue( "Failed to add metadata.",
                timeCoverageEnhancer.addMetadata( ds, crDs ));
  }

View Full Code Here

                    .getInstanceToMatchOnDatasetName(
                            matchPattern, substitutionPattern, duration );
    assertTrue( timeCoverageEnhancer != null );


    InvDatasetImpl ds = new InvDatasetImpl( null, dsName );
    CrawlableDataset crDs = new MockCrawlableDataset( dsName, false );


    assertTrue( "Unexpected success adding metadata.",
                ! timeCoverageEnhancer.addMetadata( ds, crDs ));
  }

View Full Code Here

                    .getInstanceToMatchOnDatasetPath(
                            matchPattern, substitutionPattern, duration );
    assertTrue( timeCoverageEnhancer != null );


    InvDatasetImpl ds = new InvDatasetImpl( null, dsName );
    CrawlableDataset crDs = new MockCrawlableDataset( dsName, false );


    assertTrue( "Failed to add metadata.",
                timeCoverageEnhancer.addMetadata( ds, crDs ) );


    // This dataset hasn't been finalized so ds.getTimeCoverage() doesn't work.

View Full Code Here

    File dataDirFile = new File( dataDir );
    InvService service = new InvService( this.serviceId, this.serviceType.toString(), this.serviceBase, null, null );
    DirectoryScanner me = new DirectoryScanner( service, this.serviceTitle, new File( dataDirHeader ), null, true );
    assertTrue( me != null );


    CrawlableDataset catalogCrDs;
    try
    {
      catalogCrDs = CrawlableDatasetFactory.createCrawlableDataset( dataDirFile.getAbsolutePath(), null, null );
    }
    catch ( ClassNotFoundException e )
    {
      throw new IllegalArgumentException( "Did not find class: " + e.getMessage() );
    }
    catch ( NoSuchMethodException e )
    {
      throw new IllegalArgumentException( "Required constructor not found in class: " + e.getMessage() );
    }
    catch ( IllegalAccessException e )
    {
      throw new IllegalArgumentException( "Did not have necessary access to class: " + e.getMessage() );
    }
    catch ( InvocationTargetException e )
    {
      throw new IllegalArgumentException( "Could not invoke required method in class: " + e.getMessage() );
    }
    catch ( InstantiationException e )
    {
      throw new IllegalArgumentException( "Could not instatiate class: " + e.getMessage() );
    }
    if ( ! catalogCrDs.isCollection() )
      throw new IllegalArgumentException( "catalog directory is not a directory <" + dataDir + ">." );


    InvCatalog catalog = me.getDirCatalog( catalogCrDs, ".*", true, "idBase", false, null, null, null);


    // Compare the resulting catalog an the expected catalog resource.

View Full Code Here

    File dataDirFile = new File( dataDir );
    InvService service = new InvService( this.serviceId, this.serviceType.toString(), this.serviceBase, null, null );
    DirectoryScanner me = new DirectoryScanner( service, this.serviceTitle, new File( dataDirHeader ), null, true );
    assertTrue( me != null );


    CrawlableDataset catalogCrDs;
    try
    {
      catalogCrDs = CrawlableDatasetFactory.createCrawlableDataset( dataDirFile.getAbsolutePath(), null, null );
    }
    catch ( ClassNotFoundException e )
    {
      throw new IllegalArgumentException( "Did not find class: " + e.getMessage() );
    }
    catch ( NoSuchMethodException e )
    {
      throw new IllegalArgumentException( "Required constructor not found in class: " + e.getMessage() );
    }
    catch ( IllegalAccessException e )
    {
      throw new IllegalArgumentException( "Did not have necessary access to class: " + e.getMessage() );
    }
    catch ( InvocationTargetException e )
    {
      throw new IllegalArgumentException( "Could not invoke required method in class: " + e.getMessage() );
    }
    catch ( InstantiationException e )
    {
      throw new IllegalArgumentException( "Could not instatiate class: " + e.getMessage() );
    }
    if ( ! catalogCrDs.isCollection() )
      throw new IllegalArgumentException( "catalog directory is not a directory <" + dataDir + ">." );


    InvCatalog catalog = me.getDirCatalog( catalogCrDs, ".*", true, "idBase", false, null, null, null);


    // Compare the resulting catalog an the expected catalog resource.

View Full Code Here

  public void testBasic() throws IOException
  {
    String collectionPath = "src/test/data/thredds/cataloggen/testData/modelNotFlat";
    String catalogPath = "src/test/data/thredds/cataloggen/testData/modelNotFlat/eta_211";


    CrawlableDataset collectionCrDs = null;
    try
    {
      collectionCrDs = CrawlableDatasetFactory.createCrawlableDataset( collectionPath, null, null );
    }
    catch ( Exception e )
    {
      assertTrue( "Failed to create collection level dataset <" + collectionPath + ">: " + e.getMessage(),
                  false );
      return;
    }


    SimpleCatalogBuilder builder = new SimpleCatalogBuilder( "", collectionCrDs, "server", "OPENDAP", "http://my.server/opendap/" );
    assertTrue( "SimpleCatalogBuilder is null.",
                builder != null );


    CrawlableDataset catalogCrDs = null;
    try
    {
      catalogCrDs = CrawlableDatasetFactory.createCrawlableDataset( catalogPath, null, null );
    }
    catch ( Exception e )

View Full Code Here

    String resultResourceName = resourcePath + "/" + resource_simpleWithEmptyServiceBase_result;
    File expectedCatalogDocFile = new File( resultResourceName );


    String collectionPath = "src/test/data/thredds/cataloggen/testData/modelNotFlat";
    String catalogPath = "src/test/data/thredds/cataloggen/testData/modelNotFlat/eta_211";
    CrawlableDataset collCrDs;
    CrawlableDataset catCrDs;
    try
    {
      collCrDs = CrawlableDatasetFactory.createCrawlableDataset( collectionPath, null, null );
      catCrDs = CrawlableDatasetFactory.createCrawlableDataset( catalogPath, null, null );
    }

View Full Code Here

    String resultResourceName = resourcePath + "/" + resource_simpleWithNotEmptyServiceBase_result;
    File expectedCatalogDocFile = new File( resultResourceName );


    String collectionPath = "src/test/data/thredds/cataloggen/testData/modelNotFlat";
    String catalogPath = "src/test/data/thredds/cataloggen/testData/modelNotFlat/eta_211";
    CrawlableDataset collCrDs;
    CrawlableDataset catCrDs;
    try
    {
      collCrDs = CrawlableDatasetFactory.createCrawlableDataset( collectionPath, null, null );
      catCrDs = CrawlableDatasetFactory.createCrawlableDataset( catalogPath, null, null );
    }

View Full Code Here

0 1 2

TOP

Related Classes of thredds.crawlabledataset.CrawlableDataset

thredds.cataloggen.CatalogBuilderHelper

thredds.cataloggen.CatGenAndWrite

thredds.cataloggen.datasetenhancer.TestRegExpAndDurationTimeCoverageEnhancer

thredds.cataloggen.DirectoryScanner

thredds.cataloggen.inserter.LatestCompleteProxyDsHandler

thredds.cataloggen.TestCatGenAndWrite

thredds.cataloggen.TestCollectionLevelScanner

thredds.cataloggen.TestDirectoryScanner

thredds.cataloggen.TestSimpleCatalogBuilder

thredds.crawlabledataset.filter.TestLogicalFilterComposer

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.