Examples of thredds.catalog.crawl.CatalogCrawler

thredds.catalog.crawl.CatalogCrawler
This crawls a catalog tree for its datasets, which are sent to a listener. You can get all or some of the datasets. A "direct" dataset is one which hasAccess() is true, meaning it has one or more access elements.
Example use:
```
 CatalogCrawler.Listener listener = new CatalogCrawler.Listener() { public void getDataset(InvDataset dd) { if (dd.isHarvest()) doHarvest(dd); } }; CatalogCrawler crawler = new CatalogCrawler( CatalogCrawler.USE_ALL_DIRECT, false, listener); 
```
@author John Caron

    if (!isValid) {
      log.warn("Catalog invalid= "+catalogUrl+" validation output= "+ buff);
      return;
    }


    CatalogCrawler crawler = new CatalogCrawler(CatalogCrawler.USE_ALL_DIRECT, false, this);
    long start = System.currentTimeMillis();
    try {
      crawler.crawl(cat, null, null, map);
    } finally {
      long took = (System.currentTimeMillis() - start);
      if (debug) System.out.format("***Done " + catalogUrl + " took = " + took + " msecs\n");
    }
  }

View Full Code Here


     System.out.println("***makeFromCatalog "+catURL);
     long startTime = System.currentTimeMillis();
     FmrcInventory fmrCollection = new FmrcInventory(fmrcDefinitionPath, collectionName);


     CatalogCrawler crawler = new CatalogCrawler(CatalogCrawler.USE_ALL_DIRECT, false,
             new MyListener(fmrCollection, maxDatasets, mode, cache));


     crawler.crawl(catURL, null, System.out, null);
     fmrCollection.finish();


     if (debugTiming) {
       long took = System.currentTimeMillis() - startTime;
       System.out.println("that took = "+took+" msecs");

View Full Code Here

      
    };


    ByteArrayOutputStream bis = new ByteArrayOutputStream();
    PrintStream ps = new PrintStream( bis);
    CatalogCrawler crawler = new CatalogCrawler( CatalogCrawler.USE_ALL, true, listener);
    crawler.crawl(cat, null, ps, null);
    mess.append("\n*********************\n");
    mess.append(bis.toString());
  }

View Full Code Here

      
    };


    ByteArrayOutputStream bis = new ByteArrayOutputStream();
    PrintStream ps = new PrintStream( bis);
    CatalogCrawler crawler = new CatalogCrawler( CatalogCrawler.USE_ALL, true, listener);
    crawler.crawl(cat, null, ps, null);
    mess.append("\n*********************\n");
    mess.append(bis.toString());
  }

View Full Code Here


    };


    LOG.log(Level.FINE, "catalogURL: " + this.mainCatalogURL);
    // Note: look for all datasets, that have either a urlPath="" attribute, or a <access> subelement
    CatalogCrawler crawler = new CatalogCrawler(CatalogCrawler.USE_ALL, false, listener);
    crawler.crawl(this.mainCatalogURL, ignore, System.out, this);
    this.allUrls = listener.getURLs();
    this.datasetMet = listener.getDatasetMet();
  }

View Full Code Here

      }


    };


    LOG.log(Level.INFO, "catalogURL: " + this.mainCatalogURL);
    CatalogCrawler crawler = new CatalogCrawler(CatalogCrawler.USE_ALL_DIRECT,
        false, listener);
    crawler.crawl(this.mainCatalogURL, ignore, System.out, this);
    this.allUrls = listener.getURLs();
    this.datasetMet = listener.getDatasetMet();
  }

View Full Code Here

TOP

Related Classes of thredds.catalog.crawl.CatalogCrawler

org.apache.oodt.opendapps.DatasetExtractor

thredds.catalog.dl.ADNWriter

thredds.catalog.dl.DIFWriter

thredds.catalog.InvCatalogFactory

thredds.catalog.InvCatalogImpl

thredds.catalog.InvCatalogRef

thredds.catalog.InvDataset

thredds.inventory.DatasetCollectionFromCatalog

ucar.nc2.dt.fmrc.FmrcInventory

java.util.Random

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.