List of harvestSource() Examples

Examples of harvestSource()

com.ikanow.infinit.e.harvest.HarvestController.harvestSource()
Handles going through what to do with a source for harvesting The process currently is: 1. Extract from source 2. Enrich with metadata from toAdd (entity, fulltext, events, etc) @param source The source to harvest

Examples of com.ikanow.infinit.e.harvest.HarvestController.harvestSource()

    BasicDBObject query = new BasicDBObject("extractType", "Feed");
    // A useful source known to work during V0S1 testing:
    query.put("key", "arstechnica.com.tech-policy.2012.10.last-android-vendor-f.147.4.");
    
    SourcePojo feedSource = SourcePojo.fromDb(DbManager.getIngest().getSource().findOne(query), SourcePojo.class);
    hc.harvestSource(feedSource, toAdd_feed, toUpdate_feed, toDelete_feed);
    System.out.println("############# Retrieved sample feed documents: " + toAdd_feed.size() + " from " + feedSource.getUrl());
    
    // 1] Test the store and index manager by itself:
    StoreAndIndexManager.setDiagnosticMode(true);

View Full Code Here

Examples of com.ikanow.infinit.e.harvest.HarvestController.harvestSource()

          source.getHarvestStatus().setHarvest_message(federatedQueryEnts + " entities extracted");
        }
        
      }//TESTED (END FEDERATED QUERY TEST MODE, WHICH IS A BIT DIFFERENT)
      else {
        harvester.harvestSource(source, toAdd, toUpdate, toRemove);
      }      
      
      // (don't parrot the old message back - v confusing)
      if (oldMessage == source.getHarvestStatus().getHarvest_message()) { // (ptr ==)
        source.getHarvestStatus().setHarvest_message("(no documents extracted - likely a source or configuration error)");

View Full Code Here

Examples of com.ikanow.infinit.e.harvest.HarvestController.harvestSource()

        List<DocumentPojo> toAdd = new LinkedList<DocumentPojo>();
        List<DocumentPojo> toUpdate = new LinkedList<DocumentPojo>();
        List<DocumentPojo> toRemove = new LinkedList<DocumentPojo>();        
        
        hc = _harvesterController.get();
        hc.harvestSource(_sourceToProcess, toAdd, toUpdate, toRemove);
          // (toAdd includes toUpdate)
        
        if (HarvestEnum.error != _sourceToProcess.getHarvestStatus().getHarvest_status()) {
          gpc = _genericController.get();
          gpc.processDocuments(SourceUtils.getHarvestType(_sourceToProcess), toAdd, toUpdate, toRemove, _sourceToProcess);

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.