Examples of Fetcher

cn.edu.hfut.dmic.webcollector.fetcher.Fetcher
抓取器 @author hu
com.volantis.xml.pipeline.sax.drivers.uri.Fetcher
Fetches content from a URL and inserts it into the pipeline.
net.azib.ipscan.fetchers.Fetcher
Interface of all IP Fetchers. Fetcher is responsible for gathering a certain type of information about the provided scanning subject (in GUI terms, Fetcher is a column in the results list). Fetchers do the actual information fetching about each scanned IP address. Instances of this classes are shared among all the threads, so implementations must be thread safe and stateless. @author Anton Keks
org.apache.hadoop.mapred.CoronaJTState.Fetcher
org.apache.nutch.fetcher.Fetcher
A queue-based fetcher.
This fetcher uses a well-known model of one producer (a QueueFeeder) and many consumers (FetcherThread-s).
QueueFeeder reads input fetchlists and populates a set of FetchItemQueue-s, which hold FetchItem-s that describe the items to be fetched. There are as many queues as there are unique hosts, but at any given time the total number of fetch items in all queues is less than a fixed number (currently set to a multiple of the number of threads).
As items are consumed from the queues, the QueueFeeder continues to add new input items, so that their total count stays fixed (FetcherThread-s may also add new items to the queues e.g. as a results of redirection) - until all input items are exhausted, at which point the number of items in the queues begins to decrease. When this number reaches 0 fetcher will finish.
This fetcher implementation handles per-host blocking itself, instead of delegating this work to protocol-specific plugins. Each per-host queue handles its own "politeness" settings, such as the maximum number of concurrent requests and crawl delay between consecutive requests - and also a list of requests in progress, and the time the last request was finished. As FetcherThread-s ask for new items to be fetched, queues may return eligible items or null if for "politeness" reasons this host's queue is not yet ready.
If there are still unfetched items in the queues, but none of the items are ready, FetcherThread-s will spin-wait until either some items become available, or a timeout is reached (at which point the Fetcher will abort, assuming the task is hung). @author Andrzej Bialecki
org.apache.tez.runtime.library.shuffle.common.Fetcher
Responsible for fetching inputs served by the ShuffleHandler for a single host. Construct using {@link FetcherBuilder}
org.fenixedu.academic.ui.struts.action.teacher.siteArchive.Fetcher
The Fetcher manages a queue of {@link org.fenixedu.academic.ui.struts.action.teacher.siteArchive.Resource} and it'sresponsible for retrieving and transforming each resource in the queue.
Each resource is retrieved by creating a new RequestDispatcher to the resource's url and by forwarding the request to that dispatcher. The current request and response are wrapped to avoid unwanted secondary effects and to allow the called to generate it's own content to the user.
If the resource is an HTML page then url's present in the page are transformed using the resource's rules. @author cfgi
org.stringtree.Fetcher

Examples of org.apache.tez.runtime.library.shuffle.common.Fetcher

              if (LOG.isDebugEnabled()) {
                LOG.debug("Processing pending host: " + inputHost.toDetailedString());
              }
              if (inputHost.getNumPendingInputs() > 0) {
                LOG.info("Scheduling fetch for inputHost: " + inputHost.getIdentifier());
                Fetcher fetcher = constructFetcherForHost(inputHost);
                numRunningFetchers.incrementAndGet();
                if (isShutdown.get()) {
                  LOG.info("hasBeenShutdown, Breaking out of ShuffleScheduler Loop");
                }
                ListenableFuture<FetchResult> future = fetcherExecutor

View Full Code Here

Examples of org.apache.tez.runtime.library.shuffle.common.Fetcher

              if (LOG.isDebugEnabled()) {
                LOG.debug("Processing pending host: " + inputHost.toDetailedString());
              }
              if (inputHost.getNumPendingInputs() > 0) {
                LOG.info("Scheduling fetch for inputHost: " + inputHost.getHost());
                Fetcher fetcher = constructFetcherForHost(inputHost);
                numRunningFetchers.incrementAndGet();
                if (isShutdown.get()) {
                  LOG.info("hasBeenShutdown, Breaking out of ShuffleScheduler Loop");
                }
                ListenableFuture<FetchResult> future = fetcherExecutor

View Full Code Here

Examples of org.fenixedu.academic.ui.struts.action.teacher.siteArchive.Fetcher

        final Project project = getProject(request);
        final List<ProjectSubmission> projectSubmissions =
                new ArrayList<ProjectSubmission>(project.getLastProjectSubmissionForEachStudentGroup());


        Archive archive = new DiskZipArchive(response, project.getName());
        Fetcher fetcher = new Fetcher(archive, request, response);


        for (ProjectSubmission submission : projectSubmissions) {
            StudentGroup group = submission.getStudentGroup();


            fetcher.queue(new Resource(group.getGroupNumber() + getStudentsISTID(group) + "/"
                    + submission.getProjectSubmissionFile().getFilename(), submission.getProjectSubmissionFile().getDownloadUrl()));
        }


        fetcher.process();
        archive.finish();


        return null;
    }

View Full Code Here

Examples of org.fenixedu.academic.ui.struts.action.teacher.siteArchive.Fetcher

        Integer finishIndex = Math.min(projectSubmissions.size(), startIndex + pageSize);


        final List<ProjectSubmission> subList = projectSubmissions.subList(startIndex, finishIndex);


        Archive archive = new DiskZipArchive(response, project.getName() + "-" + (startIndex + 1) + "-" + finishIndex);
        Fetcher fetcher = new Fetcher(archive, request, response);


        for (ProjectSubmission submission : subList) {
            StudentGroup group = submission.getStudentGroup();


            fetcher.queue(new Resource(group.getGroupNumber() + getStudentsISTID(group) + "/"
                    + submission.getProjectSubmissionFile().getFilename(), submission.getProjectSubmissionFile().getDownloadUrl()));
        }


        fetcher.process();
        archive.finish();


        return null;
    }

View Full Code Here

Examples of org.stringtree.Fetcher

            query = req.getContent();
            req.put(Tract.CONTENT, "");
        }
        String url = req.get(MojasefConstants.REQUEST_URI);
        put(MojasefConstants.REQUEST_SELF, withoutQuery(url));
        Fetcher headers = req.getUnderlyingFetcher();
        if (headers instanceof Listable) {
            Listable<String> listable = (Listable<String>)headers;
            Iterator<String> it = listable.list();
            while (it.hasNext()) {
                String key = it.next();
                Object object = headers.getObject(key);
                if (object instanceof String) {
                    String value = (String)object;
                    if (key.startsWith(HTTPConstants.REQUEST_HEADER)) {
                        key = key.substring(HTTPConstants.REQUEST_HEADER.length());
                    }

View Full Code Here

Examples of org.stringtree.Fetcher


public class CommonContextFallbackTest extends TestCase {
    
    public void testGetFromContext() throws IOException {
        CommonContext context = new RemoteSpecCommonContext("src/test/files/fb/fa.spec");
        Fetcher tpls = (Fetcher)context.getObject(MojasefConstants.TEMPLATE_SOURCE);
        assertEquals("I am aa: ${*hello}", tpls.getObject("home"));
        assertEquals("Hello, ${value}!", tpls.getObject("hello"));
    }

View Full Code Here

Examples of org.stringtree.Fetcher

      new TractDirectoryRepository(pageDir, false, context), 
      new TractDirectoryRepository(tplDir, false, context));
  }


  public void setup(StringFinder context, Fetcher tracts, Fetcher templates) {
    Fetcher oldTemplates = (Fetcher)context.getObject(Templater.TEMPLATE);
    if (oldTemplates != null) {
      templates = new FallbackFetcher(
            templates, oldTemplates);
    } else {
      templates = oldTemplates;

View Full Code Here

Examples of org.stringtree.Fetcher

  }


    private static Object getTemplate(StringFinder context, String name) {
        Object ret = null;
        
        Fetcher fetcher = (Fetcher) context.getObject(Templater.TEMPLATE_SOURCE);
        if (null != fetcher) ret = fetcher.getObject(name);
        if (null != ret) return ret;
        
        String fullname = Templater.TEMPLATE + "." + name;
        return FetcherHelper.getPeelback(context.getUnderlyingFetcher(), fullname);
    }

View Full Code Here

Examples of org.stringtree.Fetcher

        return context.get(MojasefConstants.REQUEST_PATHOBJECT);
  }


  public static StringFinder createSubcontext(StringFinder context, String path, String mountpoint) {
      MapFetcher repository = new MapFetcher();
      Fetcher subcontext = new FallbackFetcher(repository, context.getUnderlyingFetcher());
      repository.put(MojasefConstants.MOUNTCONTEXT, getMountContext(context));
      repository.put(MojasefConstants.MOUNTPOINT, mountpoint);
        repository.put(MojasefConstants.REQUEST_LOCALPATH, path);
      return new FetcherStringKeeper(subcontext);
  }

View Full Code Here

Examples of org.stringtree.Fetcher

      StringFinder context, Object application, Object dfl) throws IOException {
    
    String content = null;
    Token token = Token.CONTINUE;


    Fetcher underlying = context.getUnderlyingFetcher();
    Storer storer = StorerHelper.find(context);
        storer.put(Templater.BASE, context);
        
    if (application != null) {
      StringCollector buffer = new ByteArrayStringCollector();

View Full Code Here

0 1 2 3 4

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.