Examples of Fetcher

cn.edu.hfut.dmic.webcollector.fetcher.Fetcher
抓取器 @author hu
com.volantis.xml.pipeline.sax.drivers.uri.Fetcher
Fetches content from a URL and inserts it into the pipeline.
net.azib.ipscan.fetchers.Fetcher
Interface of all IP Fetchers. Fetcher is responsible for gathering a certain type of information about the provided scanning subject (in GUI terms, Fetcher is a column in the results list). Fetchers do the actual information fetching about each scanned IP address. Instances of this classes are shared among all the threads, so implementations must be thread safe and stateless. @author Anton Keks
org.apache.hadoop.mapred.CoronaJTState.Fetcher
org.apache.nutch.fetcher.Fetcher
A queue-based fetcher.
This fetcher uses a well-known model of one producer (a QueueFeeder) and many consumers (FetcherThread-s).
QueueFeeder reads input fetchlists and populates a set of FetchItemQueue-s, which hold FetchItem-s that describe the items to be fetched. There are as many queues as there are unique hosts, but at any given time the total number of fetch items in all queues is less than a fixed number (currently set to a multiple of the number of threads).
As items are consumed from the queues, the QueueFeeder continues to add new input items, so that their total count stays fixed (FetcherThread-s may also add new items to the queues e.g. as a results of redirection) - until all input items are exhausted, at which point the number of items in the queues begins to decrease. When this number reaches 0 fetcher will finish.
This fetcher implementation handles per-host blocking itself, instead of delegating this work to protocol-specific plugins. Each per-host queue handles its own "politeness" settings, such as the maximum number of concurrent requests and crawl delay between consecutive requests - and also a list of requests in progress, and the time the last request was finished. As FetcherThread-s ask for new items to be fetched, queues may return eligible items or null if for "politeness" reasons this host's queue is not yet ready.
If there are still unfetched items in the queues, but none of the items are ready, FetcherThread-s will spin-wait until either some items become available, or a timeout is reached (at which point the Fetcher will abort, assuming the task is hung). @author Andrzej Bialecki
org.apache.tez.runtime.library.shuffle.common.Fetcher
Responsible for fetching inputs served by the ShuffleHandler for a single host. Construct using {@link FetcherBuilder}
org.fenixedu.academic.ui.struts.action.teacher.siteArchive.Fetcher
The Fetcher manages a queue of {@link org.fenixedu.academic.ui.struts.action.teacher.siteArchive.Resource} and it'sresponsible for retrieving and transforming each resource in the queue.
Each resource is retrieved by creating a new RequestDispatcher to the resource's url and by forwarding the request to that dispatcher. The current request and response are wrapped to avoid unwanted secondary effects and to allow the called to generate it's own content to the user.
If the resource is an HTML page then url's present in the page are transformed using the resource's rules. @author cfgi
org.stringtree.Fetcher

Examples of org.stringtree.Fetcher

        context.put("value", "thing");
    }
    
    public void select(StringKeeper context) {
        String type = context.get("type");
        Fetcher source = (Fetcher) context.getObject(MojasefConstants.TEMPLATE_SOURCE + "." + type);
        context.put(Templater.TEMPLATE_SOURCE, source);
        context.put("value", "wotsit");
        context.put(MojasefConstants.PAGE_TEMPLATE, "home");
    }

View Full Code Here

Examples of org.stringtree.Fetcher

  private String getString(String name) {
        return FetcherStringHelper.getString(this, name);
    }


    private Iterator<String> listCookies() {
    Fetcher cookies = ListableHelper.subset(this, new StringPrefixFilter(HTTPConstants.RESPONSE_COOKIE));
    return ListableHelper.list(cookies);
  }

View Full Code Here

Examples of org.stringtree.Fetcher

    Fetcher cookies = ListableHelper.subset(this, new StringPrefixFilter(HTTPConstants.RESPONSE_COOKIE));
    return ListableHelper.list(cookies);
  }


  private Iterator<String> listHeaders() {
    Fetcher headers = ListableHelper.subset(this, new StringPrefixFilter(HTTPConstants.RESPONSE_HEADER));
    return ListableHelper.list(headers);
  }

View Full Code Here

Examples of org.stringtree.Fetcher

            System.err.println("Server.appCall unable to locate " + MojasefConstants.HTTP_APPLICATION);
        }
    }
    
    public void request(RequestContext req, OutputCollector res, Object dfl) {
        Fetcher context = new FallbackFetcher(req, common);
        
        try {
            Object file = null;
            res.start();
            String leaf = (String)context.getObject(MojasefConstants.REQUEST_LOCALPATH);
            if (leaf.indexOf(".") > 0) {
                Fetcher pub = (Fetcher)context.getObject(MojasefConstants.FILE_FETCHER);
                file = pub.getObject(leaf);
            }
            if (file != null) {
                String type = mimetype(leaf);
                req.put(HTTPConstants.RESPONSE_CONTENT_TYPE, type);
                res.write((byte[])file);

View Full Code Here

Examples of org.stringtree.Fetcher

  }


  public Object getObject(String name) {
    Object ret = context.getObject(Templater.TEMPLATE + "." + name);
        if (null == ret) {
            Fetcher fetcher = (Fetcher)context.getObject(MojasefConstants.TEMPLATE_SOURCE);
            if (null != fetcher) {
                ret = fetcher.getObject(name);
            }
        }
        
        return ret;
  }

View Full Code Here

Examples of org.stringtree.Fetcher


      put(MojasefConstants.SYSTEM_TIMESTAMP, new LiveDate());
  }
  
  protected void setTemplateFinder() {
        Fetcher source = (Fetcher) getObject(MojasefConstants.TEMPLATE_SOURCE);
        if (null == source) {
          if (null != templates) {
              source = templates;
          } else {
            source = this;

View Full Code Here

Examples of org.stringtree.Fetcher

        put(Templater.TEMPLATE, new ContextTemplateFetcher(this));
        put(Templater.TEMPLATER, templater);
  }
    
    protected void setFileFinder() {
        Fetcher fileFetcher = (Fetcher) getObject(MojasefConstants.FILE_FETCHER);
        if (null == fileFetcher) {
            if (pub == null) {
                pub = new ContextFileFetcher(this);
            }
            put(MojasefConstants.FILE_FETCHER, pub);

View Full Code Here

Examples of org.stringtree.Fetcher

    }


    @SuppressWarnings("unchecked")
    private void setApplicationContext() {
        Repository original = realRepository();
        Fetcher peelback = new PeelbackFetcher(original);
        Object object = peelback.getObject("http.application.context");
        if (null != object) {
            if (object instanceof Fetcher) {
                setOther(new FallbackRepository((Fetcher)object, original));
            } else if (object instanceof Map) {
                setOther(new FallbackRepository(new MapFetcher((Map<String, Object>)object), original));

View Full Code Here

0 1 2 3 4

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.