Examples of Fetcher

cn.edu.hfut.dmic.webcollector.fetcher.Fetcher
抓取器 @author hu
com.volantis.xml.pipeline.sax.drivers.uri.Fetcher
Fetches content from a URL and inserts it into the pipeline.
net.azib.ipscan.fetchers.Fetcher
Interface of all IP Fetchers. Fetcher is responsible for gathering a certain type of information about the provided scanning subject (in GUI terms, Fetcher is a column in the results list). Fetchers do the actual information fetching about each scanned IP address. Instances of this classes are shared among all the threads, so implementations must be thread safe and stateless. @author Anton Keks
org.apache.hadoop.mapred.CoronaJTState.Fetcher
org.apache.nutch.fetcher.Fetcher
A queue-based fetcher.
This fetcher uses a well-known model of one producer (a QueueFeeder) and many consumers (FetcherThread-s).
QueueFeeder reads input fetchlists and populates a set of FetchItemQueue-s, which hold FetchItem-s that describe the items to be fetched. There are as many queues as there are unique hosts, but at any given time the total number of fetch items in all queues is less than a fixed number (currently set to a multiple of the number of threads).
As items are consumed from the queues, the QueueFeeder continues to add new input items, so that their total count stays fixed (FetcherThread-s may also add new items to the queues e.g. as a results of redirection) - until all input items are exhausted, at which point the number of items in the queues begins to decrease. When this number reaches 0 fetcher will finish.
This fetcher implementation handles per-host blocking itself, instead of delegating this work to protocol-specific plugins. Each per-host queue handles its own "politeness" settings, such as the maximum number of concurrent requests and crawl delay between consecutive requests - and also a list of requests in progress, and the time the last request was finished. As FetcherThread-s ask for new items to be fetched, queues may return eligible items or null if for "politeness" reasons this host's queue is not yet ready.
If there are still unfetched items in the queues, but none of the items are ready, FetcherThread-s will spin-wait until either some items become available, or a timeout is reached (at which point the Fetcher will abort, assuming the task is hung). @author Andrzej Bialecki
org.apache.tez.runtime.library.shuffle.common.Fetcher
Responsible for fetching inputs served by the ShuffleHandler for a single host. Construct using {@link FetcherBuilder}
org.fenixedu.academic.ui.struts.action.teacher.siteArchive.Fetcher
The Fetcher manages a queue of {@link org.fenixedu.academic.ui.struts.action.teacher.siteArchive.Resource} and it'sresponsible for retrieving and transforming each resource in the queue.
Each resource is retrieved by creating a new RequestDispatcher to the resource's url and by forwarding the request to that dispatcher. The current request and response are wrapped to avoid unwanted secondary effects and to allow the called to generate it's own content to the user.
If the resource is an HTML page then url's present in the page are transformed using the resource's rules. @author cfgi
org.stringtree.Fetcher

Examples of cn.edu.hfut.dmic.webcollector.fetcher.Fetcher

    }
    
    
    @Override
    public Fetcher createFetcher() {
        Fetcher fetcher = new Fetcher();
        fetcher.setNeedUpdateDb(true);
        fetcher.setIsContentStored(isContentStored);
        conconfig = new CommonConnectionConfig(useragent, cookie);
        fetcher.setThreads(getThreads());
        return fetcher;
    }

View Full Code Here

Examples of com.volantis.xml.pipeline.sax.drivers.uri.Fetcher


            XMLPipeline pipeline = dynamicProcess.getPipeline();


            // In order to get the required template definition to be processed
            // we need to invoke a URI inclusion based on the given href.
            Fetcher fetcher = new Fetcher(pipeline);
            fetcher.setHref(href);


            fetcher.doInclude();
        }


        super.doAction(dynamicProcess);
    }

View Full Code Here

Examples of com.volantis.xml.pipeline.sax.drivers.uri.Fetcher


        if (!"".equals(queryString)) {
            hrefAttr.append("?").append(queryString);
        }


        Fetcher operation = new Fetcher(pipeline);


        // OK to pass null strings to operations attribute setters
        // Operation process can validate the attributes whenever
        // it performs its operation
        operation.setHref(hrefAttr.toString());
        operation.setParse(PARSE_ATTRIBUTE_VALUE);
        operation.setEncoding(ENCODING_ATTRIBUTE_VALUE);


        // Handle the timeout setting. Firstly, we need to see if there is a
        // timeout attribute. We default to -1 to indicate no timeout
        String timeoutAttr = TIMEOUT_ATTRIBUTE_VALUE;
        Period timeout = null;


        if (timeoutAttr != null) {
            // The attribute should contain an integer number of seconds
            int timeoutInSecs;
            try {
                timeoutInSecs = Integer.parseInt(timeoutAttr);
            } catch (NumberFormatException e) {
                LOGGER.warn("invalid-timeout", timeoutAttr);
                throw e;
            }


            timeout = Period.treatNonPositiveAsIndefinitely(
                    timeoutInSecs * 1000);
    }


        operation.setTimeout(timeout);


        operation.doInclude();


        return null;
    }

View Full Code Here

Examples of com.volantis.xml.pipeline.sax.drivers.uri.Fetcher

        }


        queryString.append("&").append(MessageFormat.format(
                FLICKR_PAGE_PARAM, pageIndex));


        Fetcher operation = new Fetcher(pipeline);


        // OK to pass null strings to operations attribute setters
        // Operation process can validate the attributes whenever
        // it performs its operation
        operation.setHref(FLICKR_REST_API_URL + queryString.toString());
        operation.setParse(PARSE_ATTRIBUTE_VALUE);
        operation.setEncoding(ENCODING_ATTRIBUTE_VALUE);


        // Handle the timeout setting. Firstly, we need to see if there is a
        // timeout attribute. We default to -1 to indicate no timeout
        String timeoutAttr = TIMEOUT_ATTRIBUTE_VALUE;
        Period timeout = null;


        if (timeoutAttr != null) {
            // The attribute should contain an integer number of seconds
            int timeoutInSecs;
            try {
                timeoutInSecs = Integer.parseInt(timeoutAttr);
            } catch (NumberFormatException e) {
                LOGGER.warn("invalid-timeout", timeoutAttr);
                throw e;
            }


            timeout = Period.treatNonPositiveAsIndefinitely(
                    timeoutInSecs * 1000);
        }


        operation.setTimeout(timeout);


        operation.doInclude();


        return null;
    }

View Full Code Here

Examples of net.azib.ipscan.fetchers.Fetcher

    selectedFetchersList = new List(shell, SWT.BORDER | SWT.MULTI | SWT.V_SCROLL);
    selectedFetchersList.setLayoutData(LayoutHelper.formData(140, 200, new FormAttachment(0), new FormAttachment(selectedLabel, 80, SWT.RIGHT), new FormAttachment(selectedLabel), null));
    Iterator<Fetcher> i = fetcherRegistry.getSelectedFetchers().iterator();
    i.next();  // skip IP
    while (i.hasNext()) {
      Fetcher fetcher = i.next();
      selectedFetchersList.add(fetcher.getName());
    }
    
    Button upButton = new Button(shell, SWT.NONE);
    upButton.setText(Labels.getLabel("button.up"));  
    
    Button downButton = new Button(shell, SWT.NONE);
    downButton.setText(Labels.getLabel("button.down"));
    
    Button addButton = new Button(shell, SWT.NONE);
    addButton.setText(Labels.getLabel("button.left"));


    Button removeButton = new Button(shell, SWT.NONE);
    removeButton.setText(Labels.getLabel("button.right"));
    
    Button prefsButton = new Button(shell, SWT.NONE);
    prefsButton.setText(Labels.getLabel("button.fetcherPrefs"));
    prefsButton.setToolTipText(Labels.getLabel("text.fetchers.preferences"));
    
    upButton.setLayoutData(LayoutHelper.formData(new FormAttachment(selectedFetchersList), new FormAttachment(downButton, 0, SWT.RIGHT), new FormAttachment(selectedLabel), null));
    downButton.setLayoutData(LayoutHelper.formData(new FormAttachment(selectedFetchersList), null, new FormAttachment(upButton), null));
    addButton.setLayoutData(LayoutHelper.formData(new FormAttachment(selectedFetchersList), new FormAttachment(downButton, 0, SWT.RIGHT), new FormAttachment(downButton, 16), null));  
    removeButton.setLayoutData(LayoutHelper.formData(new FormAttachment(selectedFetchersList), new FormAttachment(downButton, 0, SWT.RIGHT), new FormAttachment(addButton), null));
    prefsButton.setLayoutData(LayoutHelper.formData(new FormAttachment(selectedFetchersList), new FormAttachment(downButton, 0, SWT.RIGHT), new FormAttachment(removeButton, 16), null));
    
    Label registeredLabel = new Label(shell, SWT.NONE);
    registeredLabel.setText(Labels.getLabel("text.fetchers.availableList"));    
    registeredLabel.setLayoutData(LayoutHelper.formData(new FormAttachment(downButton, 10), null, new FormAttachment(messageLabel, 5), null));
    
    registeredFetchersList = new List(shell, SWT.BORDER | SWT.MULTI | SWT.V_SCROLL);
    registeredFetchersList.setLayoutData(LayoutHelper.formData(140, 200, new FormAttachment(downButton, 10), null, new FormAttachment(registeredLabel), null));
    i = fetcherRegistry.getRegisteredFetchers().iterator();
    i.next(); // skip IP
    while (i.hasNext()) {
      Fetcher fetcher = i.next();
      String fetcherName = fetcher.getName();
      registeredFetcherIdsByNames.put(fetcherName, fetcher.getId());
      if (selectedFetchersList.indexOf(fetcherName) < 0)
        registeredFetchersList.add(fetcherName);
    }


    Button okButton = new Button(shell, SWT.NONE);

View Full Code Here

Examples of net.azib.ipscan.fetchers.Fetcher

    }


    public void handleEvent(Event e) {
      // modify menu text a bit
      TableColumn tableColumn = (TableColumn) e.widget;
      Fetcher fetcher = (Fetcher) tableColumn.getData();
      
      MenuItem sortMenuItem = columnsMenu.getItem(0);
      MenuItem preferencesMenuItem = columnsMenu.getItem(1);
      MenuItem aboutMenuItem = columnsMenu.getItem(2);


      if (tableColumn.getParent().getSortColumn() == tableColumn)
        sortMenuItem.setText(Labels.getLabel("menu.columns.sortDirection"));
      else
        sortMenuItem.setText(Labels.getLabel("menu.columns.sortBy") + " " + fetcher.getName());


      // disable these menu items if scanning
      sortMenuItem.setEnabled(stateMachine.inState(ScanningState.IDLE));


      preferencesMenuItem.setText(fetcher.getName() + " " + Labels.getLabel("menu.columns.preferences"));
      preferencesMenuItem.setEnabled(fetcher.getPreferencesClass() != null && stateMachine.inState(ScanningState.IDLE));


      aboutMenuItem.setText(Labels.getLabel("menu.columns.about") + " " + fetcher.getName());
      
      // focus the table to make Enter work after using the menu
      tableColumn.getParent().forceFocus();
      
      // remember the clicked column (see SortBy, FetcherPreferences, and AboutFetcher below)

View Full Code Here

Examples of net.azib.ipscan.fetchers.Fetcher


    public void handleEvent(Event event) {
      // retrieve the clicked column (see ColumnClick above)
      TableColumn tableColumn = (TableColumn) ((MenuItem)event.widget).getParent().getData();
      
      Fetcher fetcher = (Fetcher) tableColumn.getData();
      
      fetcherRegistry.openPreferencesEditor(fetcher);
      
      // update name if preferences changed
      tableColumn.setText(fetcher.getFullName());
    }

View Full Code Here

Examples of net.azib.ipscan.fetchers.Fetcher

    
    public void handleEvent(Event event) {
      // retrieve the clicked column (see ColumnClick above)
      TableColumn tableColumn = (TableColumn) ((MenuItem)event.widget).getParent().getData();
      
      Fetcher fetcher = (Fetcher) tableColumn.getData();


      MessageBox messageBox = new MessageBox(tableColumn.getParent().getShell(), SWT.ICON_INFORMATION | SWT.OK);
      messageBox.setText(Labels.getLabel("text.fetchers.info") + fetcher.getName());
      String info = fetcher.getInfo();
      if (info == null) {
        info = Labels.getLabel("text.fetchers.info.notAvailable");
      }
      messageBox.setMessage(info);
      messageBox.open();

View Full Code Here

Examples of net.azib.ipscan.fetchers.Fetcher

    Thread.sleep(10);
    assertEquals(scanTime2, scanInfo.getScanTime());
  }
  
  private Fetcher mockFetcher(String name) {
    Fetcher fetcher = mock(Fetcher.class);
    when(fetcher.getName()).thenReturn(name);
    return fetcher;
  }

View Full Code Here

Examples of net.azib.ipscan.fetchers.Fetcher

    assertEquals(33, preferences.getInt("windowWidth", 0));
  }


  @Test
  public void columnWidths() throws Exception {
    Fetcher fetcher = mock(Fetcher.class);
    when(fetcher.getId()).thenReturn("fetcher.abc");


    config.setColumnWidth(fetcher, 35);
    assertEquals(35, config.getColumnWidth(fetcher));
    assertEquals(35, preferences.getInt("columnWidth." + fetcher.getId(), 0));
  }

View Full Code Here

0 1 2 3 4

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.