Examples of Crawler

ca.forklabs.javaxpcom.Crawler
Class {@code Crawler} provides a skeleton to build web crawlers usingMozilla and SWT.
This crawler gives access to a {@link Browser} object inside a simple{@link Shell} that can be interacted with to navigate the web and access theunderlying {@link nsIDOMDocument}. @author Daniel Léonard @version $Revision$
com.crawl.control.Crawler
SINGLETON: The main crawler method to extract the oinformation from the Craiglists web page. @author Team Kappa
com.crawljax.core.Crawler
crawler.Crawler
net.sf.regain.crawler.Crawler
Durchsucht alle konfigurierten Startseiten nach URLs. Die gefundenen Seiten werden je nach Einstellung nur geladen, in den Suchindex aufgenommen oder wiederum nach URLs durchsucht.
für jede URL wird Anhand der Schwarzen und der Weißen Liste entschieden, ob sie ignoriert oder bearbeitet wird. Wenn loadUnparsedUrls auf false gesetzt wurde, dann werden auch URLs ignoriert, die weder durchsucht noch indiziert werden. @author Til Schneider, www.murfman.de
org.apache.cocoon.bean.helpers.Crawler
A simple Cocoon crawler @author Upayavira @version CVS $Id: Crawler.java,v 1.4 2004/03/05 13:02:45 bdelacretaz Exp $
placeholder.Crawler
Represents the crawler. @author Heiko Bornholdt
se.fidde.detective.crawlers.Crawler
websphinx.Crawler
Web crawler.
To write a crawler, extend this class and override shouldVisit () and visit() to create your own crawler.
To use a crawler:
1. Initialize the crawler by calling setRoot() (or one of its variants) and setting other crawl parameters.
2. Register any classifiers you need with addClassifier().
3. Connect event listeners to monitor the crawler, such as websphinx.EventLog, websphinx.workbench.WebGraph, or websphinx.workbench.Statistics.
4. Call run() to start the crawler.
A running crawler consists of a priority queue of Links waiting to be visited and a set of threads retrieving pages in parallel. When a page is downloaded, it is processed as follows:
1. classify(): The page is passed to the classify() method of every registered classifier, in increasing order of their priority values. Classifiers typically attach informative labels to the page and its links, such as "homepage" or "root page".
2. visit(): The page is passed to the crawler's visit() method for user-defined processing.
3. expand(): The page is passed to the crawler's expand() method to be expanded. The default implementation tests every unvisited hyperlink on the page with shouldVisit(), and puts each link approved by shouldVisit() into the crawling queue.
By default, when expanding the links of a page, the crawler only considers hyperlinks (not applets or inline images, for instance) that point to Web pages (not mailto: links, for instance). If you want shouldVisit() to test every link on the page, use setLinkType(Crawler.ALL_LINKS).

Examples of placeholder.Crawler

      }          
    }, 0, 100);
    
    // start crawler
    PrintStream out = new PrintStream(new TextAreaOutputStream(this.textArea));
    new Crawler(out, queue, resultQueue).start();
  }

View Full Code Here

Examples of se.fidde.detective.crawlers.Crawler


  private static void invokeTasks(){
    ForkJoinPool pool = new ForkJoinPool();
    
    for(File child : foldersToScan){
      pool.invoke(new Crawler(child));
        
    }
  }

View Full Code Here

Examples of websphinx.Crawler

            
        if ((param = getParameter ("log")) != null)
            workbench.setLoggerVisible (isTrue (param));
        */


        Crawler crawler = workbench.getCrawler();


        String action = getParameter ("action");
        if (action != null) {
            String filename = getParameter ("filename");
            String pattern = getParameter ("pattern");
            
            if (action.equalsIgnoreCase ("concatenate"))
                crawler.setAction (new ConcatAction (filename, true));
            else if (action.equalsIgnoreCase ("save"))
                crawler.setAction (new MirrorAction (filename, true));
            else if (action.equalsIgnoreCase ("visualize")) {
                crawler.setAction (null);
                //workbench.setGraphVisible (true);
            }
            else if (action.equalsIgnoreCase ("extract"))
                crawler.setAction (new ExtractAction (new Tagexp (pattern), 
                                                  true, filename, false));
            else if (action.equalsIgnoreCase ("none"))
                crawler.setAction (null);
            else
                throw new RuntimeException ("unknown action: " +action);
        }
        
        String urls = getParameter ("urls");
        if (urls != null)
            try {
                crawler.setRootHrefs (urls);
            } catch (java.net.MalformedURLException e) {
                throw new RuntimeException (e.toString());
            }
        
        String domain = getParameter ("domain");
        if (domain != null) {
            if (domain.equalsIgnoreCase ("server"))
                crawler.setDomain (Crawler.SERVER);
            else if (domain.equalsIgnoreCase ("subtree"))
                crawler.setDomain (Crawler.SUBTREE);
            else
                crawler.setDomain (Crawler.WEB);
        }
        
        String type = getParameter ("type");
        if (type != null) {
            if (type.equalsIgnoreCase ("images+hyperlinks"))
                crawler.setLinkType (Crawler.HYPERLINKS_AND_IMAGES);
            else if (type.equalsIgnoreCase ("all"))
                crawler.setLinkType (Crawler.ALL_LINKS);
            else
                crawler.setLinkType (Crawler.WEB);
        }


        String depth = getParameter ("depth");
        if (depth != null)
            crawler.setMaxDepth (Integer.parseInt (depth));


        String dfs = getParameter ("depthfirst");
        if (dfs != null)
            crawler.setDepthFirst (isTrue (dfs));


        workbench.setCrawler (crawler);


        setLayout (new BorderLayout ());
        add ("Center", workbench);

View Full Code Here

0 1 2 3

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.