Package com.googlecode.flaxcrawler

Examples of com.googlecode.flaxcrawler.DefaultCrawler

15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
*/
public class TestDefaultCrawler {

    @Test
    public void testDefaultCrawler() throws Exception {
        DefaultCrawler crawler = new DefaultCrawler();
        crawler.setDownloaderController(new DefaultDownloaderController());
        crawler.setParserController(new DefaultParserController());

        CrawlerTask crawlerTask = new CrawlerTask("http://www.wikipedia.org/", 0);
        Page page = crawler.crawl(crawlerTask);

        assertNotNull(page);
        assertTrue(page.getLinks().size() > 0);
    }
View Full Code Here
34
35
36
37
38
39
40
41
42
43
44
45
46
        CrawlerConfiguration configuration = new CrawlerConfiguration();

        // Creating five crawlers (to work with 5 threads)
        for (int i = 0; i < 5; i++) {
            // Creating crawler and setting downloader and parser controllers
            DefaultCrawler crawler = new ExampleCrawler();
            crawler.setDownloaderController(downloaderController);
            crawler.setParserController(parserController);
            // Adding crawler to the configuration object
            configuration.addCrawler(crawler);
        }

        // Setting maximum parallel requests to a single site limit
View Full Code Here
TOP

Related Classes of com.googlecode.flaxcrawler.DefaultCrawler

  • com.googlecode.flaxcrawler.download.Downloader
  • com.googlecode.flaxcrawler.model.Page
  • com.googlecode.flaxcrawler.parse.Parser
  • com.googlecode.flaxcrawler.samples.TestSimpleSample
  • com.googlecode.flaxcrawler.TestDefaultCrawler
  • java.util.Date
  • java.net.URL
    • Copyright © 2018 www.massapicom. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.