Package com.googlecode.flaxcrawler

Examples of com.googlecode.flaxcrawler.CrawlerController$CrawlerWorker


        crawlerConfiguration.addCrawler(new TestCrawler());
        crawlerConfiguration.addCrawler(new TestCrawler());
        crawlerConfiguration.addCrawler(new TestCrawler());
        crawlerConfiguration.setPolitenessPeriod(1000);
        crawlerConfiguration.setMaxParallelRequests(1);
        CrawlerController crawlerController = new CrawlerController(crawlerConfiguration);
        crawlerController.addSeed(new URL("http://lenta.ru/"));
        crawlerController.start();
        crawlerController.join();
    }
View Full Code Here


        configuration.setMaxHttpErrors(HttpURLConnection.HTTP_BAD_GATEWAY, 10);
        // Setting period between two requests to a single site (in milliseconds)
        configuration.setPolitenessPeriod(500);

        // Initializing crawler controller
        CrawlerController crawlerController = new CrawlerController(configuration);
        // Adding crawler seed
        crawlerController.addSeed(new URL("http://en.wikipedia.org/"));
        // Starting and joining our crawler
        crawlerController.start();
        System.out.println(new Date() + "Waiting for 60 seconds");
        crawlerController.join(60000);
        System.out.println(new Date() + "Stopping crawler");
        crawlerController.stop();
    }
View Full Code Here

TOP

Related Classes of com.googlecode.flaxcrawler.CrawlerController$CrawlerWorker

Copyright © 2018 www.massapicom. All rights reserved.
All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.