Package de.anomic.crawler

Examples of de.anomic.crawler.RobotsTxt


        }.start();
        */

        // load the robots.txt db
        this.log.logConfig("Initializing robots.txt DB");
        this.robots = new RobotsTxt(this.tables.getHeap(WorkTables.TABLE_ROBOTS_NAME));
        this.log.logConfig("Loaded robots.txt DB: " this.robots.size() + " entries");

        // start a cache manager
        this.log.logConfig("Starting HT Cache Manager");

View Full Code Here


        }.start();
        */

        // load the robots.txt db
        this.log.logConfig("Initializing robots.txt DB");
        this.robots = new RobotsTxt(this.tables);
        this.log.logConfig("Loaded robots.txt DB: " this.robots.size() + " entries");

        // start a cache manager
        this.log.logConfig("Starting HT Cache Manager");

View Full Code Here

TOP

Related Classes of de.anomic.crawler.RobotsTxt

Copyright © 2018 www.massapicom. All rights reserved.
All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.