Examples of us.codecraft.webmagic.scheduler.Scheduler

us.codecraft.webmagic.scheduler.Scheduler
Scheduler is the part of url management.
You can implement interface Scheduler to do: manage urls to fetch remove duplicate urls @author code4crafter@gmail.com
@since 0.1.0


            @Override
            public void setThread(int threadNum) {


            }
        }).setScheduler(new Scheduler() {


            private AtomicInteger count = new AtomicInteger();


            private Random random = new Random();

View Full Code Here

     * @see Scheduler
     * @since 0.2.1
     */
    public Spider setScheduler(Scheduler scheduler) {
        checkIfRunning();
        Scheduler oldScheduler = this.scheduler;
        this.scheduler = scheduler;
        if (oldScheduler != null) {
            Request request;
            while ((request = oldScheduler.poll(this)) != null) {
                this.scheduler.push(request, this);
            }
        }
        return this;
    }

View Full Code Here

TOP

Related Classes of us.codecraft.webmagic.scheduler.Scheduler

us.codecraft.webmagic.Spider

us.codecraft.webmagic.SpiderTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.