Package cn.edu.hfut.dmic.webcollector.generator

Examples of cn.edu.hfut.dmic.webcollector.generator.Generator

89
90
91
92
93
94
95
96
97
98
99
        for (int i = 0; i < depth; i++) {
            if (status == STOPED) {
                break;
            }
            LogUtils.getLogger().info("starting depth " + (i + 1));
            Generator generator = createGenerator();
            fetcher = createFetcher();
            fetcher = updateFetcher(fetcher);
            if (fetcher == null) {
                return;
            }
View Full Code Here
65
66
67
68
69
70
71
72
73
   
    @Override
    public Generator createGenerator() {

        Generator generator = new FSGenerator(crawlPath);
        generator = new UniqueFilter(new IntervalFilter(new URLRegexFilter(generator, getRegexs())));
        return generator;
    }
View Full Code Here
51
52
53
54
55
56
57
58
    }
   
   
    @Override
    public Generator createGenerator() {
        Generator generator=new RedisGenerator(tableName, ip, port);
        return new URLRegexFilter(new IntervalFilter(generator),getRegexs());
    }
View Full Code Here
TOP

Related Classes of cn.edu.hfut.dmic.webcollector.generator.Generator

  • cn.edu.hfut.dmic.webcollector.crawler.BreadthCrawler
  • cn.edu.hfut.dmic.webcollector.crawler.Crawler
  • cn.edu.hfut.dmic.webcollector.plugin.redis.RedisCrawler
    • Copyright © 2018 www.massapicom. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.