Package net.vidageek.crawler

Examples of net.vidageek.crawler.Url

48
49
50
51
52
53
54
55
56
57
58
                visitor.visit(page);
            }

            for (String l : page.getLinks()) {
                String link = normalizer.normalize(l);
                final Url url = new Url(link, urlToCrawl.depth() + 1);
                if (visitor.followUrl(url)) {
                    executor.execute(new PageCrawlerExecutor(url, executor, counter, downloader, normalizer, visitor));
                }
            }
View Full Code Here
19
20
21
22
23
24
25
    // implementation
    private final Map<Url, String> visitedUrls = new ConcurrentHashMap<Url, String>();

    public DoesNotFollowVisitedUrlVisitor(final String beginUrl, final PageVisitor visitor) {
        this.visitor = visitor;
        visitedUrls.put(new Url(beginUrl, 0), "");
    }
View Full Code Here
TOP

Related Classes of net.vidageek.crawler.Url

  • net.vidageek.crawler.component.PageCrawlerExecutor
  • net.vidageek.crawler.visitor.DoesNotFollowVisitedUrlVisitor
    • Copyright © 2018 www.massapicom. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.