Package bixo.fetcher

Examples of bixo.fetcher.SimpleHttpFetcher

93
94
95
96
97
98
99
100
101
102
103
104
        FetcherPolicy policy = new FetcherPolicy();
        policy.setRedirectMode(RedirectMode.FOLLOW_NONE);
        policy.setMaxRedirects(MAX_REDIRECTS);
        policy.setMaxConnectionsPerHost(maxThreads);

        SimpleHttpFetcher result = new SimpleHttpFetcher(maxThreads, policy, userAgent);
        result.setDefaultMaxContentSize(MAX_CONTENT_SIZE);
       
        // We don't want any encoding (compression) of the data.
        result.setAcceptEncoding("");
        return result;
    }
View Full Code Here
65
66
67
68
69
70
71
72
73
74
75
76
        // TODO KKr - add static createRobotsFetcher method somewhere that
        // I can use here, and also in SimpleGroupingKeyGenerator
        FetcherPolicy policy = new FetcherPolicy();
        policy.setMaxContentSize(MAX_ROBOTS_SIZE);
        policy.setMaxConnectionsPerHost(MAX_CONNECTIONS_PER_HOST);
        SimpleHttpFetcher fetcher = new SimpleHttpFetcher(maxThreads, policy, userAgent);
        fetcher.setMaxRetryCount(ROBOTS_RETRY_COUNT);
        fetcher.setConnectionTimeout(ROBOTS_CONNECTION_TIMEOUT);
        fetcher.setSocketTimeout(ROBOTS_SOCKET_TIMEOUT);
       
        return fetcher;
    }
View Full Code Here
TOP

Related Classes of bixo.fetcher.SimpleHttpFetcher

  • bixo.config.FetcherPolicy
  • bixo.datum.ContentBytes
  • bixo.datum.FetchedDatum
  • bixo.datum.HttpHeaders
  • bixo.examples.crawl.DemoCrawlWorkflow
  • bixo.examples.webmining.DemoWebMiningWorkflow
  • bixo.exceptions.RedirectFetchException.RedirectExceptionReason
  • bixo.fetcher.http.SimpleHttpFetcherIntegrationTest
  • bixo.operations.FilterAndScoreByUrlAndRobotsTest
  • bixo.operations.UrlLengthener
    • Copyright © 2018 www.massapicom. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.