Package us.codecraft.webmagic.pipeline

Examples of us.codecraft.webmagic.pipeline.PageModelPipeline

27
28
29
30
31
32
33
34
35
36
37
    public static void main(String[] args) {
        OOSpider.create(Site.me()
                .setUserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36").addStartUrl("http://my.oschina.net/flashsword/blog")
                .setSleepTime(0)
                .setRetryTimes(3)
                ,new PageModelPipeline() {
            @Override
            public void process(Object o, Task task) {

            }
        }, OschinaBlog.class).thread(10).run();
View Full Code Here
30
31
32
33
34
35
36
37
38
39
40
    @ExtractByUrl
    private String url;

    public static void main(String[] args) throws IOException, JMException {
        //Just for benchmark
        Spider thread = OOSpider.create(Site.me().addStartUrl("http://www.36kr.com/").setSleepTime(0), new PageModelPipeline() {
            @Override
            public void process(Object o, Task task) {

            }
        }, Kr36NewsModel.class).thread(20);
View Full Code Here
TOP

Related Classes of us.codecraft.webmagic.pipeline.PageModelPipeline

  • us.codecraft.webmagic.model.samples.Kr36NewsModel
  • us.codecraft.webmagic.model.samples.OschinaBlog
    • Copyright © 2018 www.massapicom. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.