Examples of PageProcessor

com.facebook.presto.operator.PageProcessor
us.codecraft.webmagic.processor.PageProcessor
Interface to be implemented to customize a crawler.

In PageProcessor, you can customize:
start urls and other settings in {@link Site}
how the urls to fetch are detected
how the data are extracted and stored
@author code4crafter@gmail.com
@see Site @see Page @since 0.1.0

Examples of com.facebook.presto.operator.PageProcessor

        filter = ExpressionTreeRewriter.rewriteWith(new SymbolToInputRewriter(ImmutableMap.<Symbol, Integer>of()), filter);


        IdentityHashMap<Expression, Type> expressionTypes = getExpressionTypesFromInput(TEST_SESSION, metadata, SQL_PARSER, INPUT_TYPES, ImmutableList.of(filter));


        try {
            PageProcessor processor = compiler.compilePageProcessor(
                    SqlToRowExpressionTranslator.translate(filter, expressionTypes, metadata, session, false),
                    ImmutableList.<RowExpression>of());


            return new FilterAndProjectOperator.FilterAndProjectOperatorFactory(0, processor, ImmutableList.<Type>of());
        }

View Full Code Here

Examples of com.facebook.presto.operator.PageProcessor


        IdentityHashMap<Expression, Type> expressionTypes = getExpressionTypesFromInput(TEST_SESSION, metadata, SQL_PARSER, INPUT_TYPES, ImmutableList.of(filter, projection));


        try {
            List<RowExpression> projections = ImmutableList.of(SqlToRowExpressionTranslator.translate(projection, expressionTypes, metadata, session, false));
            PageProcessor processor = compiler.compilePageProcessor(
                    SqlToRowExpressionTranslator.translate(filter, expressionTypes, metadata, session, false),
                    projections);


            return new FilterAndProjectOperator.FilterAndProjectOperatorFactory(0, processor, ImmutableList.of(expressionTypes.get(projection)));
        }

View Full Code Here

Examples of com.facebook.presto.operator.PageProcessor

            CursorProcessor cursorProcessor = compiler.compileCursorProcessor(
                    SqlToRowExpressionTranslator.translate(filter, expressionTypes, metadata, session, false),
                    ImmutableList.of(SqlToRowExpressionTranslator.translate(projection, expressionTypes, metadata, session, false)),
                    SOURCE_ID);


            PageProcessor pageProcessor = compiler.compilePageProcessor(
                    SqlToRowExpressionTranslator.translate(filter, expressionTypes, metadata, session, false),
                    ImmutableList.of(SqlToRowExpressionTranslator.translate(projection, expressionTypes, metadata, session, false)));


            return new ScanFilterAndProjectOperator.ScanFilterAndProjectOperatorFactory(
                    0,

View Full Code Here

Examples of com.facebook.presto.operator.PageProcessor

        filter = ExpressionTreeRewriter.rewriteWith(new SymbolToInputRewriter(ImmutableMap.<Symbol, Integer>of()), filter);


        IdentityHashMap<Expression, Type> expressionTypes = getExpressionTypesFromInput(SESSION, metadata, SQL_PARSER, INPUT_TYPES, ImmutableList.of(filter));


        try {
            PageProcessor processor = compiler.compilePageProcessor(
                    SqlToRowExpressionTranslator.translate(filter, expressionTypes, metadata, session, false),
                    ImmutableList.<RowExpression>of());


            return new FilterAndProjectOperator.FilterAndProjectOperatorFactory(0, processor, ImmutableList.<Type>of());
        }

View Full Code Here

Examples of com.facebook.presto.operator.PageProcessor


        IdentityHashMap<Expression, Type> expressionTypes = getExpressionTypesFromInput(SESSION, metadata, SQL_PARSER, INPUT_TYPES, ImmutableList.of(filter, projection));


        try {
            List<RowExpression> projections = ImmutableList.of(SqlToRowExpressionTranslator.translate(projection, expressionTypes, metadata, session, false));
            PageProcessor processor = compiler.compilePageProcessor(
                    SqlToRowExpressionTranslator.translate(filter, expressionTypes, metadata, session, false),
                    projections);


            return new FilterAndProjectOperator.FilterAndProjectOperatorFactory(0, processor, ImmutableList.of(expressionTypes.get(projection)));
        }

View Full Code Here

Examples of com.facebook.presto.operator.PageProcessor

            CursorProcessor cursorProcessor = compiler.compileCursorProcessor(
                    SqlToRowExpressionTranslator.translate(filter, expressionTypes, metadata, session, false),
                    ImmutableList.of(SqlToRowExpressionTranslator.translate(projection, expressionTypes, metadata, session, false)),
                    SOURCE_ID);


            PageProcessor pageProcessor = compiler.compilePageProcessor(
                    SqlToRowExpressionTranslator.translate(filter, expressionTypes, metadata, session, false),
                    ImmutableList.of(SqlToRowExpressionTranslator.translate(projection, expressionTypes, metadata, session, false)));


            return new ScanFilterAndProjectOperator.ScanFilterAndProjectOperatorFactory(
                    0,

View Full Code Here

Examples of com.facebook.presto.operator.PageProcessor

                RowExpression traslatedFilter = SqlToRowExpressionTranslator.translate(rewrittenFilter, expressionTypes, metadata, session, true);
                List<RowExpression> translatedProjections = SqlToRowExpressionTranslator.translate(rewrittenProjections, expressionTypes, metadata, session, true);


                if (columns != null) {
                    CursorProcessor cursorProcessor = compiler.compileCursorProcessor(traslatedFilter, translatedProjections, sourceNode.getId());
                    PageProcessor pageProcessor = compiler.compilePageProcessor(traslatedFilter, translatedProjections);


                    SourceOperatorFactory operatorFactory = new ScanFilterAndProjectOperator.ScanFilterAndProjectOperatorFactory(
                            context.getNextOperatorId(),
                            sourceNode.getId(),
                            dataStreamProvider,
                            cursorProcessor,
                            pageProcessor,
                            columns,
                            Lists.transform(rewrittenProjections, forMap(expressionTypes)));


                    return new PhysicalOperation(operatorFactory, outputMappings);
                }
                else {
                    PageProcessor processor = compiler.compilePageProcessor(traslatedFilter, translatedProjections);


                    OperatorFactory operatorFactory = new FilterAndProjectOperator.FilterAndProjectOperatorFactory(
                            context.getNextOperatorId(),
                            processor,
                            Lists.transform(rewrittenProjections, forMap(expressionTypes)));

View Full Code Here

Examples of com.facebook.presto.operator.PageProcessor

                RowExpression traslatedFilter = SqlToRowExpressionTranslator.translate(rewrittenFilter, expressionTypes, metadata, session, true);
                List<RowExpression> translatedProjections = SqlToRowExpressionTranslator.translate(rewrittenProjections, expressionTypes, metadata, session, true);


                if (columns != null) {
                    CursorProcessor cursorProcessor = compiler.compileCursorProcessor(traslatedFilter, translatedProjections, sourceNode.getId());
                    PageProcessor pageProcessor = compiler.compilePageProcessor(traslatedFilter, translatedProjections);


                    SourceOperatorFactory operatorFactory = new ScanFilterAndProjectOperator.ScanFilterAndProjectOperatorFactory(
                            context.getNextOperatorId(),
                            sourceNode.getId(),
                            pageSourceProvider,
                            cursorProcessor,
                            pageProcessor,
                            columns,
                            Lists.transform(rewrittenProjections, forMap(expressionTypes)));


                    return new PhysicalOperation(operatorFactory, outputMappings);
                }
                else {
                    PageProcessor processor = compiler.compilePageProcessor(traslatedFilter, translatedProjections);


                    OperatorFactory operatorFactory = new FilterAndProjectOperator.FilterAndProjectOperatorFactory(
                            context.getNextOperatorId(),
                            processor,
                            Lists.transform(rewrittenProjections, forMap(expressionTypes)));

View Full Code Here

Examples of us.codecraft.webmagic.processor.PageProcessor

 */
public class WorkerTest {


    @Test
    public void testWorkerAsSpiderContains() throws Exception {
        PageProcessor pageProcessor = mock(PageProcessor.class);
        Site site = mock(Site.class);
        when(pageProcessor.getSite()).thenReturn(site);
        when(site.getDomain()).thenReturn("codecraft.us");
        Worker worker = new Worker();
        Spider spider = Spider.create(pageProcessor);
        worker.addSpider(spider);
        assertThat(worker.getSpider("codecraft.us")).isEqualTo(spider);

View Full Code Here

Examples of us.codecraft.webmagic.processor.PageProcessor

            testRound();
        }
    }


    private void testRound() {
        Spider spider = Spider.create(new PageProcessor() {


            private AtomicInteger count = new AtomicInteger();


            @Override
            public void process(Page page) {

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.