List of crawlWebPages() Examples

Examples of crawlWebPages()

com.crawl.control.Crawler.crawlWebPages()
The main extraction method, to call the Craigslist page and get all needed information from there. @param inCraigslistCategoryEnum Which category? @param inCraigslistAreasEnum Which area @param inSearchItem Which item do you search "iPhone", "Apple IIc", etc. @param inputIntOffers how many offers 100 = 1 Craigslist page @return

Examples of com.crawl.control.Crawler.crawlWebPages()

        String craigslistURL = aCrawl.createUrl(
                this.getCategoryCode(),
                this.getLocationURL(),
                this.getKeyword());
        // Get all items
        crawlResult = aCrawl.crawlWebPages(craigslistURL, 2500);
        logger.debug("aResultColl Size=" + crawlResult.size());
        // Create analyzer object
        analyzer = new AnaCrack();
        // Do the analyzing and return the offers
        analyzerOffers = analyzer.analyse(

View Full Code Here

Examples of com.crawl.control.Crawler.crawlWebPages()

                CraigslistCategoryEnum.FOR_SALE__COMPUTER,
                CraigslistAreasEnum.SAN_FRANCISCO, 
                "Apple");
        
        // 2. Step get all offers
        Collection<CrawlResultPackage> aResultColl =aCrawl.crawlWebPages(aUrl, 5000);


        logger.debug("aResultColl Size=" + aResultColl.size());


        // 3. Create analyzer object
        AnaCrack aAnaCrack = new AnaCrack();

View Full Code Here

Examples of com.crawl.control.Crawler.crawlWebPages()

        String aUrl=aCrawl.createUrl(
                CraigslistCategoryEnum.FOR_SALE__COMPUTER,
                CraigslistAreasEnum.PENINSULA, 
                "Apple");
               
        Collection<CrawlResultPackage> aCrawlResultColl=aCrawl.crawlWebPages(aUrl, 100);
        
        logger.debug("aCrawlResultColl.size()="+aCrawlResultColl.size());
        
        assertTrue(aCrawlResultColl.size()==100);
    }

View Full Code Here

Examples of com.crawl.control.Crawler.crawlWebPages()

        String aUrl=aCrawl.createUrl(
                CraigslistCategoryEnum.FOR_SALE__COMPUTER,
                CraigslistAreasEnum.PENINSULA, 
                "Apple");
               
        Collection<CrawlResultPackage> aCrawlResultColl=aCrawl.crawlWebPages(aUrl, 100);


        String aString=aCrawl.getMatchPattern();
        
        logger.debug("MatchPatter="+aString);

View Full Code Here

Examples of com.crawl.control.Crawler.crawlWebPages()

        String aUrl=aCrawl.createUrl(
                CraigslistCategoryEnum.FOR_SALE__COMPUTER,
                CraigslistAreasEnum.PENINSULA, 
                "Apple");
               
        Collection<CrawlResultPackage> aCrawlResultColl=aCrawl.crawlWebPages(aUrl, 100);


        String aString=aCrawl.getMatchPattern();
        
        logger.debug("MatchPatter="+aString);

View Full Code Here

Examples of com.crawl.control.Crawler.crawlWebPages()

        String aUrl=aCrawl.createUrl(
                CraigslistCategoryEnum.FOR_SALE__COMPUTER,
                CraigslistAreasEnum.PENINSULA, 
                "Apple");
               
        Collection<CrawlResultPackage> aCrawlResultColl=aCrawl.crawlWebPages(aUrl, 100);


        String aString=aCrawl.getMatchPattern();
        
        logger.info("MatchPatter="+aString+" length="+aString.length());

View Full Code Here

Examples of com.crawl.control.Crawler.crawlWebPages()

        String aUrl=aCrawl.createUrl(
                CraigslistCategoryEnum.FOR_SALE__COMPUTER,
                CraigslistAreasEnum.PENINSULA, 
                "Apple");
               
        Collection<CrawlResultPackage> aCrawlResultColl=aCrawl.crawlWebPages(aUrl, 100);
        
        logger.info("URL="+aUrl+" length="+aUrl.length());
        
        assertTrue(aUrl != null);
    }

View Full Code Here

Examples of com.crawl.control.Crawler.crawlWebPages()

        String aUrl=aCrawl.createUrl(
                CraigslistCategoryEnum.FOR_SALE__COMPUTER,
                CraigslistAreasEnum.PENINSULA, 
                "Apple");
               
        Collection<CrawlResultPackage> aCrawlResultColl=aCrawl.crawlWebPages(aUrl, 100);
        
        logger.info("URL="+aUrl+" length="+aUrl.length());
        
        assertTrue(aUrl.compareTo("http://sfbay.craigslist.org/search/sya/pen?query=Apple&maxAsk=100000&sort=pricedsc&srchType=A&s=") ==0);    
    }

View Full Code Here

Examples of com.crawl.control.Crawler.crawlWebPages()

        String aUrl=aCrawl.createUrl(
                CraigslistCategoryEnum.FOR_SALE__COMPUTER,
                CraigslistAreasEnum.PENINSULA, 
                "Apple");
               
        Collection<CrawlResultPackage> aCrawlResultColl=aCrawl.crawlWebPages(aUrl, 100);
                
        logger.debug("URL="+aUrl+" length="+aUrl.length());
        
        assertTrue(aUrl.length()==96);    
    }

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.