Examples of bixo.urls.BaseUrlNormalizer.normalize()

bixo.urls.BaseUrlNormalizer.normalize()
Convert into a normalized format, where unimportant differences between two URLs have been removed. @param url - URL to normalize. Might not be valid, e.g. missing a protocol @return - normalized URL. Still might not be valid, if input URL (for example)uses an unknown protocol and thus no checks can be done.

            List<String> lines = FileUtils.readLines(new File(args[0]));


            BaseUrlNormalizer urlNormalizer = new SimpleUrlNormalizer();
            for (String url : lines) {
                curUrl = url;
                String normalized = urlNormalizer.normalize(curUrl);
                if (!normalized.equalsIgnoreCase(curUrl)) {
                    System.out.println(curUrl + " ==> " + normalized);
                }
            }
        } catch (Throwable t) {

View Full Code Here

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.