Examples of org.archive.crawler.prefetch.PreconditionEnforcer.credentialPrecondition()

org.archive.crawler.prefetch.PreconditionEnforcer.credentialPrecondition()

Consider credential preconditions. Looks to see if any credential preconditions (e.g. html form login credentials) for this CrawlServer. If there are, have they been run already? If not, make the running of these logins a precondition of accessing any other url on this CrawlServer.

One day, do optimization and avoid running the bulk of the code below. Argument for running the code everytime is that overrides and refinements may change what comes back from credential store. @param curi CrawlURI we're checking for any required preconditions. @return True, if this curi has a precondition that needs tobe met before we can proceed. False if we can precede to process this url.

        assertTrue(curi.getHttpResponseHeader("Location").startsWith("http://localhost:7779/login.html"));


        PreconditionEnforcer preconditionEnforcer = new PreconditionEnforcer();
        preconditionEnforcer.setServerCache(getFetcher().getServerCache());
        preconditionEnforcer.setCredentialStore(getFetcher().getCredentialStore());
        boolean result = preconditionEnforcer.credentialPrecondition(curi);
        assertTrue(result);


        CrawlURI loginUri = curi.getPrerequisiteUri();
        assertEquals("http://localhost:7779/j_security_check", loginUri.toString());


        CrawlURI loginUri = curi.getPrerequisiteUri();
        assertEquals("http://localhost:7779/j_security_check", loginUri.toString());


        // there's some special logic with side effects in here for the login uri itself
        result = preconditionEnforcer.credentialPrecondition(loginUri);
        assertFalse(result);


        loginUri.setRecorder(getRecorder());
        getFetcher().process(loginUri);
        logger.info('\n' + httpRequestString(loginUri) + "\n\n" + rawResponseString(loginUri));

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.