Examples of org.archive.crawler.Heritrix

org.archive.crawler.Heritrix

Main class for Heritrix crawler. Heritrix is usually launched by a shell script that backgrounds heritrix that redirects all stdout and stderr emitted by heritrix to a log file. So that startup messages emitted subsequent to the redirection of stdout and stderr show on the console, this class prints usage or startup output such as where the web UI can be found, etc., to a STARTLOG that the shell script is waiting on. As soon as the shell script sees output in this file, it prints its content and breaks out of its wait. See ${HERITRIX_HOME}/bin/heritrix.

Heritrix can also be embedded or launched by webapp initialization or by JMX bootstrapping. So far I count 4 methods of instantiation:

From this classes main -- the method usually used;
From the Heritrix UI (The local-instances.jsp) page;
A creation by a JMX agent at the behest of a remote JMX client; and
A container such as tomcat or jboss.

@author gojomo @author Kristinn Sigurdsson @author Stack

    protected void startHeritrix(String path) throws Exception {
        String authPassword = 
            (new BigInteger(SecureRandom.getSeed(16))).abs().toString(16);
        String[] args = { "-j", path + "/jobs", "-a", authPassword };
        // TODO: add auth password? 
        heritrix = new Heritrix();
        heritrix.instanceMain(args);
        
        configureHeritrix();


        heritrix.getEngine().requestLaunch("selftest-job");

Examples of org.archive.crawler.Heritrix

Related Classes of org.archive.crawler.Heritrix