Examples of edu.ucla.sspace.hadoop.WordCooccurrenceCountingJob.execute()

edu.ucla.sspace.hadoop.WordCooccurrenceCountingJob.execute()

Exceutes the word co-occurrence counting job on the corpus files in the input directory using the current Hadoop instance, returning an iterator over all the occurrences frequences found in the corpus. @param inputDirs a list of directories on the Hadoop distributed filesystem containing all the corpus files that will be processed @return an iterator over the unique {@link WordCooccurrence} counts foundin the corpus. Note that if two words co-occur the same distance apart multiple times, only one {@code WordCooccurrence} isreturned, where the number of co-occurrences is reflected by the the {@link WordCooccurrence#getCount() getCount()} method. @throws Exception if Hadoop throws an {@code Exception} during itsexecution or if the resulting output files cannot be read.

        LOGGER.info("Beginning Hadoop corpus processing");
        // Construct the counting job that will use Hadoop to count the
        // co-occurrences
        WordCooccurrenceCountingJob job = 
            new WordCooccurrenceCountingJob(props);
        Iterator<WordCooccurrence> occurrences = job.execute(inputDirs);
        LOGGER.info("Finished Hadoop corpus processing; calculating sspace");
        int wordCount =  0;
        // Local state variables for updating the current word's vector.
        String curWord = null;
        IntegerVector semantics = null;

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.