Examples of com.flaptor.hounder.classifier.util.WhoHasPersistence

Package com.flaptor.hounder.classifier.util

Examples of com.flaptor.hounder.classifier.util.WhoHasPersistence

com.flaptor.hounder.classifier.util.WhoHasPersistence
In previous versions, we added the data to a Map(token->TokenCounter) in memory. However such aproach consumes too much RAM and was discarded. In this version we add the data to a file that is lately transformed into a Map backed by a file. This class implements a file where partial data is saved, for late sort and add to the map @author rafa @see BayesCalculator#computeProbabilities() @see BayesCalculator#addData(java.util.Map,boolean,String)

    void addData(Map<String,int[]>documentTokenCount, 
            String url) throws IOException {
        Set<String> tokens = documentTokenCount.keySet();
        WhoHasPersistence.WHRecord whr;
        for (String token : tokens) {
            whr= new WhoHasPersistence().newRecord(token, url);
            whoHasFile.writeRecord(whr);
        }
    }

View Full Code Here

        logger.info("Closing WhoHasPersistence.... done");
        
        logger.info("Sorting WhoHasPersistence....");
        File beforeSort= new File(WHO_HAS_FILE_UNSORTED);
        File afterSort= new File(WHO_HAS_FILE_SORTED);
        WhoHasPersistence whp= new WhoHasPersistence();
        MergeSort.sort(beforeSort, afterSort, whp, null);
        logger.info("Sorting WhoHasPersistence... done");


        logger.info("Writing WhoHas Total to ... " + WHO_HAS_FILE_DONE);
        RecordReader whrr= whp.newRecordReader(afterSort); 
        WhoHasPersistence.WHRecord whr= (WhoHasPersistence.WHRecord) whrr.readRecord();        
        if (null==whr) return;
        
//        FileUtil.deleteFile(WHO_HAS_FILE_DONE);
        FileWriter whDone= new FileWriter(WHO_HAS_FILE_DONE);

View Full Code Here

    
    private void resetFiles(){
        FileUtil.deleteFile(WHO_HAS_FILE_SORTED);
        FileUtil.deleteFile(WHO_HAS_FILE_UNSORTED);
        try {
            whoHasFile= new WhoHasPersistence().newRecordWriter(new File (
                    WHO_HAS_FILE_UNSORTED), true);  // This appends to existing file
        } catch (IOException e) {
            e.printStackTrace();
        }                        
    }

View Full Code Here

TOP

Related Classes of com.flaptor.hounder.classifier.util.WhoHasPersistence

com.flaptor.hounder.classifier.WhoHasBean

java.io.InputStreamReader

java.io.FileInputStream

java.io.BufferedReader

java.io.OutputStreamWriter

java.io.FileOutputStream

java.io.BufferedWriter

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.