Examples of edu.ucla.sspace.util.WorkQueue

edu.ucla.sspace.util.WorkQueue
to do goes here q.run(tasks);
Alternately, a use may register a task group identifier and then iteratively add new tasks associated with that identifier. At some point in the future, the user can then wait for all the tasks associated with that identifier to finish. This second method allows for the iterative construction of tasks, or for cases where not all of the data for the tasks is availabe at once (although the number of tasks is known).
```
 WorkQueue q = WorkQueue.getWorkQueue(); Object taskGroupId = Thread.currentThread(); // a unique id q.registerTaskGroup(taskGroupId, numTasks); for (int i = 0; i < numTasks; ++i) q.add(taskGroupId, new Runnable() { }); // job to do goes here q.await(taskGroupId); 
```
In the above example, the current thread is used as the group identifier, which ensures that any other thread executing the same code won't use the same identifier, which could result in either thread returning prematurely before its tasks have finished. However, a shared group identifier can allow multiple threads to add tasks for a common goal, with each being able await until all the tasks are finished. @author David Jurgens

     * {@inheritDoc}
     */
    public void processSpace(Properties props) {
        final double mergeThreshold = .15;


        WorkQueue workQueue = WorkQueue.getWorkQueue();
        Object key = workQueue.registerTaskGroup(clusterMap.size());


        // Iterate through all of the clusters and perform an agglomerative
        // cluster over the learned word senses.  If there is a reporter, the
        // cluster assignments are reported.
        for (Map.Entry<String, OnlineClustering<SparseDoubleVector>> entry :
                 clusterMap.entrySet()) {
            final String primaryKey = entry.getKey();
            final OnlineClustering<SparseDoubleVector> contexts =
                entry.getValue();
            workQueue.add(key, new Runnable() {
                public void run() {
                    clusterAndAssignSenses(contexts,primaryKey,mergeThreshold);
                }
            });
        }
        workQueue.await(key);


        // Null out the cluster map so that the garbage collector can reclaim it
        // and any data associated with the Clusters.
        clusterMap = null;

View Full Code Here


        Collection<Thread> threads = new LinkedList<Thread>();


        final AtomicInteger count = new AtomicInteger(0);
        
        WorkQueue queue = WorkQueue.getWorkQueue(numThreads);
        Object key = queue.registerTaskGroup(numThreads);


        long processStart = System.currentTimeMillis();
        verbose("Beginning processing using %d threads", numThreads);
        for (int i = 0; i < numThreads; ++i) {
            queue.add(key, new Runnable() {
                public void run() {
                    // repeatedly try to process documents while some still
                    // remain
                    while (docIter.hasNext()) {
                        long startTime = System.currentTimeMillis();
                        Document doc = docIter.next();
                        int docNumber = count.incrementAndGet();
                        int terms = 0;
                        try {
                            sspace.processDocument(doc.reader());
                        } catch (Throwable t) {
                            t.printStackTrace();
                        }
                        long endTime = System.currentTimeMillis();
                        verbose("parsed document #%d in %.3f seconds",
                                docNumber, ((endTime - startTime) / 1000d));
                    }
                }
            });;
        }


        queue.await(key);
        
        verbose("Processed all %d documents in %.3f total seconds",
                count.get(),
                ((System.currentTimeMillis() - processStart) / 1000d));            
    }

View Full Code Here


    /**
     * {@inheritDoc}
     */
    public void processSpace(final Properties props) {
        WorkQueue workQueue = WorkQueue.getWorkQueue();


        Object key = workQueue.registerTaskGroup(dataVectors.size());
        // Process each word's context set in a worker thread.
        for (Map.Entry<String, List<SparseDoubleVector>> entry :
                dataVectors.entrySet()) {
            // Get the root word being discriminated and list of observed
            // contexts.
            final String senseName = entry.getKey();


            List<SparseDoubleVector> contextsWithNoLength = entry.getValue();
            final List<SparseDoubleVector> contextSet = 
                new ArrayList<SparseDoubleVector>(contextsWithNoLength.size());
            for (SparseDoubleVector v : contextsWithNoLength)
                contextSet.add(Vectors.subview(v, 0, getVectorLength()));
            
            workQueue.add(key, new Runnable() {
                public void run() {
                    clusterTerm(senseName, contextSet, props);
                }
            });
        }
        workQueue.await(key);
        LOG.info("Finished processing all terms");
    }

View Full Code Here

TOP

Related Classes of edu.ucla.sspace.util.WorkQueue

edu.ucla.sspace.mains.GenericMain

edu.ucla.sspace.wordsi.StreamingWordsi

edu.ucla.sspace.wordsi.WaitingWordsi

java.util.concurrent.CountDownLatch

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.