Examples of java.util.concurrent.ThreadPoolExecutor

java.util.concurrent.ThreadPoolExecutor
An {@link ExecutorService} that executes each submitted task usingone of possibly several pooled threads, normally configured using {@link Executors} factory methods.
Thread pools address two different problems: they usually provide improved performance when executing large numbers of asynchronous tasks, due to reduced per-task invocation overhead, and they provide a means of bounding and managing the resources, including threads, consumed when executing a collection of tasks. Each ThreadPoolExecutor also maintains some basic statistics, such as the number of completed tasks.
To be useful across a wide range of contexts, this class provides many adjustable parameters and extensibility hooks. However, programmers are urged to use the more convenient {@link Executors} factory methods {@link Executors#newCachedThreadPool} (unbounded thread pool, withautomatic thread reclamation), {@link Executors#newFixedThreadPool}(fixed size thread pool) and {@link Executors#newSingleThreadExecutor} (single background thread), thatpreconfigure settings for the most common usage scenarios. Otherwise, use the following guide when manually configuring and tuning this class:
Core and maximum pool sizes

A ThreadPoolExecutor will automatically adjust the pool size (see {@link ThreadPoolExecutor#getPoolSize}) according to the bounds set by corePoolSize (see {@link ThreadPoolExecutor#getCorePoolSize}) and maximumPoolSize (see {@link ThreadPoolExecutor#getMaximumPoolSize}). When a new task is submitted in method {@link ThreadPoolExecutor#execute}, and fewer than corePoolSize threads are running, a new thread is created to handle the request, even if other worker threads are idle. If there are more than corePoolSize but less than maximumPoolSize threads running, a new thread will be created only if the queue is full. By setting corePoolSize and maximumPoolSize the same, you create a fixed-size thread pool. By setting maximumPoolSize to an essentially unbounded value such as Integer.MAX_VALUE, you allow the pool to accommodate an arbitrary number of concurrent tasks. Most typically, core and maximum pool sizes are set only upon construction, but they may also be changed dynamically using {@link ThreadPoolExecutor#setCorePoolSize} and {@link ThreadPoolExecutor#setMaximumPoolSize}.

On-demand construction

By default, even core threads are initially created and started only when new tasks arrive, but this can be overridden dynamically using method {@link ThreadPoolExecutor#prestartCoreThread} or{@link ThreadPoolExecutor#prestartAllCoreThreads}. You probably want to prestart threads if you construct the pool with a non-empty queue.

Creating new threads

New threads are created using a {@link java.util.concurrent.ThreadFactory}. If not otherwise specified, a {@link Executors#defaultThreadFactory} is used, that creates threads to allbe in the same {@link ThreadGroup} and with the sameNORM_PRIORITY priority and non-daemon status. By supplying a different ThreadFactory, you can alter the thread's name, thread group, priority, daemon status, etc. If a ThreadFactory fails to create a thread when asked by returning null from newThread, the executor will continue, but might not be able to execute any tasks.

Keep-alive times

If the pool currently has more than corePoolSize threads, excess threads will be terminated if they have been idle for more than the keepAliveTime (see {@link ThreadPoolExecutor#getKeepAliveTime}). This provides a means of reducing resource consumption when the pool is not being actively used. If the pool becomes more active later, new threads will be constructed. This parameter can also be changed dynamically using method {@link ThreadPoolExecutor#setKeepAliveTime}. Using a value of Long.MAX_VALUE {@link TimeUnit#NANOSECONDS} effectivelydisables idle threads from ever terminating prior to shut down. By default, the keep-alive policy applies only when there are more than corePoolSizeThreads. But method {@link ThreadPoolExecutor#allowCoreThreadTimeOut(boolean)} can be used to applythis time-out policy to core threads as well, so long as the keepAliveTime value is non-zero.

Queuing
Any {@link BlockingQueue} may be used to transfer and holdsubmitted tasks. The use of this queue interacts with pool sizing:
- If fewer than corePoolSize threads are running, the Executor always prefers adding a new thread rather than queuing.
- If corePoolSize or more threads are running, the Executor always prefers queuing a request rather than adding a new thread.
- If a request cannot be queued, a new thread is created unless this would exceed maximumPoolSize, in which case, the task will be rejected.
There are three general strategies for queuing:
1. Direct handoffs. A good default choice for a work queue is a {@link SynchronousQueue} that hands off tasks to threadswithout otherwise holding them. Here, an attempt to queue a task will fail if no threads are immediately available to run it, so a new thread will be constructed. This policy avoids lockups when handling sets of requests that might have internal dependencies. Direct handoffs generally require unbounded maximumPoolSizes to avoid rejection of new submitted tasks. This in turn admits the possibility of unbounded thread growth when commands continue to arrive on average faster than they can be processed.
2. Unbounded queues. Using an unbounded queue (for example a {@link LinkedBlockingQueue} without a predefinedcapacity) will cause new tasks to wait in the queue when all corePoolSize threads are busy. Thus, no more than corePoolSize threads will ever be created. (And the value of the maximumPoolSize therefore doesn't have any effect.) This may be appropriate when each task is completely independent of others, so tasks cannot affect each others execution; for example, in a web page server. While this style of queuing can be useful in smoothing out transient bursts of requests, it admits the possibility of unbounded work queue growth when commands continue to arrive on average faster than they can be processed.
3. Bounded queues. A bounded queue (for example, an {@link ArrayBlockingQueue}) helps prevent resource exhaustion when used with finite maximumPoolSizes, but can be more difficult to tune and control. Queue sizes and maximum pool sizes may be traded off for each other: Using large queues and small pools minimizes CPU usage, OS resources, and context-switching overhead, but can lead to artificially low throughput. If tasks frequently block (for example if they are I/O bound), a system may be able to schedule time for more threads than you otherwise allow. Use of small queues generally requires larger pool sizes, which keeps CPUs busier but may encounter unacceptable scheduling overhead, which also decreases throughput.
Rejected tasks
New tasks submitted in method {@link ThreadPoolExecutor#execute} will be rejected when theExecutor has been shut down, and also when the Executor uses finite bounds for both maximum threads and work queue capacity, and is saturated. In either case, the execute method invokes the {@link RejectedExecutionHandler#rejectedExecution} method of its{@link RejectedExecutionHandler}. Four predefined handler policies are provided:
1. In the default {@link ThreadPoolExecutor.AbortPolicy}, the handler throws a runtime {@link RejectedExecutionException} upon rejection.
2. In {@link ThreadPoolExecutor.CallerRunsPolicy}, the thread that invokes execute itself runs the task. This provides a simple feedback control mechanism that will slow down the rate that new tasks are submitted.
3. In {@link ThreadPoolExecutor.DiscardPolicy}, a task that cannot be executed is simply dropped.
4. In {@link ThreadPoolExecutor.DiscardOldestPolicy}, if the executor is not shut down, the task at the head of the work queue is dropped, and then execution is retried (which can fail again, causing this to be repeated.)
It is possible to define and use other kinds of {@link RejectedExecutionHandler} classes. Doing so requires some careespecially when policies are designed to work only under particular capacity or queuing policies.
Hook methods

This class provides protected overridable {@link ThreadPoolExecutor#beforeExecute} and {@link ThreadPoolExecutor#afterExecute} methods that are called before andafter execution of each task. These can be used to manipulate the execution environment; for example, reinitializing ThreadLocals, gathering statistics, or adding log entries. Additionally, method {@link ThreadPoolExecutor#terminated} can be overridden to performany special processing that needs to be done once the Executor has fully terminated.
If hook or callback methods throw exceptions, internal worker threads may in turn fail and abruptly terminate.

Queue maintenance

Method {@link ThreadPoolExecutor#getQueue} allows access tothe work queue for purposes of monitoring and debugging. Use of this method for any other purpose is strongly discouraged. Two supplied methods, {@link ThreadPoolExecutor#remove} and {@link ThreadPoolExecutor#purge} are available to assist in storagereclamation when large numbers of queued tasks become cancelled.

Finalization

A pool that is no longer referenced in a program AND has no remaining threads will be shutdown automatically. If you would like to ensure that unreferenced pools are reclaimed even if users forget to call {@link ThreadPoolExecutor#shutdown}, then you must arrange that unused threads eventually die, by setting appropriate keep-alive times, using a lower bound of zero core threads and/or setting {@link ThreadPoolExecutor#allowCoreThreadTimeOut(boolean)}.
Extension example. Most extensions of this class override one or more of the protected hook methods. For example, here is a subclass that adds a simple pause/resume feature:
```
 class PausableThreadPoolExecutor extends ThreadPoolExecutor { private boolean isPaused; private ReentrantLock pauseLock = new ReentrantLock(); private Condition unpaused = pauseLock.newCondition(); public PausableThreadPoolExecutor(...) { super(...); } protected void beforeExecute(Thread t, Runnable r) { super.beforeExecute(t, r); pauseLock.lock(); try { while (isPaused) unpaused.await(); } catch (InterruptedException ie) { t.interrupt(); } finally { pauseLock.unlock(); } } public void pause() { pauseLock.lock(); try { isPaused = true; } finally { pauseLock.unlock(); } } public void resume() { pauseLock.lock(); try { isPaused = false; unpaused.signalAll(); } finally { pauseLock.unlock(); } } } 
```
@since 1.5 @author Doug Lea

      return;
    }
    ThreadFactoryBuilder builder = new ThreadFactoryBuilder();
    builder.setNameFormat("StoreFileSplitter-%1$d");
    ThreadFactory factory = builder.build();
    ThreadPoolExecutor threadPool =
      (ThreadPoolExecutor) Executors.newFixedThreadPool(nbFiles, factory);
    List<Future<Void>> futures = new ArrayList<Future<Void>>(nbFiles);


     // Split each store file.
    for (StoreFile sf: hstoreFilesToSplit) {
      //splitStoreFile(sf, splitdir);
      StoreFileSplitter sfs = new StoreFileSplitter(sf, splitdir);
      futures.add(threadPool.submit(sfs));
    }
    // Shutdown the pool
    threadPool.shutdown();


    // Wait for all the tasks to finish
    try {
      boolean stillRunning = !threadPool.awaitTermination(
          this.fileSplitTimeout, TimeUnit.MILLISECONDS);
      if (stillRunning) {
        threadPool.shutdownNow();
        // wait for the thread to shutdown completely.
        while (!threadPool.isTerminated()) {
          Thread.sleep(50);
        }
        throw new IOException("Took too long to split the" +
            " files and create the references, aborting split");
      }

View Full Code Here

  ExecutorService createExecutor(BlockingQueue<Runnable> callQueue,
                                 int workerThreads) {
    ThreadFactoryBuilder tfb = new ThreadFactoryBuilder();
    tfb.setDaemon(true);
    tfb.setNameFormat("thrift-worker-%d");
    return new ThreadPoolExecutor(workerThreads, workerThreads,
            Long.MAX_VALUE, TimeUnit.SECONDS, callQueue, tfb.build());
  }

View Full Code Here

  public static List<HRegionInfo> createRegions(final Configuration conf, final Path rootDir,
      final HTableDescriptor hTableDescriptor, final HRegionInfo[] newRegions,
      final RegionFillTask task) throws IOException {
    if (newRegions == null) return null;
    int regionNumber = newRegions.length;
    ThreadPoolExecutor regionOpenAndInitThreadPool = getRegionOpenAndInitThreadPool(conf,
        "RegionOpenAndInitThread-" + hTableDescriptor.getNameAsString(), regionNumber);
    CompletionService<HRegionInfo> completionService = new ExecutorCompletionService<HRegionInfo>(
        regionOpenAndInitThreadPool);
    List<HRegionInfo> regionInfos = new ArrayList<HRegionInfo>();
    for (final HRegionInfo newRegion : newRegions) {
      completionService.submit(new Callable<HRegionInfo>() {
        public HRegionInfo call() throws IOException {
          // 1. Create HRegion
          HRegion region = HRegion.createHRegion(newRegion,
              rootDir, conf, hTableDescriptor, null,
              false, true);
          try {
            // 2. Custom user code to interact with the created region
            if (task != null) {
              task.fillRegion(region);
            }
          } finally {
            // 3. Close the new region to flush to disk. Close log file too.
            region.close();
          }
          return region.getRegionInfo();
        }
      });
    }
    try {
      // 4. wait for all regions to finish creation
      for (int i = 0; i < regionNumber; i++) {
        Future<HRegionInfo> future = completionService.take();
        HRegionInfo regionInfo = future.get();
        regionInfos.add(regionInfo);
      }
    } catch (InterruptedException e) {
      LOG.error("Caught " + e + " during region creation");
      throw new InterruptedIOException(e.getMessage());
    } catch (ExecutionException e) {
      throw new IOException(e);
    } finally {
      regionOpenAndInitThreadPool.shutdownNow();
    }
    return regionInfos;
  }

View Full Code Here

   */
  static ThreadPoolExecutor getRegionOpenAndInitThreadPool(final Configuration conf,
      final String threadNamePrefix, int regionNumber) {
    int maxThreads = Math.min(regionNumber, conf.getInt(
        "hbase.hregion.open.and.init.threads.max", 10));
    ThreadPoolExecutor regionOpenAndInitThreadPool = Threads
    .getBoundedCachedThreadPool(maxThreads, 30L, TimeUnit.SECONDS,
        new ThreadFactory() {
          private int count = 1;


          public Thread newThread(Runnable r) {

View Full Code Here


    if (files == null || files.length == 0) {
      return results;
    }
    // initialize the thread pool for opening store files in parallel..
    ThreadPoolExecutor storeFileOpenerThreadPool =
      this.region.getStoreFileOpenAndCloseThreadPool("StoreFileOpenerThread-" +
          this.family.getNameAsString());
    CompletionService<StoreFile> completionService =
      new ExecutorCompletionService<StoreFile>(storeFileOpenerThreadPool);


    int totalValidStoreFile = 0;
    for (int i = 0; i < files.length; i++) {
      // Skip directories.
      if (files[i].isDir()) {
        continue;
      }
      final Path p = files[i].getPath();
      // Check for empty hfile. Should never be the case but can happen
      // after data loss in hdfs for whatever reason (upgrade, etc.): HBASE-646
      // NOTE: that the HFileLink is just a name, so it's an empty file.
      if (!HFileLink.isHFileLink(p) && this.fs.getFileStatus(p).getLen() <= 0) {
        LOG.warn("Skipping " + p + " because its empty. HBASE-646 DATA LOSS?");
        continue;
      }


      // open each store file in parallel
      completionService.submit(new Callable<StoreFile>() {
        public StoreFile call() throws IOException {
          StoreFile storeFile = new StoreFile(fs, p, conf, cacheConf,
              family.getBloomFilterType(), dataBlockEncoder);
          passSchemaMetricsTo(storeFile);
          storeFile.createReader();
          return storeFile;
        }
      });
      totalValidStoreFile++;
    }


    IOException ioe = null;
    try {
      for (int i = 0; i < totalValidStoreFile; i++) {
        try {
          Future<StoreFile> future = completionService.take();
          StoreFile storeFile = future.get();
          long length = storeFile.getReader().length();
          this.storeSize += length;
          this.totalUncompressedBytes +=
              storeFile.getReader().getTotalUncompressedBytes();
          if (LOG.isDebugEnabled()) {
            LOG.debug("loaded " + storeFile.toStringDetailed());
          }
          results.add(storeFile);
        } catch (InterruptedException e) {
          if (ioe == null) ioe = new InterruptedIOException(e.getMessage());
        } catch (ExecutionException e) {
          if (ioe == null) ioe = new IOException(e.getCause());
        } 
      } 
    } finally {
      storeFileOpenerThreadPool.shutdownNow();
    }
    if (ioe != null) {
      // close StoreFile readers
      try {
        for (StoreFile file : results) {

View Full Code Here

      // Clear so metrics doesn't find them.
      storefiles = ImmutableList.of();


      if (!result.isEmpty()) {
        // initialize the thread pool for closing store files in parallel.
        ThreadPoolExecutor storeFileCloserThreadPool = this.region
            .getStoreFileOpenAndCloseThreadPool("StoreFileCloserThread-"
                + this.family.getNameAsString());


        // close each store file in parallel
        CompletionService<Void> completionService =
          new ExecutorCompletionService<Void>(storeFileCloserThreadPool);
        for (final StoreFile f : result) {
          completionService.submit(new Callable<Void>() {
            public Void call() throws IOException {
              f.closeReader(true);
              return null;
            }
          });
        }


        IOException ioe = null;
        try {
          for (int i = 0; i < result.size(); i++) {
            try {
              Future<Void> future = completionService.take();
              future.get();
            } catch (InterruptedException e) {
              if (ioe == null) {
                ioe = new InterruptedIOException();
                ioe.initCause(e);
              }
            } catch (ExecutionException e) {
              if (ioe == null) ioe = new IOException(e.getCause());
            }
          }
        } finally {
          storeFileCloserThreadPool.shutdownNow();
        }
        if (ioe != null) throw ioe;
      }
      LOG.info("Closed " + this);
      return result;

View Full Code Here

    long wakeMillis = conf.getLong(SNAPSHOT_REQUEST_WAKE_MILLIS_KEY, SNAPSHOT_REQUEST_WAKE_MILLIS_DEFAULT);
    long keepAlive = conf.getLong(SNAPSHOT_TIMEOUT_MILLIS_KEY, SNAPSHOT_TIMEOUT_MILLIS_DEFAULT);
    int opThreads = conf.getInt(SNAPSHOT_REQUEST_THREADS_KEY, SNAPSHOT_REQUEST_THREADS_DEFAULT);


    // create the actual snapshot procedure member
    ThreadPoolExecutor pool = ProcedureMember.defaultPool(wakeMillis, keepAlive, opThreads, nodeName);
    this.member = new ProcedureMember(memberRpcs, pool, new SnapshotSubprocedureBuilder());
  }

View Full Code Here

      long keepAlive = conf.getLong(
        RegionServerSnapshotManager.SNAPSHOT_TIMEOUT_MILLIS_KEY,
        RegionServerSnapshotManager.SNAPSHOT_TIMEOUT_MILLIS_DEFAULT);
      int threads = conf.getInt(CONCURENT_SNAPSHOT_TASKS_KEY, DEFAULT_CONCURRENT_SNAPSHOT_TASKS);
      this.name = name;
      executor = new ThreadPoolExecutor(1, threads, keepAlive, TimeUnit.MILLISECONDS,
          new LinkedBlockingQueue<Runnable>(), new DaemonThreadFactory("rs("
              + name + ")-snapshot-pool"));
      taskPool = new ExecutorCompletionService<Void>(executor);
    }

View Full Code Here

                    log.warn("Concurrent download resources selected, "// $NON-NLS-1$
                            + "but pool size value is bad. Use default value");// $NON-NLS-1$
                }
                // Thread pool Executor to get resources 
                // use a LinkedBlockingQueue, note: max pool size doesn't effect
                final ThreadPoolExecutor exec = new ThreadPoolExecutor(
                        poolSize, poolSize, KEEPALIVETIME, TimeUnit.SECONDS,
                        new LinkedBlockingQueue<Runnable>());


                try {
                    // sample all resources with threadpool
                    final List<Future<HTTPSampleResult>> retExec = exec.invokeAll(liste);
                    // call normal shutdown (wait ending all tasks)
                    exec.shutdown();
                    // put a timeout if tasks couldn't terminate
                    exec.awaitTermination(AWAIT_TERMINATION_TIMEOUT, TimeUnit.SECONDS);


                    // add result to main sampleResult
                    for (Future<HTTPSampleResult> future : retExec) {
                        final HTTPSampleResult binRes = future.get();
                        res.addSubResult(binRes);

View Full Code Here

    this.wakeFrequency = conf.getInt(SNAPSHOT_WAKE_MILLIS_KEY, SNAPSHOT_WAKE_MILLIS_DEFAULT);
    long keepAliveTime = conf.getLong(SNAPSHOT_TIMEOUT_MILLIS_KEY, SNAPSHOT_TIMEOUT_MILLIS_DEFAULT);


    // setup the default procedure coordinator
    String name = master.getServerName().toString();
    ThreadPoolExecutor tpool = ProcedureCoordinator.defaultPool(name, keepAliveTime, opThreads, wakeFrequency);
    ProcedureCoordinatorRpcs comms = new ZKProcedureCoordinatorRpcs(
        master.getZooKeeper(), SnapshotManager.ONLINE_SNAPSHOT_CONTROLLER_DESCRIPTION, name);
    this.coordinator = new ProcedureCoordinator(comms, tpool);
    this.rootDir = master.getMasterFileSystem().getRootDir();
    this.executorService = master.getExecutorService();

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of java.util.concurrent.ThreadPoolExecutor

ch.cmbntr.modulizer.bootstrap.util.Resources$BasicPool

co.paralleluniverse.common.monitoring.ThreadPoolExecutorMonitor

com.alibaba.dubbo.remoting.handler.ConnectChannelHandlerTest

com.alibaba.dubbo.remoting.transport.dispather.connection.ConnectionOrderedChannelHandler

com.alibaba.otter.node.etl.common.pipe.impl.http.archive.ArchiveBean

com.alibaba.otter.node.etl.extract.extractor.DatabaseExtractor

com.alibaba.otter.node.etl.load.loader.db.DbLoadAction

com.alibaba.otter.node.etl.load.loader.db.FileLoadAction

com.alibaba.otter.node.etl.OtterController

com.alibaba.otter.shared.common.utils.thread.ExecutorTemplate

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.