Examples of com.facebook.presto.hive.util.BoundedExecutor

com.facebook.presto.hive.util.BoundedExecutor
THIS CLASS IS COPIED FROM com.facebook.presto.util.BoundedExecutor
Guarantees that no more than maxThreads will be used to execute tasks submitted through execute().
There are a few interesting properties: - Multiple BoundedExecutors over a single coreExecutor will have fair sharing of the coreExecutor threads proportional to their relative maxThread counts, but can use less if not as active. - Tasks submitted to a BoundedExecutor is guaranteed to have those tasks handed to threads in that order. - Will not encounter starvation

        this(connectorId,
                metastore,
                namenodeStats,
                hdfsEnvironment,
                directoryLister,
                new BoundedExecutor(executorService, hiveClientConfig.getMaxGlobalSplitIteratorThreads()),
                hiveClientConfig.getMaxSplitSize(),
                hiveClientConfig.getMaxOutstandingSplits(),
                hiveClientConfig.getMaxSplitIteratorThreads(),
                hiveClientConfig.getMinPartitionBatchSize(),
                hiveClientConfig.getMaxPartitionBatchSize());

View Full Code Here

    }


    public SplitSource get()
    {
        // Each iterator has its own bounded executor and can be independently suspended
        final SuspendingExecutor suspendingExecutor = new SuspendingExecutor(new BoundedExecutor(executor, maxThreads));
        final HiveSplitSource hiveSplitSource = new HiveSplitSource(connectorId, maxOutstandingSplits, suspendingExecutor);
        executor.execute(new Runnable()
        {
            @Override
            public void run()

View Full Code Here


    @Override
    public Iterator<Split> iterator()
    {
        // Each iterator has its own bounded executor and can be independently suspended
        final SuspendingExecutor suspendingExecutor = new SuspendingExecutor(new BoundedExecutor(executor, maxThreads));
        final HiveSplitQueue hiveSplitQueue = new HiveSplitQueue(maxOutstandingSplits, suspendingExecutor);
        executor.submit(new Callable<Void>()
        {
            @Override
            public Void call()

View Full Code Here

    }


    public SplitSource get()
    {
        // Each iterator has its own bounded executor and can be independently suspended
        final SuspendingExecutor suspendingExecutor = new SuspendingExecutor(new BoundedExecutor(executor, maxThreads));
        final HiveSplitSource hiveSplitSource = new HiveSplitSource(connectorId, maxOutstandingSplits, suspendingExecutor);
        executor.execute(new Runnable()
        {
            @Override
            public void run()

View Full Code Here

            @ForHiveClient ExecutorService executorService)
    {
        this(connectorId,
                metastore,
                hdfsEnvironment,
                new BoundedExecutor(executorService, hiveClientConfig.getMaxGlobalSplitIteratorThreads()),
                hiveClientConfig.getMaxSplitSize(),
                hiveClientConfig.getMaxOutstandingSplits(),
                hiveClientConfig.getMaxSplitIteratorThreads(),
                hiveClientConfig.getMinPartitionBatchSize(),
                hiveClientConfig.getMaxPartitionBatchSize());

View Full Code Here

                metastore,
                namenodeStats,
                hdfsEnvironment,
                directoryLister,
                DateTimeZone.forTimeZone(hiveClientConfig.getTimeZone()),
                new BoundedExecutor(executorService, hiveClientConfig.getMaxGlobalSplitIteratorThreads()),
                hiveClientConfig.getMaxSplitSize(),
                hiveClientConfig.getMaxOutstandingSplits(),
                hiveClientConfig.getMaxSplitIteratorThreads(),
                hiveClientConfig.getMinPartitionBatchSize(),
                hiveClientConfig.getMaxPartitionBatchSize(),

View Full Code Here

    }


    public SplitSource get()
    {
        // Each iterator has its own bounded executor and can be independently suspended
        final SuspendingExecutor suspendingExecutor = new SuspendingExecutor(new BoundedExecutor(executor, maxThreads));
        final HiveSplitSource hiveSplitSource = new HiveSplitSource(connectorId, maxOutstandingSplits, suspendingExecutor);
        executor.execute(new Runnable()
        {
            @Override
            public void run()

View Full Code Here

            @ForHiveClient ExecutorService executorService)
    {
        this(connectorId,
                metastore,
                hdfsEnvironment,
                new BoundedExecutor(executorService, hiveClientConfig.getMaxGlobalSplitIteratorThreads()),
                hiveClientConfig.getMaxSplitSize(),
                hiveClientConfig.getMaxOutstandingSplits(),
                hiveClientConfig.getMaxSplitIteratorThreads(),
                hiveClientConfig.getMinPartitionBatchSize(),
                hiveClientConfig.getMaxPartitionBatchSize());

View Full Code Here

    }


    public ConnectorSplitSource get()
    {
        // Each iterator has its own bounded executor and can be independently suspended
        final SuspendingExecutor suspendingExecutor = new SuspendingExecutor(new BoundedExecutor(executor, maxThreads));
        final HiveSplitSource splitSource = new HiveSplitSource(connectorId, maxOutstandingSplits, suspendingExecutor);


        FutureTask<?> producer = new FutureTask<>(new Runnable()
        {
            @Override

View Full Code Here

                metastore,
                namenodeStats,
                hdfsEnvironment,
                directoryLister,
                DateTimeZone.forTimeZone(hiveClientConfig.getTimeZone()),
                new BoundedExecutor(executorService, hiveClientConfig.getMaxGlobalSplitIteratorThreads()),
                hiveClientConfig.getMaxSplitSize(),
                hiveClientConfig.getMaxOutstandingSplits(),
                hiveClientConfig.getMaxSplitIteratorThreads(),
                hiveClientConfig.getMinPartitionBatchSize(),
                hiveClientConfig.getMaxPartitionBatchSize(),

View Full Code Here

0 1 2 3

TOP

Related Classes of com.facebook.presto.hive.util.BoundedExecutor

com.facebook.presto.hive.HiveClient

com.facebook.presto.hive.HiveSplitIterable

com.facebook.presto.hive.HiveSplitSourceProvider

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.