Preface

Previous article " Java Thread Pool Executor><Analysis of ThreadPoolExecutor Principle ThreadPoolExecutor describes the concepts and principles of ThreadPoolExecutor, and today we'll look at its application in the Dubbo framework.

ThreadFactory and AbortPolicy

Dubbo provides us with several different types of thread pool implementations, all of which use the ThreadPoolExecutor thread pool in the JDK at the bottom.ThreadPoolExecutor is already familiar to us and has several very important parameters in its constructor.These include the rejection policy (ThreadPoolExecutor.AbortPolicy) and the ThreadFactory, which customizes ThreadPoolExecutor.AbortPolicy and ThreadFactory in Dubbo.Before learning about thread pools, let's look at the implementation of both for a better understanding.

NamedInternalThreadFactory is a subclass of ThreadFactory for custom threads in Dubbo.Its class diagram is as follows:

Where the NamedInternalThreadFactory class is implemented as follows:

public class NamedInternalThreadFactory extends NamedThreadFactory {

    public NamedInternalThreadFactory() {
        super();
    }

    public NamedInternalThreadFactory(String prefix) {
        super(prefix, false);
    }

    public NamedInternalThreadFactory(String prefix, boolean daemon) {
        super(prefix, daemon);
    }

    @Override
    public Thread newThread(Runnable runnable) {
        String name = mPrefix + mThreadNum.getAndIncrement();
        InternalThread ret = new InternalThread(mGroup, runnable, name, 0);
        ret.setDaemon(mDaemon);
        return ret;
    }
}

The implementation of the NamedThreadFactory class is as follows:

public class NamedThreadFactory implements ThreadFactory {

    protected static final AtomicInteger POOL_SEQ = new AtomicInteger(1);

    protected final AtomicInteger mThreadNum = new AtomicInteger(1);

    protected final String mPrefix;

    protected final boolean mDaemon;

    protected final ThreadGroup mGroup;

    public NamedThreadFactory() {
        this("pool-" + POOL_SEQ.getAndIncrement(), false);
    }

    public NamedThreadFactory(String prefix) {
        this(prefix, false);
    }

    public NamedThreadFactory(String prefix, boolean daemon) {
        mPrefix = prefix + "-thread-";
        mDaemon = daemon;
        SecurityManager s = System.getSecurityManager();
        mGroup = (s == null) ? Thread.currentThread().getThreadGroup() : s.getThreadGroup();
    }

    @Override
    public Thread newThread(Runnable runnable) {
        String name = mPrefix + mThreadNum.getAndIncrement();
        Thread ret = new Thread(mGroup, runnable, name, 0);
        ret.setDaemon(mDaemon);
        return ret;
    }

    public ThreadGroup getThreadGroup() {
        return mGroup;
    }

Here, the code above describes Dubbo's naming rules for threads in the thread pool, which are used to facilitate tracking information.

Next, let's look at the implementation of the rejection policy AbortPolicyWithReport class, whose class diagram looks like the following:

The source code is as follows:

public class AbortPolicyWithReport extends ThreadPoolExecutor.AbortPolicy {

    protected static final Logger logger = LoggerFactory.getLogger(AbortPolicyWithReport.class);

    private final String threadName;

    private final URL url;

    private static volatile long lastPrintTime = 0;

    private static final long TEN_MINUTES_MILLS = 10 * 60 * 1000;

    private static final String OS_WIN_PREFIX = "win";

    private static final String OS_NAME_KEY = "os.name";

    private static final String WIN_DATETIME_FORMAT = "yyyy-MM-dd_HH-mm-ss";

    private static final String DEFAULT_DATETIME_FORMAT = "yyyy-MM-dd_HH:mm:ss";

    private static Semaphore guard = new Semaphore(1);

    public AbortPolicyWithReport(String threadName, URL url) {
        this.threadName = threadName;
        this.url = url;
    }

    // Overrides the rejectedExecution method of the parent ThreadPoolExecutor.AbortPolicy.
    @Override
    public void rejectedExecution(Runnable r, ThreadPoolExecutor e) {
       // Construct warn parameters, which include information on thread status, number of thread pools, active number, number of core thread pools, maximum number of thread pools, and so on.
        String msg = String.format("Thread pool is EXHAUSTED!" +
                " Thread Name: %s, Pool Size: %d (active: %d, core: %d, max: %d, largest: %d), Task: %d (completed: "
                + "%d)," +
                " Executor status:(isShutdown:%s, isTerminated:%s, isTerminating:%s), in %s://%s:%d!",
            threadName, e.getPoolSize(), e.getActiveCount(), e.getCorePoolSize(), e.getMaximumPoolSize(),
            e.getLargestPoolSize(),
            e.getTaskCount(), e.getCompletedTaskCount(), e.isShutdown(), e.isTerminated(), e.isTerminating(),
            url.getProtocol(), url.getIp(), url.getPort());
        logger.warn(msg);
        //  dump stack information
        dumpJStack();
        throw new RejectedExecutionException(msg);
    }

 // When the rejectedExecution method is executed, it is executed.dump stack information is sent to the DUMP_DIRECTORY directory, which defaults to the user.name directory.
    private void dumpJStack() {
        long now = System.currentTimeMillis();

        //dump every 10 minutes
        if (now - lastPrintTime < TEN_MINUTES_MILLS) {
            return;
        }

        if (!guard.tryAcquire()) {
            return;
        }

        ExecutorService pool = Executors.newSingleThreadExecutor();
        pool.execute(() -> {
            String dumpPath = url.getParameter(DUMP_DIRECTORY, System.getProperty("user.home"));

            SimpleDateFormat sdf;

            String os = System.getProperty(OS_NAME_KEY).toLowerCase();

            // window system don't support ":" in file name
            if (os.contains(OS_WIN_PREFIX)) {
                sdf = new SimpleDateFormat(WIN_DATETIME_FORMAT);
            } else {
                sdf = new SimpleDateFormat(DEFAULT_DATETIME_FORMAT);
            }

            String dateStr = sdf.format(new Date());
            //try-with-resources
            try (FileOutputStream jStackStream = new FileOutputStream(
                new File(dumpPath, "Dubbo_JStack.log" + "." + dateStr))) {
                // Tool class, omitted here, interesting to see.
                JVMUtil.jstack(jStackStream);
            } catch (Throwable t) {
                logger.error("dump jStack error", t);
            } finally {
                guard.release();
            }
            lastPrintTime = System.currentTimeMillis();
        });
        //must shutdown thread pool ,if not will lead to OOM
        pool.shutdown();
    }
}

The above code is not difficult, it is log, dump stack information. Its purpose is to record the on-site information when the thread pool is full, that is, when AbortPolicy is executed, mainly for later analysis and problem investigation.

Implementation of Thread Pool

The custom ThreadFactory and AbortPolicyWithReport classes in the Dubbo thread pool are described above.Next, we continue with the different thread pool implementations provided by Dubbo, whose class diagrams are as follows:

 

1. LimitedThreadPool Thread Pool

The source code is as follows:

public class LimitedThreadPool implements ThreadPool {

    @Override
    public Executor getExecutor(URL url) {
        String name = url.getParameter(THREAD_NAME_KEY, DEFAULT_THREAD_NAME);
        int cores = url.getParameter(CORE_THREADS_KEY, DEFAULT_CORE_THREADS);
        int threads = url.getParameter(THREADS_KEY, DEFAULT_THREADS);
        int queues = url.getParameter(QUEUES_KEY, DEFAULT_QUEUES);
        return new ThreadPoolExecutor(cores, threads, Long.MAX_VALUE, TimeUnit.MILLISECONDS,
                queues == 0 ? new SynchronousQueue<Runnable>() :
                        (queues < 0 ? new LinkedBlockingQueue<Runnable>()
                                : new LinkedBlockingQueue<Runnable>(queues)),
                new NamedInternalThreadFactory(name, true), new AbortPolicyWithReport(name, url));
    }

Where:

1. The THREAD_NAME_KEY value is threadname, denoted as threadname, and its default value is Dubbo.

2. The CORE_THREADS_KEY value is: corethreads, which indicates the number of core thread pools with a default value of: 0.

3. The THREADS_KEY value is threads for the maximum number of threads and the default value is 200.

4. The QUEUES_KEY value is: queues denotes the size of the blocking queue and the default value is: 0.

    

Remarks:

1. The cores, threads parameters in the thread pool are determined externally, where the keepAliveTime value is Long.MAX_VALUE and the TimeUnit is TimeUnit.MILLISECONDS (milliseconds).(This means that all threads in the thread pool never expire, and theoretically larger than Long.MAX_VALUE will expire because it is large enough to be considered never to expire here).

2. Trinomial operators are used here:

   When queues = 0, BlockingQueue is SynchronousQueue.

   When queues < 0, construct a new LinkedBlockingQueue.

   When queues > 0, construct a LinkedBlockingQueue for the specified element.

   queues == 0 ? new SynchronousQueue<Runnable>() :
     (queues < 0 ? new LinkedBlockingQueue<Runnable>()
         : new LinkedBlockingQueue<Runnable>(queues)

The thread pool is characterized by the ability to create several threads, with a default value of 200, and a very long thread life cycle in the thread pool, which can even be seen as never expiring.

2. CachedThreadPool Thread Pool

Source code:

 public class CachedThreadPool implements ThreadPool {

    @Override
    public Executor getExecutor(URL url) {
        String name = url.getParameter(THREAD_NAME_KEY, DEFAULT_THREAD_NAME);
        int cores = url.getParameter(CORE_THREADS_KEY, DEFAULT_CORE_THREADS);
        int threads = url.getParameter(THREADS_KEY, Integer.MAX_VALUE);
        int queues = url.getParameter(QUEUES_KEY, DEFAULT_QUEUES);
        int alive = url.getParameter(ALIVE_KEY, DEFAULT_ALIVE);
        return new ThreadPoolExecutor(cores, threads, alive, TimeUnit.MILLISECONDS,
                queues == 0 ? new SynchronousQueue<Runnable>() :
                        (queues < 0 ? new LinkedBlockingQueue<Runnable>()
                                : new LinkedBlockingQueue<Runnable>(queues)),
                new NamedInternalThreadFactory(name, true), new AbortPolicyWithReport(name, url));
    }
}

Where:

1. The THREAD_NAME_KEY value is threadname, denoted as threadname, and its default value is Dubbo.

2. The CORE_THREADS_KEY value is: corethreads, expressed as: Number of core thread pools, with a default value of: 0.

3. The THREADS_KEY value is threads, which means the maximum number of threads and the default value is Integer.MAX_VALUE.

4. The QUEUES_KEY value is: queues, meaning: blocked queue size, with a default value of: 0.

5. The ALIVE_KEY value is: alive, meaning: keep AliveTime represents the lifetime of threads in the thread pool, and its default value is: 60 * 1000 (milliseconds) or one minute.

 

The thread pool is characterized by the ability to create an infinite number of threads (much lower than the Integer.MAX_VALUE value, which is considered infinite under operating system constraints), with a default maximum thread lifetime of 1 minute.That means you can create an infinite number of threads, but by default the lifecycle of threads is shorter!

3. FixedThreadPool Thread Pool

public class FixedThreadPool implements ThreadPool {

    @Override
    public Executor getExecutor(URL url) {
        String name = url.getParameter(THREAD_NAME_KEY, DEFAULT_THREAD_NAME);
        int threads = url.getParameter(THREADS_KEY, DEFAULT_THREADS);
        int queues = url.getParameter(QUEUES_KEY, DEFAULT_QUEUES);
        return new ThreadPoolExecutor(threads, threads, 0, TimeUnit.MILLISECONDS,
                queues == 0 ? new SynchronousQueue<Runnable>() :
                        (queues < 0 ? new LinkedBlockingQueue<Runnable>()
                                : new LinkedBlockingQueue<Runnable>(queues)),
                new NamedInternalThreadFactory(name, true), new AbortPolicyWithReport(name, url));
    }

Where:

1. The THREADS_KEY value is threads, which means the maximum number of threads and the default value is 200.

2. The QUEUES_KEY value is: queues, meaning: blocked queue size, with a default value of: 0.

3. The number of threads for corePoolSize and maximumPoolSize are: threads, (meaning that the number of core threads equals the maximum number of threads).

4. The default value for keepAliveTime is 0, and when the number of threads is greater than corePoolSize, the extra idle threads terminate immediately.

    

This thread pool is characterized by the same number of corePoolSize s as maxinumPoolSize in the thread pool, and when a submitted task is larger than the core thread pool, it is placed in the LinkedBlockingQueue queue for execution, which is also the default thread pool used in Dubbo.

4. EagerThreadPool Thread Pool

Source code:

public class EagerThreadPool implements ThreadPool {

        @Override
        public Executor getExecutor(URL url) {
            String name = url.getParameter(THREAD_NAME_KEY, DEFAULT_THREAD_NAME);
            int cores = url.getParameter(CORE_THREADS_KEY, DEFAULT_CORE_THREADS);
            int threads = url.getParameter(THREADS_KEY, Integer.MAX_VALUE);
            int queues = url.getParameter(QUEUES_KEY, DEFAULT_QUEUES);
            int alive = url.getParameter(ALIVE_KEY, DEFAULT_ALIVE);

            // init queue and executor
            TaskQueue<Runnable> taskQueue = new TaskQueue<Runnable>(queues <= 0 ? 1 : queues);
            EagerThreadPoolExecutor executor = new EagerThreadPoolExecutor(cores,
                    threads,
                    alive,
                    TimeUnit.MILLISECONDS,
                    taskQueue,
                    new NamedInternalThreadFactory(name, true),
                    new AbortPolicyWithReport(name, url));
            taskQueue.setExecutor(executor);
            return executor;
        }
}

Remarks:

This thread pool is implemented somewhat differently from the thread pool above, where the constructor of the ThreadPoolExecutor class is used directly.In this thread pool implementation, a custom EagerThreadPoolExecutor thread pool is first constructed, and its underlying implementation is also based on the ThreadPoolExecutor class, with the following code:

public class EagerThreadPoolExecutor extends ThreadPoolExecutor {

    /**
     * task count
     */
    private final AtomicInteger submittedTaskCount = new AtomicInteger(0);

    public EagerThreadPoolExecutor(int corePoolSize,
                                   int maximumPoolSize,
                                   long keepAliveTime,
                                   TimeUnit unit, TaskQueue<Runnable> workQueue,
                                   ThreadFactory threadFactory,
                                   RejectedExecutionHandler handler) {
        super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue, threadFactory, handler);
    }

    /**
     * @return current tasks which are executed
     */
    public int getSubmittedTaskCount() {
        return submittedTaskCount.get();
    }

    @Override
    protected void afterExecute(Runnable r, Throwable t) {
        // The number of tasks executed decreases in turn
        submittedTaskCount.decrementAndGet();
    }

    @Override
    public void execute(Runnable command) {
        if (command == null) {
            throw new NullPointerException();
        }
        // do not increment in method beforeExecute!
        // The submissions are incremented in turn.
        submittedTaskCount.incrementAndGet();
        try {
          // Call parent method to execute thread task
            super.execute(command);
        } catch (RejectedExecutionException rx) {
            // Re-add task to queue
            final TaskQueue queue = (TaskQueue) super.getQueue();
            try {
               //If the addition fails, reduce the number of tasks and throw an exception.
                if (!queue.retryOffer(command, 0, TimeUnit.MILLISECONDS)) {
                    submittedTaskCount.decrementAndGet();
                    throw new RejectedExecutionException("Queue capacity is full.", rx);
                }
            } catch (InterruptedException x) {
                submittedTaskCount.decrementAndGet();
                throw new RejectedExecutionException(x);
            }
        } catch (Throwable t) {
            // decrease any way
            submittedTaskCount.decrementAndGet();
            throw t;
        }
    }

Here we find that several methods of the parent ThreadPoolExecutor class are overloaded in the EagerThreadPoolExecutor class, as follows: afterExecute, execute method.The submittedTaskCount property is added to the task statistics separately. When the execute method of the parent class throws a RejectedExecutionException exception, the task is put back into the queue to execute with the following TaskQueue code:

public class TaskQueue<R extends Runnable> extends LinkedBlockingQueue<Runnable> {

    private static final long serialVersionUID = -2635853580887179627L;

    private EagerThreadPoolExecutor executor;

    public TaskQueue(int capacity) {
        super(capacity);
    }

    public void setExecutor(EagerThreadPoolExecutor exec) {
        executor = exec;
    }

    @Override
    public boolean offer(Runnable runnable) {
        if (executor == null) {
            throw new RejectedExecutionException("The task queue does not have executor!");
        }

        int currentPoolThreadSize = executor.getPoolSize();
        // have free worker. put task into queue to let the worker deal with task.
        // When the number of tasks submitted is less than the current thread, the offer method of the parent class is invoked between them.
        if (executor.getSubmittedTaskCount() < currentPoolThreadSize) {
            return super.offer(runnable);
        }

        // return false to let executor create new worker.
        // When the current number of threads is smaller than the maximum number of threads, a worker is created by returning false directly.
        if (currentPoolThreadSize < executor.getMaximumPoolSize()) {
            return false;
        }

        // currentPoolThreadSize >= max
        return super.offer(runnable);
    }

    /**
     * retry offer task
     *
     * @param o task
     * @return offer success or not
     * @throws RejectedExecutionException if executor is terminated.
     */
    public boolean retryOffer(Runnable o, long timeout, TimeUnit unit) throws InterruptedException {
        if (executor.isShutdown()) {
            throw new RejectedExecutionException("Executor is shutdown!");
        }
        return super.offer(o, timeout, unit);
    }
}

This thread pool is characterized by the ability to re-execute rejected task s and re-add work queue s.It's equivalent to having a retry mechanism!

epilogue

From the above analysis, I believe you should know something about thread pools in Dubbo.If you don't know anything yet, you can track it through debug.In fact, in many open source frameworks, there is a custom thread pool, but the underlying thread pool is the ThreadPoolExecutor thread pool. This knowledge point suggests that you must master it. It is a common knowledge point for both actual work and interviews.

 

Related reading:

<BigDecimal you don't know>

<Analysis of ThreadPoolExecutor Principle>

<Java Thread Pool Executor>

<Use Mybatis to be honest and not lazy!>