preface

RejectedExecutionException exception thrown by thread pool submission task on line

That is, the task submission performs the operation of reject policy. Check the business situation and thread pool configuration. It is found that the number of tasks executed in parallel is less than the maximum number of threads in the thread pool. The following is the troubleshooting process

1, Business scenario

1.1. Task description

Each time a group of tasks is executed, there are at most 15 tasks in a group, which are executed by multiple threads, and each thread processes one task; After each group of tasks is executed, the next group can be executed. There is no case that the tasks of the previous group and the next group are executed together.

1.2. Task submission process

① Task start ➠

② Get a set of tasks ➠

③ Submit task to thread pool ➠

④ Future#get() blocks waiting until the execution of the whole set of tasks is completed ➠

⑤ Go to ② to get the next batch of executable tasks

1.3. Thread pool configuration

<bean id="executor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
    <property name="corePoolSize" value="14"/>
    <property name="maxPoolSize" value="30"/>
    <property name="queueCapacity" value="1"/>
</bean>

2, Something went wrong

RejectedExecutionException exception occurred during execution because the default rejection policy AbortPolicy is adopted

Therefore, you can clearly know that after the task is submitted to the thread pool, the thread pool resources are full, resulting in the task being rejected.

3, Troubleshooting

3.1. Check thread pool configuration

There are at most 15 tasks in a group, 14 core threads, 1 blocking queue and 30 maximum threads

In theory, 14 core threads + 1 blocking queue can complete a group of tasks without even using non core threads. Why are threads full?

3.2. Check business code

Check whether the thread pool is used in multiple places or multiple batches of tasks are executed at the same time, and no errors are found;

3.3. Offline reproduction

• Configure thread pool executebeanconfig xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

<!--Custom configuration bean-->
    <bean id="executor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
        <property name="corePoolSize" value="14"/>
        <property name="maxPoolSize" value="30"/>
        <property name="queueCapacity" value="1"/>
    </bean>
</beans>

• Configure startup class

@SpringBootApplication
@ImportResource(locations = {"classpath:executeBeanConfig.xml"}) // Scan xml
public class DemoApplication {
    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }
}

• Create Tests code

@RunWith(SpringRunner.class)
@SpringBootTest
public class Tests {

    @Resource
    private ThreadPoolTaskExecutor executor;

    @Test
    public void contextLoads() throws Exception {
        // A total of 10 batches of tasks
        for(int i = 0; i < 10; i++) {
            // Perform a batch of tasks at a time
            doOnceTasks();
            System.out.println("---------------------------------------" + i);
        }
    }

    /**
     * After completing 15 tasks at a time, proceed to the next task
     */
    private void doOnceTasks(){
        List<Future> futureList = Lists.newArrayListWithCapacity(15);
        for(int i = 0; i < 15; ++i){
            Future future = executor.submit(()->{
                // Sleep randomly for 0-5 seconds
                int sec = new Double(Math.random() * 5).intValue();
                LockSupport.parkNanos(sec * 1000 * 1000 * 1000);
                System.out.println(Thread.currentThread().getName() + "  end");
            });
            futureList.add(future);
        }

        // Wait until all tasks are completed
        for(Future future : futureList){
            try {
                future.get();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }
}

Use the executor The submit () method submits a task, mainly to obtain the task execution result by using Future
:: tip supplementary knowledge points

• The execute() method is actually a method declared in the Executor. It is specifically implemented in ThreadPoolExecutor. This method is the core method of ThreadPoolExecutor. Through this method, you can submit a task to the thread pool for execution.

• submit() method is a method declared in ExecutorService. AbstractExecutorService already has a specific implementation. It is not rewritten in ThreadPoolExecutor. This method is also used to submit tasks to the thread pool, but it is different from execute() method. It can return the results of task execution. See the implementation of submit() method, You will find that it is actually the execute() method called, but it uses Future to obtain the task execution results.
  :::

• Abnormal recurrence

4, Thread pool source code reading

4.1. Thread pool execution task flow

• When the number of working threads < corepoolsize, a new thread is created to perform a new task submission for the core thread, even if there are idle threads in the thread pool at this time;
• When the number of working threads = = corePoolSize, the newly submitted task will be put into the workQueue;
• When the workQueue is full and the number of working threads is < maximumpoolsize, the newly submitted task will create a new non core thread to execute the task;
• When the workQueue is full and the number of working threads = = maximumPoolSize, the newly submitted task is processed by RejectedExecutionHandler;

Some parameters:

• corePoolSize: the size of the core pool. This parameter is closely related to the implementation principle of the thread pool described later. After creating the thread pool
  • By default, there are no threads in the thread pool. - > wait for a task to arrive before creating a thread to execute the task
  • prestartAllCoreThreads() or prestartCoreThread() method is called - > pre create thread

That is, create a corePoolSize thread or a thread before the task arrives.
By default, after the thread pool is created, the number of threads in the thread pool is 0. When a task comes, a thread will be created to execute the task. When the number of threads in the thread pool reaches the corePoolSize, the arriving task will be placed in the cache queue;

• maximumPoolSize: the maximum number of threads in the thread pool. This parameter is also a very important parameter. It indicates the maximum number of threads that can be created in the thread pool;
• keepAliveTime: indicates how long the thread will terminate if it has no task to execute.
  • By default, keepAliveTime works only when the number of threads in the thread pool is greater than corePoolSize
  • Until the number of threads in the thread pool is no more than corePoolSize, if a thread is idle for keepAliveTime, it will terminate
  • Until the number of threads in the thread pool does not exceed corePoolSize. However, if the allowCoreThreadTimeOut(boolean) method is called and the number of threads in the thread pool is not greater than corePoolSize, the keepAliveTime parameter will also work until the number of threads in the thread pool is 0;
• Unit: the time unit of the parameter keepAliveTime. There are seven values
• workQueue: a blocking queue used to store tasks waiting to be executed. The selection of this parameter is also very important and will have a significant impact on the running process of the thread pool. Generally speaking, the blocking queue here has the following options:
  • ArrayBlockingQueue;
  • LinkedBlockingQueue;
  • PriorityBlockingQueue;
  • SynchronousQueue;

ArrayBlockingQueue and PriorityBlockingQueue are rarely used, and LinkedBlockingQueue and Synchronous are generally used. The queuing policy of thread pool is related to BlockingQueue. [selection of blocking queue is explained below]

• threadFactory: thread factory, mainly used to create threads;
• handler: indicates the policy when processing a task is rejected. There are four values:
  • ThreadPoolExecutor.AbortPolicy: discards the task and throws a RejectedExecutionException exception
  • ThreadPoolExecutor.DiscardPolicy: also discards tasks without throwing exceptions.
  • ThreadPoolExecutor.DiscardOldestPolicy: discard the task at the top of the queue and try to execute the task again (repeat this process)
  • ThreadPoolExecutor.CallerRunsPolicy: this task is handled by the calling thread

4.2. execute thread pool submission task source code

class ThreadPoolExecutor{
    public void execute(Runnable command) {
        // Submit task cannot be null
        if (command == null)
            throw new NullPointerException();

        // Gets the value of the control bit ctl
        int c = ctl.get();
        // Number of work threads < number of core threads
        if (workerCountOf(c) < corePoolSize) {
            // Directly create the core thread and execute the task
            if (addWorker(command, true))
                return;

            /*
                Because locks are not used, concurrent creation of core threads may occur;
                When you get to this point, it means that the core thread is full. At this time, re obtain the value of the control bit ctl
              */
            c = ctl.get();
        }

        // If the thread pool is still RUNNING and the task is successfully submitted to the blocking queue
        if (isRunning(c) && workQueue.offer(command)) {
            int recheck = ctl.get();
            // Double check to check the thread pool status again
            // If the thread pool becomes non RUNNING, the newly added task will be rolled back
            if (! isRunning(recheck) && remove(command))
                // The task is successfully removed from the blocking queue, and the task is executed using the reject policy
                reject(command);

            // If the number of worker threads = = 0, add a thread
            // It is mainly the case that the number of compatible core threads = = 0
            else if (workerCountOf(recheck) == 0)
                addWorker(null, false);
        }

        /*
            If you get here, the number of core threads is full and the blocking queue is full
            An attempt was made to create a non core thread to perform a task
        */
        else if (!addWorker(command, false))
            // The creation of non core threads fails, which means that the number of threads reaches maximumPoolSize. At this time, the reject policy is executed
            reject(command);
    }
}

4.3. addWorker add worker thread

class ThreadPoolExecutor{
    /**
     * Add a worker thread
     * @param firstTask The first task to execute
     * @param core Is it a core thread
     * @return Creation success or failure
     */
    private boolean addWorker(Runnable firstTask, boolean core) {
        // A retry tag is defined
        retry:
        for (;;) {
            // Get control bit
            int c = ctl.get();
            // Get running status
            int rs = runStateOf(c);

            /**
             * rs >= SHUTDOWN: That is, it is not in the RUNNING state, only RUNNING < shutdown
             * ! (rs == SHUTDOWN && firstTask == null && ! workQueue.isEmpty())
             *      Equivalent to non SHUTDOWN state | firsttask= null || workQueue. isEmpty()
             *      Non SHUTDOWN state = = true: no worker thread can be added after SHUTDOWN state, and false is returned directly;
             *      Non SHUTDOWN state = = false | (firsttask! = null) = = true: in SHUTDOWN state, no more tasks are allowed to be added, and false is returned;
             *      Non SHUTDOWN state = = false | (firsttask! = null) = = false | workqueue Isempty() = = true: SHUTDOWN status, no new task is submitted, and the blocking queue is empty. There is no need to add another thread
             */
            if (rs >= SHUTDOWN &&
                ! (rs == SHUTDOWN &&
                   firstTask == null &&
                   ! workQueue.isEmpty()))
                return false;

            // CAS create worker thread
            for (;;) {
                // Get number of threads
                int wc = workerCountOf(c);
                /*
                The current number of threads is greater than the maximum
                    or
                Currently, core threads are created, but the number of threads has been > = the number of core threads
                    or
                Non core threads are currently created, but the number of threads has been > = maximumpoolsize
                */
                if (wc >= CAPACITY ||
                    wc >= (core ? corePoolSize : maximumPoolSize))
                    // false is returned directly without creating
                    return false;

                // cas modifies the number of threads in ctl. The number of threads is + 1
                if (compareAndIncrementWorkerCount(c))
                    // cas is modified successfully, and break goto ends the loop (it will not enter the loop under the tag again)
                    break retry;

                // This indicates that cas failed to increase the number of threads by 1. Try again at this time
                c = ctl.get();
                // First, judge whether the state of the process pool has changed. If it has changed, continue goto (will enter the cycle under the label again)
                // Jump to the outermost loop and re detect the status value of the thread pool
                if (runStateOf(c) != rs)
                    continue retry;
            }
        }

        boolean workerStarted = false;
        boolean workerAdded = false;
        Worker w = null;
        try {
            // Create worker object
            w = new Worker(firstTask);
            // Get worker's thread
            final Thread t = w.thread;
            if (t != null) {
                // Lock
                final ReentrantLock mainLock = this.mainLock;
                mainLock.lock();
                try {
                    // Get thread pool status
                    int rs = runStateOf(ctl.get());

                    /*
                     The thread pool is RUNNING
                        or
                     SHUTDOWN State and firstTask == null (in this case, you need to create a thread to consume the remaining tasks in the queue)
                      */
                    if (rs < SHUTDOWN ||
                        (rs == SHUTDOWN && firstTask == null)) {
                        // If the thread is in the active state, it is illegal because the thread is just created and should be in the NEW state
                        if (t.isAlive())
                            throw new IllegalThreadStateException();

                        // Add worker to the list
                        workers.add(w);
                        // largestPoolSize records the maximum number of threads reached during the use of the thread pool
                        int s = workers.size();
                        if (s > largestPoolSize)
                            largestPoolSize = s;
                        // worker is added successfully, and workerAdded is set to true
                        workerAdded = true;
                    }
                } finally {
                    mainLock.unlock();
                }

                // After the worker is added successfully, the thread can be started
                if (workerAdded) {
                    t.start();
                    // The thread is started successfully, and workerStarted is set to true
                    workerStarted = true;
                }
            }
        } finally {
            // If the worker fails to start, remove it
            if (! workerStarted)
                // workers remove the newly added worker and set the number of work threads in ctl to - 1
                addWorkerFailed(w);
        }
        return workerStarted;
    }
}

5, Problem location

5.1. Locate the execute reject policy entry

There are only two places where the rejection policy can be executed. Place a breakpoint in these two places and execute the demo. It is found that the rejection policy is executed in the second place;

5.2. Locate the reason for executing the reject policy

After entering the addWorker method, false is returned only in these two places. The thread creation fails, the interrupt point is interrupted, and the demo is executed. It is found that false is returned in the second place;

6, Problem confirmation

It is true that the created worker thread has reached the maximum number of threads and can no longer be created, and then execute the reject policy

Why is it created to the maximum? There are only 15 tasks in each group. Why do you use non core threads?

7, Positioning reason

7.1. Analyze execute method

Before adding a non core thread, try to put the task into the blocking queue. If the blocking queue is full, try to add a non core thread, that is, when creating a non core thread: workqueue Offer (command) = = false, that is, the blocking queue is full;

7.2. Guess the reason

Because our blocking queue is only 1, will the speed of submitting tasks be faster than that of fetching tasks from the blocking queue, resulting in the creation of non core threads to execute tasks? The final result is that after multiple batches of tasks, no non core threads can be created, resulting in the execution of the rejection policy.

7.3. Cause verification

Blocking queue selection
Check the source code of Spring's ThreadPoolTaskExecutor and find

If the number of blocked queues is > 0, LinkedBlockingQueue is used; otherwise, synchronous queue is used.

LinkedBlockingQueue

• Check the LinkedBlockingQueue#take method. If the queue is empty, all threads fetching elements will be blocked on a Lock notEmpty waiting condition. When elements are queued, only the signal method will be called to wake up one thread fetching elements, not all threads.

class LinkedBlockingQueue{
    private void signalNotEmpty() {
        final ReentrantLock takeLock = this.takeLock;
        // Lock
        takeLock.lock();
        try {
            // Wake up a take thread
            notEmpty.signal();
        } finally {
            takeLock.unlock();
        }
    }
}

• Because a thread has a time interval from wake-up to execution, after the blocking is awakened, it has to wait to obtain the cpu time slice, and the main thread has been publishing the task. At this time, it will cause the elements in the queue to be consumed too late and can only be consumed by non core threads.

8, Solution

8.1. Using SynchronousQueue

Use synchronous queue, that is, the blocking queue size is set to 0
The reason is that the dimensions of SynchronousQueue and LinkedBlockingQueue are inconsistent
The SynchronousQueue determines whether to join the queue successfully according to whether there is a waiting thread
The LinkedBlockingQueue is based on the buffer, regardless of whether there is already a waiting thread.
​

• SynchronousQueue

• LinkedBlockingQueue

8.2. Configure the blocking queue according to the business situation

For our business situation, since there are only 15 tasks at most, set the blocking queue size to 15, so as to ensure that no task will be rejected.