During the project two days ago, I wanted to improve the performance optimization of inserting tables, because there are two tables. Insert the old table first, and then insert the new table. More than 10000 data are a little slow

The thread pool ThreadPoolExecutor comes to mind later, and the Spring Boot project is used. The thread pool ThreadPoolTaskExecutor encapsulated by ThreadPoolExecutor provided by Spring can be directly enabled with annotations

Use steps

First create a thread pool Configuration and let Spring Boot load it to define how to create a ThreadPoolTaskExecutor. Use @ Configuration and @ EnableAsync annotations to indicate that this is a Configuration class and a thread pool Configuration class

@Configuration
@EnableAsync
public class ExecutorConfig {
   
     

    private static final Logger logger = LoggerFactory.getLogger(ExecutorConfig.class);

    @Value("${async.executor.thread.core_pool_size}")
    private int corePoolSize;
    @Value("${async.executor.thread.max_pool_size}")
    private int maxPoolSize;
    @Value("${async.executor.thread.queue_capacity}")
    private int queueCapacity;
    @Value("${async.executor.thread.name.prefix}")
    private String namePrefix;

    @Bean(name = "asyncServiceExecutor")
    public Executor asyncServiceExecutor() {
        logger.info("start asyncServiceExecutor");
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        //Configure the number of core threads
        executor.setCorePoolSize(corePoolSize);
        //Configure maximum threads
        executor.setMaxPoolSize(maxPoolSize);
        //Configure queue size
        executor.setQueueCapacity(queueCapacity);
        //Configure the name prefix of threads in the thread pool
        executor.setThreadNamePrefix(namePrefix);

        // Rejection policy: how to handle new tasks when the pool has reached max size
        // CALLER_RUNS: the task is not executed in the new thread, but in the thread of the caller
        executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
        //Perform initialization
        executor.initialize();
        return executor;
    }
}

@Value is configured in application.properties. You can refer to the configuration and define it freely  

# Asynchronous thread configuration
# Configure the number of core threads
async.executor.thread.core_pool_size = 5
# Configure maximum threads
async.executor.thread.max_pool_size = 5
# Configure queue size
async.executor.thread.queue_capacity = 99999
# Configure the name prefix of threads in the thread pool
async.executor.thread.name.prefix = async-service-

Create a Service interface, which is the interface of asynchronous threads  

public interface AsyncService {
   
     
    /**
     * Perform asynchronous tasks
     * You can add parameters according to your needs. I'll make a test demonstration here
     */
    void executeAsync();
}

  Implementation class

@Service
public class AsyncServiceImpl implements AsyncService {
   
     
    private static final Logger logger = LoggerFactory.getLogger(AsyncServiceImpl.class);

    @Override
    @Async("asyncServiceExecutor")
    public void executeAsync() {
        logger.info("start executeAsync");

        System.out.println("What asynchronous threads do");
        System.out.println("You can perform time-consuming things such as batch insertion here");

        logger.info("end executeAsync");
    }
}

 

Asynchronize the services of the Service layer, and add the annotation @ Async("asyncServiceExecutor") on the executeAsync() method. The asyncServiceExecutor method is the method name in the previous ExecutorConfig.java, indicating that the thread pool entered by the executeAsync method is created by the asyncServiceExecutor method

The next step is to inject the Service in the Controller or where through the annotation @ Autowired

@Autowired
private AsyncService asyncService;

@GetMapping("/async")
public void async(){
    asyncService.executeAsync();
}

 

  Use postmain or other tools to test the request multiple times

 2021-04-16 22:15:47.655  INFO 10516 --- [async-service-5] c.u.d.e.executor.impl.AsyncServiceImpl   : start executeAsync
 What asynchronous threads do
 You can perform time-consuming things such as batch insertion here
2021-04-16 22:15:47.655  INFO 10516 --- [async-service-5] c.u.d.e.executor.impl.AsyncServiceImpl   : end executeAsync
2021-04-16 22:15:47.770  INFO 10516 --- [async-service-1] c.u.d.e.executor.impl.AsyncServiceImpl   : start executeAsync
 What asynchronous threads do
 You can perform time-consuming things such as batch insertion here
2021-04-16 22:15:47.770  INFO 10516 --- [async-service-1] c.u.d.e.executor.impl.AsyncServiceImpl   : end executeAsync
2021-04-16 22:15:47.816  INFO 10516 --- [async-service-2] c.u.d.e.executor.impl.AsyncServiceImpl   : start executeAsync
 What asynchronous threads do
 You can perform time-consuming things such as batch insertion here
2021-04-16 22:15:47.816  INFO 10516 --- [async-service-2] c.u.d.e.executor.impl.AsyncServiceImpl   : end executeAsync
2021-04-16 22:15:48.833  INFO 10516 --- [async-service-3] c.u.d.e.executor.impl.AsyncServiceImpl   : start executeAsync
 What asynchronous threads do
 You can perform time-consuming things such as batch insertion here
2021-04-16 22:15:48.834  INFO 10516 --- [async-service-3] c.u.d.e.executor.impl.AsyncServiceImpl   : end executeAsync
2021-04-16 22:15:48.986  INFO 10516 --- [async-service-4] c.u.d.e.executor.impl.AsyncServiceImpl   : start executeAsync
 What asynchronous threads do
 You can perform time-consuming things such as batch insertion here
2021-04-16 22:15:48.987  INFO 10516 --- [async-service-4] c.u.d.e.executor.impl.AsyncServiceImpl   : end executeAsync

 

It can be found from the above logs that [async Service -] has multiple threads, which have obviously been executed in the thread pool configured by us, and the start and end logs of the controller are printed continuously in each request, indicating that each request is responded quickly, and the time-consuming operations are left to the threads in the thread pool to execute asynchronously;

Although we have used the thread pool, it is not clear how many threads are executing and how many are waiting in the queue? Here, I created a subclass of ThreadPoolTaskExecutor, which will print out the current thread pool health every time a thread is submitted

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
import org.springframework.util.concurrent.ListenableFuture;

import java.util.concurrent.Callable;
import java.util.concurrent.Future;
import java.util.concurrent.ThreadPoolExecutor;

/**
 * @Author: ChenBin
 */
public class VisiableThreadPoolTaskExecutor extends ThreadPoolTaskExecutor {
   
     

    private static final Logger logger = LoggerFactory.getLogger(VisiableThreadPoolTaskExecutor.class);

    private void showThreadPoolInfo(String prefix) {
        ThreadPoolExecutor threadPoolExecutor = getThreadPoolExecutor();

        if (null == threadPoolExecutor) {
            return;
        }

        logger.info("{}, {},taskCount [{}], completedTaskCount [{}], activeCount [{}], queueSize [{}]",
                this.getThreadNamePrefix(),
                prefix,
                threadPoolExecutor.getTaskCount(),
                threadPoolExecutor.getCompletedTaskCount(),
                threadPoolExecutor.getActiveCount(),
                threadPoolExecutor.getQueue().size());
    }

    @Override
    public void execute(Runnable task) {
        showThreadPoolInfo("1. do execute");
        super.execute(task);
    }

    @Override
    public void execute(Runnable task, long startTimeout) {
        showThreadPoolInfo("2. do execute");
        super.execute(task, startTimeout);
    }

    @Override
    public Future<?> submit(Runnable task) {
        showThreadPoolInfo("1. do submit");
        return super.submit(task);
    }

    @Override
    public <T> Future<T> submit(Callable<T> task) {
        showThreadPoolInfo("2. do submit");
        return super.submit(task);
    }

    @Override
    public ListenableFuture<?> submitListenable(Runnable task) {
        showThreadPoolInfo("1. do submitListenable");
        return super.submitListenable(task);
    }

    @Override
    public <T> ListenableFuture<T> submitListenable(Callable<T> task) {
        showThreadPoolInfo("2. do submitListenable");
        return super.submitListenable(task);
    }
}

  As shown above, the showThreadPoolInfo method prints out the total number of tasks, the number of completed threads, the number of active threads, and the queue size, and then overrides the execute, submit, and other methods of the parent class, where the showThreadPoolInfo method is called, so that each time a task is submitted to the thread pool, the basic information of the current thread pool will be printed to the log;

Modify the asyncServiceExecutor method of ExecutorConfig.java and change threadpooltaskexecutor = new threadpooltaskexecutor() to threadpooltaskexecutor = new visiblethreadpooltaskexecutor()  

@Bean(name = "asyncServiceExecutor")
    public Executor asyncServiceExecutor() {
        logger.info("start asyncServiceExecutor");
        //Modify here
        ThreadPoolTaskExecutor executor = new VisiableThreadPoolTaskExecutor();
        //Configure the number of core threads
        executor.setCorePoolSize(corePoolSize);
        //Configure maximum threads
        executor.setMaxPoolSize(maxPoolSize);
        //Configure queue size
        executor.setQueueCapacity(queueCapacity);
        //Configure the name prefix of threads in the thread pool
        executor.setThreadNamePrefix(namePrefix);

        // Rejection policy: how to handle new tasks when the pool has reached max size
        // CALLER_RUNS: the task is not executed in the new thread, but in the thread of the caller
        executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
        //Perform initialization
        executor.initialize();
        return executor;
    }

  Start the project test again

 2021-04-16 22:23:30.951  INFO 14088 --- [nio-8087-exec-2] u.d.e.e.i.VisiableThreadPoolTaskExecutor : async-service-, 2. do submit,taskCount [0], completedTaskCount [0], activeCount [0], queueSize [0]
2021-04-16 22:23:30.952  INFO 14088 --- [async-service-1] c.u.d.e.executor.impl.AsyncServiceImpl   : start executeAsync
What asynchronous threads do
You can perform time-consuming things such as batch insertion here
2021-04-16 22:23:30.953  INFO 14088 --- [async-service-1] c.u.d.e.executor.impl.AsyncServiceImpl   : end executeAsync
2021-04-16 22:23:31.351  INFO 14088 --- [nio-8087-exec-3] u.d.e.e.i.VisiableThreadPoolTaskExecutor : async-service-, 2. do submit,taskCount [1], completedTaskCount [1], activeCount [0], queueSize [0]
2021-04-16 22:23:31.353  INFO 14088 --- [async-service-2] c.u.d.e.executor.impl.AsyncServiceImpl   : start executeAsync
What asynchronous threads do
You can perform time-consuming things such as batch insertion here
2021-04-16 22:23:31.353  INFO 14088 --- [async-service-2] c.u.d.e.executor.impl.AsyncServiceImpl   : end executeAsync
2021-04-16 22:23:31.927  INFO 14088 --- [nio-8087-exec-5] u.d.e.e.i.VisiableThreadPoolTaskExecutor : async-service-, 2. do submit,taskCount [2], completedTaskCount [2], activeCount [0], queueSize [0]
2021-04-16 22:23:31.929  INFO 14088 --- [async-service-3] c.u.d.e.executor.impl.AsyncServiceImpl   : start executeAsync
What asynchronous threads do
You can perform time-consuming things such as batch insertion here
2021-04-16 22:23:31.930  INFO 14088 --- [async-service-3] c.u.d.e.executor.impl.AsyncServiceImpl   : end executeAsync
2021-04-16 22:23:32.496  INFO 14088 --- [nio-8087-exec-7] u.d.e.e.i.VisiableThreadPoolTaskExecutor : async-service-, 2. do submit,taskCount [3], completedTaskCount [3], activeCount [0], queueSize [0]
2021-04-16 22:23:32.498  INFO 14088 --- [async-service-4] c.u.d.e.executor.impl.AsyncServiceImpl   : start executeAsync
What asynchronous threads do
You can perform time-consuming things such as batch insertion here
2021-04-16 22:23:32.499  INFO 14088 --- [async-service-4] c.u.d.e.executor.impl.AsyncServiceImpl   : end executeAsync

  Note this line of log:

2021-04-16 22:23:32.496  INFO 14088 --- [nio-8087-exec-7] u.d.e.e.i.VisiableThreadPoolTaskExecutor : async-service-, 2. do submit,taskCount [3], completedTaskCount [3], activeCount [0], queueSize [0]

  This indicates that when submitting a task to the thread pool, the method submit(Callable task) is called. Currently, three tasks have been submitted and completed. At present, there are 0 threads processing tasks, and there are 0 tasks waiting in the queue. The basic situation of the thread pool has been all the way;