Mainly based on the recent multi-threaded project of Xiaomi, extract the multi-threaded instances inside.
Yesterday, I was shocked to hear the news of the death of old yuan. I was shocked by the fall of a generation of superstars. I hope old yuan can be well in heaven. I posted a picture of old yuan. This picture also reminds me of my grandfather who is already in heaven.
preface
I have been learning Java multithreading for more than half a month. When I started learning Java multithreading, I set myself a small goal. I hope to write a multithreaded Demo. Today, I mainly fulfill this small goal.
This example of multithreading is actually combined with a recent project of Xiaomi's multithreaded asynchronous task. I extracted the code involving multithreading and then made some modifications. The reason why I didn't rewrite one myself is that my ability is not enough on the one hand, and I want to learn the implementation posture of multithreading in the current project on the other hand, At least this example is applied in a real project. First learn how others make wheels, and then you will know how to make wheels yourself.
Business requirements
We do this multithreaded asynchronous task mainly because we have many asynchronous tasks that are always moving. What is always moving? After the task runs, it needs to run all the time. For example, the message Push task needs to consume the non pushed messages in the DB because there are messages all the time, so it needs a whole Push permanent asynchronous task.
In fact, our needs are not difficult. To sum up briefly:
-
It can execute multiple permanent asynchronous tasks at the same time;
-
Each asynchronous task supports opening multiple threads to consume the data of this task;
-
Supports graceful shutdown of perpetual asynchronous tasks, that is, after shutdown, all data needs to be consumed before shutdown.
To complete the above requirements, you need to pay attention to several points:
-
For each perpetual task, you can open a thread to execute;
-
Each subtask needs to be controlled by thread pool because it needs to support concurrency;
-
To close a persistent task, you need to notify the concurrent threads of subtasks, and support the graceful closing of persistent tasks and concurrent subtasks.
Project example
Thread pool
For subtasks, concurrency needs to be supported. If one thread is opened for each concurrency, it will be closed when it is used up, which will consume too much resources, so the thread pool is introduced:
public class TaskProcessUtil {
//Each task has its own thread pool
private static Map<String, ExecutorService> executors = new ConcurrentHashMap<>();
//Initialize a thread pool
private static ExecutorService init(String poolName, int poolSize) {
return new ThreadPoolExecutor(poolSize, poolSize,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>(),
new ThreadFactoryBuilder().setNameFormat("Pool-" + poolName).setDaemon(false).build(),
new ThreadPoolExecutor.CallerRunsPolicy());
}
//Get thread pool
public static ExecutorService getOrInitExecutors(String poolName,int poolSize) {
ExecutorService executorService = executors.get(poolName);
if (null == executorService) {
synchronized (TaskProcessUtil.class) {
executorService = executors.get(poolName);
if (null == executorService) {
executorService = init(poolName, poolSize);
executors.put(poolName, executorService);
}
}
}
return executorService;
}
//Reclaim thread resources
public static void releaseExecutors(String poolName) {
ExecutorService executorService = executors.remove(poolName);
if (executorService != null) {
executorService.shutdown();
}
}
}
This is a tool class of thread pool. It is very simple to initialize thread pool and recycle thread resources here. We mainly discuss obtaining thread pool. There may be concurrency when obtaining the thread pool, so you need to add a synchronized lock. After locking, you need to perform a secondary null determination check on the executorService. This is very similar to the implementation of Java singleton. For details, please refer to the article [design pattern series 5] singleton mode.
Single task
In order to better explain the implementation of a single task, our main task is to print cat data. Cat is defined as follows:
@Data
@Service
public class Cat {
private String catName;
public Cat setCatName(String name) {
this.catName = name;
return this;
}
}
A single task mainly includes the following functions:
-
Get persistent task data: here is generally scanning DB. I simply use queryData() instead.
-
Multithreaded task execution: you need to split the data into 4 copies, and then execute them concurrently by multithreads. Here, it can be supported by thread pool;
-
Graceful shutdown of perpetual task: when the task needs to be shut down, it needs to complete the remaining task data, recover thread resources and exit the task;
-
Permanent execution: if the shutdown command is not received, the task needs to be executed continuously.
Look directly at the code:
public class ChildTask {
private final int POOL_SIZE = 3; //Thread pool size
private final int SPLIT_SIZE = 4; //Data split size
private String taskName;
//Receive jvm shutdown signal to realize graceful shutdown
protected volatile boolean terminal = false;
public ChildTask(String taskName) {
this.taskName = taskName;
}
//Program execution entry
public void doExecute() {
int i = 0;
while(true) {
System.out.println(taskName + ":Cycle-" + i + "-Begin");
//Get data
List<Cat> datas = queryData();
//Processing data
taskExecute(datas);
System.out.println(taskName + ":Cycle-" + i + "-End");
if (terminal) {
//Only when the application is closed will it come here to realize elegant offline
break;
}
i++;
}
//Reclaim thread pool resources
TaskProcessUtil.releaseExecutors(taskName);
}
//Graceful shutdown
public void terminal() {
//Shut down
terminal = true;
System.out.println(taskName + " shut down");
}
//Processing data
private void doProcessData(List<Cat> datas, CountDownLatch latch) {
try {
for (Cat cat : datas) {
System.out.println(taskName + ":" + cat.toString() + ",ThreadName:" + Thread.currentThread().getName());
Thread.sleep(1000L);
}
} catch (Exception e) {
System.out.println(e.getStackTrace());
} finally {
if (latch != null) {
latch.countDown();
}
}
}
//Processing single task data
private void taskExecute(List<Cat> sourceDatas) {
if (CollectionUtils.isEmpty(sourceDatas)) {
return;
}
//Split the data into 4 copies
List<List<Cat>> splitDatas = Lists.partition(sourceDatas, SPLIT_SIZE);
final CountDownLatch latch = new CountDownLatch(splitDatas.size());
//Concurrent processing of split data, sharing a thread pool
for (final List<Cat> datas : splitDatas) {
ExecutorService executorService = TaskProcessUtil.getOrInitExecutors(taskName, POOL_SIZE);
executorService.submit(new Runnable() {
@Override
public void run() {
doProcessData(datas, latch);
}
});
}
try {
latch.await();
} catch (Exception e) {
System.out.println(e.getStackTrace());
}
}
//Obtain Yongdong task data
private List<Cat> queryData() {
List<Cat> datas = new ArrayList<>();
for (int i = 0; i < 5; i ++) {
datas.add(new Cat().setCatName("Luo Xiaohei" + i));
}
return datas;
}
}
Briefly explain:
-
queryData: used to obtain data. In practical applications, queryData needs to be defined as an abstract method, and then each task implements its own method.
-
doProcessData: data processing logic. In practical application, doProcessData needs to be defined as an abstract method, and then each task implements its own method.
-
taskExecute: split the data into 4 copies, obtain the thread pool of the task, hand it to the thread pool for concurrent execution, and then click latch Await() is blocked. When all four data are executed successfully, the blocking ends and the method returns.
-
terminal: it is only used to accept the shutdown command. Here, the variable is defined as volatile, so the multi-threaded memory is visible. See [Java Concurrent Programming Series 2] volatile for details;
-
doExecute: the program execution entry encapsulates the execution process of each task. When terminal=true, first execute the task data, then recycle the thread pool, and finally exit.
Task entry
Direct code:
public class LoopTask {
private List<ChildTask> childTasks;
public void initLoopTask() {
childTasks = new ArrayList();
childTasks.add(new ChildTask("childTask1"));
childTasks.add(new ChildTask("childTask2"));
for (final ChildTask childTask : childTasks) {
new Thread(new Runnable() {
@Override
public void run() {
childTask.doExecute();
}
}).start();
}
}
public void shutdownLoopTask() {
if (!CollectionUtils.isEmpty(childTasks)) {
for (ChildTask childTask : childTasks) {
childTask.terminal();
}
}
}
public static void main(String args[]) throws Exception{
LoopTask loopTask = new LoopTask();
loopTask.initLoopTask();
Thread.sleep(5000L);
loopTask.shutdownLoopTask();
}
}
Each task opens a separate Thread. Here I initialize two persistent tasks, childTask1 and childTask2, and then execute them respectively. After 5 seconds of Sleep, close the task. We can see whether we can exit gracefully according to our expectations.
Result analysis
The results are as follows:
childTask1:Cycle-0-Begin childTask2:Cycle-0-Begin childTask1:Cat(catName=Luo Xiaohei 0),ThreadName:Pool-childTask1 childTask1:Cat(catName=Luo Xiaohei 4),ThreadName:Pool-childTask1 childTask2:Cat(catName=Luo Xiaohei 4),ThreadName:Pool-childTask2 childTask2:Cat(catName=Luo Xiaohei 0),ThreadName:Pool-childTask2 childTask1:Cat(catName=Luo Xiaohei 1),ThreadName:Pool-childTask1 childTask2:Cat(catName=Luo Xiaohei 1),ThreadName:Pool-childTask2 childTask2:Cat(catName=Luo Xiaohei 2),ThreadName:Pool-childTask2 childTask1:Cat(catName=Luo Xiaohei 2),ThreadName:Pool-childTask1 childTask2:Cat(catName=Luo Xiaohei 3),ThreadName:Pool-childTask2 childTask1:Cat(catName=Luo Xiaohei 3),ThreadName:Pool-childTask1 childTask2:Cycle-0-End childTask2:Cycle-1-Begin childTask1:Cycle-0-End childTask1:Cycle-1-Begin childTask2:Cat(catName=Luo Xiaohei 0),ThreadName:Pool-childTask2 childTask2:Cat(catName=Luo Xiaohei 4),ThreadName:Pool-childTask2 childTask1:Cat(catName=Luo Xiaohei 4),ThreadName:Pool-childTask1 childTask1:Cat(catName=Luo Xiaohei 0),ThreadName:Pool-childTask1 childTask1 shut down childTask2 shut down childTask2:Cat(catName=Luo Xiaohei 1),ThreadName:Pool-childTask2 childTask1:Cat(catName=Luo Xiaohei 1),ThreadName:Pool-childTask1 childTask1:Cat(catName=Luo Xiaohei 2),ThreadName:Pool-childTask1 childTask2:Cat(catName=Luo Xiaohei 2),ThreadName:Pool-childTask2 childTask1:Cat(catName=Luo Xiaohei 3),ThreadName:Pool-childTask1 childTask2:Cat(catName=Luo Xiaohei 3),ThreadName:Pool-childTask2 childTask1:Cycle-1-End childTask2:Cycle-1-End
In the output data, "pool childTask" is the thread pool name, "childTask" is the task name, "Cat(catName = Luo Xiaohei)" is the execution result, "childTask shutdown" is the closing flag, "childTask:Cycle-X-Begin" and "childTask:Cycle-X-End" are the start and end flags of each cycle.
Let's analyze the execution results: childTask1 and childTask2 were executed respectively. In the first round of the cycle, five Luo Xiaohei data were normally output. In the second round of execution, I started the close instruction. This time, the second round of execution did not stop directly, but executed the data in the task first and then exited, so it is completely in line with our elegant exit conclusion.
epilogue
In fact, this is a classic example of using thread pool, which was written by a colleague of our company. I feel that the whole process has no problems and the implementation is very elegant, which is worth learning.
Then, in the process of learning java multithreading, I feel that my current mastery speed is relatively fast. From the JAVA memory model, to the basic knowledge and common tools of Java multithreading, to the final multithreading practice, a total of 8 articles can really let you from Java white to write more robust multithreaded programs.
In fact, before learning language or technology, I prefer to read some eight part essays. In fact, eight part essays need to be read. More importantly, I practice myself and need to write more. Therefore, many of the previous articles are pure theory. Now, they are more a combination of theory and practice. Even if I see some examples on the Internet, I will Copy them down and let the program run again.
For the Java multithreading part, I plan to write another 1-2 articles later. This series will be suspended first, because my goal is to learn all the relevant technologies of Java ecology, so I will eat it as soon as possible, and then focus on learning more in-depth knowledge after learning all.
More articles, please pay attention to WeChat official account "Lou Zi's advanced road".