1. Why to write this article

When NoSQL became popular a few years ago, like other teams, our team was keen on exciting new things and planned to replace an application's database. However, when we go deep into details, we think of what a wise man once said: "details determine success or failure.". In the end, we realized that NoSQL is not a silver bullet to solve all problems, and the answer of NoSQL vs RDMS is "depending on the situation".

Similarly, last year, concurrent libraries like RxJava and Spring Reactor added exciting statements, such as asynchronous non blocking methods. To avoid making the same mistake again, we try to evaluate the differences between concurrent frameworks such as ExecutorService, RxJava, Disruptor and Akka, and how to determine the correct usage of each framework.

The terms used in this article are described in more detail here.

2. Analyze the example use cases of concurrency framework

3. Quickly update thread configuration

Before we start to compare concurrency frameworks, let's quickly review how to configure the optimal number of threads to improve the performance of parallel tasks. This theory applies to all frameworks, and uses the same thread configuration to measure performance in all frameworks.

• For memory tasks, the number of threads is approximately equal to the number of cores with the best performance, although it can make some changes based on the hyper threading features in its respective processors.

• For example, in an 8-core machine, if each request to an application must execute four tasks in parallel in memory, the load on this machine should be maintained at @ 2 req/sec and eight threads in the ThreadPool.

• For I/O tasks, the number of threads configured in ExecutorService should depend on the latency of the external service.

• Unlike in memory tasks, threads involved in I/O tasks are blocked and waiting until the external service responds or times out. Therefore, when the threads involved in I/O tasks are blocked, the number of threads should be increased to handle the additional load from concurrent requests.

• The number of threads for I/O tasks should be increased in a conservative way, because many threads in the active state bring the cost of context switching, which will affect the performance of the application. To avoid this situation, you should increase the exact number of threads on this machine and the load proportionally based on the waiting time of the threads involved in the I/O task.

Reference: http://baddotrobot.com/blog/2013/06/01/optimum-number-of-threads/

4. Performance test results

Performance test configuration GCP - > processor: Intel(R) Xeon(R) CPU @ 2.30GHz; architecture: x86 ʄ CPU core: 8 (Note: These results are only meaningful for this configuration, and do not mean that one framework is better than the other).

5. Use actuator service to parallelize IO tasks

5.1 when to use?

If an application is deployed on multiple nodes, and the req/sec of each node is less than the number of cores available, ExecutorService can be used to parallelize tasks and execute code faster.

5.2 when does it apply?

If an application is deployed on multiple nodes, and the req/sec of each node is much higher than the number of cores available, further parallelization with ExecutorService will only make the situation worse.

When the external service delay increases to 400ms, the performance test results are as follows (request rate @50 req/sec, 8 cores).

5.3 example of sequential execution of all tasks

// I/O tasks: invoking external services
String posts = JsonService.getPosts();
String comments = JsonService.getComments();
String albums = JsonService.getAlbums();
String photos = JsonService.getPhotos();

// Merge responses from external services
// (tasks in memory will be performed as part of this operation)
int userId = new Random().nextInt(10) + 1;
String postsAndCommentsOfRandomUser = ResponseUtil.getPostsAndCommentsOfRandomUser(userId, posts, comments);
String albumsAndPhotosOfRandomUser = ResponseUtil.getAlbumsAndPhotosOfRandomUser(userId, albums, photos);

// Build the final response and send it back to the client
String response = postsAndCommentsOfRandomUser + albumsAndPhotosOfRandomUser;
return response;

5.4 code example of parallel execution of I / O task and ExecutorService

// Add I/O tasks
List<Callable<String>> ioCallableTasks = new ArrayList<>();
ioCallableTasks.add(JsonService::getPosts);
ioCallableTasks.add(JsonService::getComments);
ioCallableTasks.add(JsonService::getAlbums);
ioCallableTasks.add(JsonService::getPhotos);

// Call all parallel tasks
ExecutorService ioExecutorService = CustomThreads.getExecutorService(ioPoolSize);
List<Future<String>> futuresOfIOTasks = ioExecutorService.invokeAll(ioCallableTasks);

// Get I/O operation (blocking call) results
String posts = futuresOfIOTasks.get(0).get();
String comments = futuresOfIOTasks.get(1).get();
String albums = futuresOfIOTasks.get(2).get();
String photos = futuresOfIOTasks.get(3).get();

// Merge responses (tasks in memory are part of this operation)
String postsAndCommentsOfRandomUser = ResponseUtil.getPostsAndCommentsOfRandomUser(userId, posts, comments);
String albumsAndPhotosOfRandomUser = ResponseUtil.getAlbumsAndPhotosOfRandomUser(userId, albums, photos);

// Build the final response and send it back to the client
return postsAndCommentsOfRandomUser + albumsAndPhotosOfRandomUser;

6. Use actuator service to parallelize IO tasks (completable future)

Similar to the above: the HTTP thread processing the incoming request is blocked, while the completefuture is used to process parallel tasks

6.1 when to use?

If there is no AsyncResponse, the performance is the same as ExecutorService. This is better if multiple API calls have to be asynchronous and linked (like promises in Node).

ExecutorService ioExecutorService = CustomThreads.getExecutorService(ioPoolSize);

// I/O tasks
CompletableFuture<String> postsFuture = CompletableFuture.supplyAsync(JsonService::getPosts, ioExecutorService);
CompletableFuture<String> commentsFuture = CompletableFuture.supplyAsync(JsonService::getComments,
    ioExecutorService);
CompletableFuture<String> albumsFuture = CompletableFuture.supplyAsync(JsonService::getAlbums,
    ioExecutorService);
CompletableFuture<String> photosFuture = CompletableFuture.supplyAsync(JsonService::getPhotos,
    ioExecutorService);
CompletableFuture.allOf(postsFuture, commentsFuture, albumsFuture, photosFuture).get();

// Get response from I/O task (blocking call)
String posts = postsFuture.get();
String comments = commentsFuture.get();
String albums = albumsFuture.get();
String photos = photosFuture.get();

// Merge response (tasks in memory will be part of this operation)
String postsAndCommentsOfRandomUser = ResponseUtil.getPostsAndCommentsOfRandomUser(userId, posts, comments);
String albumsAndPhotosOfRandomUser = ResponseUtil.getAlbumsAndPhotosOfRandomUser(userId, albums, photos);

// Build the final response and send it back to the client
return postsAndCommentsOfRandomUser + albumsAndPhotosOfRandomUser;

7. Use ExecutorService to process all tasks in parallel

Use ExecutorService to process all tasks in parallel and @ suspended AsyncResponse response to send the response in a non blocking manner.

Picture from http://tutorials.jenkov.com/java-nio/nio-vs-io.html

• The HTTP thread processes the connection of the incoming request and passes the processing to the Executor Pool. When all tasks are completed, another HTTP thread will send the response back to the client (asynchronous and non blocking).

• Reason for performance degradation:

• In synchronous communication, although the threads involved in I/O tasks are blocked, as long as the process has additional threads to bear the concurrent request load, it is still running.

• As a result, the benefits of keeping threads nonblocking are minimal, and the cost of processing requests in this pattern seems high.

• In general, using asynchronous non blocking methods for the examples discussed here can degrade application performance.

7.1 when to use?

If the use case is similar to a server-side chat application, and the thread does not need to keep the connection before the client responds, the asynchronous, non blocking method is more popular than synchronous communication. In these cases, system resources can be better utilized by asynchronous and non blocking methods, rather than just waiting.

// Submit parallel tasks for asynchronous execution
ExecutorService ioExecutorService = CustomThreads.getExecutorService(ioPoolSize);
CompletableFuture<String> postsFuture = CompletableFuture.supplyAsync(JsonService::getPosts, ioExecutorService);
CompletableFuture<String> commentsFuture = CompletableFuture.supplyAsync(JsonService::getComments,
ioExecutorService);
CompletableFuture<String> albumsFuture = CompletableFuture.supplyAsync(JsonService::getAlbums,
ioExecutorService);
CompletableFuture<String> photosFuture = CompletableFuture.supplyAsync(JsonService::getPhotos,
ioExecutorService);

// When the / posts API returns a response, it will be combined with the response from the / comments API
// As part of this operation, some tasks in memory will be performed
CompletableFuture<String> postsAndCommentsFuture = postsFuture.thenCombineAsync(commentsFuture,
(posts, comments) -> ResponseUtil.getPostsAndCommentsOfRandomUser(userId, posts, comments),
ioExecutorService);

// When the / albums API returns a response, it will be combined with the response from the / photos API
// As part of this operation, some tasks in memory will be performed
CompletableFuture<String> albumsAndPhotosFuture = albumsFuture.thenCombineAsync(photosFuture,
(albums, photos) -> ResponseUtil.getAlbumsAndPhotosOfRandomUser(userId, albums, photos),
ioExecutorService);

// Build the final response and restore the http connection and send the response back to the client
postsAndCommentsFuture.thenAcceptBothAsync(albumsAndPhotosFuture, (s1, s2) -> {
LOG.info("Building Async Response in Thread " + Thread.currentThread().getName());
String response = s1 + s2;
asyncHttpResponse.resume(response);
}, ioExecutorService);

8. RxJava

• This is similar to the above situation. The only difference is that RxJava provides a better DSL for streaming programming, which is not shown in the following example.

• The performance is better than that of completable future.

8.1 when to use?

If the coding scenario is suitable for asynchronous non blocking mode, RxJava or any responsive development library can be preferred. It also has additional functions such as back pressure, which can balance the load between producers and consumers.

int userId = new Random().nextInt(10) + 1;
ExecutorService executor = CustomThreads.getExecutorService(8);

// I/O tasks
Observable<String> postsObservable = Observable.just(userId).map(o -> JsonService.getPosts())
.subscribeOn(Schedulers.from(executor));
Observable<String> commentsObservable = Observable.just(userId).map(o -> JsonService.getComments())
.subscribeOn(Schedulers.from(executor));
Observable<String> albumsObservable = Observable.just(userId).map(o -> JsonService.getAlbums())
.subscribeOn(Schedulers.from(executor));
Observable<String> photosObservable = Observable.just(userId).map(o -> JsonService.getPhotos())
.subscribeOn(Schedulers.from(executor));

// Merge responses from / posts and / comments API s
// As part of this operation, some tasks in memory will be performed
Observable<String> postsAndCommentsObservable = Observable
.zip(postsObservable, commentsObservable,
(posts, comments) -> ResponseUtil.getPostsAndCommentsOfRandomUser(userId, posts, comments))
.subscribeOn(Schedulers.from(executor));

// Merge responses from / albums and / photos API s
// As part of this operation, some tasks in memory will be performed
Observable<String> albumsAndPhotosObservable = Observable
.zip(albumsObservable, photosObservable,
(albums, photos) -> ResponseUtil.getAlbumsAndPhotosOfRandomUser(userId, albums, photos))
.subscribeOn(Schedulers.from(executor));

// Build final response
Observable.zip(postsAndCommentsObservable, albumsAndPhotosObservable, (r1, r2) -> r1 + r2)
.subscribeOn(Schedulers.from(executor))
.subscribe((response) -> asyncResponse.resume(response), e -> asyncResponse.resume("error"));

9. Disruptor

[Queue vs RingBuffer]

Picture 1: http://tutorials.jenkov.com/java-concurrency/blocking-queues.html

Picture 2: https://www.baeldung.com/lmax-disruptor-concurrency

• In this case, the HTTP thread will be blocked until the disruptor completes the task, and use countdownload to synchronize the HTTP thread with the thread in ExecutorService.

• The main feature of this framework is to handle inter thread communication without any locks. In ExecutorService, the data between producers and consumers will be passed through the Queue, and a lock is involved in the data transmission between producers and consumers. The Disruptor framework handles this producer consumer communication through a data structure called Ring Buffer, which is an extended version of circular array queues, and does not require any locks.

• This library is not applicable to the use cases we discussed here. Add just out of curiosity.

9.1 when to use?

The Disruptor framework performs better when used with an event driven architecture, or when focused on a single producer and multiple consumers of memory tasks.

static {
    int userId = new Random().nextInt(10) + 1;

    // Example event handler; count down latch is used to synchronize threads with http threads
    EventHandler<Event> postsApiHandler = (event, sequence, endOfBatch) -> {
        event.posts = JsonService.getPosts();
        event.countDownLatch.countDown();
    };

    // Configure the dispatcher to handle events
    DISRUPTOR.handleEventsWith(postsApiHandler, commentsApiHandler, albumsApiHandler)
    .handleEventsWithWorkerPool(photosApiHandler1, photosApiHandler2)
    .thenHandleEventsWithWorkerPool(postsAndCommentsResponseHandler1, postsAndCommentsResponseHandler2)
    .handleEventsWithWorkerPool(albumsAndPhotosResponseHandler1, albumsAndPhotosResponseHandler2);
    DISRUPTOR.start();
}

// For each request, an event is published in RingBuffer:
Event event = null;
RingBuffer<Event> ringBuffer = DISRUPTOR.getRingBuffer();
long sequence = ringBuffer.next();
CountDownLatch countDownLatch = new CountDownLatch(6);
try {
    event = ringBuffer.get(sequence);
    event.countDownLatch = countDownLatch;
    event.startTime = System.currentTimeMillis();
} finally {
    ringBuffer.publish(sequence);
}
try {
    event.countDownLatch.await();
} catch (InterruptedException e) {
    e.printStackTrace();
}

10. Akka

Image from: https://blog.codecentric.de/en/2015/08/introduction-to-akka-actors/

• The main advantage of Akka library is that it has local support for building distributed systems.

• It runs on a system called Actor System. This system abstracts the concept of thread, and the Actor in Actor System communicates through asynchronous messages, which is similar to the communication between producers and consumers.

• This extra level of abstraction helps the Actor System provide features such as fault tolerance, location transparency, and so on.

• With the correct actor to thread strategy, the framework can be optimized to perform better than the results shown in the table above. Although it can not match the performance of traditional methods on a single node, it is still the first choice because of its ability to build distributed and elastic systems.

10.1 example code

// From controller:
Actors.masterActor.tell(new Master.Request("Get Response", event, Actors.workerActor), ActorRef.noSender());

// handler :
public Receive createReceive() {
    return receiveBuilder().match(Request.class, request -> {
    Event event = request.event; // Ideally, immutable data structures should be used here.
    request.worker.tell(new JsonServiceWorker.Request("posts", event), getSelf());
    request.worker.tell(new JsonServiceWorker.Request("comments", event), getSelf());
    request.worker.tell(new JsonServiceWorker.Request("albums", event), getSelf());
    request.worker.tell(new JsonServiceWorker.Request("photos", event), getSelf());
    }).match(Event.class, e -> {
    if (e.posts != null && e.comments != null & e.albums != null & e.photos != null) {
    int userId = new Random().nextInt(10) + 1;
    String postsAndCommentsOfRandomUser = ResponseUtil.getPostsAndCommentsOfRandomUser(userId, e.posts,
    e.comments);
    String albumsAndPhotosOfRandomUser = ResponseUtil.getAlbumsAndPhotosOfRandomUser(userId, e.albums,
    e.photos);
    String response = postsAndCommentsOfRandomUser + albumsAndPhotosOfRandomUser;
    e.response = response;
    e.countDownLatch.countDown();
    }
    }).build();
}

11. Summary

• Determine the configuration of the Executor framework based on the load of the machine, and check whether the load can be balanced according to the number of parallel tasks in the application.

• For most traditional applications, using a reactive development library or any asynchronous library can degrade performance. This pattern is useful only if the use case is similar to a server-side chat application, where the thread does not need to keep the connection until the client responds.

• The Disruptor framework performs well when used with event driven architecture patterns; however, when used in combination with traditional architectures, it does not meet the standards for the use cases we discuss here. Note here that the Akka and Disruptor libraries deserve a separate article on how to use them to implement event driven architectural patterns.

• The source code for this article can be found on GitHub.

Recommend to my blog to read more:

1.Java JVM, collection, multithreading, new features series

2.Spring MVC, Spring Boot, Spring Cloud series tutorials

3.Maven, Git, Eclipse, Intellij IDEA series tools tutorial

4.Latest interview questions of Java, backend, architecture, Alibaba and other large factories

Feel good, don't forget to like + forward!