Recently, in the process of studying the source code of goreplay, I feel that some ideas are still worth learning from. So I set up a flag to realize a ten million log playback function. However, there is a thorny problem in the process of this implementation: the LinkedBlockingQueue of Java is difficult to directly meet the demand scenarios and performance requirements.
Testers who are familiar with goreplay should be aware that Go language chanel is widely used in the framework of goreplay. Coupled with the high performance of Go language itself, it can be said that the two swords are combined. So I also want to write a similar idea according to the gourd painting ladle. There will be a special topic on this later.
Based on this, I found the high-performance queue of disruptor. Disruptor is a high-performance queue developed by LMAX, a British foreign exchange trading company. Its original intention is to solve the delay problem of memory queue (it is found in the performance test that it is in the same order of magnitude as I/O operation). The system developed based on disruptor can support 6 million orders per second in a single thread.
When testing using Disruptor, you don't need to create various objects and abstract various object methods as in the Springboot framework. My principle is how to be simple and how to do it. Let's share the basic practice and simple case demonstration of Disruptor in testing.
rely on
// https://mvnrepository.com/artifact/com.lmax/disruptor implementation group: 'com.lmax', name: 'disruptor', version: '3.4.2'
Only Gradle versions are listed. 3 + is recommended. Some Lambda syntax support is required.
Event object
First, we need to define an Event type. Of course, we can also directly use Java Lang.string uses existing classes, but when setting the Event object, you need to use the new keyword and construct a new Event. The consistency of using the set method is better than that of direct = assignment. Writing an Event class alone can be simpler. Yes, the code is more logical and does not need to be affected by other classes.
Here I define a simple Event class:
public static class FunEvent {
String id;
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
}
Disruptor creation
To create a Disruptor object, we first need to define an event type and call the construction method. There are five parameters. 1.Event initialization method; 2.ringbuffsize, which should be an integer multiple of 2; 3.threadfactory, a factory class for creating threads. Here I use COM funtester. frame. execute. ThreadPoolUtil#getFactory()； 4. Producer mode, which is divided into single producer and multi producer enumeration categories; 5. Waiting strategy. The official provides some implementation classes for selection. I use COM lmax. Disruptor. YieldingWaitStrategy.
The creation method is as follows:
Disruptor<FunEvent> disruptor = new Disruptor<FunEvent>(
FunEvent::new,
1024 * 1024,
ThreadPoolUtil.getFactory(),
ProducerType.MULTI,
new YieldingWaitStrategy()
);producer
For message queues, there are two important roles, producer and consumer. First, I'll talk about the Disruptor producer. I found a lot of data. It's necessary to create a producer's class and then implement a method. The content of this method is basically the same. The content is as follows:
long sequence = ringBuffer.next();
try {
FunEvent funEvent = ringBuffer.get(sequence);
funEvent.setId(orderId);
} finally {
ringBuffer.publish(sequence);
}Then I think it's a bit superfluous to use the producer object to call this method. Fortunately, an article introduced some new features of Disruptor and mentioned that it supports Lambda syntax, so you don't need to create producer object. The syntax is as follows:
ringBuffer.publishEvent((Event, sequence) -> Event.setId(StringUtil.getString(10)));
consumer
Consumers need to implement two interfaces to create com lmax. disruptor. EventHandler and com lmax. disruptor. Workhandler, one is dealing with single consumer mode and the other is dealing with multi consumer mode.
The creation method is as follows:
/**
* consumer
*/
private static class FunEventHandler implements EventHandler<FunEvent>, WorkHandler<FunEvent> {
public void onEvent(FunEvent Event, long sequence, boolean endOfBatch) {
output("Consumption news:" + Event.getId() + TAB + sequence);
}
public void onEvent(FunEvent Event) {
output("Consumption news:" + Event.getId());
}
}Configure handler
There are two types: configuring a single consumer and configuring multiple consumers.
Individual consumers:
disruptor.handleEventsWith(new FunEventHandler());
Multiple consumers:
disruptor.handleEventsWithWorkerPool(new FunEventHandler(), new FunEventHandler());
Whether single or multiple, each call will produce a com lmax. disruptor. dsl. Eventhandlergroup, each com lmax. disruptor. dsl. Eventhandlergroup will completely consume the events generated by each producer. If it is set for 5 times, an Event will be consumed for 5 times, each com lmax. disruptor. dsl. Eventhandlergroup consumes once and is blocked. Join a com lmax. disruptor. dsl. The consumption of eventhandlergroup object is slow, which will block other consumers from consuming the next Event.
start-up
After the assembly is completed, you can start the Disruptor. The syntax is as follows: Disruptor start();， Close the syntax Disruptor shutdown();， Closing here will not empty the existing events of getRingBuffer. According to the official document, it should stop production and wait for consumption.
Demo demo
Java version
public static void main(String[] args) {
Disruptor<FunEvent> disruptor = new Disruptor<FunEvent>(
FunEvent::new,
1024 * 1024,
ThreadPoolUtil.getFactory(),
ProducerType.MULTI,
new YieldingWaitStrategy()
);
disruptor.handleEventsWithWorkerPool(new FunEventHandler(), new FunEventHandler());
disruptor.handleEventsWith(new FunEventHandler());
disruptor.start();
RingBuffer<FunEvent> ringBuffer = disruptor.getRingBuffer();
for (int i = 0; i < 3; i++) {
ringBuffer.publishEvent((event, sequence) -> event.setId(StringUtil.getString(10)));
}
sleep(5.0);
disruptor.shutdown();
}Console output:
INFO-> main Current user: oker，Working directory:/Users/oker/IdeaProjects/funtester/,System coding format:UTF-8,system Mac OS X edition:10.16 INFO-> main ###### # # # # ####### ###### ##### ####### ###### ##### # # # ## # # # # # # # # #### # # # # # # #### ##### # #### ##### # # # # # # # # # # # # # # ##### # # # ###### ##### # ###### # # INFO-> F-3 Consumption news:i3OrH2ZnxD 0 INFO-> F-1 Consumption news:i3OrH2ZnxD INFO-> F-2 Consumption news:whhoxoMxmR INFO-> F-3 Consumption news:whhoxoMxmR 1 INFO-> F-2 Consumption news:IeP9fIRpKp INFO-> F-3 Consumption news:IeP9fIRpKp 2 Process finished with exit code 0
As you can see, each message will be consumed twice. Where F-3 thread consumption = the sum of F-1 and F-2 thread consumption, which is understood by home. Com lmax. disruptor. dsl. Functions of eventhandlergroup.
Groovy + asynchronous version
public static void main(String[] args) {
Disruptor<FunEvent> disruptor = new Disruptor<FunEvent>(
FunEvent::new,
1024 * 1024,
ThreadPoolUtil.getFactory(),
ProducerType.MULTI,
new YieldingWaitStrategy()
)
disruptor.handleEventsWithWorkerPool(new FunEventHandler(), new FunEventHandler())
disruptor.handleEventsWith(new FunEventHandler())
disruptor.start()
RingBuffer<FunEvent> ringBuffer = disruptor.getRingBuffer();
def funtester = {
fun {
100.times {ringBuffer.publishEvent((event, sequence) -> event.setId(StringUtil.getString(10)));}
}
}
10.times {funtester()}
sleep(5.0)
disruptor.shutdown()
}