At present, the scheme used by the pressure measurement system is completed by the secondary development of goreplay. Because the whole is Java technology stack, there are two problems in using goreplay: first, compatibility, language and development framework increase the complexity of use case creation and execution; The secondary development cost of replay cannot meet the secondary performance test. If two sets of pressure testing engines are maintained, it will bring more workload.
Therefore, in order to solve these two problems as much as possible, I received a job to investigate the log playback function implemented by Java. I mainly read the source code of goreplay and its design idea, and reproduce it in Java.
Here we use what we shared two days ago Demo of common API for Disruptor high performance queue,Application of high performance queue Disruptor in testing , you can turn it over again if you are interested. In addition, the video version is still in production and will meet you in the coming year.
thinking
The overall design idea is as follows:
Ten million log playback design
PS: traffic increment and dynamic increase and decrease have not been realized, and the source code of goreplay is still being studied.
Log pulling and parsing
The logic in the original project is still adopted for the log pulling and preliminary analysis. The log is pulled from the gateway log through SQL statement, the log content is preliminarily analyzed, put into the cloud OSS, and the link is stored in the database (this step is put after the traffic is recorded successfully).
PS: Currently, the only useful information retained by log parsing is URL
The log format is as follows:
/v1/level,funtester.com,-,token,-,1622611469,- /v1/level,funtester.com,-,token,-,1622611469,- /v1/level,funtester.com,-,token,-,1622611469,- /v1/level,funtester.com,-,token,-,1622611469,- /v1/level,funtester.com,-,token,-,1622611469,- /v1/level,funtester.com,-,token,-,1622611469,- /v1/level,funtester.com,-,token,-,1622611469,-
Implementation steps
- First, put the useful information (URL) and token in the log into memory
- Configure the host, read the URL, and assemble the HTTP request in response to the header (token, pressure test ID, common header, analog disk ID).
- Create the Disruptor object and create the producer asynchronously
- The log traffic reply function of the HTTP interface is achieved through the consumer consumption (sending request) message (HTTP request object).
performance index
- Local 6C16G configuration test data
- The measured reading speed of 10 million URL s is about 9s ~ 13s, and there is no pressure on the memory. If the subsequent demand for a larger amount of logs, the logs can be read asynchronously through stream. The measured log reading speed is more than 800000 / s, which meets the current demand.
- Single producer speed 250000 QPS
- The single machine test QPS is 88000, the CPU runs full and touches the physical limit. This data has little difference from the pressure test compared with the previous tools.
risk
- Consumers store messages asynchronously. Messages will be discarded if they exceed a certain number. This problem is triggered when the consumer speed is less than the producer speed.
- The number of consumers needs to be set before starting. If the parameter setting is unreasonable, it will lead to the bottleneck of consumer pressure and can not dynamically increase consumers.
PS: these risks will be solved one by one in the future.
code implementation
Producer Demo:
def ft = {
output("Create thread")
fun {
int i = 0
while (key) {
def url = logs.get(i % logs.size())
def get = getHttpGet(HOST + url)
get.addHeader("token", tokens.get(i % tokens.size()))
get.addHeader(HttpClientConstant.USER_AGENT)
ringBuffer.publishEvent {e, s ->
e.setRequest(get)
}
i++
}
}
}
ft()
Read file code
/**
* Read part of the contents of the super large file through the closure incoming method
*
* @param filePath
* @param function
* @return
*/
public static List<String> readByLine(String filePath, Function<String, String> function) {
if (StringUtils.isEmpty(filePath) || !new File(filePath).exists() || new File(filePath).isDirectory())
ParamException.fail("File information error!" + filePath);
logger.debug("Read file name:{}", filePath);
List<String> lines = new ArrayList<>();
File file = new File(filePath);
if (file.isFile() && file.exists()) { // Determine whether the file exists
try (FileInputStream fileInputStream = new FileInputStream(file);
InputStreamReader read = new InputStreamReader(fileInputStream, DEFAULT_CHARSET);
BufferedReader bufferedReader = new BufferedReader(read, 3 * 1024 * 1024);) {
String line = null;
while ((line = bufferedReader.readLine()) != null) {
String apply = function.apply(line);
if (StringUtils.isNotBlank(apply)) lines.add(apply);
}
} catch (Exception e) {
logger.warn("Error reading file contents", e);
}
} else {
logger.warn("The specified file cannot be found:{}", filePath);
}
return lines;
}
Demo demo
package com.funtest.groovytest
import com.funtester.base.constaint.FixedThread
import com.funtester.config.HttpClientConstant
import com.funtester.frame.execute.Concurrent
import com.funtester.frame.execute.ThreadPoolUtil
import com.funtester.httpclient.ClientManage
import com.funtester.httpclient.FunLibrary
import com.funtester.utils.ArgsUtil
import com.funtester.utils.RWUtil
import com.lmax.disruptor.EventHandler
import com.lmax.disruptor.RingBuffer
import com.lmax.disruptor.WorkHandler
import com.lmax.disruptor.YieldingWaitStrategy
import com.lmax.disruptor.dsl.Disruptor
import com.lmax.disruptor.dsl.ProducerType
import org.apache.http.client.methods.HttpGet
import org.apache.http.client.methods.HttpRequestBase
import org.junit.platform.commons.util.StringUtils
import java.util.concurrent.LinkedBlockingDeque
import java.util.function.Function
class ReplayTest extends FunLibrary {
static String url = "http://localhost:12345/test";
static HttpGet httpGet = getHttpGet(url);
// static LinkedBlockingQueue<HttpRequestBase> requests = new LinkedBlockingQueue<>()
static def HOST = "http://localhost:12345"
static def key = true
static Disruptor<RequestEvent> disruptor
public static void main(String[] args) {
def logfile = "/Users/oker/Desktop/log.csv"
// def logfile = "/Users/oker/Desktop/fun.csv"
//10 million logs
def tokenfile = "/Users/oker/Desktop/token.csv"
//20000 user token
List<String> logs = RWUtil.readByLine(logfile, new Function<String, String>() {
@Override
String apply(String s) {
return StringUtils.isNotBlank(s) && s.startsWith("/") ? s.split(COMMA)[0] : null
}
});
List<String> tokens = RWUtil.readByLine(tokenfile, new Function<String, String>() {
@Override
String apply(String s) {
return StringUtils.isNotBlank(s) ? s.split(COMMA)[4] : null
}
});
output("total ${formatLong(logs.size())} Log")
disruptor = new Disruptor<RequestEvent>(
RequestEvent::new,
512 * 512,
ThreadPoolUtil.getFactory(),
ProducerType.MULTI,
new YieldingWaitStrategy()
);
RingBuffer<RequestEvent> ringBuffer = disruptor.getRingBuffer();
def ft = {
output("Create thread")
fun {
int i = 0
while (key) {
def url = logs.get(i % logs.size())
def get = getHttpGet(HOST + url)
get.addHeader("token", tokens.get(i % tokens.size()))
get.addHeader(HttpClientConstant.USER_AGENT)
ringBuffer.publishEvent {e, s ->
e.setRequest(get)
}
i++
}
}
}
ft()
disruptor.handleEventsWith(new FunTester(10))
// 5.times {ft()}
//Let's start the test
ClientManage.init(10, 5, 0, "", 0)
def util = new ArgsUtil(args)
def thread = util.getIntOrdefault(0, 20)
def times = util.getIntOrdefault(1, 60000)
RUNUP_TIME = util.getIntOrdefault(2, 0)
def tasks = []
thread.times {
def tester = new FunTester(times)
disruptor.handleEventsWith(tester);
tasks << tester
}
disruptor.start();
new Concurrent(tasks, "This is a ten million level log playback demonstration Demo").start()
}
private static class FunTester extends FixedThread implements EventHandler<RequestEvent>, WorkHandler<RequestEvent> {
LinkedBlockingDeque<HttpRequestBase> reqs = new LinkedBlockingDeque<HttpRequestBase>()
FunTester(int limit) {
super(null, limit, true)
}
@Override
protected void doing() throws Exception {
FunLibrary.executeOnly(reqs.take())
}
@Override
FixedThread clone() {
return new FunTester(limit)
}
@Override
protected void after() {
super.after()
key = false
disruptor.shutdown()
}
@Override
void onEvent(RequestEvent event, long sequence, boolean endOfBatch) throws Exception {
if (reqs.size() < 100000) reqs.add(event.getRequest())
}
@Override
void onEvent(RequestEvent event) throws Exception {
if (reqs.size() < 100000) reqs.add(event.getRequest())
}
}
private static class RequestEvent {
HttpRequestBase request;
public HttpRequestBase getRequest() {
return request;
}
public void setRequest(HttpRequestBase request) {
this.request = request;
}
}
}
PS: multiple group s are used here. The reason is marked in the design draft.