background
End to end testing is used to verify the overall behavior of the application.
Compared with Unit Testing, which focuses on function verification, e2e is more prone to external dependencies, such as relying on external HTTP interface data, MYSQL data, Redis data, etc. These can be understood as external data dependencies, which affect the behavior of the application, but are not controlled by the application.
If the application as a whole is abstracted as an Input ouput, these external dependencies are the factors that affect the application Output.
Therefore, as long as we can mock these dependencies, we can run e2e test cases stably.
Problem introduction
Here, we focus on the Node application, isolate other HTTP dependencies, and stably run the test cases of each interface.
Example of user login interface test:
describe('user', () => {
it('login by phone number', async (done) => {
const res = await userLogin(axios, {
phoneNumber: '15555555555',
});
assert.ok(Array.isArray(res.data?.data));
done();
});
it('login by phone id', async (done) => {
const res = await userLogin(axios, {
id: 1,
});
assert.ok(Array.isArray(res.data?.data));
done();
});
it('login by account & pwd', async (done) => {
const res = await userLogin(axios as any, {
account: 'test',
pwd: 'test',
});
assert.ok(Array.isArray(res.data?.data));
done();
});
it('login by account & pwd after change pwd', async (done) => {
const res = await userLogin(axios as any, {
account: '1',
password: '2',
});
assert.ok(Array.isArray(res.data?.data));
done();
});
}Example shows that userLogin invokes the user login interface provided by the Node service. The Node service performs some data validation and preprocessing, and then calls another HTTP API to perform the login.
We found that the user login interface has a variety of possible parameters and different performances. Isolate the external HTTP API (referred to by Dep0) behind the Node server login interface. You need to record multiple requests and return records of Dep0 and match them with the application examples.
So the question here is:
- An external HTTP dependency records the request parameters / return data of multiple scenarios;
- The recorded data matches the test case;
- When the test case is running, it can record external HTTP requests or read recorded HTTP requests according to commands.
realization
How to record
Here, axios interceptor is used for recording:
import { AxiosRequestConfig, AxiosResponse } from 'axios';
import 'url';
import fs from 'fs-extra';
import { getCtxLogger } from '../../services/context';
import crypto from 'crypto';
// Some constants
const defaultFilePath = `${process.cwd()}/deps/net/`;
const ext = '.json';
// Record request input parameters before request
export const requestInterceptor = async (config: AxiosRequestConfig) => {
const { url, method } = config;
const urlObj = new URL(url as string);
const { pathname } = urlObj;
const filePath = getFilePathFromConfig(config);
try {
await fs.ensureFile(filePath);
const rawCurrentContent = await fs.readFile(filePath, 'utf8');
let currentContent = {};
try {
if (rawCurrentContent) {
currentContent = JSON.parse(rawCurrentContent);
}
} catch (e) {
console.error(`JSON.parse ${filePath} error, use empty object`, e);
}
if (!currentContent[pathname]) {
currentContent[pathname] = {};
}
if (!currentContent[pathname][method as string]) {
currentContent[pathname][method as string] = {};
}
const hash = getHashFromConfig(config);
if (!currentContent[pathname][method as string][hash]) {
currentContent[pathname][method as string][hash] = {};
}
// force update
currentContent[pathname][method as string][hash].request = config;
await fs.writeJSON(filePath, currentContent);
} catch (e) {
console.error('deps/net requestInterceptor error', e);
}
return config;
};
// After the response, the record returns
export const responseInterceptor = async (response: AxiosResponse) => {
const { config } = response;
const { url, method } = config;
const urlObj = new URL(url as string);
const { host, pathname } = urlObj;
const filePath = `${defaultFilePath}${host}${ext}`;
try {
await fs.ensureFile(filePath);
const rawCurrentContent = await fs.readFile(filePath, 'utf8');
const currentContent = JSON.parse(rawCurrentContent);
const hash = getHashFromConfig(config);
// force update
if (!currentContent[pathname][method as string][hash]) {
currentContent[pathname][method as string][hash] = {};
}
currentContent[pathname][method as string][hash].response = response.data;
await fs.writeJSON(filePath, currentContent);
} catch (e) {
console.error('deps/net responseInterceptor error', e);
}
return response;
};
// Auxiliary function to calculate the path of the record file from the config object
export function getFilePathFromConfig(config: AxiosRequestConfig): string {
const { url } = config;
const urlObj = new URL(url as string);
const { host } = urlObj;
const filePath = `${defaultFilePath}${host}${ext}`;
return filePath;
}Here is one thing to note:
How to calculate an identity according to the request input parameters. After the request returns, the same identity can be calculated again according to the response. Only in this way can an HTTP request and response be corresponding. The function to complete this function is getHashFromConfig above:
function getHashFromConfig(config: AxiosRequestConfig): string {
const pure = {
url: config.url,
params: config.params,
cookie: config.headers?.cookie,
data: config.data,
};
const strData = JSON.stringify(pure);
// hash config
const hash = crypto.createHash('md5').update(strData)
.digest('hex');
return hash;
}Instead of hash ing AxiosRequestConfig directly, only a few key fields are extracted. Because the test found that:
The response config is inconsistent at the string level before and after the request.
Results recorded after execution:
{
"/ws/district/v1/list": {
"get": {
"e4473f2e67634485d3b6defd93a502f1": {
"request": {
"transitional": {
"silentJSONParsing": true,
"forcedJSONParsing": true,
"clarifyTimeoutError": false
},
"transformRequest": [
null
],
"transformResponse": [
null
],
"timeout": 0,
"xsrfCookieName": "XSRF-TOKEN",
"xsrfHeaderName": "X-XSRF-TOKEN",
"maxContentLength": -1,
"maxBodyLength": -1,
"headers": {
"common": {
"Accept": "application/json, text/plain, */*"
},
"delete": {},
"get": {},
"head": {},
"post": {
"Content-Type": "application/x-www-form-urlencoded"
},
"put": {
"Content-Type": "application/x-www-form-urlencoded"
},
"patch": {
"Content-Type": "application/x-www-form-urlencoded"
}
},
"method": "get",
"url": "https://apis.map.qq.com/ws/district/v1/list"
},
"response": {
"status": 301,
"message": "Required fields are missing key"
}
}
}
}
}Summary
So far, we have completed the recording of HTTP requests. It also solves the problem of matching the record content with the use case:
According to the request input parameter hash, the request with the same input parameter can be uniquely identified once.
How does the request function read mock data
It is also implemented based on axios interceptor:
export const mockedRequest = axios.create();
// reference resources: https://stackoverflow.com/questions/62686283/axios-how-to-intercept-and-respond-to-axios-request
mockedRequest.interceptors.request.use((config) => { const { url, method } = config;
const urlObj = new URL(url as string);
const { pathname } = urlObj;
throw {
// Calculate the hash to match the recorded data
hash: getHashFromConfig(config),
filePath: getFilePathFromConfig(config),
pathname,
method,
}; // <- this will stop request
});
mockedRequest.interceptors.response.use(
response => response,
async (error) => {
const hash = error?.hash;
const filePath = error?.filePath;
const {
pathname,
method,
} = error;
const rawCurrentContent = await fs.readFile(filePath, 'utf8');
let currentContent = {};
try {
if (rawCurrentContent) {
currentContent = JSON.parse(rawCurrentContent);
}
} catch (e) {
console.error(`JSON.parse ${filePath} error, use empty object`, e);
}
const res = {
data: currentContent[pathname][method][hash]?.response,
};
return res;
} // <- sends as successful response
);Here, the axios request interceptor throw feature is used to terminate the request and redirect to reading the data recorded locally.
Finally, add whether to perform recording automatically according to the environment variable:
// Use the environment variable RECORD to turn on the recording mode
if (process.env.RECORD) {
request.interceptors.request.use(requestInterceptor);
request.interceptors.response.use(responseInterceptor);
}Summary
The recorded JSON follows git management. When the mockRequest implemented above is applied in e2e testing, it can stably replay the HTTP response and ensure the normal operation of the test case during CI.
summary
Through the axios interceptor, the isolation of external HTTP dependencies is completed.
In fact, the request and response data of the external HTTP interface we recorded can be used not only for mock, but also for deriving types, eliminating the need to write the interface manually.
import mockData from '../deps/net/apis.map.qq.com.json'; d type ListResponse = typeof mockData['/ws/district/v1/list'].get.e4473f2e67634485d3b6defd93a502f1.response;
Finally, a complete implementation is attached: https://github.com/xiaoshude/node-common
extend
Here, HTTP dependency isolation is implemented from within the application. If a general scheme is considered, you can also try to directly overwrite the Node http module for traffic interception.