SpringCloud Upgrade Path 2020.0.x-45. Implementing Public Logging

Posted by seventheyejosh on Thu, 02 Dec 2021 04:10:02 +0100

This series of code addresses: https://github.com/JoJoTec/sp...

This section builds on the previous Publisher factory implementation with link information to implement the GlobalFilter for public logging. Review our needs:

We need to record each request at the gateway:

  • HTTP related elements:

    • URL-related information
    • Request information, such as HTTP HEADER, request time, etc.
    • Certain types of requestors
    • Response information, such as response codes
    • Response body for some types of responses
  • Link Information

Notes for Body recording requests and responses

As mentioned in the previous section, body processing of requests and responses can result in loss of link information for Spring Cloud Sleuth if put into the main link with its results. Two other things to note are:

  • TCP sticky package unpacking causes a request body to be split into several parts or a package to contain several requests
  • DataBuffer to release the original request body read after reading

Why release the DataBuffer read by the original request body? This is because if the read-out DataBuffer is not manually released, the underlying count will remain non-zero, causing a memory leak. You can see from the framework code that the DataBuffer here needs to be released manually, referring to the source code:


public ByteBuffer decode(DataBuffer dataBuffer, ResolvableType elementType,
        @Nullable MimeType mimeType, @Nullable Map<String, Object> hints) {

    int byteCount = dataBuffer.readableByteCount();
    ByteBuffer copy = ByteBuffer.allocate(byteCount);
    if (logger.isDebugEnabled()) {
        logger.debug(Hints.getLogPrefix(hints) + "Read " + byteCount + " bytes");
    return copy;

We want to convert a body that can be output to a log into a string for output. To keep the code simple and prevent errors, we use a tool class to read and release a DataBuffer as a string:

package com.github.jojotech.spring.cloud.apigateway.common;

import com.google.common.base.Charsets;

import org.springframework.core.io.buffer.DataBuffer;
import org.springframework.core.io.buffer.DataBufferUtils;

public class BufferUtil {
    public static String dataBufferToString(DataBuffer dataBuffer) {
        byte[] content = new byte[dataBuffer.readableByteCount()];
        return new String(content, Charsets.UTF_8);

Write GlobalFilter to implement public logging

With so much paving in front of us, we can finally get started writing this blog, GlobalFilter:

package com.github.jojotech.spring.cloud.apigateway.filter;

import java.net.URI;
import java.util.Set;

import com.alibaba.fastjson.JSON;
import com.github.jojotech.spring.cloud.apigateway.common.BufferUtil;
import com.github.jojotech.spring.cloud.apigateway.common.TracedPublisherFactory;
import lombok.extern.log4j.Log4j2;
import org.reactivestreams.Publisher;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cloud.gateway.filter.GatewayFilterChain;
import org.springframework.cloud.gateway.filter.GlobalFilter;
import org.springframework.core.Ordered;
import org.springframework.core.io.buffer.DataBuffer;
import org.springframework.core.io.buffer.DataBufferFactory;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.http.server.reactive.ServerHttpRequest;
import org.springframework.http.server.reactive.ServerHttpRequestDecorator;
import org.springframework.http.server.reactive.ServerHttpResponse;
import org.springframework.http.server.reactive.ServerHttpResponseDecorator;
import org.springframework.stereotype.Component;
import org.springframework.web.server.ServerWebExchange;

public class CommonLogFilter implements GlobalFilter, Ordered {
    //Output body format
    public static final Set<MediaType> legalLogMediaTypes = Set.of(

    private TracedPublisherFactory tracedPublisherFactory;

    public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
        long startTime = System.currentTimeMillis();
        ServerHttpRequest request = exchange.getRequest();
        ServerHttpResponse response = exchange.getResponse();
        //Get factory for unpacking buffer s to process aggregate read requests and responses to body s
        DataBufferFactory dataBufferFactory = response.bufferFactory();
        //Request http header
        HttpHeaders requestHeaders = request.getHeaders();
        //Request body type
        MediaType requestContentType = requestHeaders.getContentType();
        //Request uri
        String uri = request.getURI().toString();
        //Request http method
        HttpMethod method = request.getMethod();
        log.info("{} -> {}: header: {}", method, uri, JSON.toJSONString(requestHeaders));
        Flux<DataBuffer> dataBufferFlux = tracedPublisherFactory.getTracedFlux(request.getBody(), exchange)
                //Use buffer to read all body s here to avoid unpacking
                .map(dataBuffers -> {
                    //Glue all buffer s together
                    DataBuffer dataBuffer = dataBufferFactory.join(dataBuffers);
                    //body will only be output when debug is turned on
                    if (log.isDebugEnabled()) {
                        //Only specific body types will output specific
                        if (legalLogMediaTypes.contains(requestContentType)) {
                            try {
                                //Convert the body to a String for output, and note that the original buffer needs to be released because the body stream has been read, but there is no place to recycle it
                                //Reference resources
                                String s = BufferUtil.dataBufferToString(dataBuffer);
                                log.debug("body: {}", s);
                                dataBuffer = dataBufferFactory.wrap(s.getBytes());
                            catch (Exception e) {
                                log.error("error read request body: {}", e.getMessage(), e);
                        else {
                            log.debug("body: {}", request);
                    return dataBuffer;
        return chain.filter(exchange.mutate().request(new ServerHttpRequestDecorator(request) {
            public Flux<DataBuffer> getBody() {
                return dataBufferFlux;
        }).response(new ServerHttpResponseDecorator(response) {
            public Mono<Void> writeWith(Publisher<? extends DataBuffer> body) {
                HttpHeaders responseHeaders = super.getHeaders();
                //Called here is a write-back to the client's HTpClientConnect, which has jumped out of Spring Cloud Sleuth's link Span, so there is no link tracking information
                //But in CommonTraceFilter we put link information in the response Header, so we don't need to manually add link information here
                log.info("response: {} -> {} {} header: {}, time: {}ms", method, uri, getStatusCode(), JSON.toJSONString(responseHeaders), System.currentTimeMillis() - startTime);
                final MediaType contentType = responseHeaders.getContentType();
                if (contentType != null && body instanceof Flux && legalLogMediaTypes.contains(contentType) && log.isDebugEnabled()) {
                    //There is a problem with TCP sticky package unpacking. This body is written many times and cannot get the complete body on one call, so it is converted to fluxBody to accept the complete body using its buffer s
                    Flux<? extends DataBuffer> fluxBody = tracedPublisherFactory.getTracedFlux(Flux.from(body), exchange);
                    return super.writeWith(fluxBody.buffer().map(buffers -> {
                        DataBuffer buffer = dataBufferFactory.join(buffers);
                        try {
                            String s = BufferUtil.dataBufferToString(buffer);
                            log.debug("response: body: {}", s);
                            return dataBufferFactory.wrap(s.getBytes());
                        } catch (Exception e) {
                            log.error("error read response body: {}", e.getMessage(), e);
                        return buffer;
                // if body is not a flux. never got there.
                return super.writeWith(body);

    public int getOrder() {
        //Specify the order after the CommonTrace Filter, which reads link information, preferably before all filters
        return new CommonTraceFilter().getOrder() + 1;

The points that need attention are clearly indicated in the notes, please refer to them.

view log

By adding the following log configuration, we open the body's log so that it's complete:

<AsyncLogger name="com.github.jojotech.spring.cloud.apigateway.filter.CommonLogFilter" level="debug" additivity="false" includeLocation="true">
    <appender-ref ref="console" />

Send a POST request with body, as you can see from the log:

2021-11-29 14:08:42,231  INFO [sports,8481ce2786b686fa,8481ce2786b686fa] [24916] [reactor-http-nio-2][com.github.jojotech.spring.cloud.apigateway.filter.CommonLogFilter:59]:POST -> header: {"Content-Type":["text/plain"],"User-Agent":["PostmanRuntime/7.28.4"],"Accept":["*/*"],"Postman-Token":["666b17c9-0789-46e6-b515-9a4538803308"],"Host":[""],"Accept-Encoding":["gzip, deflate, br"],"Connection":["keep-alive"],"content-length":["8"]}
2021-11-29 14:08:42,233 DEBUG [sports,8481ce2786b686fa,8481ce2786b686fa] [24916] [reactor-http-nio-2][com.github.jojotech.spring.cloud.apigateway.filter.CommonLogFilter:74]:body: ifasdasd
2021-11-29 14:08:42,463  INFO [sports,,] [24916] [reactor-http-nio-2][com.github.jojotech.spring.cloud.apigateway.filter.CommonLogFilter$1:96]:response: POST -> 200 OK header: {"traceId":["8481ce2786b686fa"],"spanId":["8481ce2786b686fa"],"Date":["Mon, 29 Nov 2021 14:08:43 GMT"],"Content-Type":["application/json"],"Server":["gunicorn/19.9.0"],"Access-Control-Allow-Origin":["*"],"Access-Control-Allow-Credentials":["true"],"content-length":["886"]}, time: 232ms
2021-11-29 14:08:42,466 DEBUG [sports,8481ce2786b686fa,8481ce2786b686fa] [24916] [reactor-http-nio-2][com.github.jojotech.spring.cloud.apigateway.filter.CommonLogFilter$1:105]:response: body: {
  "args": {
    "test": "1"
  "data": "ifasdasd", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Content-Length": "8", 
    "Content-Type": "text/plain", 
    "Forwarded": "proto=http;host=\"\";for=\"\"", 
    "Host": "httpbin.org", 
    "Postman-Token": "666b17c9-0789-46e6-b515-9a4538803308", 
    "User-Agent": "PostmanRuntime/7.28.4", 
    "X-Amzn-Trace-Id": "Root=1-61a4deeb-3d016ff729306d862edcca0b", 
    "X-B3-Parentspanid": "8481ce2786b686fa", 
    "X-B3-Sampled": "0", 
    "X-B3-Spanid": "5def545b28a7a842", 
    "X-B3-Traceid": "8481ce2786b686fa", 
    "X-Forwarded-Host": "", 
    "X-Forwarded-Prefix": "/test-ss"
  "json": null, 
  "method": "POST", 
  "origin": ",", 
  "url": ""

2021-11-29 14:08:42,474  INFO [sports,,] [24916] [reactor-http-nio-2][reactor.util.Loggers$Slf4JLogger:269]:8481ce2786b686fa,8481ce2786b686fa -> - - [2021-11-29T14:08:42.230008Z[Etc/GMT]] "POST /test-ss/anything?test=1 HTTP/1.1" 200 886 243 ms

WeChat Search "My Programming Miao" pays attention to public numbers, brushes every day, easily upgrades technology, and chops various offer s:

Topics: spring-cloud