1. background
With the popularity of microservice architecture, services are divided according to different dimensions. A request often involves multiple services. And many services may be distributed in thousands of servers, across multiple different data centers. In order to locate and solve the fault quickly and analyze the application performance, the whole link monitoring component is produced under the background of this problem. The most famous is the Google Dapper mentioned in Google's published paper. To understand the behavior of distributed systems in this context, we need to monitor the associated actions across different applications and different servers.
1.1 principle of full link
By adding and passing the call chain ID in the process of service call, the link data between applications can be generated and finally connected into a complete call chain. In the whole call process, each request needs to pass through TxId, SpanId and pSpanId.
1.2 Spring Cloud Gateway
As the second generation of gateway framework officially launched by Spring Cloud, Spring cloud gateway is a gateway developed based on Spring 5.0, Spring Boot2.0 and Reactor technologies, and uses NIO model for communication.
1.2.1 Spring WebFlux
Spring Boot 2.0 includes a new spring webplus module. The Flux in the name comes from the class Flux in Reactor. The module includes support for responsive HTTP and WebSocket clients, as well as support for REST, HTML and WebSocket interaction programs. In general, Spring MVC is used for synchronous processing; Spring Webflux is used for asynchronous processing.
1.2.2 Mono and Flux
Mono represents an asynchronous sequence of 0 or 1 elements, i.e. either elements are published successfully or errors are made
Flux represents an asynchronous sequence of 0 to N elements, that is, either 0 to N elements are published successfully or errors are made
Flux and Mono can be converted to each other. For example, if you count a flux sequence, the result is a Mono object, or if you combine two Mono sequences, you get a flux object.
2. Spring Cloud Gateway does not monitor
Spring Cloud Gateway, as the gateway, is mainly responsible for the routing and forwarding of services. If the gateway is not monitored, the gateway node will be lost in the whole link, which will be directly displayed as the user accessing the subsequent application; it can not effectively locate whether the slow user request is a gateway problem or a subsequent node.
3. Monitoring by spring cloud gateway
Because Spring Cloud Gateway adopts the non blocking call of Reactor framework, tasks will be executed across threads, resulting in poor delivery of the call chain ID required for HTTP header information.
- Gateway receive thread
- Gateway return thread
- Routing and forwarding thread
3.1 Spring Cloud Gateway process
Now, the process of Spring Cloud Gateway is sorted out. Because this article only involves cross thread service scheduling, the routing process is not discussed.
3.1.1 request entry of spring cloud gateway
Org.springframework.http.server.reactive.reactorhttpahandleradapter? First, convert the received HttpServerRequest or the final HttpServerResponse package that needs to be returned to reactor server httprequest and reactor server httpresponse, and then process the request.
public Mono<Void> apply(HttpServerRequest request, HttpServerResponse response) { NettyDataBufferFactory bufferFactory = new NettyDataBufferFactory(response.alloc()); ServerHttpRequest adaptedRequest; ServerHttpResponse adaptedResponse; try { adaptedRequest = new ReactorServerHttpRequest(request, bufferFactory); adaptedResponse = new ReactorServerHttpResponse(response, bufferFactory); } catch (URISyntaxException ex) { logger.error("Invalid URL " + ex.getMessage(), ex); response.status(HttpResponseStatus.BAD_REQUEST); return Mono.empty(); } if (adaptedRequest.getMethod() == HttpMethod.HEAD) { adaptedResponse = new HttpHeadResponseDecorator(adaptedResponse); } return this.httpHandler.handle(adaptedRequest, adaptedResponse) .doOnError(ex -> logger.error("Handling completed with error", ex)) .doOnSuccess(aVoid -> logger.debug("Handling completed with success")); }
3.1.2 construct gateway context
org.springframework.web.server.adapter.HttpWebHandlerAdapter#handle
- createExchange() constructs the gateway context ServerWebExchange
- getDelegate() obtains a series of webhandlers to be processed through delegation
public Mono<Void> handle(ServerHttpRequest request, ServerHttpResponse response) { ServerWebExchange exchange = createExchange(request, response); return getDelegate().handle(exchange) .onErrorResume(ex -> handleFailure(request, response, ex)) .then(Mono.defer(response::setComplete)); } protected ServerWebExchange createExchange(ServerHttpRequest request, ServerHttpResponse response) { return new DefaultServerWebExchange(request, response, this.sessionManager, getCodecConfigurer(), getLocaleContextResolver(), this.applicationContext); }
3.1.3 enter the Filter chain
Org.springframework.cloud.gateway.handler.filteringwehandler? Get the GatewayFilter array, and create a DefaultGatewayFilterChain according to the obtained GatewayFilter array to filter and process the request.
public Mono<Void> handle(ServerWebExchange exchange) { Route route = exchange.getRequiredAttribute(GATEWAY_ROUTE_ATTR); List<GatewayFilter> gatewayFilters = route.getFilters(); List<GatewayFilter> combined = new ArrayList<>(this.globalFilters); combined.addAll(gatewayFilters); AnnotationAwareOrderComparator.sort(combined); logger.debug("Sorted gatewayFilterFactories: "+ combined); return new DefaultGatewayFilterChain(combined).filter(exchange); }
3.1.4 execute Filter chain
Org.springframework.cloud.gateway.handler.filteringwehandler $defaultgatewayfilterchain - chain call to filter
public Mono<Void> filter(ServerWebExchange exchange) { return Mono.defer(() -> { if (this.index < filters.size()) { GatewayFilter filter = filters.get(this.index); DefaultGatewayFilterChain chain = new DefaultGatewayFilterChain(this, this.index + 1); return filter.filter(exchange, chain); } else { return Mono.empty(); // complete } }); }
3.1.5 Gateway Filter adapter
Org.springframework.cloud.gateway.handler.filteringwehandler $gatewayfilteradapter - filter gatewayfilteradapter is the wrapper class of GlobalFilter, which is finally executed by Global Filter.
private static class GatewayFilterAdapter implements GatewayFilter { private final GlobalFilter delegate; public GatewayFilterAdapter(GlobalFilter delegate) { this.delegate = delegate; } public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) { return this.delegate.filter(exchange, chain); } }
3.1.6 Netty routing gateway filter
org.springframework.cloud.gateway.filter.NettyRoutingFilter#filter 
There are many implementations of GlobalFilter. Only nettyroutingfilter and NettyWriteResponseFilter are analyzed here. Nettyroutingfilter is responsible for using the Netty HttpClient agent for downstream requests.
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) { // Get requestUrl URI requestUrl = exchange.getRequiredAttribute(GATEWAY_REQUEST_URL_ATTR); // Determine whether it can be processed, http or https prefix String scheme = requestUrl.getScheme(); if (isAlreadyRouted(exchange) || (!"http".equals(scheme) && !"https".equals(scheme))) { return chain.filter(exchange); } // Set routed setAlreadyRouted(exchange); ServerHttpRequest request = exchange.getRequest(); // Create Netty Request Method object final HttpMethod method = HttpMethod.valueOf(request.getMethod().toString()); final String url = requestUrl.toString(); HttpHeaders filtered = filterRequest(this.headersFilters.getIfAvailable(), exchange); final DefaultHttpHeaders httpHeaders = new DefaultHttpHeaders(); filtered.forEach(httpHeaders::set); String transferEncoding = request.getHeaders().getFirst(HttpHeaders.TRANSFER_ENCODING); boolean chunkedTransfer = "chunked".equalsIgnoreCase(transferEncoding); boolean preserveHost = exchange.getAttributeOrDefault(PRESERVE_HOST_HEADER_ATTRIBUTE, false); // Request back end service return this.httpClient.request(method, url, req -> { final HttpClientRequest proxyRequest = req.options(NettyPipeline.SendOptions::flushOnEach) .headers(httpHeaders) .chunkedTransfer(chunkedTransfer) .failOnServerError(false) .failOnClientError(false); if (preserveHost) { String host = request.getHeaders().getFirst(HttpHeaders.HOST); proxyRequest.header(HttpHeaders.HOST, host); } return proxyRequest.sendHeaders() //Send request header .send(request.getBody().map(dataBuffer -> // Send request Body ((NettyDataBuffer)dataBuffer).getNativeBuffer())); }).doOnNext(res -> { ServerHttpResponse response = exchange.getResponse(); // put headers and status so filters can modify the response HttpHeaders headers = new HttpHeaders(); res.responseHeaders().forEach(entry -> headers.add(entry.getKey(), entry.getValue())); exchange.getAttributes().put("original_response_content_type", headers.getContentType()); HttpHeaders filteredResponseHeaders = HttpHeadersFilter.filter( this.headersFilters.getIfAvailable(), headers, exchange, Type.RESPONSE); response.getHeaders().putAll(filteredResponseHeaders); HttpStatus status = HttpStatus.resolve(res.status().code()); if (status != null) { response.setStatusCode(status); } else if (response instanceof AbstractServerHttpResponse) { // https://jira.spring.io/browse/SPR-16748 ((AbstractServerHttpResponse) response).setStatusCodeValue(res.status().code()); } else { throw new IllegalStateException("Unable to set status code on response: " +res.status().code()+", "+response.getClass()); } // Defer committing the response until all route filters have run // Put client response as ServerWebExchange attribute and write response later NettyWriteResponseFilter exchange.getAttributes().put(CLIENT_RESPONSE_ATTR, res); }).then(chain.filter(exchange)); }
3.1.7 Netty write back response gateway filter
Org.springframework.cloud.gateway.filter.nettywriteresponsefilter - the filter nettywriteresponsefilter appears in pairs with NettyRoutingFilter, which is responsible for writing the proxy response back to the gateway client response.
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) { // then method to implement After Filter logic return chain.filter(exchange).then(Mono.defer(() -> { // Get Netty Response HttpClientResponse clientResponse = exchange.getAttribute(CLIENT_RESPONSE_ATTR); if (clientResponse == null) { return Mono.empty(); } log.trace("NettyWriteResponseFilter start"); ServerHttpResponse response = exchange.getResponse(); // Write Netty Response back to the client NettyDataBufferFactory factory = (NettyDataBufferFactory) response.bufferFactory(); //TODO: what if it's not netty final Flux<NettyDataBuffer> body = clientResponse.receive() .retain() // ByteBufFlux => ByteBufFlux .map(factory::wrap); // ByteBufFlux => Flux<NettyDataBuffer> MediaType contentType = response.getHeaders().getContentType(); return (isStreamingMediaType(contentType) ? response.writeAndFlushWith(body.map(Flux::just)) : response.writeWith(body)); })); }
4. Monitoring by spring cloud gateway
When the gateway is finally monitored, you can see the application request flow, the gateway request business flow, and the call load of each application.
Or scan the WeChat official account below the two-dimensional code query
