1. Preface

The previous article analyzes how the Consumer initiates an RPC call, and how the Request object Request is encoded from the client, then sent to the server through the network, and then decoded by the server. Next, it starts to analyze how the service Provider handles RPC call requests.
​

This paper will analyze from two dimensions. The first is ChannelHandler. Since it is processing network requests, it must process network IO events. From the moment the Provider receives the byte sequence, how to convert them into Request objects and process them step by step. Then there is the Invoker. After obtaining the Request object, how does the Provider execute the local call, and then respond the method result to the client.
​

Remember that when analyzing the Consumer, the proxy object generated by Dubbo is wrapped in layers of invokers, and finally realizes a set of complex functions. For providers, ChannelHandler and Invoker are the same. They have been packaged layer by layer. The responsibilities of the class are very clear, that is, the source code will look a little dizzy.

2. ChannelHandler

ChannelHandler is the interface used by Dubbo to handle network IO events. The corresponding functions of the method are as follows:

method	function
connected	Connection event
disconnected	Disconnect event
sent	Message sending event
received	Message receiving event
caught	Abnormal event

Note that this is the interface defined by Dubbo, not the interface provided by Netty. Don't get confused!

For example, when the Provider receives a message, it triggers the received() method. However, message processing is a big project, including complex logic such as message decoding, heartbeat processing and message distribution. Therefore, all the code will not be written in a ChannelHandler class. Dubbo adopts decorator mode to package channelhandlers layer by layer, and finally realizes this set of complex functions.
​

Dubbo uses Netty as the network transport layer framework by default. Here we also take Netty as an example. Let's take a look at the packaging process of ChannelHandler. As analyzed in the previous article, when the Provider exposes the service, it will create a server-side ExchangeServer. The code is as follows:

ExchangeServer server = Exchangers.bind(url, requestHandler);

This requestHandler is the anonymous internal class object of DubboProtocol. Its responsibility is to match the Invoker according to the Invocation and call, and then return the result. It is the lowest ChannelHandler.
The bind() method code is as follows, which will be wrapped by HeaderExchangeHandler and DecodeHandler.

public ExchangeServer bind(URL url, ExchangeHandler handler) throws RemotingException {
    return new HeaderExchangeServer(Transporters.bind(url, new DecodeHandler(new HeaderExchangeHandler(handler))));
}

Transporters#bind() will eventually create NettyServer, and its constructor will wrap ChannelHandler in three layers.

public NettyServer(URL url, ChannelHandler handler) throws RemotingException {
    super(ExecutorUtil.setThreadName(url, SERVER_THREAD_POOL_NAME), 
          ChannelHandlers.wrap(handler, url));
}

The ChannelHandlers#wrap() method wraps the ChannelHandler through Dispatcher, HeartbeatHandler and MultiMessageHandler, and their functions will be described later.

protected ChannelHandler wrapInternal(ChannelHandler handler, URL url) {
    return new MultiMessageHandler(new HeartbeatHandler(ExtensionLoader.getExtensionLoader(Dispatcher.class)
                                                        .getAdaptiveExtension().dispatch(handler, url)));
}

To sum up, the packaging structure of the final ChannelHandler is as follows, which we analyze one by one.

AbstractPeer#received
>>MultiMessageHandler#received
>>>>HeartbeatHandler#received
>>>>>>AllChannelHandler#received
>>>>>>>>DecodeHandler#received
>>>>>>>>>>HeaderExchangeHandler#received
>>>>>>>>>>>>DubboProtocol.requestHandler#reply

2.1 AbstractPeer

AbstractPeer is the parent class of NettyServer. Why does it exist? It is because when NettyServer arranges the ChannelHandlerPipeline, the tail is NettyServerHandler, which depends on the ChannelHandler, which is the NettyServer itself.
​

NettyServerHandler inherits from ChannelDuplexHandler, which means that it can handle inbound and outbound events at the same time. It is just an empty shell or an adapter. It will trigger ChannelHandler#received() when there are readable events in the Channel.

public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
    NettyChannel channel = NettyChannel.getOrAddChannel(ctx.channel(), url, handler);
    handler.received(channel, msg);
}

At this time, AbstractPeer#received() will be triggered. The logic of the parent class is very simple to ensure that the Channel is handed over to the next Handler for processing without being closed.

public void received(Channel ch, Object msg) throws RemotingException {
    if (closed) {
        return;
    }
    handler.received(ch, msg);
}

2.2 MultiMessageHandler

The ChannelHandler#received() method is designed to receive and process a single message. However, it was mentioned earlier when analyzing the server decoding that the server may receive multiple messages at one time due to network reasons. At this time, Dubbo will create multiple messages to store, and the bottom layer only uses a List to store multiple messages.
It can also be seen from the name that the Handler has the ability to process multiple messages, and the code is also very simple. If it is multiple messages, it will be processed circularly, otherwise it will be processed directly.

public void received(Channel channel, Object message) throws RemotingException {
    if (message instanceof MultiMessage) {
        MultiMessage list = (MultiMessage) message;
        for (Object obj : list) {
            // Multiple messages, loop by step
            handler.received(channel, obj);
        }
    } else {
        handler.received(channel, message);
    }
}

2.3 HeartbeatHandler

It can still be seen from the name that this Handler is specially used to handle heartbeat. If it is a heartbeat request / response, this class will be handled directly without following the follow-up process. If it is an RPC call request, it will be handed over to the next Handler for processing.

public void received(Channel channel, Object message) throws RemotingException {
    setReadTimestamp(channel);
    if (isHeartbeatRequest(message)) {// Heartbeat request
        Request req = (Request) message;
        if (req.isTwoWay()) {// I look forward to receiving your reply
            Response res = new Response(req.getId(), req.getVersion());
            res.setEvent(HEARTBEAT_EVENT);
            channel.send(res);// Direct build Response sending
            if (logger.isInfoEnabled()) {
                int heartbeat = channel.getUrl().getParameter(Constants.HEARTBEAT_KEY, 0);
                if (logger.isDebugEnabled()) {
                    logger.debug("Received heartbeat from remote channel " + channel.getRemoteAddress()
                                 + ", cause: The channel has no data-transmission exceeds a heartbeat period"
                                 + (heartbeat > 0 ? ": " + heartbeat + "ms" : ""));
                }
            }
        }
        return;
    }
    if (isHeartbeatResponse(message)) {// Heartbeat response
        if (logger.isDebugEnabled()) {
            logger.debug("Receive heartbeat response in thread " + Thread.currentThread().getName());
        }
        return;
    }
    handler.received(channel, message);
}

2.4 AllChannelHandler

This class is responsible for sending messages to a specific thread for processing. This thread may be an IO thread or a business thread.
Dubbo refers to the thread receiving requests in the underlying communication framework as IO thread. If the event processing logic is very simple and all are pure memory operations, it can be considered to execute directly in the IO thread to avoid the overhead of thread switching. However, if the event processing logic is complex and involves operations such as database query, it is strongly recommended to send it to the business thread pool for execution, because IO threads are very valuable. Once IO threads are full due to blocking, new requests will not be received!
​

Dubbo supports a variety of thread distribution strategies, as follows:

strategy	explain
all	All messages are sent to the thread pool, including request, response, connection event, disconnection event, etc
direct	All messages are not sent to the thread pool and are executed directly on the IO thread
message	Only request and response messages are sent to the thread pool, and other messages are executed on the IO thread
execution	Only request messages are sent to the thread pool, and other messages are executed on the IO thread
connection	On the IO thread, put the disconnection events into the queue, execute them one by one in order, and send other messages to the thread pool

The default policy is all, and the corresponding class is AllChannelHandler, which will send messages to the business thread pool for execution.

public void received(Channel channel, Object message) throws RemotingException {
    // Get thread pool
    ExecutorService executor = getPreferredExecutorService(message);
    try {
        // Submit asynchronous tasks, process messages
        executor.execute(new ChannelEventRunnable(channel, handler, ChannelState.RECEIVED, message));
    } catch (Throwable t) {}
}

After the ChannelEventRunnable is created, it will be submitted to the thread pool for scheduling execution. Finally, the run() method will be executed. If the state is to receive messages, it will be handed over to the next Handler for processing.

public void run() {
    if (state == ChannelState.RECEIVED) {
        try {
            handler.received(channel, message);
        } catch (Exception e) {
            logger.warn("ChannelEventRunnable handle " + state + " operation error, channel is " + channel
                        + ", message is " + message, e);
        }
    }
}

2.5 DecodeHandler

As can be seen from its name, it is responsible for decoding messages. The decoding here may not be accurate. Strictly speaking, it should be the deserialization of the Body. The byte sequence decoded as a Request object is executed on the IO thread. Dubbo allows you to configure whether the deserialization process is also executed on the IO thread. The default is false. If the IO thread does not complete deserialization, this class will deserialize on the business thread and then hand it over to the next Handler for processing.

public void received(Channel channel, Object message) throws RemotingException {
    if (message instanceof Decodeable) {
        decode(message);
    }
    // Received a request
    if (message instanceof Request) {
        decode(((Request) message).getData());
    }
    // A response was received
    if (message instanceof Response) {
        decode(((Response) message).getResult());
    }
    handler.received(channel, message);
}

2.6 HeaderExchangeHandler

This class does partial preprocessing of the message. For the request object Request, it will determine whether it is an event message, whether the desired response is received, and so on, and then invokes the corresponding method for processing. In the case of RPC requests, the handleRequest() method will eventually be called.

public void received(Channel channel, Object message) throws RemotingException {
    final ExchangeChannel exchangeChannel = HeaderExchangeChannel.getOrAddChannel(channel);
    if (message instanceof Request) {
        // Processing requests
        Request request = (Request) message;
        if (request.isEvent()) {
            // Processing event messages
            handlerEvent(channel, request);
        } else {
            if (request.isTwoWay()) {
                // Expect a response and need a response
                handleRequest(exchangeChannel, request);
            } else {
                // No response result required
                handler.received(exchangeChannel, request.getData());
            }
        }
    }
}

The handleRequest() method will create a Response object, then the last ChannelHandler will process the request and return the result, and finally respond to the client.

void handleRequest(final ExchangeChannel channel, Request req) throws RemotingException {
    Response res = new Response(req.getId(), req.getVersion());
    Object msg = req.getData();
    // The last Handler handles the request and returns the result
    CompletionStage<Object> future = handler.reply(channel, msg);
    future.whenComplete((appResult, t) -> {
        try {
            if (t == null) {
                res.setStatus(Response.OK);
                res.setResult(appResult);
            } else {
                res.setStatus(Response.SERVICE_ERROR);
                res.setErrorMessage(StringUtils.toString(t));
            }
            // Response results
            channel.send(res);
        } catch (RemotingException e) {
            logger.warn("Send result to consumer failed, channel is " + channel + ", msg is " + e);
        }
    });
}

2.7 DubboProtocol internal class

DubboProtocol has an anonymous internal class object of ExchangeHandler, which is specially used to process the RPC call request of dubbo protocol. The method is reply(), and the logic is very simple. The parameter Invocation has told us which method of which Service to call. We just need to match the corresponding Invoker, execute the invoke local call, and return the result.

public CompletableFuture<Object> reply(ExchangeChannel channel, Object message) throws RemotingException {
    Invocation inv = (Invocation) message;
    // Get the corresponding Invoker from exporterMap
    Invoker<?> invoker = getInvoker(channel, inv);
    // Remote address
    RpcContext.getContext().setRemoteAddress(channel.getRemoteAddress());
    // Execute local call
    Result result = invoker.invoke(inv);
    // Return results
    return result.thenApply(Function.identity());
}

At this point, all ChannelHandler processes are completed.

3. Invoker

The ChannelHandler finally matches the invoker according to the request parameter Invocation, and then starts to execute the local call to obtain the result. Invoker has also been encapsulated layer by layer, but don't worry. Most of Dubbo's logic is implemented on the client side, and the invoker on the Provider side is not complex.
​

The Invoker packaging structure on the Provider side is summarized as follows:

ProtocolFilterWrapper Perform various Filter...
>>DelegateProviderMetaDataInvoker
>>>>AbstractProxyInvoker
>>>>>>JavassistProxyFactory Proxy object

3.1 ProtocolFilterWrapper

This class is a wrapper class. It will wrap a layer of Filter chain FilterChain for the Invoker object. The method is buildInvokerChain(). Use SPI to load all activated filters, and then string them into a one-way linked list. The Invoker is placed at the end. The final invoke method can be executed only after passing through all the previous filters.

After all filters are executed in turn, they will eventually be handed over to the DelegateProviderMetaDataInvoker for execution.

3.2 DelegateProviderMetaDataInvoker

It is a pure wrapper class without any logic. It only holds metadata service objects on the basis of referencing Invoker.

public class DelegateProviderMetaDataInvoker<T> implements Invoker {
    protected final Invoker<T> invoker;
    private ServiceConfig<?> metadata;
    
    public Result invoke(Invocation invocation) throws RpcException {
        return invoker.invoke(invocation);
    }
}

3.3 AbstractProxyInvoker

Proxy Invoker's parent class. It will call the abstract method doInvoke() to initiate a local call. After getting the result, it will be encapsulated into an AppResponse and returned

public Result invoke(Invocation invocation) throws RpcException {
    Object value = doInvoke(proxy, invocation.getMethodName(), invocation.getParameterTypes(), invocation.getArguments());
    CompletableFuture<Object> future = wrapWithFuture(value);
    CompletableFuture<AppResponse> appResponseFuture = future.handle((obj, t) -> {
        AppResponse result = new AppResponse();
        result.setValue(obj);
        return result;
    });
    return new AsyncRpcResult(appResponseFuture, invocation);
}

doInvoke() is implemented by subclasses. There are two ways to call locally. One is because of Java reflection, and the other is that classes dynamically generated by bytecode technology will make direct method calls. In terms of performance, the latter is better. Dubbo uses the latter by default, and the proxy class is generated by JavassistProxyFactory class.

public <T> Invoker<T> getInvoker(T proxy, Class<T> type, URL url) {
    // Improve reflection efficiency
    final Wrapper wrapper = Wrapper.getWrapper(proxy.getClass().getName().indexOf('$') < 0 ? proxy.getClass() : type);
    return new AbstractProxyInvoker<T>(proxy, type, url) {
        @Override
        protected Object doInvoke(T proxy, String methodName,
                                  Class<?>[] parameterTypes,
                                  Object[] arguments) throws Throwable {
            // The dynamically generated Wrapper object makes direct method calls at the bytecode level
            return wrapper.invokeMethod(proxy, methodName, parameterTypes, arguments);
        }
    };
}

4. Summary

When a Provider processes an RPC call request from a Consumer, it needs to go through two important classes: ChannelHandler and Invoker.
ChannelHandler is used to handle network IO events. The received() method is used to handle received messages. Receiving messages is a big project, which requires decoding, multi message processing, message distribution and other operations. It is impossible to write all codes in one class. Therefore, Dubbo uses decorator mode to wrap ChannelHandler layer by layer, and each packaging class performs its own duties, Finally, a complete set of complex processes is realized. For RPC requests, ChannelHandler will eventually match the Invoker according to the Invocation parameter, initiate a local call, and then respond to the result.
Invoker has also been packaged layer by layer, but the logic is not complex. The core is ProtocolFilterWrapper, which is used to execute Filter logic. Finally, it performs local calls according to the proxy object. The methods include Java reflection and bytecode technology to generate proxy object, and direct method calls at bytecode level.