1, Demand analysis

At present, there are a lot of information about the principle and mechanism of KCP and the implementation of various versions of KCP on the Internet. I did the KCP correlation analysis of two articles before, which are the principle mechanism and performance testing practice.
Our current project is a game with high real-time requirements. Theoretically, according to the traditional real-time game practice, the performance of TCP is also sufficient. However, in order to pursue better effect and smoother experience, we decided to use KCP as network layer communication in combat.

According to the investigation and performance test, the reasons are summarized as follows:

• When the network environment is poor, TCP has a high packet loss rate and is very unstable; KCP performs very stably in the internal and external network environment.
• Unreasonable RTO delay calculation of TCP will cause too long retransmission time; KCP has a more friendly algorithm design for retransmission and RTO delay calculation.
• TCP is a protocol designed to control network bandwidth, but in the current network environment, bandwidth is not particularly important; KCP is designed to control flow rate.

2, First edition implementation

At the beginning, we analyzed the open source java version implementation library of GitHub boss( https://github.com/l42111996/java-Kcp )In theory, it is completely sufficient. This version of the open source library optimizes the flush strategy of KCP messages on the basis of the original C, combined with the event driven of Netty in Java and the utilization of multi-core CPU

    /**
     * Execute flush
     * flush strategy
     * 1,Check the buffer after the send call. If you can send, directly call update to get the time and store it in uktucp
     * 2,When the scheduled task reaches the time to check uktucp and its own timing, if it can be sent, it will be sent directly. If the time is delayed, it will be re timed. After the scheduled task is sent successfully, it will detect whether the buffer triggers the sending time
     * 3,After the read event is triggered, the detection buffer triggers the write event
     */

1. Design and Implementation

From the perspective of source code, the author is very familiar with Netty and KCP, and has made a good integration of the two. According to the author's document, this version is verified by 5 online projects in Tencent.
Therefore, after I have a general understanding of this framework, I will combine the original author's suggestions on the use of KCP in version C( https://github.com/skywind3000/kcp/wiki/Cooperate-With-Tcp-Server ), the network layer is transformed in the first edition, and the following design is made:

Retain the original TCP channel, add a KCP/UDP channel, and do network switching logic in the game logic layer. When we detect which channel the client sends messages through, we switch the player network to the current channel.

Based on this implementation, we can realize the following functions in the project:

• Using the security and reliability of TCP, the TCP channel is used to handle the login of players in the battle suit, as well as the process of creating, entering, clearing and exiting the battle scene
• Taking advantage of the network stability of KCP and the low delay characteristics in poor network environment, the KCP/UDP channel is used to process the status synchronization messages of players in combat (our combat is status synchronization, but it can also be used in the project of frame synchronization. This is not important, it can be understood as the communication messages in combat)
• In some networks where UDP packets are unreachable, the network channel can be flexibly switched in battle and returned to the TCP standby channel

2. Problem analysis

In the intra battle communication in the case of single network, no matter the player uses TCP or KCP/UDP channel, there is no problem.
However, we soon found some disadvantages of this mode in multi network handoff. Once some boundary conditions of network handoff are involved, problems may occur

As shown in the above sequence diagram:

• During the 2.1 operation, the client found that the TCP network was blocked, and the ack packet of 2.2 and the status synchronization packet of 2.3 were blocked in the network and returned after a long time
  To the client (or the operation package of the client may be blocked when it is sent to the server. Any package in 2.1, 2.2 and 2.3 in the figure may be blocked)
• At this time, the client switches to the KCP network and then sends the subsequent player operations. The combat state of the server may have changed a lot.
• After a long time, the client received a status synchronization package 2.3 from the server, telling you that the strange blood deduction is 100, but it is likely that the strange has been killed long ago.
  Question: will the client process or not process this status synchronization packet.

3. Solutions

Of course, based on this communication architecture, the solutions also include:

1. Add a network packet serial number to each message packet. Packets lower than the current serial number will not be processed, but this method can only be applied to state synchronization games. State synchronization allows packet loss, as long as the final state of players is consistent; However, if it is a frame synchronization game, it will be a little headache. Packet loss is absolutely not allowed in the frame synchronization game, because the client will perform logical operation according to the frame data, rather than a simple synchronization state, so the frame synchronization game must need a perfect packet sequence management mechanism.
2. Add a timestamp to each message packet. The principle is the same as adding the packet sequence number, and it can only be applied to state synchronization games.
3. Whether it is TCP or KCP network, the server makes a reply mechanism for each packet of the client. The client needs to know whether its message has been received by the server. Because the application layer cannot perceive that the network bottom layer receives ACK packets, it is necessary to implement a set of ACK like mechanism in the application layer.

Of the three schemes, it is obvious that the first two are simpler to implement, but it is only conceivable that there will be problems in network switching. In practice, there may be more problems.

Since the problem lies in network switching and packet sequence management, why not deal with network message packets at a unified entrance and exit?

3, Multi channel network implementation

This dividing line means that the following contents begin to get to the point

After some discussion with the bosses, they decided to use a more radical but once and for all approach. That is, the communication architecture takes TCP and UDP as dual channel network communication and KCP as unified packet management model.

It took about a week. I'm based on This open source library A set of dual channel network layer with KCP as the application layer and TCP and UDP as the underlying communication protocol is realized. Under such an architecture, the sequence, fragmentation, window size, flow control, etc. of message packets are completely left to the KCP, while the underlying network thinks of what it wants to use. Because the management of network messages is uniformly handed over to KCP, it no longer has the problem of switching between networks.

In order to make it more flexible and easy to expand, I Abstract This set of network layer into an open source library and modify it to support multi-channel network at the lower layer. For convenience, I modify more parameters in the interface of the original framework to be configurable, which is not only convenient for myself, but also open to everyone, Make this framework open the configuration of KCP application layer and underlying network communication to the greatest extent.

The framework has released release 1 Version 1. github and maven central warehouse are uploaded. You can import and use them through maven according to my instructions
github: https://github.com/hjcenry/ktucp-netty
You are welcome to contribute a little star. Opening it is not only convenient for you, but also convenient for you. If you have any problems in the use process, you can mention issues in github, and I will solve them.

4, Ktucp netty

The name is simple and crude. Because the communication architecture is based on kcp/tcp/udp, it is simply three in one ktucp. The suffix netty represents that the whole framework is implemented based on netty

The following content is extracted from README in github project and gives a brief introduction to the architecture and use of this framework.

Modification of open source project based on original author: https://github.com/l42111996/java-Kcp.git

Original project:
Communication architecture

application layer <--> UDP <--> KCP

Realize function

• Basic implementation of kcp in java
• Optimize the flush strategy of kcp
• Based on event driven, using multi-core performance
• Multiple kcp parameters can be configured
• It supports configuring conv or address(ip+port) to determine the unique connection
• Support fec (reduce latency)
• Support crc32 verification

New addition and Optimization Based on the original project:
Communication architecture

 application layer
  ┌┴┐          
UDP TCP  ...(N Networks)
  └┬┘
  KCP

Optimization and addition

• Supports configuring multiple TCP/UDP underlying network services
• Supports TCP and UDP channel switching
• Support custom configuration of Netty parameters of the underlying network
• Supports adding custom handlers for the underlying network
• Support custom codec
• It supports switching the network at the lower layer of the KCP
• It supports mandatory use of a network to send data
• Support the use of custom time service (you can use your own system's cache time system instead of the System.currentTimeMillis method)

5, Why use multiple networks

According to the original author's suggestions on the use of KCP( https://github.com/skywind3000/kcp/wiki/Cooperate-With-Tcp-Server )
In actual use, it is best to use TCP and UDP:

1. The domestic network situation is special, and UDP packets may be blocked by the firewall
2. When LB is used in TCP network, one end of the two ends may not perceive the disconnection of the other end
3. The reliable connection through TCP can be used as the standby line, and the standby TCP can be used when UDP is not available

Combined with the above requirements, the purpose of this set of open source library is to integrate TCP and UDP networks into the same KCP mechanism, and even support the start of multi TCP and multi UDP services.
And open the underlying Netty configuration permissions to the greatest extent, and users can customize their own network framework according to their own needs

You are welcome to use it. If there are any bug s and optimization requirements, you are welcome to mention the issue for discussion

6, Quick start

Well, there's no more nonsense. Let's get started and see how it works

maven address

<dependency>
    <groupId>io.github.hjcenry</groupId>
    <artifactId>ktucp-net</artifactId>
    <version>1.1</version>
</dependency>

Server

1. Create ChannelConfig

ChannelConfig channelConfig = new ChannelConfig();
channelConfig.nodelay(true, 40, 2, true);
channelConfig.setSndWnd(512);
channelConfig.setRcvWnd(512);
channelConfig.setMtu(512);
channelConfig.setTimeoutMillis(10000);
channelConfig.setUseConvChannel(true);
// Most parameters can be configured here
// ...

2. Create KtucpListener to listen to network events

KtucpListener ktucpListener = new KtucpListener() {
    @Override
    public void onConnected(int netId, Uktucp uktucp) {
        System.out.println("onConnected:" + uktucp);
    }

    @Override
    public void handleReceive(Object object, Uktucp uktucp) throws Exception {
        System.out.println("handleReceive:" + uktucp);
        ByteBuf byteBuf = (ByteBuf) object;
        // TODO read byteBuf
    }

    @Override
    public void handleException(Throwable ex, Uktucp uktucp) {
        System.out.println("handleException:" + uktucp);
        ex.printStackTrace();
    }

    @Override
    public void handleClose(Uktucp uktucp) {
        System.out.println("handleClose:" + uktucp);
        System.out.println("snmp:" + uktucp.getSnmp());
    }

    @Override
    public void handleIdleTimeout(Uktucp uktucp) {
        System.out.println("handleIdleTimeout:" + uktucp);
    }
};

3. Create and start KtucpServer

KtucpServer ktucpServer = new KtucpServer();
// A UDP port is started by default
ktucpServer.init(ktucpListener, channelConfig, 8888);

4. Observation log

[main] INFO com.hjcenry.log.KtucpLog - KtucpServer Start :
===========================================================
TcpNetServer{bindPort=8888, bossGroup.num=1, ioGroup.num=8}
UdpNetServer{bindPort=8888, bossGroup.num=8, ioGroup.num=0}
===========================================================

client

1. Create ChannelConfig

ChannelConfig channelConfig = new ChannelConfig();
// The client has one more convId than the server
channelConfig.setConv(1);
channelConfig.nodelay(true, 40, 2, true);
channelConfig.setSndWnd(512);
channelConfig.setRcvWnd(512);
channelConfig.setMtu(512);
channelConfig.setTimeoutMillis(10000);
channelConfig.setUseConvChannel(true);
// Most parameters can be configured here
// ...

2. Create KtucpListener to listen to network events

KtucpListener ktucpListener = new KtucpListener() {
    @Override
    public void onConnected(int netId, Uktucp uktucp) {
        System.out.println("onConnected:" + uktucp);
    }

    @Override
    public void handleReceive(Object object, Uktucp uktucp) throws Exception {
        System.out.println("handleReceive:" + uktucp);
        ByteBuf byteBuf = (ByteBuf) object;
        // TODO read byteBuf
    }

    @Override
    public void handleException(Throwable ex, Uktucp uktucp) {
        System.out.println("handleException:" + uktucp);
        ex.printStackTrace();
    }

    @Override
    public void handleClose(Uktucp uktucp) {
        System.out.println("handleClose:" + uktucp);
        System.out.println("snmp:" + uktucp.getSnmp());
    }

    @Override
    public void handleIdleTimeout(Uktucp uktucp) {
        System.out.println("handleIdleTimeout:" + uktucp);
    }
};

3. Create and start KtucpClient

// A UDP port is started by default
KtucpClient ktucpClient = new KtucpClient();
ktucpClient.init(ktucpListener, channelConfig, new InetSocketAddress("127.0.0.1", 8888));

4. Observation log

[main] INFO com.hjcenry.log.KtucpLog - KtucpClient Connect :
===========================================================
TcpNetClient{connect= local:null -> remote:/127.0.0.1:8888, ioGroup.num=8}
UdpNetClient{connect= local:null -> remote:/127.0.0.1:8888, ioGroup.num=0}
===========================================================

The above is a simple example to quickly start ktucp services and clients. For detailed usage of multi network, please refer to examples 3 and 4 below

7, Use attention

• Client corresponding implementation: the framework only implements the Java version, and other versions of clients need to be implemented according to this communication architecture (if UDP channel is used only, it can also be compatible with the original KCP)
• Uniqueness of convId: because the udp address or tcp channel cannot be verified, only convId can be used to obtain the unique uktup object
• Validity verification of convId: it is necessary to judge the source of convId to prevent forgery. Because convId is read from the message packet, the bottom layer of the framework judges the Channel uniqueness of the message packet of TCP connection, but UDP has no good judgment method for the time being. If there are security requirements, the application layer needs to do its own anti-counterfeiting detection. For example, the server assigns a token to the client, the client brings the token in each message packet header, and the server verifies the token in each packet header
• Handle a lot of network connection management: because the underlying configuration is relatively open, the default is KCP timeout, that is, disconnect all connections. If there are other configurations, please pay attention to the connection release time

8, Usage and examples

Some directly cite the examples of the original author

1. server side example
2. client side instance
3. Multi network server example
4. Multi network client instance
5. Best practices
6. A lot of information
7. C# compatible server
8. C# client
9. Compatible with KCP go

9, Relevant information

1. https://github.com/skywind3000/kcp Original c version of KCP
2. https://github.com/xtaci/kcp-go go version KCP, with a lot of optimization
3. https://github.com/Backblaze/JavaReedSolomon java version fec
4. https://github.com/LMAX-Exchange/disruptor High performance inter thread messaging Library
5. https://github.com/JCTools/JCTools High performance concurrent Library
6. https://github.com/szhnet/kcp-netty A KCP of java version
7. https://github.com/l42111996/csharp-kcp dotNetty based c# version KCP, perfect compatibility
8. https://github.com/l42111996/java-Kcp.git The original version of this open source library

10, Follow up plan

The length is a little too long, and the rest is included in the plan

• Do a performance test and analysis of network handoff for the framework
• Write a detailed wiki document in github, and present all configurable parameters and call interfaces in a document situation
• Based on this framework, some network related examples are implemented, such as rpc call, game server and so on
• If you have time, you can consider implementing clients in other languages (or see if you have any requirements in this regard)
• Do you have any better ideas or optimization schemes for this framework? Welcome to make comments or join in the development

Wechat: hjcenry welcome to exchange and discuss