Codec

Each network application must define how to parse the original bytes transmitted back and forth between two nodes and how to convert them to and from the data format of the target application. This conversion logic is processed by a codec, which consists of an encoder and a decoder, each of which can convert a byte stream from one format to another

• The encoder converts the message into a format suitable for transmission (most likely a byte stream)
• The decoder converts the network byte stream back to the message format of the application

Therefore, the encoder operates on outbound data while the decoder processes inbound data

1. Decoder

In this section, we will study the decoder classes provided by Netty and provide specific examples of when and how to use them. These classes cover two different use cases:

• Decode bytes into messages -- ByteToMessageDecoder and ReplayingDecoder
• Decode one message type into another -- MessageToMessageDecoder

When will the decoder be used? Very simple, it is used whenever you need to convert inbound data for the next ChannelInboundHandler in the ChannelPipeline. In addition, thanks to the design of ChannelPipeline, multiple decoders can be linked together to realize any complex conversion logic

1.1 abstract class ByteToMessageDecoder

Decoding bytes into messages is a common task. Netty provides an abstract base class ByteToMessageDecoder, which buffers inbound data until it is ready for processing

The following is an example of how to use this class. Suppose you receive a byte stream containing a simple int, and each int needs to be processed separately. In this case, you need to read each int from the inbound ByteBuf and pass it to the next ChannelInboundHandler in the ChannelPipeline. In order to decode this byte stream, you need to extend the ByteToMessageDecoder class (note that when an int of atomic type is added to the List, it will be automatically boxed as Integer)

// Extend ByteToMessageDecoder to decode bytes into a specific format
public class ToIntegerDecoder extends ByteToMessageDecoder {
    @Override
    protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {
        //Check if there are at least 4 bytes readable (byte length of 1 int)
        if (in.readableBytes() >= 4) {
            //Read an int from the inbound ByteBuf and add it to the List of decoded messages
            out.add(in.readInt());
        }
    }
}

Although ByteToMessageDecoder makes it easy to implement this pattern, you may find it a little cumbersome to have to verify whether the input ByteBuf has enough data before calling the readInt() method. The replaying decoder, which is a special decoder, eliminates this step with a small amount of overhead

1.2 abstract class ReplayingDecoder

ReplayingDecoder extends the ByteToMessageDecoder class so that we don't have to call the readableBytes() method. It does this by using a custom ByteBuf implementation, ReplayingDecoderByteBuf, to wrap the incoming ByteBuf, which will execute the call internally

The complete declaration of this class is:

public abstract class ReplayingDecoder<S> extends ByteToMessageDecoder

The type parameter S specifies the type used for state management, where Void represents that state management is not required. The following code shows the ToIntegerDecoder re implemented based on ReplayingDecoder

// Extend replayingdecoder < void > to decode bytes into messages
public class ToIntegerDecoder2 extends ReplayingDecoder<Void> {
    // The ByteBuf passed in is ReplayingDecoderByteBuf
    @Override
    protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {
        // Read an int from the inbound ByteBuf and add it to the List of decoded messages
        out.add(in.readInt());
    }
}

As before, the int extracted from ByteBuf will be added to the List. If not enough bytes are available, the implementation of the readInt() method will throw an Error, which will be captured and processed in the base class. When more data is available for reading, the decode() method will be called again

Note the following aspects of ReplayingDecoderByteBuf:

• Not all ByteBuf operations are supported. If an unsupported method is called, an unsupported operationexception will be thrown
• ReplayingDecoder is slightly slower than ByteToMessageDecoder

The following classes are used to handle more complex use cases:

• io.netty.handler.codec.LineBasedFrameDecoder - this class is also used inside Netty. It uses end of line control characters (\ n or \ r\n) to parse message data
• io.netty.handler.codec.http.HttpObjectDecoder -- HTTP data decoder

1.3 abstract class MessageToMessageDecoder

In this section, we will explain how to convert between two message formats, for example, from one POJO type to another

public abstract class MessageToMessageDecoder<I> extends ChannelInboundHandlerAdapter

Parameter type I specifies the type of input parameter msg of the decode() method, which is the only method you must implement

We will write an IntegerToStringDecoder decoder to extend MessageToMessageDecoder. Its decode() method will convert Integer parameters to String representation. As before, the decoded String will be added to the outgoing List and forwarded to the next ChannelInboundHandler

public class IntegerToStringDecoder extends MessageToMessageEncoder<Integer> {
    @Override
    protected void encode(ChannelHandlerContext ctx, Integer msg, List<Object> out) throws Exception {
        //Converts the Integer message to its String representation and adds it to the output List
        out.add(String.valueOf(msg));
    }
}

1.4 TooLongFrameException

Since Netty is an asynchronous framework, you need to buffer bytes in memory before they can be decoded. Therefore, the decoder cannot buffer a large amount of data so that the available memory is exhausted. To address this common concern, Netty provides the TooLongFrameException class, which will be thrown by the decoder when the frame exceeds the specified size limit

To avoid this, you can set a threshold for the maximum number of bytes. If the threshold is exceeded, a TooLongFrameException will be thrown (which will then be captured by the channelhandler. Exceptionguess() method). Then, how to handle the exception depends entirely on the user of the decoder. Some protocols (such as HTTP) may allow you to return a special response. In other cases, the only option may be to close the corresponding connection

The following example uses TooLongFrameException to notify other channelhandlers in ChannelPipeline of frame size overflow. It should be noted that this protection is particularly important if you are using a variable frame size protocol

public class SafeByteToMessageDecoder extends ByteToMessageDecoder {
    public static final int MAX_FRAME_SIZE = 1024;
    @Override
    protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {
        int readable = in.readableBytes();
        // Check whether the buffer exceeds MAX_FRAME_SIZE bytes
        if (readable > MAX_FRAME_SIZE) {
            // Skip all readable bytes, throw TooLongFrameException and notify ChannelHandler
            in.skipBytes(readable);
            throw new TooLongFrameException("Frame too big!");
        }
        //do something
    }
}

2. Encoder

The encoder implements ChannelOutboundHandler and converts outbound data from one format to another, which is just opposite to the function of the decoder we just learned. Netty provides a set of classes to help you write encoders with the following functions:

• Encode the message into bytes
• Encode messages as messages

2.1 abstract class MessageToByteEncoder

Previously, we saw how to use ByteToMessageDecoder to convert bytes into messages. Now we use MessageToByteEncoder to do the reverse

This class has only one method, while the decoder has two. The reason is that the decoder usually needs to generate the last message after the Channel is closed (therefore, there is a decodeLast() method. Obviously, this does not apply to the encoder scenario - it is meaningless to generate a message after the connection is closed

The following code shows ShortToByteEncoder, which accepts an instance of Short type as a message, encodes it as the atomic type value of Short, writes it into ByteBuf, and then forwards it to the next ChannelOutboundHandler in ChannelPipeline. Each outgoing Short value will occupy 2 bytes in ByteBuf.

public class ShortToByteEncoder extends MessageToByteEncoder<Short> {
    @Override
    protected void encode(ChannelHandlerContext ctx, Short msg, ByteBuf out) throws Exception {
        // Write Short to ByteBuf
        out.writeShort(msg);
    }
}

2.2 abstract class MessageToMessageEncoder

The encode() method of the MessageToMessageEncoder class provides the ability to decode inbound data from one message format to another

The following code extends MessageToMessageEncoder with IntegerToStringEncoder, which adds the String representation of each outbound Integer to the List

public class IntegerToStringEncoder extends MessageToMessageEncoder {
    @Override
    protected void encode(ChannelHandlerContext ctx, Object msg, List out) throws Exception {
        out.add(String.valueOf(msg));
    }
}

Abstract codec class

Although we have been discussing decoders and encoders as separate entities, you will sometimes find it useful to manage the conversion of inbound and outbound data and messages in the same class. Netty's abstract codec classes are just used for this purpose, because each of them will be bundled with a decoder / encoder pair to handle the two types of operations we have been learning. As you might have guessed, these classes implement both ChannelInboundHandler and ChannelOutboundHandler interfaces

Why don't we always use these composite classes before separate decoders and encoders? Because by separating the two functions as much as possible, it maximizes the reusability and scalability of the code, which is a basic principle of Netty design

1. Abstract class ByteToMessageCodec

Let's study a scenario where we need to decode bytes into some form of message, possibly POJO, and then encode it again. ByteToMessageCodec will handle all this for us because it combines ByteToMessageDecoder and its reverse -- MessageToByteEncoder

Any request / response protocol can be an ideal choice for using ByteToMessageCodec. For example, in an SMTP implementation, the codec will read the incoming bytes and decode them into a custom message type, such as SmtpRequest. At the receiving end, when a response is created, a SmtpResponse will be generated, which will be encoded back to bytes for transmission

2. Abstract class MessageToMessageCodec

By using MessageToMessageCodec, we can implement the round trip process of this transformation in a single class. MessageToMessageCodec is a parameterized class, which is defined as follows:

public abstract class MessageToMessageCodec<INBOUND_IN,OUTBOUND_IN>

The decode() method is to add inbound_ A message of type in is converted to outbound_ The in type message, while the encode() method performs its reverse operation. Add inbound_ Messages of type in are regarded as types sent over the network, while messages of type outbound_ It may be helpful to think of messages of type in as the type that the application is processing

WebSocket protocol

The following example of MessageToMessageCodec refers to a new WebSocket protocol, which can realize full two-way communication between Web browser and server

Our WebSocketConvertHandler will use inbound when parameterizing MessageToMessageCodec_ WebSocketFrame of type in and outbound_ MyWebSocketFrame of type in, which is a static nested class of WebSocketConvertHandler itself

public class WebSocketConvertHandler 
        extends MessageToMessageCodec<WebSocketFrame, WebSocketConvertHandler.MyWebSocketFrame> {


    @Override
    protected void encode(ChannelHandlerContext ctx, MyWebSocketFrame msg, List<Object> out) throws Exception {
        // Instantiate a WebSocketFrame of the specified subtype
        ByteBuf payload = msg.getData().duplicate().retain();
        switch (msg.getType()) {
            case BINARY:
                out.add(new BinaryWebSocketFrame(payload));
                break;
            case TEXT:
                out.add(new TextWebSocketFrame(payload));
                break;
            case CLOSE:
                out.add(new CloseWebSocketFrame(true, 0, payload));
                break;
            case CONTINUATION:
                out.add(new ContinuationWebSocketFrame(payload));
                break;
            case PONG:
                out.add(new PongWebSocketFrame(payload));
                break;
            case PING:
                out.add(new PingWebSocketFrame(payload));
                break;
            default:
                throw new IllegalStateException("Unsupported websocket msg " + msg);
        }
    }

    // Decode WebSocketFrame into MyWebSocketFrame and set FrameType
    @Override
    protected void decode(ChannelHandlerContext ctx, WebSocketFrame msg, List<Object> out) throws Exception {
        ByteBuf paload = msg.content().duplicate().retain();
        if (msg instanceof  BinaryWebSocketFrame) {
            out.add(new MyWebSocketFrame(MyWebSocketFrame.FrameType.BINARY, paload));
        } else
        if (msg instanceof  CloseWebSocketFrame) {
            out.add(new MyWebSocketFrame(MyWebSocketFrame.FrameType.CLOSE, paload));
        } else
        if (msg instanceof  PingWebSocketFrame) {
            out.add(new MyWebSocketFrame(MyWebSocketFrame.FrameType.PING, paload));
        } else
        if (msg instanceof  PongWebSocketFrame) {
            out.add(new MyWebSocketFrame(MyWebSocketFrame.FrameType.PONG, paload));
        } else
        if (msg instanceof  TextWebSocketFrame) {
            out.add(new MyWebSocketFrame(MyWebSocketFrame.FrameType.TEXT, paload));
        } else
        if (msg instanceof  ContinuationWebSocketFrame) {
            out.add(new MyWebSocketFrame(MyWebSocketFrame.FrameType.CONTINUATION, paload));
        } else {
            throw new IllegalStateException("Unsupported websocket msg " + msg);
        }
    }

    public static final class MyWebSocketFrame {
        public enum FrameType {
            BINARY,
            CLOSE,
            PING,
            PONG,
            TEXT,
            CONTINUATION
        }
        private final FrameType type;
        private final ByteBuf data;

        public MyWebSocketFrame(FrameType type, ByteBuf data) {
            this.type = type;
            this.data = data;
        }

        public FrameType getType() {
            return type;
        }

        public ByteBuf getData() {
            return data;
        }
    }
}

3. CombinedChannelDuplexHandler class

As we mentioned earlier, combining a decoder and encoder can have an impact on reusability. However, there is a way to avoid this penalty without sacrificing the convenience of deploying a decoder and an encoder as a separate unit. CombinedChannelDuplexHandler provides this solution, which is declared as:

public class CombinedChannelDuplexHandler
	<I extends ChannelInboundHandler, O extends ChannelOutboundHandler>

This class acts as a container for ChannelInboundHandler and ChannelOutboundHandler (the type parameters I and O of this class). By providing types that inherit the decoder class and encoder class respectively, we can implement a codec without directly extending the abstract codec class

First, let's look at the following code, which extends ByteToMessageDecoder because it reads characters from ByteBuf

public class ByteToCharDecoder extends ByteToMessageDecoder {
    @Override
    protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) throws Exception {
        while (in.readableBytes() >= 2) {
            out.add(in.readChar());
        }
    }
}

The decode() method here will extract 2 bytes from ByteBuf at a time and write them to the List as characters, which will be automatically boxed into Character objects

The following code converts Character back to bytes. This class extends MessageToByteEncoder because it needs to encode char messages into ByteBuf. This is done by writing directly to ByteBuf

public class CharToByteEncoder extends MessageToByteEncoder<Character> {
    @Override
    protected void encode(ChannelHandlerContext ctx, Character msg, ByteBuf out) throws Exception {
        out.writeChar(msg);
    }
}

Now that we have a decoder and an encoder, we can combine them to build a codec

// The parameterized CombinedByteCharCodec is realized through the decoder and encoder
public class CombinedChannelDuplexHandler extends
        io.netty.channel.CombinedChannelDuplexHandler<ByteToCharDecoder, CharToByteEncoder> {
    public CombinedChannelDuplexHandler() {
        // Pass the delegate instance to the parent class
        super(new ByteToCharDecoder(), new CharToByteEncoder());
    }
}