In article 08, I would like to praise the UDP communication function of Nacos

Posted by jijodasgupta on Tue, 07 Dec 2021 21:35:42 +0100

Learning is not so utilitarian. The second senior brother will take you to easily read the source code from a higher dimension ~

Nacos uses UDP communication mode in the service registration function. Its main function is to assist in notifying the client when the service instance changes. However, for most programmers using Nacos, they may not know this function, let alone use it flexibly.

Looking at the implementation of the complete source code, I still want to praise this function. It can be said that it is very ingenious and practical. However, there are some deficiencies in the implementation, which will be pointed out at the end of the paper.

This article will take you from the source level to analyze how Nacos 2.0 realizes the notification of service instance change based on UDP protocol.

UDP notification Fundamentals

Before analyzing the source code, let's take a look at the implementation principle of UDP in Nacos as a whole.

As we know, UDP protocol communication is bidirectional, and there is no so-called client and server. Therefore, UDP monitoring will be enabled on both the client and server. The client starts a separate thread to process UDP messages. When communicating with the registry using HTTP protocol, when the client calls the service subscription interface, the UPD information (IP and port) of the client will be sent to the registry, which is encapsulated and stored with PushClient object.

When there is an instance change in the registry, a ServiceChangeEvent event will be published. After listening to this event, the registry will traverse the stored PushClient and notify the client based on UDP protocol. When the client receives the UDP notification, it can update the instance list of the local cache.

As we know earlier, there will be a time difference in instance update when registering a service based on HTTP protocol, because the client regularly pulls the instance list in the server. If the pull is too frequent, the pressure on the registration center is relatively large. If the pull cycle is relatively long, the changes of instances cannot be quickly perceived. The notification of UDP protocol just makes up for this disadvantage. Therefore, we should like the function of UDP based notification.

Let's take a look at how the source level is implemented.

Client UDP notification listening and processing

When the client instantiates NamingHttpClientProxy, PushReceiver will be initialized in its constructor.

public NamingHttpClientProxy(String namespaceId, SecurityProxy securityProxy, ServerListManager serverListManager,
        Properties properties, ServiceInfoHolder serviceInfoHolder) {
    // ...
    // Build BeatReactor
    this.beatReactor = new BeatReactor(this, properties);
    // Build UDP port listening
    this.pushReceiver = new PushReceiver(serviceInfoHolder);
    // ...
}

The construction method of PushReceiver is as follows:

public PushReceiver(ServiceInfoHolder serviceInfoHolder) {
    try {
        // Hold ServiceInfoHolder reference
        this.serviceInfoHolder = serviceInfoHolder;
        // Get UDP port
        String udpPort = getPushReceiverUdpPort();
        // Build a datagram socket according to the port. If no port is set, the random port is used
        if (StringUtils.isEmpty(udpPort)) {
            this.udpSocket = new DatagramSocket();
        } else {
            this.udpSocket = new DatagramSocket(new InetSocketAddress(Integer.parseInt(udpPort)));
        }
        // Create a ScheduledExecutorService with only one thread
        this.executorService = new ScheduledThreadPoolExecutor(1, new ThreadFactory() {
            @Override
            public Thread newThread(Runnable r) {
                Thread thread = new Thread(r);
                thread.setDaemon(true);
                thread.setName("com.alibaba.nacos.naming.push.receiver");
                return thread;
            }
        });

        // The PushReceiver implements the Runnable interface
        this.executorService.execute(this);
    } catch (Exception e) {
        NAMING_LOGGER.error("[NA] init udp socket failed", e);
    }
}

The construction method of PushReceiver does the following operations:

  • First, hold the ServiceInfoHolder object reference;
  • Second, obtain the UDP port;
  • Third, instantiate the DatagramSocket object to send and receive Socket data;
  • Fourth, create a thread pool and execute PushReceiver (implement the Runnable interface);

Since PushReceiver implements the Runnable interface, the run method must be re implemented:

@Override
public void run() {
    while (!closed) {
        try {
            
            // byte[] is initialized with 0 full filled by default
            byte[] buffer = new byte[UDP_MSS];
            // Create datagram packet to store the received message
            DatagramPacket packet = new DatagramPacket(buffer, buffer.length);
            // When receiving a message, thread blocking will occur when the message is not received
            udpSocket.receive(packet);
            // Convert message to json format
            String json = new String(IoUtils.tryDecompress(packet.getData()), UTF_8).trim();
            NAMING_LOGGER.info("received push data: " + json + " from " + packet.getAddress().toString());
            // Convert message in json format into PushPacket object
            PushPacket pushPacket = JacksonUtils.toObj(json, PushPacket.class);
            String ack;
            // If the conditions are met, call ServiceInfoHolder to process the received message and return the response message
            if (PUSH_PACKAGE_TYPE_DOM.equals(pushPacket.type) || PUSH_PACKAGE_TYPE_SERVICE.equals(pushPacket.type)) {
                serviceInfoHolder.processServiceInfo(pushPacket.data);
                
                // send ack to server
                ack = "{\"type\": \"push-ack\"" + ", \"lastRefTime\":\"" + pushPacket.lastRefTime + "\", \"data\":"
                        + "\"\"}";
            } else if (PUSH_PACKAGE_TYPE_DUMP.equals(pushPacket.type)) {
                // dump data to server
                ack = "{\"type\": \"dump-ack\"" + ", \"lastRefTime\": \"" + pushPacket.lastRefTime + "\", \"data\":"
                        + "\"" + StringUtils.escapeJavaScript(JacksonUtils.toJson(serviceInfoHolder.getServiceInfoMap()))
                        + "\"}";
            } else {
                // do nothing send ack only
                ack = "{\"type\": \"unknown-ack\"" + ", \"lastRefTime\":\"" + pushPacket.lastRefTime
                        + "\", \"data\":" + "\"\"}";
            }
            // Send response message
            udpSocket.send(new DatagramPacket(ack.getBytes(UTF_8), ack.getBytes(UTF_8).length,
                    packet.getSocketAddress()));
        } catch (Exception e) {
            if (closed) {
                return;
            }
            NAMING_LOGGER.error("[NA] error while receiving push data", e);
        }
    }
}

The PushReceiver#run method mainly handles the following operations:

  • First, construct datagram packet for receiving message data;
  • Second, block and wait for the arrival of the message through the datagram socket#receive method;
  • Third, after DatagramSocket#receive receives the message, the method continues to execute;
  • Fourth, parse the message in JSON format as PushPacket object;
  • Fifthly, judge the message type, call ServiceInfoHolder#processServiceInfo to process the received message information, and in this method, the PushPacket will be transformed into a ServiceInfo object;
  • Sixth, encapsulate ACK information (i.e. response message information);
  • Seventh, send response message through datagram socket;

Above, we saw how the Nacos client monitors and processes messages based on UDP, but did not find how the client sends UDP information to the registry. Let's sort out the logic of sending UDP information.

Sending UDP information on the client

UDP is stored in NamingHttpClientProxy_ PORT_ Param is the port parameter information of UDP.

UDP port information is transmitted through instance query class interfaces, such as query instance list, query single health instance, query all instances, subscription interface, subscribed update task UpdateTask and other interfaces. In these methods, the NamingClientProxy#queryInstancesOfService method is called.

Implementation of queryInstancesOfService method in NamingHttpClientProxy:

@Override
public ServiceInfo queryInstancesOfService(String serviceName, String groupName, String clusters, int udpPort,
        boolean healthyOnly) throws NacosException {
    final Map<String, String> params = new HashMap<String, String>(8);
    params.put(CommonParams.NAMESPACE_ID, namespaceId);
    params.put(CommonParams.SERVICE_NAME, NamingUtils.getGroupedName(serviceName, groupName));
    params.put(CLUSTERS_PARAM, clusters);
    // Get UDP port
    params.put(UDP_PORT_PARAM, String.valueOf(udpPort));
    params.put(CLIENT_IP_PARAM, NetUtils.localIP());
    params.put(HEALTHY_ONLY_PARAM, String.valueOf(healthyOnly));
    String result = reqApi(UtilAndComs.nacosUrlBase + "/instance/list", params, HttpMethod.GET);
    if (StringUtils.isNotEmpty(result)) {
        return JacksonUtils.toObj(result, ServiceInfo.class);
    }
    return new ServiceInfo(NamingUtils.getGroupedName(serviceName, groupName), clusters);
}

However, when viewing the source code, you will find that the parameter values passed by the UDP port are 0 in the UpdateTask of querying the instance list, querying a single health instance, querying all instances and subscribing update tasks. Only the subscription interface of HTTP protocol is taken as the UDP port number in PushReceiver.

@Override
public ServiceInfo subscribe(String serviceName, String groupName, String clusters) throws NacosException {
    return queryInstancesOfService(serviceName, groupName, clusters, pushReceiver.getUdpPort(), false);
}

In the above code, we already know that there is a method of getPushReceiverUdpPort in PushReceiver:

public static String getPushReceiverUdpPort() {
    return System.getenv(PropertyKeyConst.PUSH_RECEIVER_UDP_PORT);
}

Obviously, the UDP port is set through the environment variable, and the corresponding key is "push.receiver.udp.port".

In version 1.4.2, the queryList method of the NamingProxy member variable in HostReactor will also pass UDP ports:

public void updateService(String serviceName, String clusters) throws NacosException {
    ServiceInfo oldService = getServiceInfo0(serviceName, clusters);
    try {
        String result = serverProxy.queryList(serviceName, clusters, pushReceiver.getUdpPort(), false);
        if (StringUtils.isNotEmpty(result)) {
            processServiceJson(result);
        }
    } finally {
        // ...
    }
}

For the implementation in version 1.4.2, you can look at the source code by yourself. It will not be expanded here.

After completing the transmission of UDP basic information on the client, let's see how the server receives and stores this information.

UDP service store

The server side processes the UDP port in the interface for obtaining the instance list.

@GetMapping("/list")
@Secured(parser = NamingResourceParser.class, action = ActionTypes.READ)
public Object list(HttpServletRequest request) throws Exception {
    // ...
    // If no UDP port information is obtained, the default port is 0
    int udpPort = Integer.parseInt(WebUtils.optional(request, "udpPort", "0"));
    // ...
    // The IP and UDP ports of the client are encapsulated in the Subscriber object
    Subscriber subscriber = new Subscriber(clientIP + ":" + udpPort, agent, app, clientIP, namespaceId, serviceName,
            udpPort, clusters);
    return getInstanceOperator().listInstance(namespaceId, serviceName, subscriber, clusters, healthyOnly);
}

In the getInstanceOperator() method, you will get which protocol is currently used, and then select the corresponding processing class:

/**
 * Determine and return the operation service using V1 version or V2 version
 * @return V1: Jraft Protocol (server side); V2: gRpc protocol (client)
 */
private InstanceOperator getInstanceOperator() {
    return upgradeJudgement.isUseGrpcFeatures() ? instanceServiceV2 : instanceServiceV1;
}

The specific implementation class here is InstanceOperatorServiceImpl:

@Override
public ServiceInfo listInstance(String namespaceId, String serviceName, Subscriber subscriber, String cluster,
        boolean healthOnly) throws Exception {
    ClientInfo clientInfo = new ClientInfo(subscriber.getAgent());
    String clientIP = subscriber.getIp();
    ServiceInfo result = new ServiceInfo(serviceName, cluster);
    Service service = serviceManager.getService(namespaceId, serviceName);
    long cacheMillis = switchDomain.getDefaultCacheMillis();
    // now try to enable the push
    try {
        // Processing client information that supports UDP protocol
        if (subscriber.getPort() > 0 && pushService.canEnablePush(subscriber.getAgent())) {
            subscriberServiceV1.addClient(namespaceId, serviceName, cluster, subscriber.getAgent(),
                    new InetSocketAddress(clientIP, subscriber.getPort()), pushDataSource, StringUtils.EMPTY,
                    StringUtils.EMPTY);
            cacheMillis = switchDomain.getPushCacheMillis(serviceName);
        }
    } catch (Exception e) {
        // ...
    }
    // ...
}

When the UDP port is greater than 0 and the client defined by the agent parameter supports UDP, the corresponding client information is encapsulated in the InetSocketAddress object and then placed in the NamingSubscriberServiceV1Impl (this class has been abandoned. See how to adjust the method implementation later).

In NamingSubscriberServiceV1Impl, the corresponding parameters will be encapsulated as PushClient and stored in the Map.

public void addClient(String namespaceId, String serviceName, String clusters, String agent,
        InetSocketAddress socketAddr, DataSource dataSource, String tenant, String app) {
    
    PushClient client = new PushClient(namespaceId, serviceName, clusters, agent, socketAddr, dataSource, tenant,
            app);
    addClient(client);
}

addClient method will store PushClient information in concurrentmap < string, concurrentmap < string, PushClient > >:

private final ConcurrentMap<String, ConcurrentMap<String, PushClient>> clientMap = new ConcurrentHashMap<>();

public void addClient(PushClient client) {
        // client is stored by key 'serviceName' because notify event is driven by serviceName change
        String serviceKey = UtilsAndCommons.assembleFullServiceName(client.getNamespaceId(), client.getServiceName());
        ConcurrentMap<String, PushClient> clients = clientMap.get(serviceKey);
        if (clients == null) {
            clientMap.putIfAbsent(serviceKey, new ConcurrentHashMap<>(1024));
            clients = clientMap.get(serviceKey);
        }
        
        PushClient oldClient = clients.get(client.toString());
        if (oldClient != null) {
            oldClient.refresh();
        } else {
            PushClient res = clients.putIfAbsent(client.toString(), client);
           // ...
        }
    }

At this time, the IP and port information of UDP has been encapsulated in PushClient and stored in the member variable of NamingSubscriberServiceV1Impl.

UDP notification for registry

When the server finds that an instance has changed, such as actively logging off, it will publish a ServiceChangeEvent event, and UdpPushService will listen to the event and carry out business processing.

In the onApplicationEvent method of UdpPushService, it will remove or send UDP notification according to the specific situation of PushClient. The core logic code in onApplicationEvent is as follows:

ConcurrentMap<String, PushClient> clients = subscriberServiceV1.getClientMap()
        .get(UtilsAndCommons.assembleFullServiceName(namespaceId, serviceName));
if (MapUtils.isEmpty(clients)) {
    return;
}

Map<String, Object> cache = new HashMap<>(16);
long lastRefTime = System.nanoTime();
for (PushClient client : clients.values()) {
    // Remove zombie client
    if (client.zombie()) {
        Loggers.PUSH.debug("client is zombie: " + client);
        clients.remove(client.toString());
        Loggers.PUSH.debug("client is zombie: " + client);
        continue;
    }
    
    AckEntry ackEntry;
    String key = getPushCacheKey(serviceName, client.getIp(), client.getAgent());
    byte[] compressData = null;
    Map<String, Object> data = null;
    if (switchDomain.getDefaultPushCacheMillis() >= 20000 && cache.containsKey(key)) {
        org.javatuples.Pair pair = (org.javatuples.Pair) cache.get(key);
        compressData = (byte[]) (pair.getValue0());
        data = (Map<String, Object>) pair.getValue1();
    }
    
    // Encapsulating AckEntry objects
    if (compressData != null) {
        ackEntry = prepareAckEntry(client, compressData, data, lastRefTime);
    } else {
        ackEntry = prepareAckEntry(client, prepareHostsData(client), lastRefTime);
        if (ackEntry != null) {
            cache.put(key, new org.javatuples.Pair<>(ackEntry.getOrigin().getData(), ackEntry.getData()));
        }
    }
    // Notify other clients via UDP
    udpPush(ackEntry);
}

The core logic of event processing is to judge the status information of PushClient first. If it is already a zombie client, it will be removed. Then the message sent to UDP and the information of the receiving client are encapsulated as AckEntry objects, and then the udpPush method is invoked to transmit UDP messages.

UDP reception for registry

When looking at the client source code, we see that the client will not only receive UDP requests, but also respond. So how does the registry receive the response? Also in the UdpPushService class, the static code block inside the class initializes a UDP datagram socket to receive messages:

static {
    try {
        udpSocket = new DatagramSocket();
        Receiver receiver = new Receiver();
        Thread inThread = new Thread(receiver);
        inThread.setDaemon(true);
        inThread.setName("com.alibaba.nacos.naming.push.receiver");
        inThread.start();
    } catch (SocketException e) {
        Loggers.SRV_LOG.error("[NACOS-PUSH] failed to init push service");
    }
}

Receiver is an internal class that implements the Runnable interface. In its run method, it mainly receives message information, then judges the message, and operates the data in the local Map according to the judgment results.

Insufficient UDP design

At the beginning of the article, it is written that the design of UDP is very good, that is, it makes up for the lack of HTTP timed pull without affecting the performance. However, at present, Nacos has some shortcomings in UDP, which may also be personal nitpicking.

First, the document does not specify how to use the UDP function, which leads many users not to know the existence of UDP function and the restrictions on use.

Second, it is not friendly to cloud services. The UDP port of the client can be customized, but the UDP port of the server is obtained randomly. In cloud services, even for intranet services, UDP ports are restricted by firewalls. If the UDP port of the server is obtained randomly (the client is also the default), the UDP communication will be directly blocked by the firewall, and the user can't see any exceptions at all (UDP protocol doesn't care whether the client receives messages).

As for these two points, they are more flawed than hidden. After reading the source code or reading my article, my friends probably know how to use them. In the follow-up, you can give the official an Issue to see if it can be improved.

Summary

This article focuses on the Nacos UDP based service instance change notification from three aspects:

First, the client listens to the UDP port. When the service instance sent by the receiving registry changes, it can update the local instance cache in time;

Second, the client sends its UDP information to the registry through the subscription interface, and the registry stores it;

Third, the instance in the registry has changed. Through the event mechanism, the change information is sent to the client through UDP protocol.

After this article, you must not only understand the notification mechanism of UDP protocol in Nacos. At the same time, it also opens up a new idea, that is, how to use UDP, what scenarios to use UDP, and the possible problems of using UDP in cloud services.

About the blogger: the author of the technical book "inside of SpringBoot technology", loves to study technology and write technical dry goods articles.

Hot Topics

©2025 Programmer Think