Principle Analysis of Distributed Message Communication ActiveMQ II

Posted by techker on Sun, 05 May 2019 08:42:22 +0200

This chapter focuses on:

1. unconsumed Message source code analysis 
2. PrefetchSize on the consumer side 
3. Message confirmation process 
4. Message Retransmitting Mechanism 
5. ActiveMQ multi-node high performance scheme

Message Consumption Flow Chart

Acquisition process of unconsumed Messages data

Let's see what's going on in ActiveMQConnectionFactory. createConnection.  

  1. Create a transport protocol dynamically
  2. Create a connection
  3. Through transport.start()
 protected ActiveMQConnection createActiveMQConnection(String userName, String password) throws JMSException {
        if (brokerURL == null) {
            throw new ConfigurationException("brokerURL not set.");
        }
        ActiveMQConnection connection = null;
        try {
            Transport transport = createTransport();
            connection = createActiveMQConnection(transport, factoryStats);

            connection.setUserName(userName);
            connection.setPassword(password);

            configureConnection(connection);

            transport.start();

            if (clientID != null) {
                connection.setDefaultClientID(clientID);
            }

            return connection;
        } catch (JMSException e) {
            // Clean up!
            try {
                connection.close();
            } catch (Throwable ignore) {
            }
            throw e;
        } catch (Exception e) {
            // Clean up!
            try {
                connection.close();
            } catch (Throwable ignore) {
            }
            throw JMSExceptionSupport.create("Could not connect to broker URL: " + brokerURL + ". Reason: " + e, e);
        }
    }

transport.start()

In previous articles, I have analyzed that transport is actually a chain call and a multi-layer wrapped object.

ResponseCorrelator(MutexTransport(WireFormatNegotiator(InactivityMonitor(TcpTransport()))) 
Finally, the TcpTransport.start() method is called, but instead of starting in this class, it is in the parent class ServiceSupport.start().

 public void start() throws Exception {
        if (started.compareAndSet(false, true)) {
            boolean success = false;
            stopped.set(false);
            try {
                preStart();
                doStart();
                success = true;
            } finally {
                started.set(success);
            }
            for(ServiceListener l:this.serviceListeners) {
                l.started(this);
            }
        }
    }

This code looks familiar. The source code of the middleware we have seen before, the communication layer, is implemented and decoupled independently. ActiveMQ also provides Transport interfaces and TransportSupport classes.

The main function of this interface is to enable the client to send messages asynchronously, synchronously and consumed.

Next, look down at doStart(), and call TcpTransport.doStart(), then through super.doStart(), call TransportThreadSupport.doStart(). Create a thread, pass in this, call the run method of the subclass, that is, TcpTransport.run().

TcpTransport.run 

The run method reads data packets from the socket, and as long as the TcpTransport does not stop, it will continue to call doRun.

@Override
    public void run() {
        LOG.trace("TCP consumer thread for " + this + " starting");
        this.runnerThread=Thread.currentThread();
        try {
            while (!isStopped()) {
                doRun();
            }
        } catch (IOException e) {
            stoppedLatch.get().countDown();
            onException(e);
        } catch (Throwable e){
            stoppedLatch.get().countDown();
            IOException ioe=new IOException("Unexpected error occurred: " + e);
            ioe.initCause(e);
            onException(ioe);
        }finally {
            stoppedLatch.get().countDown();
        }
    }

TcpTransport.run 
The run method reads data packets from the socket, and as long as the TcpTransport does not stop, it will continue to call doRun.

protected void doRun() throws IOException {
        try {
            Object command = readCommand();
            doConsume(command);
        } catch (SocketTimeoutException e) {
        } catch (InterruptedIOException e) {
        }
    }

TcpTransport.readCommand 
In this case, Data Formatting through wireFormat can be considered as a deserialization process. The default implementation of wireFormat is OpenWireFormat, an activeMQ custom cross-language
wire protocol.

 protected Object readCommand() throws IOException {
        return wireFormat.unmarshal(dataIn);
    }

From this analysis, we can almost understand that the main task of the transport layer is to obtain data and convert data into objects, and then transfer the object objects to ActiveMQ Connection.

TransportSupport.doConsume 
The most important method in the TransportSupport class is doConsume, which is used to "consume messages".

    public void doConsume(Object command) {
        if (command != null) {
            if (transportListener != null) {
                transportListener.onCommand(command);
            } else {
                LOG.error("No transportListener available to process inbound command: " + command);
            }
        }
    }

The only member variable in the TransportSupport class is TransportListener, which means that a Transport support class binds a transport listener class, and the most important way to transfer the transport listener interface is void onCommand(Object command); it is used to process commands, where is the transportListener assigned? Back to the construction method of ActiveMQ Connection. The ActiveMQConnection itself is passed (ActiveMQConnection is one of the implementation classes of the TransportListener interface), so the message goes from the transport layer to our connection layer.

 protected ActiveMQConnection(final Transport transport, IdGenerator clientIdGenerator, IdGenerator connectionIdGenerator, JMSStatsImpl factoryStats) throws Exception {
        
        //Binding transport objects
        this.transport = transport;
        this.clientIdGenerator = clientIdGenerator;
        this.factoryStats = factoryStats;

        // Configure a single threaded executor who's core thread can timeout if
        // idle
        executor = new ThreadPoolExecutor(1, 1, 5, TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(), new ThreadFactory() {
            @Override
            public Thread newThread(Runnable r) {
                Thread thread = new Thread(r, "ActiveMQ Connection Executor: " + transport);
                //Don't make these daemon threads - see https://issues.apache.org/jira/browse/AMQ-796
                //thread.setDaemon(true);
                return thread;
            }
        });
        // asyncConnectionThread.allowCoreThreadTimeOut(true);
        String uniqueId = connectionIdGenerator.generateId();
        this.info = new ConnectionInfo(new ConnectionId(uniqueId));
        this.info.setManageable(true);
        this.info.setFaultTolerant(transport.isFaultTolerant());
        this.connectionSessionId = new SessionId(info.getConnectionId(), -1);

        //transport binds itself
        this.transport.setTransportListener(this);

        this.stats = new JMSConnectionStatsImpl(sessions, this instanceof XAConnection);
        this.factoryStats.addConnection(this);
        this.timeCreated = System.currentTimeMillis();
        this.connectionAudit.setCheckForDuplicates(transport.isFaultTolerant());
    }

As can be seen from the constructor, when creating an ActiveMQConnection object, in addition to binding with Transport, the thread pool executor is initialized. Let's look at the core methods of this class.

onCommand 
In this case, different messages will be distributed. For example, the incoming command is MessageDispatch, and the visit ing method of this command will call the processMessageDispatch method.

public void onCommand(final Object o) {
        final Command command = (Command)o;
        if (!closed.get() && command != null) {
            try {
                command.visit(new CommandVisitorAdapter() {
                    @Override
                    public Response processMessageDispatch(MessageDispatch md) throws Exception {
                        //Waiting for Transport interrupt processing to complete
                        waitForTransportInterruptionProcessingToComplete();
                        //Here, the consumer object to be distributed is retrieved through the consumer id
                        ActiveMQDispatcher dispatcher = dispatchers.get(md.getConsumerId());
                        if (dispatcher != null) {
                            // Copy in case a embedded broker is dispatching via
                            // vm://
                            // md.getMessage() == null to signal end of queue
                            // browse.
                            Message msg = md.getMessage();
                            if (msg != null) {
                                msg = msg.copy();
                                msg.setReadOnlyBody(true);
                                msg.setReadOnlyProperties(true);
                                msg.setRedeliveryCounter(md.getRedeliveryCounter());
                                msg.setConnection(ActiveMQConnection.this);
                                msg.setMemoryUsage(null);
                                md.setMessage(msg);
                            }
                            //Call the session ActiveMQSession's own dispatch method to process this message
                            dispatcher.dispatch(md);
                        } else {
                            LOG.debug("{} no dispatcher for {} in {}", this, md, dispatchers);
                        }
                        return null;
                    }
                     //If the ProducerAck is passed in, then the following method is called, and here we focus only on MessageDispatch 
                       //All right. 
                    @Override
                    public Response processProducerAck(ProducerAck pa) throws Exception {
                        if (pa != null && pa.getProducerId() != null) {
                            ActiveMQMessageProducer producer = producers.get(pa.getProducerId());
                            if (producer != null) {
                                producer.onProducerAck(pa);
                            }
                        }
                        return null;
                    }

In this scenario, we only focus on the processMessageDispatch method, which simply calls the dispatch method of ActiveMQSession to process messages.
tips: command.visit, where the adapter pattern is used, if command is a Message Dispatch, then it calls the processMessage Dispatch method, and the others
No, he doesn't care. The code is as follows: MessageDispatch.visit.

@Override 
 
public Response visit(CommandVisitor visitor) throws Exception { 
 
   return visitor.processMessageDispatch(this); 
 
} 

ActiveMQSession.dispatch(md) 
The executor is actually a member object, ActiveMQSessionExecutor, which handles message distribution.

 public void dispatch(MessageDispatch messageDispatch) {
        try {
            executor.execute(messageDispatch);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            connection.onClientInternalException(e);
        }
    }

ActiveMQSessionExecutor.execute 
The core function of this method is to process the distribution of messages.  

 void execute(MessageDispatch message) throws InterruptedException {

        if (!startedOrWarnedThatNotStarted) {

            ActiveMQConnection connection = session.connection;
            long aboutUnstartedConnectionTimeout = connection.getWarnAboutUnstartedConnectionTimeout();
            if (connection.isStarted() || aboutUnstartedConnectionTimeout < 0L) {
                startedOrWarnedThatNotStarted = true;
            } else {
                long elapsedTime = System.currentTimeMillis() - connection.getTimeCreated();

                // lets only warn when a significant amount of time has passed
                // just in case its normal operation
                if (elapsedTime > aboutUnstartedConnectionTimeout) {
                    LOG.warn("Received a message on a connection which is not yet started. Have you forgotten to call Connection.start()? Connection: " + connection
                             + " Received: " + message);
                    startedOrWarnedThatNotStarted = true;
                }
            }
        }
   
        //If the session is not distributed asynchronously and the session pool branch is not used, dispatch is called to send the message
        if (!session.isSessionAsyncDispatch() && !dispatchedBySessionPool) {
            dispatch(message);
        } else {
            //Put messages directly into the queue
            messageQueue.enqueue(message);
            wakeup();
        }
    }

The default is asynchronous message distribution. So, call messageQueue.enqueue directly, put the message in the queue, and call the wakeup method.

Asynchronous distribution process

    public void wakeup() {
        //Further verification
        if (!dispatchedBySessionPool) {
            //Determine whether session is distributed asynchronously
            if (session.isSessionAsyncDispatch()) {
                try {
                    TaskRunner taskRunner = this.taskRunner;
                    if (taskRunner == null) {
                        synchronized (this) {
                            if (this.taskRunner == null) {
                                if (!isRunning()) {
                                    // stop has been called
                                    return;
                                }
                                //TaskRunner Factory creates a task runtime class taskRunner, which takes itself as a 
                                //A task is introduced into the createTaskRunner to illustrate the current situation 
                                //The class of Task must have implemented the Task interface. Simply speaking, it implements a task through thread pool. 
                                //Asynchronous scheduling, simple 
                                this.taskRunner = session.connection.getSessionTaskRunner().createTaskRunner(this,
                                        "ActiveMQ Session: " + session.getSessionId());
                            }
                            taskRunner = this.taskRunner;
                        }
                    }
                    taskRunner.wakeup();
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            } else {
                //Synchronous distribution
                while (iterate()) {
                }
            }
        }
    }

So, for asynchronous distribution, the iterate method in ActiveMQSessionExecutor is called. Let's look at the code for this method.

iterate 
There are two things to do in this method.

  1. Transfer all messages listened by consumers to the queue to be consumed.
  2. If there are legacy messages in messageQueue, distribute them as well.
public boolean iterate() {

        // Deliver any messages queued on the consumer to their listeners.
        for (ActiveMQMessageConsumer consumer : this.session.consumers) {
            if (consumer.iterate()) {
                return true;
            }
        }

        // No messages left queued on the listeners.. so now dispatch messages
        // queued on the session
        MessageDispatch message = messageQueue.dequeueNoWait();
        if (message == null) {
            return false;
        } else {
            dispatch(message);
            return !messageQueue.isEmpty();
        }
    }

ActiveMQMessageConsumer.iterate 

 public boolean iterate() {
        MessageListener listener = this.messageListener.get();
        if (listener != null) {
            MessageDispatch md = unconsumedMessages.dequeueNoWait();
            if (md != null) {
                dispatch(md);
                return true;
            }
        }
        return false;
    }

Synchronized distribution process
The dispatch method in ActiveMQSession Excutor is called directly by the process of synchronous distribution. The code is as follows.

public void dispatch(MessageDispatch md) {
        MessageListener listener = this.messageListener.get();
        try {
            clearMessagesInProgress();
            clearDeliveredList();
            synchronized (unconsumedMessages.getMutex()) {
                if (!unconsumedMessages.isClosed()) {
                    if (this.info.isBrowser() || !session.connection.isDuplicate(this, md.getMessage())) {
                        if (listener != null && unconsumedMessages.isRunning()) {
                            if (redeliveryExceeded(md)) {
                                posionAck(md, "listener dispatch[" + md.getRedeliveryCounter() + "] to " + getConsumerId() + " exceeds redelivery policy limit:" + redeliveryPolicy);
                                return;
                            }
                            ActiveMQMessage message = createActiveMQMessage(md);
                            beforeMessageIsConsumed(md);
                            try {
                                boolean expired = isConsumerExpiryCheckEnabled() && message.isExpired();
                                if (!expired) {
                                    listener.onMessage(message);
                                }
                                afterMessageIsConsumed(md, expired);
                            } catch (RuntimeException e) {
                                LOG.error("{} Exception while processing message: {}", getConsumerId(), md.getMessage().getMessageId(), e);
                                md.setRollbackCause(e);
                                if (isAutoAcknowledgeBatch() || isAutoAcknowledgeEach() || session.isIndividualAcknowledge()) {
                                    // schedual redelivery and possible dlq processing
                                    rollback();
                                } else {
                                    // Transacted or Client ack: Deliver the next message.
                                    afterMessageIsConsumed(md, false);
                                }
                            }
                        } else {
                            if (!unconsumedMessages.isRunning()) {
                                // delayed redelivery, ensure it can be re delivered
                                session.connection.rollbackDuplicate(this, md.getMessage());
                            }

                            if (md.getMessage() == null) {
                                // End of browse or pull request timeout.
                                unconsumedMessages.enqueue(md);
                            } else {
                                if (!consumeExpiredMessage(md)) {
                                    unconsumedMessages.enqueue(md);
                                    if (availableListener != null) {
                                        availableListener.onMessageAvailable(this);
                                    }
                                } else {
                                    beforeMessageIsConsumed(md);
                                    afterMessageIsConsumed(md, true);

                                    // Pull consumer needs to check if pull timed out and send
                                    // a new pull command if not.
                                    if (info.getCurrentPrefetchSize() == 0) {
                                        unconsumedMessages.enqueue(null);
                                    }
                                }
                            }
                        }
                    } else {
                        // deal with duplicate delivery
                        ConsumerId consumerWithPendingTransaction;
                        if (redeliveryExpectedInCurrentTransaction(md, true)) {
                            LOG.debug("{} tracking transacted redelivery {}", getConsumerId(), md.getMessage());
                            if (transactedIndividualAck) {
                                immediateIndividualTransactedAck(md);
                            } else {
                                session.sendAck(new MessageAck(md, MessageAck.DELIVERED_ACK_TYPE, 1));
                            }
                        } else if ((consumerWithPendingTransaction = redeliveryPendingInCompetingTransaction(md)) != null) {
                            LOG.warn("{} delivering duplicate {}, pending transaction completion on {} will rollback", getConsumerId(), md.getMessage(), consumerWithPendingTransaction);
                            session.getConnection().rollbackDuplicate(this, md.getMessage());
                            dispatch(md);
                        } else {
                            LOG.warn("{} suppressing duplicate delivery on connection, poison acking: {}", getConsumerId(), md);
                            posionAck(md, "Suppressing duplicate delivery on connection, consumer " + getConsumerId());
                        }
                    }
                }
            }
            if (++dispatchedCount % 1000 == 0) {
                dispatchedCount = 0;
                Thread.yield();
            }
        } catch (Exception e) {
            session.connection.onClientInternalException(e);
        }
    }

Up to now, we have made it clear how to accept the message and how to process it. We hope it will be helpful for us to understand the core mechanism of activeMQ.

Consumer-side PrefetchSize
Remember the prefetchsize we talked about when we analyzed the source code at the consumer end? What does this prefetchsize do? Let's look at it next.

Principle analysis

  1. The consumer side of activemq also has a window mechanism, through prefetchSize you can set the window size. Different types of queues have different default values for prefetchSize. The default values for persistent and non-persistent queues are 1000, 100 for persistent topic, and Short.MAX_VALUE-1 for non-persistent queues.
  2. With the example above, we should basically know the role of prefetchSize. The consumer will get data in batches according to the size of prefetchSize, such as the default value of 1000, then the consumer will pre-load 1000 pieces of data into local memory.  

The setting method of prefetchSize
Add consumer.prefetchSize to createQueue to see the effect of Destination destination=session.createQueue("myQueue?consumer.prefetchSize=10");
Since there is batch loading, there must be batch validation, which is a thorough optimization.
optimizeAcknowledge 

  1. ActiveMQ provides optimized Acknowledge to optimize validation, which indicates whether "optimized ACK" is turned on or not. PreetchSize and optimized AcknowledgeTime parameters are meaningful only for true.
  2. Optimized confirmation can reduce client burden (no frequent confirmation messages are needed) and communication overhead. On the other hand, because of delayed confirmation (0.65*prefetchSize messages are ack nowledged by default), broker can send messages in batches when it sends them again.
  3. If only prefetchSize is turned on and every message is confirmed, broker will send only one message after receiving confirmation, not a batch release. Of course, it can also manually delay confirmation by setting DUPS_OK_ACK. We need to specify the Optimized ACK option Connection Factory= New ActiveMQ Connection Factory in brokerUrl ("tcp://192.1"). 68.11.153:61616?Jms.optimizeAcknowledge=true&jms.optimizeAcknowledgeTimeOut=10000";
  4. Note that if optimizeAcknowledge is true, prefetchSize must be greater than 0. When prefetchSize=0, it means that consumer retrieves messages from broker through PULL.

Sum up

  1. So far, we know the role of optimize Acknowledge and prefetch Size, which work together to achieve an efficient message consumption model by obtaining messages in batches and delaying batch validation. It not only reduces the number of blocking times when the client acquires the message, but also reduces the network communication overhead each time it acquires the message.
  2. It should be noted that if the consumption speed of the consumer side is relatively high, the performance of consumer can be greatly improved through the combination of the two. If consumer's consumption performance itself is slow, the larger prefetchSize settings can not effectively achieve the purpose of improving consumption performance. Because too large prefetchSize is not conducive to load balancing of consumer-side messages. Because in general, we deploy multiple consumer nodes to improve consumer performance.  
  3. There is another potential risk in this optimization scheme. If the client side fails before the message can be confirmed after being consumed, the message may be re-sent to other consumer s. This risk requires that the client side can tolerate "duplicate" messages.  

 

To be continued...

Topics: Programming Session socket Apache network