An InterruptException occurred when the RocketMQConsumer service was shut down

Posted by akumakeenta on Tue, 01 Mar 2022 11:00:42 +0100

An InterruptException occurred when the RocketMQConsumer service was shut down

Background summary

The main problem is the version upgrade

  1. Old version core rocketmq dependency

    <dependency>
        <groupId>org.apache.rocketmq</groupId>
        <artifactId>spring-boot-starter-rocketmq</artifactId>
        <version>${vesion}</version>
    </dependency>
     <dependency>
    	<groupId>org.apache.rocketmq</groupId>
    	<artifactId>rocketmq-client</artifactId>
    	<version>4.3.2</version>
    </dependency>
    
  2. New version of core rocketmq dependency

    <dependency>
        <groupId>org.apache.rocketmq</groupId>
        <artifactId>rocketmq-client</artifactId>
        <version>4.9.2</version>
    </dependency>
    <dependency>
    	<groupId>org.apache.rocketmq</groupId>
    	<artifactId>rocketmq-spring-boot-starter</artifactId>
    	<version>2.2.1</version>
    </dependency>
    

java.lang.InterruptedException

Simply enumerate an InterruptedException

java.sql.SQLException: interrupt
	at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1430) ~[druid-1.1.12.jar!/:1.1.12]
	at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1272) ~[druid-1.1.12.jar!/:1.1.12]
	at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5007) ~[druid-1.1.12.jar!/:1.1.12]
	at com.alibaba.druid.filter.FilterAdapter.dataSource_getConnection(FilterAdapter.java:2745) ~[druid-1.1.12.jar!/:1.1.12]
	at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5003) ~[druid-1.1.12.jar!/:1.1.12]
	at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:680) ~[druid-1.1.12.jar!/:1.1.12]
	at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:5003) ~[druid-1.1.12.jar!/:1.1.12]
	at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1250) ~[druid-1.1.12.jar!/:1.1.12]
	at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1242) ~[druid-1.1.12.jar!/:1.1.12]
	at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:89) ~[druid-1.1.12.jar!/:1.1.12]
	// Omit some stack information
	at org.apache.rocketmq.spring.support.DefaultRocketMQListenerContainer.handleMessage(DefaultRocketMQListenerContainer.java:399) [rocketmq-spring-boot-2.2.1.jar!/:2.2.1]
	at org.apache.rocketmq.spring.support.DefaultRocketMQListenerContainer.access$100(DefaultRocketMQListenerContainer.java:71) [rocketmq-spring-boot-2.2.1.jar!/:2.2.1]
	at org.apache.rocketmq.spring.support.DefaultRocketMQListenerContainer$DefaultMessageListenerConcurrently.consumeMessage(DefaultRocketMQListenerContainer.java:359) [rocketmq-spring-boot-2.2.1.jar!/:2.2.1]
	at cn.techwolf.trace.rocketmq.spring.TracingMessageListenerConcurrently.consumeMessage(TracingMessageListenerConcurrently.java:37) [instrument-rocketmq-spring-1.101.jar!/:1.101]
	at org.apache.rocketmq.client.impl.consumer.ConsumeMessageConcurrentlyService$ConsumeRequest.run(ConsumeMessageConcurrentlyService.java:392) [rocketmq-client-4.9.2.jar!/:4.9.2]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_202]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
Caused by: java.lang.InterruptedException

be careful

The following analysis is only a personal point of view and is not necessarily correct. If there is a bar essence, please do not continue to watch it. Welcome to leave a message for discussion

Tip: some classes involved in this article are A NullPointerException problem was recorded when the RocketMQ service was started This article does not do some detailed explanation

spring shutdown & rocketmq shutdown timing (interleaving is not important)

  1. spring

  2. rocketmq inherits SmartCycle and calls back its stop method when the container is closed

rocketmq shutdown analysis

First, confirm the implementation of our RocketMQConsumer. Defaultrocketmqllistenercontainer:

// Class full path org apache. rocketmq. spring. support. DefaultRocketMQListenerContainer
public class DefaultRocketMQListenerContainer implements InitializingBean, RocketMQListenerContainer, SmartLifecycle, ApplicationContextAware {
    private DefaultMQPushConsumer consumer;
    @Override
    public void destroy() {// Called when the DisposableBean callback destroys the bean
        this.setRunning(false);
        if (Objects.nonNull(consumer)) {
            consumer.shutdown();
        }
        log.info("container destroyed, {}", this.toString());
    }
    @Override
    public void stop() { // SmartLifecycle callback callback before closing container
        if (this.isRunning()) {
            if (Objects.nonNull(consumer)) {
                consumer.shutdown();
            }
            setRunning(false);
        }
    }
}

It can be seen that the defaultrocketmqllistenercontainer is called first if it is closed Stop method. The next step is to look at consumer The shutdown () method:

// Class full path org apache. rocketmq. client. consumer. DefaultMQPushConsumer
public class DefaultMQPushConsumer extends ClientConfig implements MQPushConsumer {
    protected final transient DefaultMQPushConsumerImpl defaultMQPushConsumerImpl;
    /**
     * Maximum time to await message consuming when shutdown consumer, 0 indicates no await.
     */
    private long awaitTerminationMillisWhenShutdown = 0;
     @Override
    public void shutdown() {
        this.defaultMQPushConsumerImpl.shutdown(awaitTerminationMillisWhenShutdown);
        if (null != traceDispatcher) {
            traceDispatcher.shutdown();
        }
    }
}
// Class full path org apache. rocketmq. client. impl. consumer. DefaultMQPushConsumerImpl
public class DefaultMQPushConsumerImpl implements MQConsumerInner {
    private ConsumeMessageService consumeMessageService;
	public synchronized void shutdown(long awaitTerminateMillis) {
        switch (this.serviceState) {
            case CREATE_JUST:
                break;
            case RUNNING:
                this.consumeMessageService.shutdown(awaitTerminateMillis);
                this.persistConsumerOffset();
                this.mQClientFactory.unregisterConsumer(this.defaultMQPushConsumer.getConsumerGroup());
                this.mQClientFactory.shutdown();
                log.info("the consumer [{}] shutdown OK", this.defaultMQPushConsumer.getConsumerGroup());
                this.rebalanceImpl.destroy();
                this.serviceState = ServiceState.SHUTDOWN_ALREADY;
                break;
            case SHUTDOWN_ALREADY:
                break;
            default:
                break;
        }
    }
}
// Consummessageservice has two implementation classes: consummessageconcurrentlyservice and consummessageorderlyservice
// The specific implementation class of consumeMessageService we use is consumemessageconcurrent service. Specific analysis of specific problems 
// Class full path org apache. rocketmq. client. impl. consumer. ConsumeMessageConcurrentlyService
public class ConsumeMessageConcurrentlyService implements ConsumeMessageService {
	private final ThreadPoolExecutor consumeExecutor;
    public void shutdown(long awaitTerminateMillis) {
        this.scheduledExecutorService.shutdown();
        ThreadUtils.shutdownGracefully(this.consumeExecutor, awaitTerminateMillis, TimeUnit.MILLISECONDS);
        this.cleanExpireMsgExecutors.shutdown();
    }
}
// Class full path org apache. rocketmq. common. utils. ThreadUtils
public final class ThreadUtils {
    public static void shutdownGracefully(ExecutorService executor, long timeout, TimeUnit timeUnit) {
        // Disable new tasks from being submitted.
        executor.shutdown();
        try {
            // Wait a while for existing tasks to terminate.
            if (!executor.awaitTermination(timeout, timeUnit)) { // Pay attention here
                executor.shutdownNow();
                // Wait a while for tasks to respond to being cancelled.
                if (!executor.awaitTermination(timeout, timeUnit)) {
                    log.warn(String.format("%s didn't terminate!", executor));
                }
            }
        } catch (InterruptedException ie) {
            // (Re-)Cancel if current thread also interrupted.
            executor.shutdownNow();
            // Preserve interrupt status.
            Thread.currentThread().interrupt();
        }
    }
}

From the above, we can clearly see that consumemessageconcurrent service During shutdown, the default value of awaitterminatemilis is 0, which is executed to threadutils When shutdowngracefully, shutdownnow will be called directly without waiting

What's the difference between shutdown and shutdown now? Just search Baidu

  • Shutdown = > gently shut down and wait for all tasks added to the thread pool to finish executing
  • Shutdown now = > close immediately, stop the executing task, and return the unexecuted task in the queue

Compare with the previous code rocketmq client 4.3.2

public class ConsumeMessageConcurrentlyService implements ConsumeMessageService {
	private final ThreadPoolExecutor consumeExecutor;
 public void shutdown(long awaitTerminateMillis) {
     this.scheduledExecutorService.shutdown();
     this.consumeExecutor.shutdown(); // here
     this.cleanExpireMsgExecutors.shutdown();
 }
}

Therefore, I think it is because the interrupt InterruptException exception occurs when the service is shut down due to the direct call to shutdownnow

Solution

From the above analysis, we can know that it is caused by calling shutdown now directly. We should be able to adjust the awaitterminatemilis parameter, that is, defaultmqpushconsumer Awaitterminationmilliswhenshutdown parameter

At present, I don't seem to see any way to support the global configuration of this parameter (official way). Here are two ideas

RocketMQPushConsumerLifecycleListener or RocketMQPushConsumerLifecycleListener

// org.apache.rocketmq.spring.support.DefaultRocketMQListenerContainer
public class DefaultRocketMQListenerContainer implements InitializingBean, RocketMQListenerContainer, SmartLifecycle, ApplicationContextAware {
	@Override
    public void afterPropertiesSet() throws Exception {
        initRocketMQPushConsumer();
        this.messageType = getMessageType();
        this.methodParameter = getMethodParameter();
        log.debug("RocketMQ messageType: {}", messageType);
    }
    private void initRocketMQPushConsumer() throws MQClientException {
        // Omit some codes
        if (Objects.nonNull(rpcHook)) {
            consumer = new DefaultMQPushConsumer(consumerGroup, rpcHook, new AllocateMessageQueueAveragely(),
                enableMsgTrace, this.applicationContext.getEnvironment().
                resolveRequiredPlaceholders(this.rocketMQMessageListener.customizedTraceTopic()));
            consumer.setVipChannelEnabled(false);
        } else {
            log.debug("Access-key or secret-key not configure in " + this + ".");
            consumer = new DefaultMQPushConsumer(consumerGroup, enableMsgTrace,
                this.applicationContext.getEnvironment().
                    resolveRequiredPlaceholders(this.rocketMQMessageListener.customizedTraceTopic()));
        }
        // Omit some consumer parameter configuration codes
      	// You can see that he will call back the prepareStart method
        if (rocketMQListener instanceof RocketMQPushConsumerLifecycleListener) {
            ((RocketMQPushConsumerLifecycleListener) rocketMQListener).prepareStart(consumer);
        } else if (rocketMQReplyListener instanceof RocketMQPushConsumerLifecycleListener) {
            ((RocketMQPushConsumerLifecycleListener) rocketMQReplyListener).prepareStart(consumer);
        }
    }
}

You can see that after the defaultrocketmqllistenercontainer is initialized, the prepareStart methods of rocketmqppushconsumerlifcyclelistener and rocketmqppushconsumerlifcyclelistener will be called back

That's easy

@Service
@RocketMQMessageListener(nameServer = "${spring.rocketmq.nameServer}",
        topic = "${topic}",
        consumerGroup = "${group}")
public class TestConsumer implements RocketMQListener<String>, RocketMQPushConsumerLifecycleListener {

    @Override
    public void onMessage(String msg) {
        // do something
    }

    @Override
    public void prepareStart(DefaultMQPushConsumer consumer) {
        consumer.setAwaitTerminationMillisWhenShutdown(1000); // set up
    }
}

It can be abstracted into a general class, and the consumer can inherit this class

DefaultRocketMQListenerContainer.getConsumer

Defaultrocketmqllistenercontainer will be registered as a bean, and the specific implementation is at org apache. rocketmq. spring. autoconfigure. In ListenerContainerConfiguration, there is no analysis here. We can try to get all the DefaultRocketMQListenerContainer and call the getConsumer method. The sample code can trigger the code in detail.

public void set() {
    List<DefaultRocketMQListenerContainer> containers = getAllBeans();
    for (DefaultRocketMQListenerContainer c : containers) {
        c.getConsumer().setAwaitTerminationMillisWhenShutdown(1000);
    }
}

Finally, the above analysis is only for me and is not necessarily correct. In the first way, there is no InterruptException. I don't know whether it is accidental;

Welcome to leave a message for discussion

Topics: Java Apache Spring Boot message queue RocketMQ