DefaultMQProducer is the default producer implementation in RocketMQ. The inheritance relationship between DefaultMQProducer classes is as follows:
You can see that the producer includes the producer's operation and configuration properties during implementation, which is a typical class object design.
Here are some core attributes:
namesrvAddr: inherits from ClientConfig and represents the Namesrv address of the RocketMQ cluster. If there are multiple, separate them with semicolons.
clientIP: the IP address of the machine where the Client program is located. It supports IPv4 and IPv6. IPv4 excludes the local loopback address and private intranet address. It should be noted that if the Client runs in the Docker container, the IP address obtained is the IP address of the container, not the IP address of the host.
instanceName: instance name. Each instance needs a unique name, because sometimes we will deploy multiple program processes on the same machine. If the name is repeated, it will lead to startup failure.
vipChannelEnabled: This is a boolean value indicating whether to open the VIP channel. The difference between VIP channel and non VIP channel is that the port number used in the communication process is different.
clientCallBackExecutorThreads: number of client callback threads. This parameter indicates the number of callback threads of Netty communication layer. The default value is Runtime.getRuntime().availableProcessors(), which indicates the effective number of current CPU s.
pollNameServerInterval: the interval length for obtaining Topic routing information, unit: ms, default: 30000ms.
Heartbeat Broker interval: the duration of heartbeat interval with Broker, unit: ms, default: 30000ms.
defaultMQProducerImpl: the implementation class of the default producer, which encapsulates various API s of the Broker. If you want to implement a producer yourself, you can add a new implementation to keep the DefaultMQProducer external interface unchanged, and the user is not aware of it at all.
Producer group: producer group name, which is a parameter that must be passed. Rocketmq way means that the behavior of producer instances in the same producer group needs to be consistent.
sendMsgTimeout: sending timeout, in ms.
compressMsgBodyOverHowmuch: the upper limit of the message body. When the upper limit is exceeded, the message will be compressed through ZIP. This value is 4MB by default. How does this parameter take effect in the Client? The specific implementation code is as follows:
private boolean tryToCompressMessage(final Message msg) { if (msg instanceof MessageBatch) { //batch dose not support compressing right now return false; } byte[] body = msg.getBody(); if (body != null) { if (body.length >= this.defaultMQProducer.getCompressMsgBodyOverHowmuch()) { try { byte[] data = UtilAll.compress(body, zipCompressLevel); if (data != null) { msg.setBody(data); return true; } } catch (IOException e) { log.error("tryToCompressMessage exception", e); log.warn(msg.toString()); } } } return false; }
Retrytimeswhensensayncfailed: the number of retries after asynchronous sending failed. The default is 2 times. Asynchronous retry is a conditional retry, not every time the sending fails.
Here are some core methods:
start(): This is the entry to start the entire producer instance. It is mainly responsible for verifying whether the producer's configuration parameters are correct, and starting communication channels, various scheduled tasks, Pull services, Rebalance services, registering producers to brokers, etc.
shutdown(): close the locally registered producer and the client registered with the Broker.
fetchPublishMessageQueue(Topic): get which queues a Topic has. It needs to be called when sending messages and Pull messages.
send(Messgae msg): send normal messages synchronously.
send(Message msg, long timeout): send ordinary messages synchronously (timeout setting)
Send (message MSG, sendcallback, sendcallback): send ordinary messages asynchronously
Send (message MSG, sendcallback, sendcallback, long timeout): send ordinary messages asynchronously (timeout setting)
sendOneway(Message msg): Send a one-way message. Only responsible for sending messages, regardless of the sending results
send(Message msg,MessageQueue mq): send messages to the specified queue synchronously
send(Message msg,MessageQueue mq, long timeout): send messages to the specified queue synchronously (timeout setting)
When sending messages to a specified queue synchronously, if there is only one sending thread, and the messages in the specified queue are in order when they are sent to a specified queue, they will be sorted according to the sending time; If this is the case for the queues of a Topic, we call all messages of the Topic partitioned and orderly.
Send (message MSG, messagequeue MQ, sendcallback, sendcallback): send the message asynchronously to the specified queue
Send (message MSG, messagequeue MQ, sendcallback, sendcallback, long timeout): asynchronous sending to the specified queue (timeout setting)
Send (message MSG, MessageQueueSelector, selector, object Arg, sendcallback, sendcallback): custom messages are sent to the specified queue. Select which queue the message is sent to by implementing the MessageQueueSelector interface.
Send (collection < message > msgs): send messages in batches
Two core management interfaces are described below
createTopic(String key,String newTopic,int queueName): creates a Topic
viewMessage(String offsetMsgId): query the message content according to the message id
The process started by the producer is simpler than that started by the consumer. Generally, users use the constructor of DefaultMQProducer to construct a producer instance and set various parameters. For example, Namesrv address, producer group, etc. call the start() method to start the producer instance. The start() method calls the start() method of the producer's default implementation class to start. Here we mainly talk about the implementation of the start() method of the implementation class:
// The first step is to call the DefaultMQProducer default constructor public DefaultMQProducer(final String namespace, final String producerGroup, RPCHook rpcHook) { // Set up namespace and producer group this.namespace = namespace; this.producerGroup = producerGroup; // Create producer default implementation defaultMQProducerImpl = new DefaultMQProducerImpl(this, rpcHook); } // The second step is to call the start method to start the producer instance public void start() throws MQClientException { this.setProducerGroup(withNamespace(this.producerGroup)); // Call the start() method of the default implementation this.defaultMQProducerImpl.start(); if (null != traceDispatcher) { try { traceDispatcher.start(this.getNamesrvAddr(), this.getAccessChannel()); } catch (MQClientException e) { log.warn("trace dispatcher start failed ", e); } } } public void start(final boolean startFactory) throws MQClientException { // The default service state is the creation state. If it is not the creation state, an exception will be thrown directly switch (this.serviceState) { case CREATE_JUST: // Setting producer status to start failed this.serviceState = ServiceState.START_FAILED; // Parameter verification this.checkConfig(); // The default producer group name is converted to the process id if (!this.defaultMQProducer.getProducerGroup().equals(MixAll.CLIENT_INNER_PRODUCER_GROUP)) { this.defaultMQProducer.changeInstanceNameToPID(); } // Create mq factory this.mQClientFactory = MQClientManager.getInstance().getOrCreateMQClientInstance(this.defaultMQProducer, rpcHook); // Registered producer boolean registerOK = mQClientFactory.registerProducer(this.defaultMQProducer.getProducerGroup(), this); if (!registerOK) { this.serviceState = ServiceState.CREATE_JUST; throw new MQClientException("The producer group[" + this.defaultMQProducer.getProducerGroup() + "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL), null); } this.topicPublishInfoTable.put(this.defaultMQProducer.getCreateTopicKey(), new TopicPublishInfo()); // Register local routing information if (startFactory) { mQClientFactory.start(); } log.info("the producer [{}] start OK. sendMessageWithVIPChannel={}", this.defaultMQProducer.getProducerGroup(), this.defaultMQProducer.isSendMessageWithVIPChannel()); // The producer status is set to running status this.serviceState = ServiceState.RUNNING; break; case RUNNING: case START_FAILED: case SHUTDOWN_ALREADY: throw new MQClientException("The producer service state not OK, maybe started once, " + this.serviceState + FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK), null); default: break; } // Send heartbeat information to all brokers this.mQClientFactory.sendHeartbeatToAllBrokerWithLock(); // Enable scheduled tasks to scan and remove expired tasks every second this.startScheduledTask(); } // mQClientFactory.start(); public void start() throws MQClientException { synchronized (this) { switch (this.serviceState) { case CREATE_JUST: // Setting the service status to start failed this.serviceState = ServiceState.START_FAILED; // If not specified,looking address from name server // If namesrv is not configured, get it remotely if (null == this.clientConfig.getNamesrvAddr()) { this.mQClientAPIImpl.fetchNameServerAddr(); } // Start request-response channel // Start communication module service this.mQClientAPIImpl.start(); // Start various schedule tasks // Start various scheduled tasks this.startScheduledTask(); // Start pull service // Start message pull service this.pullMessageService.start(); // Start rebalance service // Start rebalanceService service service this.rebalanceService.start(); // Start push service // Start message push service this.defaultMQProducer.getDefaultMQProducerImpl().start(false); log.info("the client factory [{}] start OK", this.clientId); // Set the service status to running this.serviceState = ServiceState.RUNNING; break; case START_FAILED: throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null); default: break; } } }
Step 1: judge the service status of the current producer through switch case. The default state is create when creating_ Just, set the default startup status to startup failure.
Step 2: execute the checkConfig() method. Verify various parameters set by the producer instance. For example, whether the producer group name is empty, whether the naming rules are met, and whether the length is met.
Step 3: execute the changeInstanceNameToPID() method. Verify instance name. If it is the default name, modify it to process id.
Step 4: execute the getAndCreateMQClientInstance() method to initialize an MQClientInstance according to the producer group name. The initialization code is as follows:
public MQClientInstance getOrCreateMQClientInstance(final ClientConfig clientConfig, RPCHook rpcHook) { String clientId = clientConfig.buildMQClientId(); MQClientInstance instance = this.factoryTable.get(clientId); if (null == instance) { // Create client instance instance = new MQClientInstance(clientConfig.cloneClientConfig(), this.factoryIndexGenerator.getAndIncrement(), clientId, rpcHook); MQClientInstance prev = this.factoryTable.putIfAbsent(clientId, instance); if (prev != null) { instance = prev; log.warn("Returned Previous MQClientInstance for clientId:[{}]", clientId); } else { log.info("Created new MQClientInstance for clientId:[{}]", clientId); } } return instance; } public String buildMQClientId() { StringBuilder sb = new StringBuilder(); sb.append(this.getClientIP()); sb.append("@"); sb.append(this.getInstanceName()); if (!UtilAll.isBlank(this.unitName)) { sb.append("@"); sb.append(this.unitName); } return sb.toString(); }
The function of MQClientInstance instance is to manage the production and consumption behavior of all producers and consumers in this instance.
The variables in this example are described below
public class MQClientInstance { ...... // The internal instance of all producers of the current client instance private final ConcurrentMap<String/* group */, MQProducerInner> producerTable = new ConcurrentHashMap<String, MQProducerInner>(); // Internal instances of all consumers of the current client instance private final ConcurrentMap<String/* group */, MQConsumerInner> consumerTable = new ConcurrentHashMap<String, MQConsumerInner>(); // All management instances of the current client instance private final ConcurrentMap<String/* group */, MQAdminExtInner> adminExtTable = new ConcurrentHashMap<String, MQAdminExtInner>(); private final NettyClientConfig nettyClientConfig; // In fact, each client is also a netty server, which also supports Broker access. All client supported interfaces are implemented here private final MQClientAPIImpl mQClientAPIImpl; // Local implementation class of management interface private final MQAdminImpl mQAdminImpl; // Locally cached routing information of all topics in the current producer and consumer private final ConcurrentMap<String/* Topic */, TopicRouteData> topicRouteTable = new ConcurrentHashMap<String, TopicRouteData>(); private final Lock lockNamesrv = new ReentrantLock(); private final Lock lockHeartbeat = new ReentrantLock(); private final ConcurrentMap<String/* Broker Name */, HashMap<Long/* brokerId */, String/* address */>> brokerAddrTable = new ConcurrentHashMap<String, HashMap<Long, String>>(); private final ConcurrentMap<String/* Broker Name */, HashMap<String/* address */, Integer>> brokerVersionTable = new ConcurrentHashMap<String, HashMap<String, Integer>>(); // Local scheduled tasks, such as regularly obtaining the current Namesrv address, regularly synchronizing Namesrv information, regularly updating Topic routing information, regularly sending heartbeat information to brokers, regularly cleaning offline brokers, regularly persisting consumption sites, and regularly adjusting the number of consumption threads private final ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(new ThreadFactory() { @Override public Thread newThread(Runnable r) { return new Thread(r, "MQClientFactoryScheduledThread"); } }); // From the processing method processRequest(), we can know which functional interfaces are currently supported private final ClientRemotingProcessor clientRemotingProcessor; // Pull service private final PullMessageService pullMessageService; // Rebalance services. Periodically specify the rebalancing method, private final RebalanceService rebalanceService; private final DefaultMQProducer defaultMQProducer; // Consumption monitoring private final ConsumerStatsManager consumerStatsManager; private final AtomicLong sendHeartbeatTimesTotal = new AtomicLong(0); private ServiceState serviceState = ServiceState.CREATE_JUST; private Random random = new Random(); public MQClientInstance(ClientConfig clientConfig, int instanceIndex, String clientId) { this(clientConfig, instanceIndex, clientId, null); } public MQClientInstance(ClientConfig clientConfig, int instanceIndex, String clientId, RPCHook rpcHook) { this.clientConfig = clientConfig; this.instanceIndex = instanceIndex; this.nettyClientConfig = new NettyClientConfig(); this.nettyClientConfig.setClientCallbackExecutorThreads(clientConfig.getClientCallbackExecutorThreads()); this.nettyClientConfig.setUseTLS(clientConfig.isUseTLS()); this.clientRemotingProcessor = new ClientRemotingProcessor(this); this.mQClientAPIImpl = new MQClientAPIImpl(this.nettyClientConfig, this.clientRemotingProcessor, rpcHook, clientConfig); if (this.clientConfig.getNamesrvAddr() != null) { this.mQClientAPIImpl.updateNameServerAddressList(this.clientConfig.getNamesrvAddr()); log.info("user specified name server address: {}", this.clientConfig.getNamesrvAddr()); } this.clientId = clientId; this.mQAdminImpl = new MQAdminImpl(this); this.pullMessageService = new PullMessageService(this); this.rebalanceService = new RebalanceService(this); this.defaultMQProducer = new DefaultMQProducer(MixAll.CLIENT_INNER_PRODUCER_GROUP); this.defaultMQProducer.resetClientConfig(clientConfig); this.consumerStatsManager = new ConsumerStatsManager(this.scheduledExecutorService); log.info("Created a new client Instance, InstanceIndex:{}, ClientID:{}, ClientConfig:{}, ClientVersion:{}, SerializerType:{}", this.instanceIndex, this.clientId, this.clientConfig, MQVersion.getVersionDesc(MQVersion.CURRENT_VERSION), RemotingCommand.getSerializeTypeConfigInThisServer()); } public static TopicPublishInfo topicRouteData2TopicPublishInfo(final String topic, final TopicRouteData route) { TopicPublishInfo info = new TopicPublishInfo(); info.setTopicRouteData(route); if (route.getOrderTopicConf() != null && route.getOrderTopicConf().length() > 0) { String[] brokers = route.getOrderTopicConf().split(";"); for (String broker : brokers) { String[] item = broker.split(":"); int nums = Integer.parseInt(item[1]); for (int i = 0; i < nums; i++) { MessageQueue mq = new MessageQueue(topic, item[0], i); info.getMessageQueueList().add(mq); } } info.setOrderTopic(true); } else { List<QueueData> qds = route.getQueueDatas(); Collections.sort(qds); for (QueueData qd : qds) { if (PermName.isWriteable(qd.getPerm())) { BrokerData brokerData = null; for (BrokerData bd : route.getBrokerDatas()) { if (bd.getBrokerName().equals(qd.getBrokerName())) { brokerData = bd; break; } } if (null == brokerData) { continue; } if (!brokerData.getBrokerAddrs().containsKey(MixAll.MASTER_ID)) { continue; } for (int i = 0; i < qd.getWriteQueueNums(); i++) { MessageQueue mq = new MessageQueue(topic, qd.getBrokerName(), i); info.getMessageQueueList().add(mq); } } } info.setOrderTopic(false); } return info; } public static Set<MessageQueue> topicRouteData2TopicSubscribeInfo(final String topic, final TopicRouteData route) { Set<MessageQueue> mqList = new HashSet<MessageQueue>(); List<QueueData> qds = route.getQueueDatas(); for (QueueData qd : qds) { if (PermName.isReadable(qd.getPerm())) { for (int i = 0; i < qd.getReadQueueNums(); i++) { MessageQueue mq = new MessageQueue(topic, qd.getBrokerName(), i); mqList.add(mq); } } } return mqList; } public void start() throws MQClientException { synchronized (this) { switch (this.serviceState) { case CREATE_JUST: this.serviceState = ServiceState.START_FAILED; // If not specified,looking address from name server if (null == this.clientConfig.getNamesrvAddr()) { this.mQClientAPIImpl.fetchNameServerAddr(); } // Start request-response channel this.mQClientAPIImpl.start(); // Start various schedule tasks this.startScheduledTask(); // Start pull service this.pullMessageService.start(); // Start rebalance service this.rebalanceService.start(); // Start push service this.defaultMQProducer.getDefaultMQProducerImpl().start(false); log.info("the client factory [{}] start OK", this.clientId); this.serviceState = ServiceState.RUNNING; break; case START_FAILED: throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null); default: break; } } } private void startScheduledTask() { if (null == this.clientConfig.getNamesrvAddr()) { this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() { @Override public void run() { try { MQClientInstance.this.mQClientAPIImpl.fetchNameServerAddr(); } catch (Exception e) { log.error("ScheduledTask fetchNameServerAddr exception", e); } } }, 1000 * 10, 1000 * 60 * 2, TimeUnit.MILLISECONDS); } this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() { @Override public void run() { try { MQClientInstance.this.updateTopicRouteInfoFromNameServer(); } catch (Exception e) { log.error("ScheduledTask updateTopicRouteInfoFromNameServer exception", e); } } }, 10, this.clientConfig.getPollNameServerInterval(), TimeUnit.MILLISECONDS); this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() { @Override public void run() { try { MQClientInstance.this.cleanOfflineBroker(); MQClientInstance.this.sendHeartbeatToAllBrokerWithLock(); } catch (Exception e) { log.error("ScheduledTask sendHeartbeatToAllBroker exception", e); } } }, 1000, this.clientConfig.getHeartbeatBrokerInterval(), TimeUnit.MILLISECONDS); this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() { @Override public void run() { try { MQClientInstance.this.persistAllConsumerOffset(); } catch (Exception e) { log.error("ScheduledTask persistAllConsumerOffset exception", e); } } }, 1000 * 10, this.clientConfig.getPersistConsumerOffsetInterval(), TimeUnit.MILLISECONDS); this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() { @Override public void run() { try { MQClientInstance.this.adjustThreadPool(); } catch (Exception e) { log.error("ScheduledTask adjustThreadPool exception", e); } } }, 1, 1, TimeUnit.MINUTES); } public String getClientId() { return clientId; } // Get the latest Topic routing information from multiple namesrvs and update the local cache public void updateTopicRouteInfoFromNameServer() { Set<String> topicList = new HashSet<String>(); // Consumer { Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator(); while (it.hasNext()) { Entry<String, MQConsumerInner> entry = it.next(); MQConsumerInner impl = entry.getValue(); if (impl != null) { Set<SubscriptionData> subList = impl.subscriptions(); if (subList != null) { for (SubscriptionData subData : subList) { topicList.add(subData.getTopic()); } } } } } // Producer { Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator(); while (it.hasNext()) { Entry<String, MQProducerInner> entry = it.next(); MQProducerInner impl = entry.getValue(); if (impl != null) { Set<String> lst = impl.getPublishTopicList(); topicList.addAll(lst); } } } for (String topic : topicList) { this.updateTopicRouteInfoFromNameServer(topic); } } /** * @param offsetTable * @param namespace * @return newOffsetTable */ public Map<MessageQueue, Long> parseOffsetTableFromBroker(Map<MessageQueue, Long> offsetTable, String namespace) { HashMap<MessageQueue, Long> newOffsetTable = new HashMap<MessageQueue, Long>(); if (StringUtils.isNotEmpty(namespace)) { for (Entry<MessageQueue, Long> entry : offsetTable.entrySet()) { MessageQueue queue = entry.getKey(); queue.setTopic(NamespaceUtil.withoutNamespace(queue.getTopic(), namespace)); newOffsetTable.put(queue, entry.getValue()); } } else { newOffsetTable.putAll(offsetTable); } return newOffsetTable; } /** * Remove offline broker Check that the Client is valid in the Broker */ private void cleanOfflineBroker() { try { if (this.lockNamesrv.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) try { ConcurrentHashMap<String, HashMap<Long, String>> updatedTable = new ConcurrentHashMap<String, HashMap<Long, String>>(); Iterator<Entry<String, HashMap<Long, String>>> itBrokerTable = this.brokerAddrTable.entrySet().iterator(); while (itBrokerTable.hasNext()) { Entry<String, HashMap<Long, String>> entry = itBrokerTable.next(); String brokerName = entry.getKey(); HashMap<Long, String> oneTable = entry.getValue(); HashMap<Long, String> cloneAddrTable = new HashMap<Long, String>(); cloneAddrTable.putAll(oneTable); Iterator<Entry<Long, String>> it = cloneAddrTable.entrySet().iterator(); while (it.hasNext()) { Entry<Long, String> ee = it.next(); String addr = ee.getValue(); if (!this.isBrokerAddrExistInTopicRouteTable(addr)) { it.remove(); log.info("the broker addr[{} {}] is offline, remove it", brokerName, addr); } } if (cloneAddrTable.isEmpty()) { itBrokerTable.remove(); log.info("the broker[{}] name's host is offline, remove it", brokerName); } else { updatedTable.put(brokerName, cloneAddrTable); } } if (!updatedTable.isEmpty()) { this.brokerAddrTable.putAll(updatedTable); } } finally { this.lockNamesrv.unlock(); } } catch (InterruptedException e) { log.warn("cleanOfflineBroker Exception", e); } } public void checkClientInBroker() throws MQClientException { Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator(); while (it.hasNext()) { Entry<String, MQConsumerInner> entry = it.next(); Set<SubscriptionData> subscriptionInner = entry.getValue().subscriptions(); if (subscriptionInner == null || subscriptionInner.isEmpty()) { return; } for (SubscriptionData subscriptionData : subscriptionInner) { if (ExpressionType.isTagType(subscriptionData.getExpressionType())) { continue; } // may need to check one broker every cluster... // assume that the configs of every broker in cluster are the the same. String addr = findBrokerAddrByTopic(subscriptionData.getTopic()); if (addr != null) { try { this.getMQClientAPIImpl().checkClientInBroker( addr, entry.getKey(), this.clientId, subscriptionData, clientConfig.getMqClientApiTimeout() ); } catch (Exception e) { if (e instanceof MQClientException) { throw (MQClientException) e; } else { throw new MQClientException("Check client in broker error, maybe because you use " + subscriptionData.getExpressionType() + " to filter message, but server has not been upgraded to support!" + "This error would not affect the launch of consumer, but may has impact on message receiving if you " + "have use the new features which are not supported by server, please check the log!", e); } } } } } } // Send the heartbeat information of the client to all brokers public void sendHeartbeatToAllBrokerWithLock() { if (this.lockHeartbeat.tryLock()) { try { this.sendHeartbeatToAllBroker(); this.uploadFilterClassSource(); } catch (final Exception e) { log.error("sendHeartbeatToAllBroker exception", e); } finally { this.lockHeartbeat.unlock(); } } else { log.warn("lock heartBeat, but failed. [{}]", this.clientId); } } private void persistAllConsumerOffset() { Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator(); while (it.hasNext()) { Entry<String, MQConsumerInner> entry = it.next(); MQConsumerInner impl = entry.getValue(); impl.persistConsumerOffset(); } } public void adjustThreadPool() { Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator(); while (it.hasNext()) { Entry<String, MQConsumerInner> entry = it.next(); MQConsumerInner impl = entry.getValue(); if (impl != null) { try { if (impl instanceof DefaultMQPushConsumerImpl) { DefaultMQPushConsumerImpl dmq = (DefaultMQPushConsumerImpl) impl; dmq.adjustThreadPool(); } } catch (Exception e) { } } } } public boolean updateTopicRouteInfoFromNameServer(final String topic) { return updateTopicRouteInfoFromNameServer(topic, false, null); } private boolean isBrokerAddrExistInTopicRouteTable(final String addr) { Iterator<Entry<String, TopicRouteData>> it = this.topicRouteTable.entrySet().iterator(); while (it.hasNext()) { Entry<String, TopicRouteData> entry = it.next(); TopicRouteData topicRouteData = entry.getValue(); List<BrokerData> bds = topicRouteData.getBrokerDatas(); for (BrokerData bd : bds) { if (bd.getBrokerAddrs() != null) { boolean exist = bd.getBrokerAddrs().containsValue(addr); if (exist) return true; } } } return false; } private void sendHeartbeatToAllBroker() { final HeartbeatData heartbeatData = this.prepareHeartbeatData(); final boolean producerEmpty = heartbeatData.getProducerDataSet().isEmpty(); final boolean consumerEmpty = heartbeatData.getConsumerDataSet().isEmpty(); if (producerEmpty && consumerEmpty) { log.warn("sending heartbeat, but no consumer and no producer. [{}]", this.clientId); return; } if (!this.brokerAddrTable.isEmpty()) { long times = this.sendHeartbeatTimesTotal.getAndIncrement(); Iterator<Entry<String, HashMap<Long, String>>> it = this.brokerAddrTable.entrySet().iterator(); while (it.hasNext()) { Entry<String, HashMap<Long, String>> entry = it.next(); String brokerName = entry.getKey(); HashMap<Long, String> oneTable = entry.getValue(); if (oneTable != null) { for (Map.Entry<Long, String> entry1 : oneTable.entrySet()) { Long id = entry1.getKey(); String addr = entry1.getValue(); if (addr != null) { if (consumerEmpty) { if (id != MixAll.MASTER_ID) continue; } try { int version = this.mQClientAPIImpl.sendHearbeat(addr, heartbeatData, clientConfig.getMqClientApiTimeout()); if (!this.brokerVersionTable.containsKey(brokerName)) { this.brokerVersionTable.put(brokerName, new HashMap<String, Integer>(4)); } this.brokerVersionTable.get(brokerName).put(addr, version); if (times % 20 == 0) { log.info("send heart beat to broker[{} {} {}] success", brokerName, id, addr); log.info(heartbeatData.toString()); } } catch (Exception e) { if (this.isBrokerInNameServer(addr)) { log.info("send heart beat to broker[{} {} {}] failed", brokerName, id, addr, e); } else { log.info("send heart beat to broker[{} {} {}] exception, because the broker not up, forget it", brokerName, id, addr, e); } } } } } } } } private void uploadFilterClassSource() { Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator(); while (it.hasNext()) { Entry<String, MQConsumerInner> next = it.next(); MQConsumerInner consumer = next.getValue(); if (ConsumeType.CONSUME_PASSIVELY == consumer.consumeType()) { Set<SubscriptionData> subscriptions = consumer.subscriptions(); for (SubscriptionData sub : subscriptions) { if (sub.isClassFilterMode() && sub.getFilterClassSource() != null) { final String consumerGroup = consumer.groupName(); final String className = sub.getSubString(); final String topic = sub.getTopic(); final String filterClassSource = sub.getFilterClassSource(); try { this.uploadFilterClassToAllFilterServer(consumerGroup, className, topic, filterClassSource); } catch (Exception e) { log.error("uploadFilterClassToAllFilterServer Exception", e); } } } } } } public boolean updateTopicRouteInfoFromNameServer(final String topic, boolean isDefault, DefaultMQProducer defaultMQProducer) { try { if (this.lockNamesrv.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) { try { TopicRouteData topicRouteData; if (isDefault && defaultMQProducer != null) { topicRouteData = this.mQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer(defaultMQProducer.getCreateTopicKey(), clientConfig.getMqClientApiTimeout()); if (topicRouteData != null) { for (QueueData data : topicRouteData.getQueueDatas()) { int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums()); data.setReadQueueNums(queueNums); data.setWriteQueueNums(queueNums); } } } else { topicRouteData = this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, clientConfig.getMqClientApiTimeout()); } if (topicRouteData != null) { TopicRouteData old = this.topicRouteTable.get(topic); boolean changed = topicRouteDataIsChange(old, topicRouteData); if (!changed) { changed = this.isNeedUpdateTopicRouteInfo(topic); } else { log.info("the topic[{}] route info changed, old[{}] ,new[{}]", topic, old, topicRouteData); } if (changed) { TopicRouteData cloneTopicRouteData = topicRouteData.cloneTopicRouteData(); for (BrokerData bd : topicRouteData.getBrokerDatas()) { this.brokerAddrTable.put(bd.getBrokerName(), bd.getBrokerAddrs()); } // Update Pub info { TopicPublishInfo publishInfo = topicRouteData2TopicPublishInfo(topic, topicRouteData); publishInfo.setHaveTopicRouterInfo(true); Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator(); while (it.hasNext()) { Entry<String, MQProducerInner> entry = it.next(); MQProducerInner impl = entry.getValue(); if (impl != null) { impl.updateTopicPublishInfo(topic, publishInfo); } } } // Update sub info { Set<MessageQueue> subscribeInfo = topicRouteData2TopicSubscribeInfo(topic, topicRouteData); Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator(); while (it.hasNext()) { Entry<String, MQConsumerInner> entry = it.next(); MQConsumerInner impl = entry.getValue(); if (impl != null) { impl.updateTopicSubscribeInfo(topic, subscribeInfo); } } } log.info("topicRouteTable.put. Topic = {}, TopicRouteData[{}]", topic, cloneTopicRouteData); this.topicRouteTable.put(topic, cloneTopicRouteData); return true; } } else { log.warn("updateTopicRouteInfoFromNameServer, getTopicRouteInfoFromNameServer return null, Topic: {}. [{}]", topic, this.clientId); } } catch (MQClientException e) { if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) { log.warn("updateTopicRouteInfoFromNameServer Exception", e); } } catch (RemotingException e) { log.error("updateTopicRouteInfoFromNameServer Exception", e); throw new IllegalStateException(e); } finally { this.lockNamesrv.unlock(); } } else { log.warn("updateTopicRouteInfoFromNameServer tryLock timeout {}ms. [{}]", LOCK_TIMEOUT_MILLIS, this.clientId); } } catch (InterruptedException e) { log.warn("updateTopicRouteInfoFromNameServer Exception", e); } return false; } private HeartbeatData prepareHeartbeatData() { HeartbeatData heartbeatData = new HeartbeatData(); // clientID heartbeatData.setClientID(this.clientId); // Consumer for (Map.Entry<String, MQConsumerInner> entry : this.consumerTable.entrySet()) { MQConsumerInner impl = entry.getValue(); if (impl != null) { ConsumerData consumerData = new ConsumerData(); consumerData.setGroupName(impl.groupName()); consumerData.setConsumeType(impl.consumeType()); consumerData.setMessageModel(impl.messageModel()); consumerData.setConsumeFromWhere(impl.consumeFromWhere()); consumerData.getSubscriptionDataSet().addAll(impl.subscriptions()); consumerData.setUnitMode(impl.isUnitMode()); heartbeatData.getConsumerDataSet().add(consumerData); } } // Producer for (Map.Entry<String/* group */, MQProducerInner> entry : this.producerTable.entrySet()) { MQProducerInner impl = entry.getValue(); if (impl != null) { ProducerData producerData = new ProducerData(); producerData.setGroupName(entry.getKey()); heartbeatData.getProducerDataSet().add(producerData); } } return heartbeatData; } private boolean isBrokerInNameServer(final String brokerAddr) { Iterator<Entry<String, TopicRouteData>> it = this.topicRouteTable.entrySet().iterator(); while (it.hasNext()) { Entry<String, TopicRouteData> itNext = it.next(); List<BrokerData> brokerDatas = itNext.getValue().getBrokerDatas(); for (BrokerData bd : brokerDatas) { boolean contain = bd.getBrokerAddrs().containsValue(brokerAddr); if (contain) return true; } } return false; } /** * This method will be removed in the version 5.0.0,because filterServer was removed,and method * <code>subscribe(final String topic, final MessageSelector messageSelector)</code> is recommended. */ @Deprecated private void uploadFilterClassToAllFilterServer(final String consumerGroup, final String fullClassName, final String topic, final String filterClassSource) throws UnsupportedEncodingException { byte[] classBody = null; int classCRC = 0; try { classBody = filterClassSource.getBytes(MixAll.DEFAULT_CHARSET); classCRC = UtilAll.crc32(classBody); } catch (Exception e1) { log.warn("uploadFilterClassToAllFilterServer Exception, ClassName: {} {}", fullClassName, RemotingHelper.exceptionSimpleDesc(e1)); } TopicRouteData topicRouteData = this.topicRouteTable.get(topic); if (topicRouteData != null && topicRouteData.getFilterServerTable() != null && !topicRouteData.getFilterServerTable().isEmpty()) { Iterator<Entry<String, List<String>>> it = topicRouteData.getFilterServerTable().entrySet().iterator(); while (it.hasNext()) { Entry<String, List<String>> next = it.next(); List<String> value = next.getValue(); for (final String fsAddr : value) { try { this.mQClientAPIImpl.registerMessageFilterClass(fsAddr, consumerGroup, topic, fullClassName, classCRC, classBody, 5000); log.info("register message class filter to {} OK, ConsumerGroup: {} Topic: {} ClassName: {}", fsAddr, consumerGroup, topic, fullClassName); } catch (Exception e) { log.error("uploadFilterClassToAllFilterServer Exception", e); } } } } else { log.warn("register message class filter failed, because no filter server, ConsumerGroup: {} Topic: {} ClassName: {}", consumerGroup, topic, fullClassName); } } private boolean topicRouteDataIsChange(TopicRouteData olddata, TopicRouteData nowdata) { if (olddata == null || nowdata == null) return true; TopicRouteData old = olddata.cloneTopicRouteData(); TopicRouteData now = nowdata.cloneTopicRouteData(); Collections.sort(old.getQueueDatas()); Collections.sort(old.getBrokerDatas()); Collections.sort(now.getQueueDatas()); Collections.sort(now.getBrokerDatas()); return !old.equals(now); } private boolean isNeedUpdateTopicRouteInfo(final String topic) { boolean result = false; { Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator(); while (it.hasNext() && !result) { Entry<String, MQProducerInner> entry = it.next(); MQProducerInner impl = entry.getValue(); if (impl != null) { result = impl.isPublishTopicNeedUpdate(topic); } } } { Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator(); while (it.hasNext() && !result) { Entry<String, MQConsumerInner> entry = it.next(); MQConsumerInner impl = entry.getValue(); if (impl != null) { result = impl.isSubscribeTopicNeedUpdate(topic); } } } return result; } public void shutdown() { // Consumer if (!this.consumerTable.isEmpty()) return; // AdminExt if (!this.adminExtTable.isEmpty()) return; // Producer if (this.producerTable.size() > 1) return; synchronized (this) { switch (this.serviceState) { case CREATE_JUST: break; case RUNNING: this.defaultMQProducer.getDefaultMQProducerImpl().shutdown(false); this.serviceState = ServiceState.SHUTDOWN_ALREADY; this.pullMessageService.shutdown(true); this.scheduledExecutorService.shutdown(); this.mQClientAPIImpl.shutdown(); this.rebalanceService.shutdown(); MQClientManager.getInstance().removeClientFactory(this.clientId); log.info("the client factory [{}] shutdown OK", this.clientId); break; case SHUTDOWN_ALREADY: break; default: break; } } } // Register a consumer locally public synchronized boolean registerConsumer(final String group, final MQConsumerInner consumer) { if (null == group || null == consumer) { return false; } MQConsumerInner prev = this.consumerTable.putIfAbsent(group, consumer); if (prev != null) { log.warn("the consumer group[" + group + "] exist already."); return false; } return true; } // Unregistered consumer public synchronized void unregisterConsumer(final String group) { this.consumerTable.remove(group); this.unregisterClient(null, group); } private void unregisterClient(final String producerGroup, final String consumerGroup) { Iterator<Entry<String, HashMap<Long, String>>> it = this.brokerAddrTable.entrySet().iterator(); while (it.hasNext()) { Entry<String, HashMap<Long, String>> entry = it.next(); String brokerName = entry.getKey(); HashMap<Long, String> oneTable = entry.getValue(); if (oneTable != null) { for (Map.Entry<Long, String> entry1 : oneTable.entrySet()) { String addr = entry1.getValue(); if (addr != null) { try { this.mQClientAPIImpl.unregisterClient(addr, this.clientId, producerGroup, consumerGroup, clientConfig.getMqClientApiTimeout()); log.info("unregister client[Producer: {} Consumer: {}] from broker[{} {} {}] success", producerGroup, consumerGroup, brokerName, entry1.getKey(), addr); } catch (RemotingException e) { log.error("unregister client exception from broker: " + addr, e); } catch (InterruptedException e) { log.error("unregister client exception from broker: " + addr, e); } catch (MQBrokerException e) { log.error("unregister client exception from broker: " + addr, e); } } } } } } // Register a producer locally public synchronized boolean registerProducer(final String group, final DefaultMQProducerImpl producer) { if (null == group || null == producer) { return false; } MQProducerInner prev = this.producerTable.putIfAbsent(group, producer); if (prev != null) { log.warn("the producer group[{}] exist already.", group); return false; } return true; } // Unregister local producers public synchronized void unregisterProducer(final String group) { this.producerTable.remove(group); this.unregisterClient(group, null); } // Register an administrative instance public boolean registerAdminExt(final String group, final MQAdminExtInner admin) { if (null == group || null == admin) { return false; } MQAdminExtInner prev = this.adminExtTable.putIfAbsent(group, admin); if (prev != null) { log.warn("the admin group[{}] exist already.", group); return false; } return true; } public void unregisterAdminExt(final String group) { this.adminExtTable.remove(group); } // Perform a rebalance immediately public void rebalanceImmediately() { this.rebalanceService.wakeup(); } // rebalance is performed once for all registered consumer instances public void doRebalance() { for (Map.Entry<String, MQConsumerInner> entry : this.consumerTable.entrySet()) { MQConsumerInner impl = entry.getValue(); if (impl != null) { try { impl.doRebalance(); } catch (Throwable e) { log.error("doRebalance exception", e); } } } } public MQProducerInner selectProducer(final String group) { return this.producerTable.get(group); } public MQConsumerInner selectConsumer(final String group) { return this.consumerTable.get(group); } // Query master or save information in local cache public FindBrokerResult findBrokerAddressInAdmin(final String brokerName) { String brokerAddr = null; boolean slave = false; boolean found = false; HashMap<Long/* brokerId */, String/* address */> map = this.brokerAddrTable.get(brokerName); if (map != null && !map.isEmpty()) { for (Map.Entry<Long, String> entry : map.entrySet()) { Long id = entry.getKey(); brokerAddr = entry.getValue(); if (brokerAddr != null) { found = true; if (MixAll.MASTER_ID == id) { slave = false; } else { slave = true; } break; } } // end of for } if (found) { return new FindBrokerResult(brokerAddr, slave, findBrokerVersion(brokerName, brokerAddr)); } return null; } // Query master information in local cache public String findBrokerAddressInPublish(final String brokerName) { HashMap<Long/* brokerId */, String/* address */> map = this.brokerAddrTable.get(brokerName); if (map != null && !map.isEmpty()) { return map.get(MixAll.MASTER_ID); } return null; } // Query the save information in the local cache public FindBrokerResult findBrokerAddressInSubscribe( final String brokerName, final long brokerId, final boolean onlyThisBroker ) { String brokerAddr = null; boolean slave = false; boolean found = false; HashMap<Long/* brokerId */, String/* address */> map = this.brokerAddrTable.get(brokerName); if (map != null && !map.isEmpty()) { brokerAddr = map.get(brokerId); slave = brokerId != MixAll.MASTER_ID; found = brokerAddr != null; if (!found && slave) { brokerAddr = map.get(brokerId + 1); found = brokerAddr != null; } if (!found && !onlyThisBroker) { Entry<Long, String> entry = map.entrySet().iterator().next(); brokerAddr = entry.getValue(); slave = entry.getKey() != MixAll.MASTER_ID; found = true; } } if (found) { return new FindBrokerResult(brokerAddr, slave, findBrokerVersion(brokerName, brokerAddr)); } return null; } public int findBrokerVersion(String brokerName, String brokerAddr) { if (this.brokerVersionTable.containsKey(brokerName)) { if (this.brokerVersionTable.get(brokerName).containsKey(brokerAddr)) { return this.brokerVersionTable.get(brokerName).get(brokerAddr); } } //To do need to fresh the version return 0; } // Find a list of consumer IDS public List<String> findConsumerIdList(final String topic, final String group) { String brokerAddr = this.findBrokerAddrByTopic(topic); if (null == brokerAddr) { this.updateTopicRouteInfoFromNameServer(topic); brokerAddr = this.findBrokerAddrByTopic(topic); } if (null != brokerAddr) { try { return this.mQClientAPIImpl.getConsumerIdListByGroup(brokerAddr, group, clientConfig.getMqClientApiTimeout()); } catch (Exception e) { log.warn("getConsumerIdListByGroup exception, " + brokerAddr + " " + group, e); } } return null; } // Find the Broker address by Topic name public String findBrokerAddrByTopic(final String topic) { TopicRouteData topicRouteData = this.topicRouteTable.get(topic); if (topicRouteData != null) { List<BrokerData> brokers = topicRouteData.getBrokerDatas(); if (!brokers.isEmpty()) { int index = random.nextInt(brokers.size()); BrokerData bd = brokers.get(index % brokers.size()); return bd.selectBrokerAddr(); } } return null; } // Reset consumption location public synchronized void resetOffset(String topic, String group, Map<MessageQueue, Long> offsetTable) { DefaultMQPushConsumerImpl consumer = null; try { MQConsumerInner impl = this.consumerTable.get(group); if (impl != null && impl instanceof DefaultMQPushConsumerImpl) { consumer = (DefaultMQPushConsumerImpl) impl; } else { log.info("[reset-offset] consumer dose not exist. group={}", group); return; } consumer.suspend(); ConcurrentMap<MessageQueue, ProcessQueue> processQueueTable = consumer.getRebalanceImpl().getProcessQueueTable(); for (Map.Entry<MessageQueue, ProcessQueue> entry : processQueueTable.entrySet()) { MessageQueue mq = entry.getKey(); if (topic.equals(mq.getTopic()) && offsetTable.containsKey(mq)) { ProcessQueue pq = entry.getValue(); pq.setDropped(true); pq.clear(); } } try { TimeUnit.SECONDS.sleep(10); } catch (InterruptedException e) { } Iterator<MessageQueue> iterator = processQueueTable.keySet().iterator(); while (iterator.hasNext()) { MessageQueue mq = iterator.next(); Long offset = offsetTable.get(mq); if (topic.equals(mq.getTopic()) && offset != null) { try { consumer.updateConsumeOffset(mq, offset); consumer.getRebalanceImpl().removeUnnecessaryMessageQueue(mq, processQueueTable.get(mq)); iterator.remove(); } catch (Exception e) { log.warn("reset offset failed. group={}, {}", group, mq, e); } } } } finally { if (consumer != null) { consumer.resume(); } } } // Gets the consumption progress of each queue in a subscription relationship public Map<MessageQueue, Long> getConsumerStatus(String topic, String group) { MQConsumerInner impl = this.consumerTable.get(group); if (impl != null && impl instanceof DefaultMQPushConsumerImpl) { DefaultMQPushConsumerImpl consumer = (DefaultMQPushConsumerImpl) impl; return consumer.getOffsetStore().cloneOffsetTable(topic); } else if (impl != null && impl instanceof DefaultMQPullConsumerImpl) { DefaultMQPullConsumerImpl consumer = (DefaultMQPullConsumerImpl) impl; return consumer.getOffsetStore().cloneOffsetTable(topic); } else { return Collections.EMPTY_MAP; } } public TopicRouteData getAnExistTopicRouteData(final String topic) { return this.topicRouteTable.get(topic); } public MQClientAPIImpl getMQClientAPIImpl() { return mQClientAPIImpl; } public MQAdminImpl getMQAdminImpl() { return mQAdminImpl; } public long getBootTimestamp() { return bootTimestamp; } public ScheduledExecutorService getScheduledExecutorService() { return scheduledExecutorService; } public PullMessageService getPullMessageService() { return pullMessageService; } public DefaultMQProducer getDefaultMQProducer() { return defaultMQProducer; } // Get local cached Topic route public ConcurrentMap<String, TopicRouteData> getTopicRouteTable() { return topicRouteTable; } // The message is directly sent to the specified consumer. Different from normal delivery, it specifies one of the subscribed consumer groups, not all. It is generally applicable to the scenario where a consumer group wants to consume again after the consumption message public ConsumeMessageDirectlyResult consumeMessageDirectly(final MessageExt msg, final String consumerGroup, final String brokerName) { MQConsumerInner mqConsumerInner = this.consumerTable.get(consumerGroup); if (null != mqConsumerInner) { DefaultMQPushConsumerImpl consumer = (DefaultMQPushConsumerImpl) mqConsumerInner; ConsumeMessageDirectlyResult result = consumer.getConsumeMessageService().consumeMessageDirectly(msg, brokerName); return result; } return null; } // Obtain the consumption statistics of consumers, including consumption RT, consumption tps, etc public ConsumerRunningInfo consumerRunningInfo(final String consumerGroup) { MQConsumerInner mqConsumerInner = this.consumerTable.get(consumerGroup); if (mqConsumerInner == null) { return null; } ConsumerRunningInfo consumerRunningInfo = mqConsumerInner.consumerRunningInfo(); List<String> nsList = this.mQClientAPIImpl.getRemotingClient().getNameServerAddressList(); StringBuilder strBuilder = new StringBuilder(); if (nsList != null) { for (String addr : nsList) { strBuilder.append(addr).append(";"); } } String nsAddr = strBuilder.toString(); consumerRunningInfo.getProperties().put(ConsumerRunningInfo.PROP_NAMESERVER_ADDR, nsAddr); consumerRunningInfo.getProperties().put(ConsumerRunningInfo.PROP_CONSUME_TYPE, mqConsumerInner.consumeType().name()); consumerRunningInfo.getProperties().put(ConsumerRunningInfo.PROP_CLIENT_VERSION, MQVersion.getVersionDesc(MQVersion.CURRENT_VERSION)); return consumerRunningInfo; } public ConsumerStatsManager getConsumerStatsManager() { return consumerStatsManager; } public NettyClientConfig getNettyClientConfig() { return nettyClientConfig; } public ClientConfig getClientConfig() { return clientConfig; } }