Recently, RocketMQ has just started its production environment. In your spare time, share something here. The main purpose is for beginners to get started with RocketMQ quickly.

What is RocketMQ

Introduction to RocketMQ on Github:
RcoketMQ is a message middleware with low latency, high reliability, scalability and ease of use.
It has the following characteristics:

1. Supports publish/subscribe (Pub/Sub) and peer-to-peer (P2P) messaging models
2. Reliable FIFO and strict sequential delivery in a queue
3. Supports pull and push message modes
4. The stacking power of a single queue of millions of messages
5. Supports multiple messaging protocols, such as JMS, MQTT, and so on
6. Distributed highly available deployment architecture that satisfies at least one messaging semantics
7. Provide docker mirrors for isolation testing and cloud cluster deployment
8. Dashboard with rich configuration, metrics, and monitoring capabilities

For these feature descriptions, you can simply look at them, and then you will understand them naturally after you learn more.

Technical Term

Producer

Message producer, producer's role is to send messages to MQ, producer itself can produce messages, such as reading text information.You can also provide an interface to the outside world, which is called by the outside application, and then sent to the MQ by the producer.

Producer Group

A producer group is simply a group of producers who send the same type of message.You don't have to worry about it here, just know it.

Consumer

Message consumers. Simply put, the application that consumes messages on MQ is the consumer. Whether messages are processed logically or stored directly in the database depends on business needs.

Consumer Group

A consumer group, similar to a producer, consists of multiple consumer instances consuming the same type of message.

Topic

Topic is a logical categorization of messages, for example, if you have an order class message and an inventory class message, then you need to categorize. One is the order Topic to store the order related message, the other is the inventory Topic to store the inventory related message.

Message

Message is the carrier of message.A message must specify a topic, which corresponds to the address to which it is sent.Message also has an optional tag setting so that consumers can filter messages based on tags.Additional key-value pairs can also be added, for example, if you need a business key to find messages on the broker to diagnose problems during development.

Tag

Tags can be thought of as further refinement of Topic.Messages for different purposes are typically tagged in the same business module by introducing tags.

Broker

Broker is the main role of the Rocket MQ system, which is what has been said before.Broker receives messages from producers, stores them, and prepares consumers for requests to cancel interest rates.

Name Server

Name Server provides routing information for producer s and consumer s.

RocketMQ architecture

From this diagram, you can see four clusters, NameServer, Broker, Producer, and Constumer.

1. NameServer: Provides lightweight service discovery and routing.Each NameServer records complete routing information, provides equivalent read and write services, and supports fast storage extensions.
2. Broker: Handles message storage by providing lightweight Topic and Queue mechanisms, while supporting push and pull modes as well as fault-tolerant mechanisms for master-slave structures.
3. Producer: Producer, the instance that produces the message, and Producer with the same Producer Group forms a cluster.
4. Consumer: Consumer, the instance that receives a message to consume, Constumers with the same Consumer Group form a cluster.

To briefly illustrate what the arrows mean in the diagram, starting with Broker, Broker Master1 and Broker Slave1 are master-slave structures that synchronize data with each other, Date Sync.All sections in each broker and NameServer cluster at the same time
Points make long connections and register Topic information periodically to all NameServer s.
Producer establishes a long connection with one of the nodes in the NameServer cluster (randomly selected), periodically obtains Topic routing information from NameServer, establishes a long connection to the Broker Master providing Topic services, and periodically sends a heartbeat to the Broker.Producer can only send messages to Broker Master, but Consumer does not. It also makes a long connection to Master and Slave, which provide Topic services, either subscribing to messages from Broker Master or from Broker Slave.

RocketMQ cluster deployment mode

1. Single master mode
   That is, there is only one master node, not a cluster. Once the master node is down, the entire service is unavailable and suitable for individual learning.
2. Multiple master mode
   Multiple master nodes form a cluster, and a single master node downtime or restart has no effect on the application.
   Benefits: Highest performance in all modes
   Disadvantages: During the downtime of a single master node, messages that are not consumed are not available until the node is restored, and the real-time nature of the messages is affected.
   Note: Using a synchronous brush disk ensures that messages are not lost, and the corresponding queue for Topic should be distributed among the nodes in the cluster, not just on one node, otherwise the downtime of that node can have an impact on the application subscribing to that top.
3. Multiple master multiple slave asynchronous replication mode
   On the basis of multiple master mode, each master node has at least one slave corresponding to it.The master node is readable and writable, but slaves are read-only and not writable, similar to mysql's master mode.
   Advantage: When master is down, consumers can read messages from slave, the real-time message is not affected, and the performance is almost the same as many masters
   Disadvantages: Synchronization with asynchronous replication can cause message loss problems.
4. Synchronized Dual Write Mode with Multiple Masters and Multiple Slives
   Similar to the multiple master multiple slave asynchronous replication mode, the difference is how data is synchronized between master and slave.
   Benefits: Synchronized dual-write synchronization mode ensures data is not lost
   Disadvantages: Sending a single message RT is slightly longer and has about 10% lower performance than asynchronous replication.
   Brush Disk Policy: Synchronous and asynchronous brush disks (refers to whether the node's own data is stored synchronously or asynchronously)
   Synchronization: Synchronized double-write and asynchronous replication (referring to the synchronization of data between a set of master s and slave s)
   Note: To ensure reliable data, synchronous brush disc and synchronous dual writing are required, but the performance will be lower than other methods.

RocketMQ Single Primary Deployment

As a quick start, I chose the first single master deployment mode.
First, explain my installation environment:

1. Centos 7.2
2. jdk 1.8
3. Maven 3.2.x
4. Git

Here git is available or not, primarily to download the source code directly on github.You can also choose to download it on GitHub yourself and upload it to the server.Take the git operation as an example.

1. clone source and compiled with maven

   > git clone https://github.com/alibaba/RocketMQ.git /opt/RocketMQ
   > cd /opt/RocketMQ && mvn -Dmaven.test.skip=true clean package install assembly:assembly -U
   > cd target/alibaba-rocketmq-broker/alibaba-rocketmq

   Possible problems here
   1. Executing git clone https://github.com/alibaba/RocketMQ.git The following prompts appear when /home/inspkgs/RocketMQ:

   fatal: unable to access 'https://github.com/alibaba/RocketMQ.git/': Could not resolve host: github.com; Unknown error

   Solution: Generally due to network reasons, execute the following commands

   > ping github.com

   Once you've determined you can ping, re-execute the git clone command.
   2. When executing mvn-Dmaven.test.skip=true clean package install assembly:assembly-U compilation, there may be a slow download of related jar s.
   This is also because the default Maven central warehouse is abroad and can be recompiled as needed after adding the following to <mirrors></mirrors>in/home/maven/conf/setting.xml:

   <mirror>
    <id>aliyun</id>
    <mirrorOf>central</mirrorOf>
    <name>aliyun maven</name>
    <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
   </mirror>

2. Start Name Server

   > nohup sh /opt/RocketMQ/bin/mqnamesrv &
   //Execute jps view process
   > jps
   25913 NamesrvStartup
   //Check the log to make sure the service is started properly
   > tail -f ~/logs/rocketmqlogs/namesrv.log
   The Name Server boot success...

3. Start broker

   > nohup sh /opt/RocketMQ/bin/mqbroker -n localhost:9876 &
   //Execute jps view process
   > jps
   25954 BrokerStartup
   //Check the log to make sure the service is started properly
   > tail -f ~/logs/rocketmqlogs/broker.log 
   The broker[broker-a, 10.1.54.121:10911] boot success...

4. Send and receive messages
   Before sending/receiving messages, we need to tell the client the NameServer address.RocketMQ offers a variety of ways to achieve this goal.For simplicity, we use the environment variable NAMESRV_ADDR

   > export NAMESRV_ADDR=localhost:9876
   > sh /opt/RocketMQ/bin/tools.sh com.alibaba.rocketmq.example.quickstart.Producer
   SendResult [sendStatus=SEND_OK, msgId= ...
   > sh /opt/RocketMQ/bin/tools.sh com.alibaba.rocketmq.example.quickstart.Consumer
   ConsumeMessageThread_%d Receive New Messages: [MessageExt...

5. Shut down services

   > sh /opt/RocketMQ/bin/mqshutdown broker
   The mqbroker(36695) is running...
   Send shutdown request to mqbroker(36695) OK
   > sh /opt/RocketMQ/bin/mqshutdown namesrv
   The mqnamesrv(36664) is running...
   Send shutdown request to mqnamesrv(36664) OK

Producer, Consumer Demo

1. Producer

   public class Producer {
    public static void main(String[] args) throws MQClientException, InterruptedException {
   
        //Declare and initialize a producer
        //A producer group name is required as an argument to the constructor, where producer1
        DefaultMQProducer producer = new DefaultMQProducer("producer1");
   
        //Set the NameServer address, which should be changed to the actual NameServer address, used between multiple addresses; separate
        //NameServer must have an address, but it can also be set as an environment variable, not necessarily in code
        producer.setNamesrvAddr("10.1.54.121:9876;10.1.54.122:9876");
   
        //Invoke the start() method to start a producer instance
        producer.start();
   
        //Send 10 messages to Topic for TopicTest, tag for TagA, and message content for i value on "Hello RocketMQ" stitch
        for (int i = 0; i < 10; i++) {
            try {
                Message msg = new Message("TopicTest",// topic
                        "TagA",// tag
                        ("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET)// body
                );
   
                //Call producer's send() method to send a message
                //This calls synchronization, so the result is returned
                SendResult sendResult = producer.send(msg);
   
                //Print the returned results to see the status of the message and some relevant information
                System.out.println(sendResult);
            } catch (Exception e) {
                e.printStackTrace();
                Thread.sleep(1000);
            }
        }
   
        //After sending the message, call the shutdown() method to close producer
        producer.shutdown();
    }
   }

2. Consumer

   public class Consumer {
   
    public static void main(String[] args) throws InterruptedException, MQClientException {
   
        //Declare and initialize a consumer
        //A consumer group name is required as an argument to the constructor, where consumer1
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("consumer1");
   
        //Also set NameServer address
        consumer.setNamesrvAddr("10.1.54.121:9876;10.1.54.122:9876");
   
        //Here's a consumer policy
        //CONSUME_FROM_LAST_OFFSET default policy, consuming from the end of the queue, skipping historical messages
        //CONSUME_FROM_FIRST_OFFSET starts consuming from the beginning of the queue, that is, all historical messages (also stored in broker) are consumed once
        //CONSUME_FROM_TIMESTAMP starts at a certain point in time and is used with setConsumeTimestamp(), which defaults to half an hour ago
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
   
        //Set the Topic and Tag that consumer subscribes to, * for all Tags
        consumer.subscribe("TopicTest", "*");
   
        //Set up a Listener for logical processing of messages
        consumer.registerMessageListener(new MessageListenerConcurrently() {
   
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
                                                            ConsumeConcurrentlyContext context) {
   
                System.out.println(Thread.currentThread().getName() + " Receive New Messages: " + msgs);
   
                //Return to consumption
                //CONSUME_SUCCESS Consumption Success
                //RECONSUME_LATER consumption failed and needs to be re-consumed later
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
   
        //Call the start() method to start consumer
        consumer.start();
   
        System.out.println("Consumer Started.");
    }
   }

At the end of this article, I am writing an article for the first time. Where I am not clear about it, I would also like to invite your little partner Haihan.If you have problems, you can communicate more.If I have time later, I will also share more dry goods, including detailed API s for producer, consumer, how to integrate with spring boot, some considerations for production environment deployment, and learning about the pits that RocketMQ has traveled through.
Finally, if you feel your writing is OK, it is a kind of affirmation that you like it or appreciate it a little. Thank you for your support.