01. MQ overview

Product categories and comparison of MQ

At present, the mainstream message queue middleware in the market mainly include Kafka, ActiveMQ, RabbitMQ, RocketMQ, etc.

ActiveMQ and RabbitMQ are only used by companies with average business volume due to throughput. RabbitMQ is developed in erlang language and we don't understand it. Therefore, the expansion and maintenance costs are very high, and it's a headache to find a problem.

Kafka and RocketMQ have been shining in their respective fields, with considerable throughput, reliability and timeliness.

kafka:

- programing language: scala
- Mainstream of big data MQ. 

RabbitMQ:

- programing language: erlang
- be based on erlang The language is not easy to modify the bottom layer. Do not find the cause of the problem. It is not recommended.

RocketMQ

- programing language: java
- Suitable for large projects. Applicable to clusters.

ActiveMQ

- programing language: java
- It is applicable to small and medium-sized projects.

Let's see the comparison of these message intermediaries through the chart:

characteristic	ActiveMQ	RabbitMQ	RocketMQ	Kafka
Single machine throughput	10000, one order of magnitude lower than RocketMQ and Kafka	Same as ActiveMQ	100000 class, supporting high throughput	100000 level, high throughput, generally cooperate with big data systems for real-time data calculation, log collection and other scenarios
Impact of topic number on throughput			Topics can reach hundreds / thousands of levels, and the throughput will decrease slightly. This is a major advantage of RocketMQ. A large number of topic s can be supported on the same machine	When topics range from tens to hundreds, the throughput will decrease significantly. Under the same machine, Kafka tries to ensure that the number of topics is not too large. If large-scale topics are to be supported, more machine resources need to be added
Timeliness	ms level	Subtle level, which is a major feature of RabbitMQ, with the lowest latency	ms level	Delay within ms level
usability	High availability based on master-slave architecture	Same as ActiveMQ	Very high, distributed architecture	Very high, distributed, multiple copies of one data, downtime of a few machines, no data loss and unavailability
Message reliability	There is a low probability of data loss	Basically not lost	After the optimized configuration of parameters, 0 can be lost	Same as RocketMQ
Functional support	The MQ domain is extremely functional	Based on erlang development, it has strong concurrency, good performance and low latency	MQ functions are relatively complete, distributed and scalable	The function is relatively simple. It mainly supports simple MQ functions. Real time calculation and log collection in the field of big data are used on a large scale
Community activity	low	in	high	high

Background of MQ

Problems with direct calls between systems?

After the microservice architecture, chain call is the general process when we write programs. In order to complete an overall function, it will be divided into multiple functions (or sub modules), such as module A calls module B, module B calls module C, and module C calls module D.

However, in large-scale distributed applications, RPC interaction between systems is complex. It is not impossible to call hundreds of interfaces behind a function. It is a general example of the transition from stand-alone architecture to distributed microservice architecture. What are the problems with these architectures?

• (1) The interface coupling between systems is serious

  For each new downstream function, the relevant upstream interfaces shall be reconstructed;

• (2) In the face of large traffic concurrency, it is easy to be washed out.

  The throughput capacity of each interface module is limited. If the upper limit capacity is a dam, it is easy to be washed out when a large flow (flood) comes.
  For example, second kill business: the upstream system initiates an order purchase operation, that is, an order is placed, which will be completed soon. However, the downstream system needs to complete all logic behind the second kill business (read order, inventory check, inventory freeze, balance check, balance freeze, order production, balance deduction, inventory reduction, generate flow, balance unfreeze, inventory unfreeze).

• (3) There is a performance problem waiting for synchronization

  RPC interfaces are basically synchronous calls. The overall service performance follows the barrel theory, that is, the time consumption of the overall system depends on the slowest interface in the link.

According to the above problems, the objectives to be achieved can be defined when designing the system:

• 1. The system shall be decoupled. When a new module is connected, the code change can be minimized and decoupled
• 2. Set the flow buffer pool to enable the back-end system to consume according to its own throughput capacity without being washed out and peak shaving
• 3. Strong and weak dependency sorting can asynchronize the operation of non critical call links and improve the throughput of the overall system

Main role of MQ

• (1) Asynchronous. The caller does not have to wait.
• (2) Decoupling. The problem of coupling calls between systems is solved.
• (3) Peak elimination. Resist the peak flow and protect the main business.

Definition of MQ

Message Oriented Middleware (Message Oriented Middleware) MOM can well solve the above problems. It refers to the use of efficient and reliable messaging mechanism for platform independent data exchange, and the integration of distributed systems based on data communication. By providing message passing and message queuing models, it provides application decoupling, elastic scaling, redundant storage, traffic peak shaving and asynchronous communication in a distributed environment Communication, data synchronization and other functions.
The general process is as follows: the sender sends the message to the message server, which stores the message in several queues / topic s. When appropriate, the message server forwards the message back to the receiver. In this process, the sending and receiving are asynchronous, that is, the sending does not need to wait, and there is no inevitable relationship between the life cycle of the sender and the receiver; Especially in the publish pub / subscribe sub mode, one to many communication can also be completed, that is, a message has multiple recipients.

Features of MQ

(1) Asynchronous processing mode is adopted

• The message sender can send a message without waiting for a response. The message sender sends the message to a virtual channel (subject or queue);

• The message receiver subscribes to or listens to the love channel. A message may eventually be forwarded to one or more message recipients, who do not need to respond synchronously to the message sender. The whole process is asynchronous.

• Case:
  In other words, when A system communicates with another system, if system A wants to send A message to system B, let him handle it. However, system A does not pay attention to how system B handles it or whether it has handled it well, so system A sends the message to MQ, regardless of the life or death of the message, and then system B can consume it from MQ for processing. As for how to handle it, whether it is finished or when it is handled, it is all A matter of system B, not system A.
  

(2) Decoupling between application systems

• The sender and receiver don't have to know each other, they just need to confirm the message.
• The sender and receiver do not have to be online at the same time.

(3) Overall architecture

(4) Disadvantages of MQ

• The two systems cannot call synchronously, reply in real time, and respond to the reply of a call.

02. RPC architecture

What is RPC architecture?

RPC (Remote Procedure Call): that is, remote procedure call. It is a common communication method in distributed systems. It allows programs to call procedures or functions in another address space (usually on another machine sharing the network), without programmers explicitly coding the details of the remote call.

In addition to RPC, common multi system data interaction schemes include distributed message queue, HTTP request call, database and distributed cache.

RPC and HTTP calls do not go through middleware. They are direct data interaction of end-to-end system.

The schematic diagram is as above, that is, two servers a and B, one application is deployed on server A. if you want to call the functions / methods provided by the application on server B, because they are not in the same memory space, they cannot be called directly. You need to express the semantics of the call and convey the called data through the network.

For example, server A wants to call A method on server B: Employee getEmployeeByName(String fullName)

The whole RPC calling process mainly goes through the following steps:

• 1. Establish communication

  First of all, we should solve the problem of communication: that is, if machine A wants to call machine B, it must first establish A communication connection.

  It is mainly through the establishment of a TCP connection between the client and the server. All the exchanged data of remote procedure calls are transmitted in this connection. The connection can be an on-demand connection, which is broken after the call is completed, or a long connection. Multiple remote procedure calls share the same connection.

• 2. Service addressing

  To solve the problem of addressing, that is, how the application on server A tells the underlying RPC framework, how to connect to server B (such as host or IP address) and specific port, and what is the name of the method.

  Usually, we need to provide the B machine (host name or IP address) and specific port, and then specify the name of the called method or function and the input / output parameter information, so as to complete a service call.

  Reliable addressing (mainly service discovery) is the cornerstone of RPC implementation. For example, redis or zookeeper can be used to register services.

  From the perspective of service provider: when the service provider starts, it needs to automatically register the service with the registry;
  When the provider service stops, it is necessary to log off the service from the registry;
  The provider needs to send heartbeat to the registry regularly. After not receiving heartbeat from the provider for a period of time, it is considered that the provider has stopped the service and removed the corresponding service from the registry.

  From the perspective of the caller: when the caller starts, he subscribes to the message of the registry and obtains the address of the provider from the registry;
  When a provider goes online or offline, the registry will inform the caller;
  Unsubscribe when the caller goes offline.

• 3. Network transmission

  serialize
  When an application on machine A initiates an RPC call, the calling method and its input parameter information need to be transmitted to machine B through the underlying network protocol such as TCP. Because the network protocol is based on binary, All parameter data we transmit need to be serialized or marshaled into binary form before it can be transmitted in the network. Then, the serialized or marshaled binary data will be sent to machine B through addressing operation and network transmission.

  Deserialization
  After machine B receives the request from the application of machine A, it needs to deserialize the received parameters and other information (reverse operation of serialization), that is, restore the binary information to the expression in memory, and then find the corresponding method (part of addressing) for local call (generally called by generating Proxy),

  There are usually JDK dynamic proxy, CGLIB dynamic proxy, Javassist bytecode generation technology, etc.), and then get the return value of the call.

• 4. Service call

  Local call to machine B After the return value is obtained (through Proxy), it is also necessary to send the return value back to machine A. similarly, it also needs to undergo serialization operation, and then send the binary data back to machine a through network transmission. When machine a receives these return values, it will conduct deserialization operation again to restore the expression in memory, and finally hand them to the expression on machine a The application performs related processing (generally business logic processing operations).

Common RPC architectures

• Thrift: thrift is a software framework for the development of extensible and cross language services. It combines a powerful software stack and code generation engine to build on C + +, Java, python, PHP, ruby, Erlang, Perl, Haskell, c#, cocoa, JavaScript, node JS, Smalltalk, and Ocaml are seamless and efficient services between these programming languages.

• Dubbo: Dubbo is a distributed service framework and SOA governance solution. Its functions mainly include: high-performance NIO communication and multi protocol integration, service dynamic addressing and routing, soft load balancing and fault tolerance, dependency analysis and degradation, etc. Dubbo is the core framework of Alibaba's internal SOA service-oriented governance scheme. Since its open source in 2011, Dubbo has been used by many non Alibaba companies.

• Spring Cloud: Spring Cloud is composed of many subprojects, such as Spring Cloud Config, Spring Cloud Netflix, Spring Cloud consult, etc. it provides common tools for building distributed systems and micro services, such as configuration management, service discovery, circuit breaker, intelligent routing, micro agent, control bus, one-time token, global lock, master selection, distributed session and cluster status, It meets all the solutions needed to build microservices. Spring Cloud is based on Spring Boot, which makes development and deployment extremely simple.

03. ActiveMQ installation

Refer to this blog: https://blog.csdn.net/weixin_45583303/article/details/119617825

04. Introduction case

queue introduction case

pom.xml import dependency

<dependencies>
  <!-- activemq Required jar package-->
  <dependency>
    <groupId>org.apache.activemq</groupId>
    <artifactId>activemq-all</artifactId>
    <version>5.15.9</version>
  </dependency>
  <!-- activemq and spring Consolidated infrastructure package -->
  <dependency>
    <groupId>org.apache.xbean</groupId>
    <artifactId>xbean-spring</artifactId>
    <version>3.16</version>
  </dependency>
</dependencies>

Basic steps of JMS development

MQ push destination

In the point-to-point messaging domain, destinations are called queues
In the publish and subscribe messaging domain, destinations are called topics

Getting started with queued message producers

public class JmsProduce {
    //  IP address of activemq deployed on linux + port number of activemq
    public static final String ACTIVEMQ_URL = "tcp://127.0.0.1:61616";
    // Name of destination
    public static final String QUEUE_NAME = "first_queue";
    
    public static void main(String[] args) throws  Exception{
        // 1 create the connection factory according to the given url, and the constructor adopts the default user name and password. Other constructors of this class can specify a user name and password.
        ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory(ACTIVEMQ_URL);
        // 2. Through the connection factory, obtain the connection and start the access.
        Connection connection = activeMQConnectionFactory.createConnection();
        connection.start();
        // 3 create a session. The first parameter is whether to start the transaction, and the second parameter is the message signing method
        Session session = connection.createSession(false,Session.AUTO_ACKNOWLEDGE);
        // 4 create destinations (two types: Queue / Topic). Destination is the parent class of Queue and Topic
        Queue queue = session.createQueue(QUEUE_NAME);
        // 5 create the producer of the message
        MessageProducer messageProducer = session.createProducer(queue);
        // 6. Three messages produced by messageProducer are sent to the message queue
        for (int i = 1; i < 4 ; i++) {
            // 7 create message
            TextMessage textMessage = session.createTextMessage("msg--" + i);
            // 8 send to mq via messageProducer
            messageProducer.send(textMessage);
        }
        // 9 close resources
        messageProducer.close();
        session.close();
        connection.close();
        System.out.println("  **** Message sent to MQ complete ****");
    }
}

Queue of ActiveMQ console

Run the above code, and the console displays as follows:

Number Of Pending Messages:

• Messages waiting to be consumed. This is the number of messages not out of the queue. The formula = total received - Total out of the queue.

Number Of Consumers:

• Number of consumers, the number of consumers on the consumer side.

Messages Enqueued:

• The number of incoming messages, the total number of messages in the queue, including those out of the queue, only increases.

Messages Dequeued:

• The number of outgoing messages can be understood as the number consumed by consumers.

Summary:

• When a message enters the queue, the message waiting to be consumed is 1 and the message entering the queue is 1.
• When the message is consumed, the message waiting to be consumed is 0, the message entering the queue is 1, and the message leaving the queue is 1.
• When there is another message, the message waiting to be consumed is 1 and the message entering the queue is 2.

Getting started with queued message consumers

// Message consumer
public class JmsConsumer {

    public static final String ACTIVEMQ_URL = "tcp://127.0.0.1:61616";
    public static final String QUEUE_NAME = "first_queue";

    public static void main(String[] args) throws Exception{
        ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory(ACTIVEMQ_URL);
        javax.jms.Connection connection = activeMQConnectionFactory.createConnection();
        connection.start();
        Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
        Queue queue = session.createQueue(QUEUE_NAME);
        // 5 consumers who create messages
        MessageConsumer messageConsumer = session.createConsumer(queue);
        while(true){
            // reveive() has been waiting for the message to be received. It will be blocked until the message can be received. The synchronous blocking method is similar to the socket accept method.
            // If no message is received after waiting for n milliseconds, the blocking ends.
            // Because the message sender is TextMessage, the message receiver is also TextMessage
            TextMessage message = (TextMessage)messageConsumer.receive();
            if (null != message){
                System.out.println("****Consumer message:"+message.getText());
            }else {
                break;
            }
        }
        messageConsumer.close();
        session.close();
        connection.close();
    }
}

Console display:

Asynchronous listening consumer (MessageListener)

// The consumer of the message is the system that answers the message
public class JmsConsumer02 {
    public static final String ACTIVEMQ_URL = "tcp://127.0.0.1:61616";

    public static final String QUEUE_NAME = "first_queue";

    public static void main(String[] args) throws Exception{
        ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory(ACTIVEMQ_URL);
        javax.jms.Connection connection = activeMQConnectionFactory.createConnection();
        connection.start();
        Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
        Queue queue = session.createQueue(QUEUE_NAME);
        MessageConsumer messageConsumer = session.createConsumer(queue);

        /*
         * Consume messages by listening, which is asynchronous and non blocking.
         * Register a listener through the setMessageListener of messageConsumer,
         * When a message is sent, the system automatically calls the onMessage method of MessageListener to process the message
         */
        messageConsumer.setMessageListener(new MessageListener() {
            public void onMessage(Message message)  {
                //  instanceof determines whether an object A is A subclass of class B
                if (null != message  && message instanceof TextMessage){
                    TextMessage textMessage = (TextMessage)message;
                    try {
                        System.out.println("****Consumer message:"+textMessage.getText());
                    }catch (JMSException e) {
                        e.printStackTrace();
                    }
                }
            }
        });
        // Let the main thread not end, because once the main thread ends, other threads (the threads listening for messages here) will also be forced to end.
        // In actual development, our program will run all the time, and this code will be omitted.
        System.in.read();
        messageConsumer.close();
        session.close();
        connection.close();
    }
}

Queue summary

(1) Two consumption modes

Synchronous blocking mode(receive)
	- Subscriber or recipient credit MessageConsumer of receive()Method to receive a message, receive Method will block until it can receive a message (or until it times out).

Asynchronous non blocking mode (listener) onMessage())
	- Subscriber or recipient via MessageConsumer of setMessageListener(MessageListener listener)register
	- A message listener. When the message arrives, the system will automatically call the listener MessageListener of onMessage(Message message)method.

(2) Characteristics of queue (characteristics of point-to-point messaging domain are as follows):

- 1,Each message can only be consumed by one consumer, similar to one to one It's like personal express to get your own
- 2,There is no time correlation between the message producer and the message consumer, regardless of whether the consumer is running when the message sender sends the message.
 	 Consumers can extract messages. For example, we send text messages. After the sender sends them, the recipient may not view the messages in time.
- 3,The consumed messages will not be stored in the queue, so consumers will not consume the consumed messages.

(3) Message consumption

Case 1: only start consumer 1.
Result: consumer 1 will consume all the data.

Situation 2: start consumer 1 first and then consumer 2.
Results: consumer 1 consumed all the data. Consumer 2 will not consume the message.

Situation 3: the producer releases 6 messages, and consumers 1 and 2 have been started before.
Result: consumer 1 and consumer 2 shared the news equally. Consume 3 messages respectively.

Question: how to divide consumers 1 and 2 unevenly? But according to their respective consumption capacity.

topic introduction case

topic overview

In the publish subscribe messaging domain, a destination is called a topic
The characteristics of publish / subscribe messaging domain are as follows:
(1) The producer publishes messages to topic. Each message can have multiple consumers, belonging to a 1: N relationship;
(2) There is a temporal correlation between producers and consumers. Consumers who subscribe to a topic can only consume messages published since its subscription.
(3) When the producer produces, the topic does not save the message. It is stateless and does not land. If no one subscribes to produce, it is a waste message. Therefore, generally start the consumer first and then the producer.

By default, as described above, but JMS The specification allows customers to create persistent subscriptions, which eases the time dependency requirement to a certain extent. Persistent subscriptions allow consumers to consume messages they send when they are not activated. In a word, it's like our WeChat official account subscription.

Producer introduction case

public class JmsProduce {
    public static final String ACTIVEMQ_URL = "tcp://127.0.0.1:61616";
    // Name of destination
    public static final String TOPIC_NAME = "first_queue";

    public static void main(String[] args) throws  Exception{
        ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory(ACTIVEMQ_URL);
        Connection connection = activeMQConnectionFactory.createConnection();
        connection.start();
        Session session = connection.createSession(false,Session.AUTO_ACKNOWLEDGE);
        // 4. Create destinations (two types: Queue / Topic). Destination is the parent class of Queue and Topic
        //Use Topic here
        Topic topic = session.createTopic(TOPIC_NAME);
        MessageProducer messageProducer = session.createProducer(topic);
        // 6. Three messages produced by messageProducer are sent to the message queue
        for (int i = 1; i < 4 ; i++) {
            // 7 create message
            TextMessage textMessage = session.createTextMessage("msg--" + i);
            // 8 send to mq via messageProducer
            messageProducer.send(textMessage);
        }
        // 9 close resources
        messageProducer.close();
        session.close();
        connection.close();
        System.out.println("  **** Message sent to MQ complete ****");
    }
}

topic of ActiveMQ console

Consumer entry cases

// Message consumer
public class JmsConsumer {
    public static final String ACTIVEMQ_URL = "tcp://127.0.0.1:61616";
    public static final String TOPIC_NAME = "first_queue";

    public static void main(String[] args) throws Exception{
        ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory(ACTIVEMQ_URL);
        javax.jms.Connection connection = activeMQConnectionFactory.createConnection();
        connection.start();
        Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
        Topic topic = session.createTopic(TOPIC_NAME);
        // 5 consumers who create messages
        MessageConsumer messageConsumer = session.createConsumer(topic);
        while(true){
            // reveive() has been waiting for the message to be received. It will be blocked until the message can be received. The synchronous blocking method is similar to the socket accept method.
            // If no message is received after waiting for n milliseconds, the blocking ends.
            // Because the message sender is TextMessage, the message receiver is also TextMessage
            TextMessage message = (TextMessage)messageConsumer.receive();
            if (null != message){
                System.out.println("****Consumer message:"+message.getText());
            }else {
                break;
            }
        }
        messageConsumer.close();
        session.close();
        connection.close();
    }
}

Asynchronous listening consumer (MessageListener)

// The consumer of the message is the system that answers the message
public class JmsConsumer02 {
    public static final String ACTIVEMQ_URL = "tcp://127.0.0.1:61616";

    public static final String TOPIC_NAME = "first_queue";

    public static void main(String[] args) throws Exception{
        ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory(ACTIVEMQ_URL);
        javax.jms.Connection connection = activeMQConnectionFactory.createConnection();
        connection.start();
        Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
        Topic topic = session.createTopic(TOPIC_NAME);
        MessageConsumer messageConsumer = session.createConsumer(topic);

        /*
         * Consume messages by listening, which is asynchronous and non blocking.
         * Register a listener through the setMessageListener of messageConsumer,
         * When a message is sent, the system automatically calls the onMessage method of MessageListener to process the message
         */
        messageConsumer.setMessageListener(new MessageListener() {
            public void onMessage(Message message)  {
                //  instanceof determines whether an object A is A subclass of class B
                if (null != message  && message instanceof TextMessage){
                    TextMessage textMessage = (TextMessage)message;
                    try {
                        System.out.println("****Consumer message:"+textMessage.getText());
                    }catch (JMSException e) {
                        e.printStackTrace();
                    }
                }
            }
        });
        // Let the main thread not end, because once the main thread ends, other threads (the threads listening for messages here) will also be forced to end.
        // In actual development, our program will run all the time, and this code will be omitted.
        System.in.read();
        messageConsumer.close();
        session.close();
        connection.close();
    }
}

Comparison between topic and queue

	Topic mode queue	Queue mode queue
Working mode	In the subscription publishing mode, if there are no subscribers, the message will be discarded. If there are multiple subscribers, these subscribers will receive the message	In the load balancing mode, if there are no consumers, the message will not be discarded. If there are multiple consumers, a message will only be sent to one of them
Yes no status	Stateless	queue data will be saved as a file on MQ by default. For example, ActiveMQ will generally be saved under:% HOME%data, or it can be configured as DB storage
Transfer integrity	If there is no subscriber, the message will be discarded	Messages are not discarded
Processing efficiency	Because messages will be copied according to the number of subscribers, the processing performance will gradually decrease with the increase of the number of subscribers, and the differences of different message protocols should be combined	Since a message will only be sent to one consumer, even if there are more consumers, the performance will not be significantly reduced. Of course, the specific performance of different message protocols is different