1 Introduction

When it comes to message oriented middleware, I think everyone should be familiar with it and have more or less contact with it. In fact, the popular understanding is that message middleware is a developed system and deployed independently, and then our business system sends and receives messages through it, so as to achieve the effect of asynchronous call. The most common implementation of message middleware is through queue, that is, the so-called Message Queue (MQ).

2 functions

2.1 asynchronous

         Message queuing was originally designed to solve asynchronous requests.

         Suppose there are two systems A and B, in which system A needs about 10 milliseconds to process business, while system B needs 100 milliseconds to process business. If it is called synchronously, it takes 110 milliseconds. However, if message oriented middleware is used, system A processes the business for 10ms, sends the message to the message queue, takes 1ms, and then returns it directly to the user. When system B goes to MQ to consume information, system A does not need to care, so it only takes 11ms for the user.

2.2 decoupling    

         Another great advantage of message queuing is the decoupling between systems.

         For example, if multiple systems need to deal with system A, system A needs to be updated every time the system is changed or added, with A high degree of coupling. After the message queue is added, the changes of other systems are imperceptible to system A and do not need to be concerned by system A, which realizes the decoupling function.

2.3 peak shaving with large flow

         A typical scenario is the second kill business.

 

         At the beginning of the second kill, there may be millions of requests, but there may be only 100 requests processed in the end. In addition, millions of requests come in at the same time, and the capacity of downstream systems to process resources is limited. Therefore, when there is a peak, it is easy to cause server downtime and users can't access it.

         So there is flow peak clipping. In essence, it means delaying user requests more and filtering user access requirements layer by layer, following the principle of "minimizing the number of requests finally landed in the database".

         There are generally two solutions.

         In scheme 1, the message queue is used to buffer the instantaneous traffic, and the synchronous direct call is changed to asynchronous push. In the middle, a queue is used to undertake the instantaneous traffic peak at one end, and the message is smoothly pushed to the downstream at the other end.

         Scheme 2 uses funnel-shaped design to process requests, which has little to do with the content discussed in this paper and will not be repeated here.

3 common message oriented middleware in the market

 

4 Comparison of various MQ

characteristic	rabbitMQ	rocketMQ	kafka
development language	erlang	java	scala
Single machine throughput	10000 / S	100000 / S	100000 / S
Timeliness	Microsecond This is the biggest advantage of rabbit, with low latency	millisecond	millisecond
usability	High, master-slave architecture	Very high, distributed architecture	Very high, distributed architecture. Multiple copies of data, no loss of data, no unavailability.
Functional characteristics	erlang development, strong concurrency, excellent performance and low latency	MQ has complete functions and good expansion	Simple function, mainly used for big data real-time calculation and log collection, factual standard
shortcoming	1. Erlang development is difficult to understand the source code. Its basic functions depend on the rapid maintenance and bug repair of the open source community, which is not conducive to secondary development and maintenance. 2 RabbitMQ does have a lower throughput because it has a heavy implementation mechanism. 3. It is necessary to learn more complex interfaces and protocols, and the cost of learning and maintenance is high.	1. There are not many supported client languages. At present, they are java and c + +, of which c + + is immature; 2. Average community activity 3. JMS and other interfaces are not implemented in the mq core. Some systems need to modify a lot of code to migrate	1. If the Kafka single machine has more than 64 queues / partitions, the load will soar obviously. The more queues, the higher the load, and the longer the response time of sending messages 2. Short polling mode is used, and the real-time performance depends on the polling interval; 3. Consumption failed and retry is not supported; 4. Message sequence is supported, but message disorder will occur after an agent goes down; 5. Slow community renewal;

5 message queue selection suggestions

5.1 Kafka

         Kafka is mainly characterized by processing message consumption based on Pull mode and pursuing high throughput. Its initial purpose is to collect and transmit logs, which is suitable for the data collection business of Internet services that generate a large amount of data.

         It is suggested that large companies can choose it. If there is log collection function, it must be the first choice kafka.

5.2 RocketMQ

         Born in the field of financial Internet and demanding high reliability, especially in the case of order deduction and business peak cutting in e-commerce, when a large number of transactions pour in, the back end may not be able to handle them in time.

         RocketMQ may be more reliable in stability. These business scenarios have been tested many times in Alibaba double 11. If your business has the above concurrent scenarios, it is recommended to choose RocketMQ.

5.3 RabbitMQ

         RabbitMQ: combined with the concurrency advantages of erlang language, it has good performance and high community activity, but it is not conducive to secondary development and maintenance. However, the RabbitMQ community is very active and can solve the bug s encountered in the development process.

         If your data volume is not so large, small companies prefer RabbitMQ with more complete functions.

6 deployment and integration of message queue

         Next, I will briefly introduce the deployment of these queues (using docker stand-alone deployment) and the integration with Spring

6.1 RabbitMQ

6.1.1 deployment configuration

rabbit.yml

version: '3.7'
services:

  rabbit:
    image: rabbitmq:3.7.14-management-alpine
    networks:
      - loc_net
    ports:
      - 5672:5672
      - 15672:15672
    volumes:
      - "/data/rabbit/data:/var/lib/rabbitmq"


networks:
  loc_net:
    external: true

Then pass http://ip:15672 To access, you can enter the web management page. The default account password is guest guest

6.1.2 Springboot + RabbitMQ

pom.xml

<!-- rabbit be based on amqp Consensual -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-amqp</artifactId>
</dependency>

application.yml

spring:
  rabbitmq:
    username: guest
    password: guest
    port: 5672
    addresses: ip

RabbitMQComponent.java

@Component
public class RabbitMQComponent {

    @Autowired
    private RabbitTemplate rabbitTemplate;

    /**
     * General message
     */
    public void sendMsg(String topic, String key, String payload) {
        rabbitTemplate.convertAndSend(topic, key, payload);
    }

}

Test.java

@Test
public void sendMsgV3() {
    rabbitMQComponent.sendMsg("demo", "demo", "{\"test\":\"test\"}");
}

Consumer DemoRabbitListener.java

@Component
public class DemoRabbitListener {

    @RabbitListener(queues = "demo")
    public void onMessage(Message message) {
        System.out.println("Receive message:" + new String(message.getBody()));
    }

}

6.2 RocketMQ

6.2.1 deployment configuration

rocketmq.yml

version: '3.7'
services:
  rocketmq-namesrv:
    image: rocketmqinc/rocketmq
    container_name: rmqnamesrv
    restart: always
    ports:
      - 9876:9876
    volumes:
      - /data/rocketmq/logs:/home/rocketmq/logs
      - /data/rocketmq/store:/home/rocketmq/store
    command: sh mqnamesrv
  rocketmq-broker:
    image: rocketmqinc/rocketmq
    container_name: rmqbroker
    restart: always
    ports:
      - 10909:10909
      - 10911:10911
      - 10912:10912
    volumes:
      - /data/rocketmq/logs:/home/rocketmq/logs
      - /data/rocketmq/store:/home/rocketmq/store
      - /data/rocketmq/conf/broker.conf:/opt/rocketmq-4.4.0/conf/broker.conf
    command: sh mqbroker -n ip:9876 -c /opt/rocketmq-4.4.0/conf/broker.conf
    networks:
      - loc_net
    environment:
      - JAVA_HOME=/usr/lib/jvm/jre
  rocketmq-console:
    image: styletang/rocketmq-console-ng
    container_name: rocketmq-console-ng
    restart: always
    ports:
      - 8076:8080
    networks:
      - loc_net
    environment:
      - JAVA_OPTS= -Dlogging.level.root=info -Drocketmq.namesrv.addr=ip:9876 -Dcom.rocketmq.sendMessageWithVIPChannel=false

networks:
  loc_net:
    external: true

Then pass http://ip:8076 To access, you can enter the web management page

6.2.2 Springboot + RocketMQ

pom.xml

<!-- https://mvnrepository.com/artifact/org.apache.rocketmq/rocketmq-spring-boot-starter -->
<dependency>
    <groupId>org.apache.rocketmq</groupId>
    <artifactId>rocketmq-spring-boot-starter</artifactId>
    <version>2.2.0</version>
</dependency>

application.yml

rocketmq:
  nameServer: ip:9876
  producer:
    group: demo-group

RocketMQComponent.java

@Component
public class RocketMQComponent {

    @Autowired
    private RocketMQTemplate rocketMQTemplate;

    /**
     * General message
     */
    public void sendMsg(String topic, String key, String payload) {
        String destination = topic;
        if (StringUtils.isNotBlank(key)) {
            destination = destination + ":" + key;
        }
        rocketMQTemplate.convertAndSend(destination, payload);
    }

}

Test.java

@Test
public void sendMsg() {
    rocketMQComponent.sendMsg("demo", "TagA", "Hello MQ");
}

Consumer DemoRocketListener.java

@Component
@RocketMQMessageListener(topic = "demo", consumerGroup = "consumer-msg-group")
public class DemoRocketListener implements RocketMQListener<String> {

    @Override
    public void onMessage(String message) {
        System.out.println("Receive message:" + message);
    }
}

6.3 Kafka

6.3.1 deployment configuration

kafka.yml

version: '3.7'
services:
  zk:
    container_name: zk
    image: wurstmeister/zookeeper
    volumes:
      - /data/zk:/data
    ports:
      - 2181:2181
    networks:
      - loc_net

  kafka:
    container_name: kafka
    image: wurstmeister/kafka
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 0
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://ip:9092
      KAFKA_ZOOKEEPER_CONNECT: ip:2181
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
    volumes:
      - /data/kafka/logs:/kafka
    networks:
      - loc_net

  kafka-web:
    container_name: kafka-web
    image: freakchicken/kafka-ui-lite
    ports:
      - 8889:8889
    networks:
      - loc_net

networks:
  loc_net:
    external: true

Then pass http://ip:8889 To access, you can enter the web management page

6.3.2 Springboot + Kafka

pom.xml

<!-- kafka -->
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

application.yml

spring:
  kafka:
    bootstrap-servers: "ip:9092"
    producer: # producer
      retries: 3 # If a value greater than 0 is set, the client will resend the failed records
      batch-size: 16384
      buffer-memory: 33554432
      acks: 1
      # Specifies the encoding and decoding method of the message key and message body
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
    consumer:
      group-id: default-group
      enable-auto-commit: false
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
    listener:
      ack-mode: manual_immediate

KafkaMQComponent.java

@Component
public class KafkaMQComponent {

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    /**
     * General message
     */
    public void sendMsg(String topic, String key, String payload) {
        kafkaTemplate.send(topic, key, payload);
    }

}

Test.java

@Test
public void sendMsgV2() {
    kafkaMQComponent.sendMsg("demo", "key", " {\"test\":\"test\"}");
}

Consumer DemoKafkaListener.java

@Component
public class DemoKafkaListener {

    @KafkaListener(topics = "demo", groupId = "kafka-consumer")
    public String onMessage(ConsumerRecord<String, String> cr, Acknowledgment acknowledgment) {
        System.out.println("Receive message:" + cr.value());
        // Manual submission indicates that the message has been received
        acknowledgment.acknowledge();
        return "successful";
    }
}

        The above demo only gives a simple way to send and receive information, and there are many advanced functions. Because of the space, we won't talk about them. In general, kafka is the simplest and rabbitmq is the most versatile.

7 Summary

        Message queue is an important component in distributed system, which is suitable for various scenarios. Each function and product list can extend a lot of content. This article has the right to throw a brick and attract jade. If there is anything wrong or insufficient, you are welcome to point out. In the future, we will have a special column to introduce the contents one by one.