1. Middleware

Middleware is the software between operating system and application program. Some people think it should be a part of operating system. When using middleware, people often integrate a group of middleware to form a platform (including development platform and operation platform), but there must be a communication middleware in this group of middleware, that is, middleware + platform + communication. This definition also limits that it can be called middleware only when used in distributed systems, At the same time, it can also be distinguished from supporting software and using software.

1.1 features

• Meet the needs of a large number of applications
• Run on a variety of hardware and OS platforms
• Support distributed computing and provide transparent interaction of applications or services across networks, hardware and OS platforms
• Support standard protocols
• Support standard interfaces

1.2 Message Oriented Middleware

The reliable message passing mechanism is used for direct communication between the system and the system. By providing message passing and message party mechanism, it can expand the communication between processes in the distributed system environment. It is a technical service with the functions of receiving data, receiving requests, storing data, sending data and so on.

1.3 message components

• agreement
• Persistence mechanism
• Distribution strategy
• fault-tolerant

1.4 agreement

1. A set of conventions that the underlying operating system of a computer and the application program follow when communicating. Only by following the common conventions and specifications can the system and the underlying operating system communicate with each other. 2 Different from general network applications, it is mainly responsible for data acceptance and transmission, so the performance is relatively high. 3 The protocol must strictly abide by the specifications for data format and data exchange between computers.

1.5 composition of the agreement

• Syntax: syntax is the structure and format of user data and control information, as well as the order in which data appears
• Semantics: semantics is to explain the meaning of each part of the control information. It specifies what kind of control information needs to be sent, and what kind of action to complete and what kind of response to make
• Timing: timing is a detailed description of the sequence of events

1.6 AMQP

AMQP: (full name: Advanced Message Queuing Protocol) is an Advanced Message Queuing Protocol. Jointly designed by JPMorgan Chase group and other companies. It is an application layer standard and Advanced Message Queuing Protocol that provides unified messaging services. It is an open standard of application layer protocol and is designed for message oriented middleware. The client and message middleware based on this protocol can deliver messages, which is not limited by different products and development languages of the client / middleware. (supporter: RabbitMQ ActiveMQ)

• Distributed transaction

• Message persistence

• High performance and high reliability

  1.6.1 producer circulation process

  

  1.6.2 consumer circulation process

1.7 MQTT

MQTT protocol (message queuing Telecommunication Transport) message queue is an open timely communication protocol of IBM and an important part of the system architecture of the Internet of things. Supporter: rabbitmq (ActiveMQ)

• Light weight
• Simple structure
• Fast transmission
• Transaction not supported
• Persistence is not supported

1.8 OpenMessage

It is an application development standard in the fields of Distributed Information Middleware and stream processing jointly founded by Alibaba, Yahoo, Didi travel, Stremalio and other companies in recent years.

• Simple structure
• Fast parsing speed
• affair
• Persistence

1.9 Kafka

Kafka protocol is a binary protocol based on TCP/IP. The message is divided by length and consists of some basic data types.

• Simple structure
• Fast parsing speed
• No transaction support
• Persistent design

2. Installation

2.1 rpm installation

https://blog.csdn.net/qq_38020915/article/details/117957570

2.2 docker installation

https://blog.csdn.net/qq_38020915/article/details/117957001

3.RabbitMQ

3.1 core concepts

• Server: also known as Broker, it receives client connections and implements AMQP services.
• Connection: the network connection between the application and the Broker. TCPIP three handshakes and four handshakes
• Channel: network channel, which corresponds to a session task and reads and writes messages
• Message: message, the data transmitted between service and application, which is composed of Properties and body
• Virtual Host: virtual address used for logical isolation
• Queue: queue
• Exchange: switch, distribute to queue according to routing key
• Bindings: virtual connections between switches and queues
• Routing key: routing rules

3.2 composition

3.3 operation process

3.4 message model

• Simple mode

• Working mode

  • Fair distribution

    • producer

      Same as simple mode

    • consumer

      • maven

        //Simple mode
        public class Consumer{
        	//3. Accepted content
            //Indicators defined
            channel.basicQos(1);
            channel.basicConsume("queue1",false,new DefaultConsumer(){
                public void handle(String consumerTag, Delivery message) throws IOException {
                  System.out.println(new String("The message is" + new String(meassage.getBody()),"UTF-8"));
                  //Change to manual response
                  channel.basicAck(delivery.getEnvelope().getDeliveryTag(),false);
                },new CancelCallback(){
                    public void handle(String consumerTag) throws IOException {
                        System.out.println("Acceptance failed");
                }
              });
            //4. Close
            channel.close();
            connection.close();
        }
        

      • springboot

            # TODO fair distribution 0: (default) polling 1: fair distribution
        #    listener:
        #      direct:
        #        prefetch: 1
        #        acknowledge-mode: manual
        
        

  • polling

    Just like the simple model.

• Publish subscribe mode

• Routing mode

• Theme mode

• Parameter mode

• Release confirmation mode

3.5 usage scenarios and problem solving

• Thread pool asynchronous processing (peak shaving)
• Decoupling (module separation, high cohesion, low coupling)
• Message reliability (persistence)
• High availability (cluster)

3.6 SpringBoot 2.x integrated use

Full code address: https://gitee.com/dingwen-gitee/stu-spr-boo-rab.git

3.6.1 dependence

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>com.dingwen</groupId>
        <artifactId>stu-spr-boo-rab</artifactId>
        <version>1.0-SNAPSHOT</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.dingwen.rabcon</groupId>
    <artifactId>rab-con</artifactId>
    <version>1.0-SNAPSHOT</version>
    <name>rab-con</name>
    <description>rab-con</description>
    <properties>
        <java.version>1.8</java.version>
    </properties>
    <dependencies>

        <!--rabbitmq -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-amqp</artifactId>
        </dependency>


        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <excludes>
                        <exclude>
                            <groupId>org.projectlombok</groupId>
                            <artifactId>lombok</artifactId>
                        </exclude>
                    </excludes>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

3.6.2 configuration (fanout)

package com.dingwen.rabcon.config;

import com.dingwen.rabcon.constant.RabbitConstant;
import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.FanoutExchange;
import org.springframework.amqp.core.Queue;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * rabbitmq Configuration class
 *
 * @author dingwen
 * 2021.06.17 10:36
 */
@Configuration
public class RabbitmqConfiguration {

    /**
     * Switch declaring fanout mode
     *
     * @return {@link FanoutExchange}
     */
    @Bean
    public FanoutExchange fanoutExchange() {
        // Persistence, turn off automatic deletion
        return new FanoutExchange(RabbitConstant.FANOUT_EXCHANGE_NAME, true, false);
    }

    /**
     * SMS task queue
     *
     * @return {@link Queue}
     */
    @Bean
    public Queue smsQueue() {
        // Persistence, exclusive queue, turn off automatic deletion
        return new Queue(RabbitConstant.FANOUT_QUEUE_SMS_NAME);
    }

    /**
     * Mail task queue
     *
     * @return {@link Queue}
     */
    @Bean
    public Queue emailQueue() {
        // Persistence, exclusive queue, turn off automatic deletion
        return new Queue(RabbitConstant.FANOUT_QUEUE_EMAIL_NAME);
    }

    /**
     * Bind SMS queue to switch
     *
     * @return {@link Binding}
     */
    @Bean
    public Binding smsBinding() {
        return BindingBuilder.bind(smsQueue()).to(fanoutExchange());
    }

    /**
     * Mailbox queue bound to switch
     *
     * @return {@link Binding}
     */
    @Bean
    public Binding emailBinding() {
        return BindingBuilder.bind(emailQueue()).to(fanoutExchange());
    }

}

3.6.3 producer

package com.dingwen.rabpro.service.impl;

import com.dingwen.rabpro.constant.RabbitConstant;
import com.dingwen.rabpro.service.OrderService;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.UUID;

/**
 * order service impl
 *
 * @author dingwen
 * 2021.06.17 11:11
 */
@Service
public class OrderServiceImpl implements OrderService {
    private final RabbitTemplate rabbitTemplate;

    @Autowired
    public OrderServiceImpl(RabbitTemplate rabbitTemplate) {
        this.rabbitTemplate = rabbitTemplate;
    }

    @Override
    public void placeOrder() {
        rabbitTemplate.convertAndSend(RabbitConstant.FANOUT_EXCHANGE_NAME, null, UUID.randomUUID().toString());
    }
}

3.6.4 consumer

package com.dingwen.rabcon.service.impl;

import com.dingwen.rabcon.constant.RabbitConstant;
import com.dingwen.rabcon.service.EmailService;
import org.springframework.amqp.rabbit.annotation.RabbitHandler;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Service;

import java.util.Date;


/**
 * email service impl
 *
 * @author dingwen
 * 2021.06.17 11:22
 */
@Service
@RabbitListener(queues = {RabbitConstant.FANOUT_QUEUE_EMAIL_NAME})
public class EmailServiceImpl implements EmailService {

    @Override
    @RabbitHandler
    public void sendEmail(String message) {
        System.out.println("E: message = " + message);
    }
}

3.6.5 topic matching rules

• "*": only one level
• "#": at least one level

3.7 persistence

Original address: https://zhuanlan.zhihu.com/p/74840278

3.7.1 when does a message need to be persisted

• When the message itself is publish ed, it is required to write the message to the disk
• Memory is tight, and some messages in memory need to be transferred to disk

3.7.2 when will messages be sent to disk

• There will be a Buffer with a size of 1M (1048576) before writing to the file. When writing to the file, the data will be written to this Buffer first. If the Buffer is full, the Buffer will be written to the file (not necessarily to the disk)
• There is a fixed disk brushing time: 25ms, that is, no matter whether the Buffer is full or not, the data in the Buffer and the contents of files that have not been refreshed to the disk will be brushed to the disk every 25ms
• After each message is written, if there is no subsequent write request, the written message will be directly brushed to the disk: it is realized by using Erlang's receive x after 0. As long as there is no message in the process's mailbox, a timeout message will be generated, and the timeout will trigger the disk brushing operation

3.7.3 message saving to disk format

The message is saved in $mnesia / MSG_ store_ In the persistent / x.rdq file, where x is the numerical number. Starting from 1, the maximum size of each file is 16M (16777216). Exceeding this size will generate a new file, and the file number will be increased by 1.

3.7.4 when to delete a file

• When the proportion of junk messages (messages that have been deleted) in all files is greater than the threshold (garage_collision = 0.5), the file merge operation (when at least three files exist) will be triggered to improve disk utilization
• The content is written in the publish message and deleted in the ack message (the useful data size of the file is updated). When the useful data of a file is equal to 0, the file is deleted

3.7.5 storage location

3.8 Exclusive Queue

Only queues visible to themselves, that is, other users are not allowed to access

• Visible only to the Connection that first declared it
• Automatically delete when its connection is disconnected (including persistent queue)
• Restrict connections, not channels

If you try to redeclare or access (e.g. publish, consume) the exclusive queue in a different connection, you will get the error esource that the resource is locked_ LOCKED - cannot obtain exclusive access to locked queue 'UserLogin2'

3.9 blocking queue

When the queue is empty, the latter waits back when the queue is slow. When the queue is empty, it will not be empty for operation.

4.RabbitMQ advanced

4.1 expiration time

The expiration time (TTL) can be set for the message, indicating that the message can be consumed within this time. After the message expires, it is deleted or put into the dead letter queue. You can set a single message or the entire queue. If the two methods are used at the same time, the smaller TTL shall prevail.

4.1.1 message TTL

package com.dingwen.rabpro.service.impl;

import com.dingwen.rabpro.constant.RabbitConstant;
import com.dingwen.rabpro.service.OrderService;
import org.springframework.amqp.core.MessagePostProcessor;
import org.springframework.amqp.core.MessageProperties;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.UUID;

/**
 * order service impl
 *
 * @author dingwen
 * 2021.06.17 11:11
 */
@Service
public class OrderServiceImpl implements OrderService {
    private final RabbitTemplate rabbitTemplate;

    @Autowired
    public OrderServiceImpl(RabbitTemplate rabbitTemplate) {
        this.rabbitTemplate = rabbitTemplate;
    }

    @Override
    public void placeOrder() {

        // Test message expiration time
        MessagePostProcessor messagePostProcessor = (message) -> {
            MessageProperties messageProperties = message.getMessageProperties();
            messageProperties.setContentEncoding("UTF-8");
            // Expiration time 5000 ms
            messageProperties.setExpiration("5000");
            return message;
        };
        // fanout
        rabbitTemplate.convertAndSend(RabbitConstant.FANOUT_EXCHANGE_NAME, null, UUID.randomUUID().toString(), messagePostProcessor);

        // direct topic
        // As above, you need to specify a routing key
    }
}

4.1.2 queue TTL

package com.dingwen.rabcon.config;

import com.dingwen.rabcon.constant.RabbitConstant;
import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.FanoutExchange;
import org.springframework.amqp.core.Queue;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.HashMap;
import java.util.Map;

/**
 * rabbitmq Configuration class
 *
 * @author dingwen
 * 2021.06.17 10:36
 */
@Configuration
public class RabbitmqConfigurationFanout {

    /**
     * Switch declaring fanout mode
     *
     * @return {@link FanoutExchange}
     */
    @Bean
    public FanoutExchange fanoutExchange() {
        // Persistence, turn off automatic deletion
        return new FanoutExchange(RabbitConstant.FANOUT_EXCHANGE_NAME, true, false);
    }

    /**
     * SMS task queue
     *
     * @return {@link Queue}
     */
    @Bean
    public Queue smsQueue() {
        // Persistence, exclusive queue, turn off automatic deletion
        return new Queue(RabbitConstant.FANOUT_QUEUE_SMS_NAME);
    }

    /**
     * Mail task queue
     *
     * @return {@link Queue}
     */
    @Bean
    public Queue emailQueue() {
        // Persistence, exclusive queue, turn off automatic deletion

        // TTL queue test
        Map<String, Object> args = new HashMap<>(1);
        // Shaping milliseconds
        args.put("x-message-ttl", 5000);
        return new Queue(RabbitConstant.FANOUT_QUEUE_EMAIL_NAME, true, false, false, args);
//        return new Queue(RabbitConstant.FANOUT_QUEUE_EMAIL_NAME);
    }

    /**
     * Bind SMS queue to switch
     *
     * @return {@link Binding}
     */
    @Bean
    public Binding smsBinding() {
        return BindingBuilder.bind(smsQueue()).to(fanoutExchange());
    }

    /**
     * Mailbox queue bound to switch
     *
     * @return {@link Binding}
     */
    @Bean
    public Binding emailBinding() {
        return BindingBuilder.bind(emailQueue()).to(fanoutExchange());
    }

}

4.2 dead letter queue

DLX,dead letter exchange. When a message in a queue becomes a dead letter, it can be re sent to the specified dead letter switch. DLX is also a normal switch, which is no different from general switches. It can be specified on any queue. In fact, it is to set the properties of a queue. When there is dead letter in this queue, Rabbitmq will automatically republish the message to the set DLX, and then route it to another queue, that is, dead letter queue. To use dead letter queue, you only need to set the queue parameter x-dead-letter-exchange to specify the switch when defining the queue.

4.2.1 reasons for bad message

• Message rejected
• Message expiration
• The queue has reached its maximum length

4.2.1.1 service switch

    public FanoutExchange fanoutExchange() {
        // Persistence, turn off automatic deletion


        return new FanoutExchange(RabbitConstant.FANOUT_EXCHANGE_NAME, true, false);
    }

4.2.1.2 dead letter switch

    @Bean
    public DirectExchange directExchange() {
        return new DirectExchange(RabbitConstant.DIRECT_QUEUE_DEAD_NAME, true, false);
    }

4.2.1.3 service queue

    /**
     * Mail task queue
     *
     * @return {@link Queue}
     */
    @Bean
    public Queue emailQueue() {
        // Persistence, exclusive queue, turn off automatic deletion

        // TTL queue test
        Map<String, Object> args = new HashMap<>(3);
        // Shaping milliseconds
        args.put("x-message-ttl", 5000);

        // Binding dead letter switch
        args.put("x-dead-letter-exchange", RabbitConstant.DIRECT_EXCHANGE_NAME_DEAD);
        args.put("x-dead-letter-routing-key", RabbitConstant.DIRECT_QUEUE__ROUTE_DEAD);
        return new Queue(RabbitConstant.FANOUT_QUEUE_EMAIL_NAME, true, false, false, args);
//        return new Queue(RabbitConstant.FANOUT_QUEUE_EMAIL_NAME);
    }

4.2.1.4 dead letter queue

    /**
     * Dead letter queue
     *
     * @return {@link Queue}
     */
    @Bean
    public Queue deadQueue() {
        return new Queue(RabbitConstant.DIRECT_QUEUE_DEAD_NAME, true, false, false);
    }

4.2.1.5 binding between service queue and service switch

    /**
     * Mailbox queue bound to switch
     *
     * @return {@link Binding}
     */
    @Bean
    public Binding emailBinding() {
        return BindingBuilder.bind(emailQueue()).to(fanoutExchange());
    }

4.2.1.6 binding between dead letter queue and service switch

    /**
     * Dead letter queue binding
     *
     * @return {@link Binding}
     */
    @Bean
    public Binding deadBinding() {
        return BindingBuilder.bind(deadQueue()).to(fanoutExchange());
    }

4.3 monitoring

4.3.1 monitoring

When the memory usage exceeds the configured threshold or the remaining disk space exceeds the configured threshold, RabbitMQ will temporarily block the connection of the client and stop receiving messages from the client, so as to avoid the collapse of the server, and the mentality detection mechanism between the client and the server will also fail.

# Relative value mode
rabbitmqctl set_vm_memory_high_watermark <fraction>   Recommendation 0.4-0.6

# Absolute value method
rabbitmqctl set_vm_memory_high_watermark absolute <memory_limit>

# You can also modify the configuration file

4.3.2 memory page change

Before a Broker node and memory blocking producer, it will try to page the messages in the queue to disk to free up memory space. Both persistent and non persistent messages will be written to disk. The persistent message itself has a copy in the disk, so the persistent messages will be cleared from memory during the transfer process. By default, when the memory reaches the threshold of 50%, page feed will be processed.

In other words, when the memory threshold is 0.4 by default, when the memory exceeds 0.4 * 0.5 = 0.2, page feed will be performed. For example, there is 1000MB of memory. When the memory utilization reaches 400MB, it has reached the limit, but because the configured page feed memory is 0.5, it will reach the limit at this time
Before reaching the limit of 400mb, 200MB in memory will be transferred to disk. So as to achieve stable operation.

vm_memory_high_watermark_paging_ratio = 0.75

4.3.4 disk alert

When the available disk space falls below the configured limit (50MB by default), an alert will be triggered and all producers will be blocked. The goal is to avoid filling the entire disk, which will cause all writes on the node to fail and may cause RabbitMQ to terminate. To reduce the risk of filling the disk, all incoming messages are blocked. In the case of insufficient memory, transient messages are still paged to disk, and will occupy the already limited disk space. If the disk alert is set too low and the message is quickly rolled out, you may run out of disk space and crash RabbitMQ between disk space checks (at least 10 seconds apart). A more conservative approach is to set the limit to the same amount of memory installed on the system

rabbitmqctl set_disk_free_limit  <disk_limit>

rabbitmqctl set_disk_free_limit memory_limit  <fraction>

4.4 cluster

https://www.jianshu.com/p/6376936845ff

4.5 distributed transactions

Distributed transaction means that the operation of the transaction is located on different nodes, and the ACID characteristics of the transaction need to be guaranteed.

Solution: using the confirmation mechanism of RabbiMQ, when the producer sends a message, it must wait until the consumer confirms the receipt of the message, otherwise the scheduled task will be retransmitted (using message redundancy) In case of abnormality at the consumer end, the test enters the dead letter queue for manual processing.

4.5.1 reliable production

4.5.2 reliable consumption

4.5.3 applicable scenarios

• Asynchronous operation
• Business tolerance delay