Spring cloud stream message driven

Posted by spartan789 on Fri, 03 Dec 2021 07:54:49 +0100

Message driven overview

What is Spring Cloud Stream? Officially, Spring Cloud Stream is a framework for building message driven microservices. Applications interact with binder objects in Spring Cloud Stream through inputs or outputs. We configure binding, and the binder object of Spring Cloud Stream is responsible for interacting with the message middleware. Therefore, we only need to figure out how to interact with Spring Cloud Stream to facilitate the use of message driven methods.   By using Spring Integration to connect the message broker middleware to realize message event driven. Spring Cloud Stream provides personalized automatic configuration implementation for some vendors' message middleware products, citing the three core concepts of publish subscribe, consumption group and partition. Currently, only RabbitMQ and Kafka are supported.

One sentence explanation: shield the differences between the underlying message middleware, reduce the switching cost, and unify the message programming model

design idea

Standard MQ

Information content is transmitted between producers / consumers through message mediaMessage
Messages must go through a specific channel  Message channel
How are messages in the message channel consumed? Who is responsible for sending and receiving messages     The interface SubscribableChannel is subscribed by the MessageHandler message processor

  Why Cloud Stream

Why can stream unify the underlying differences?

Without the concept of binder, when our SpringBoot application needs to directly interact with message oriented middleware,
Because the original intention of each message middleware is different, there will be great differences in their implementation details
By defining the binder as the middle layer, the isolation between application and message middleware details is perfectly realized.
By exposing the unified Channel channel to the application, the application does not need to consider a variety of different message middleware implementations.
 
By defining Binder as the middle layer, the isolation between application and message middleware details is realized.
 Binder

  Without the concept of binder, when our SpringBoot application wants to directly interact with the message middleware, due to the different original intention of each message middleware, there will be great differences in their implementation details. By defining the binder as the middle layer, it perfectly realizes the isolation between the application and the message middleware details. The further encapsulation of message oriented middleware by Stream can make the code layer insensitive to the middleware, and even dynamically switch the middleware (rabbitmq to kafka), which makes the development of micro services highly decoupled, and services can pay more attention to their own business processes
 
By defining Binder as the middle layer, the isolation between application and message middleware details is realized.
 
Binder can generate binding. Binding is used to bind producers and consumers of message containers. It has two types: INPUT and OUTPUT. INPUT corresponds to consumers and OUTPUT corresponds to producers.
 
 
  The message communication mode in Stream follows the publish subscribe mode

Broadcast Topic          RabbitMQ is Exchange    In Kakfa, it is Topic

Spring Cloud Stream standard process routine


 

BinderIt is very convenient to connect middleware to shield differences
ChannelChannel is an abstraction of Queue. In the message communication system, it is the medium for storage and forwarding. The Queue is configured through channel
Source and SinkSimply understood, the reference object is the Spring Cloud Stream itself,
Publishing a message from Stream is output, and receiving a message is input.

Coding API and common annotations  


  Case description

RabbitMQ environment is OK
Three new sub modules are built in the project

         Cloud stream rabbitmq provider 8801, as a producer, sends messages

         Cloud stream rabbitmq consumer 8802 as a message receiving module

         Cloud stream rabbitmq consumer 8803 as a message receiving module

Message driven producer

cloud-stream-rabbitmq-provider8801

POM

<dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
        </dependency>
        <!--Basic configuration-->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

yml

server:
  port: 8801

spring:
  application:
    name: cloud-stream-provider
  cloud:
      stream:
        binders: # Configure the service information of rabbitmq to be bound here;
          defaultRabbit: # Represents the name of the definition, which is used for binding integration
            type: rabbit # Message component type
            environment: # Set the environment configuration related to rabbitmq
              spring:
                rabbitmq:
                  host: localhost
                  port: 5672
                  username: guest
                  password: guest
        bindings: # Integration of services
          output: # This name is the name of a channel
            destination: studyExchange # Represents the Exchange name definition to use
            content-type: application/json # Set the message type, json this time, and "text/plain" for text
            binder: defaultRabbit # Set the specific settings of the message service to be bound

eureka:
  client: # Configure Eureka registration on the client
    service-url:
      defaultZone: http://localhost:7001/eureka
  instance:
    lease-renewal-interval-in-seconds: 2 # Set the heartbeat interval (30 seconds by default)
    lease-expiration-duration-in-seconds: 5 # If the interval of 5 seconds is exceeded now (the default is 90 seconds)
    instance-id: send-8801.com  # Displays the host name in the information list
    prefer-ip-address: true     # The access path becomes an IP address



 
 
 

Main startup class streamqmain8801

@SpringBootApplication
public class StreamMQMain8801
{
    public static void main(String[] args)
    {
        SpringApplication.run(StreamMQMain8801.class,args);
    }
}

Business class
          Send message interface

public interface IMessageProvider
{
    public String send() ;
}

         Send message interface implementation class

@EnableBinding(Source.class) // It can be understood as the definition of a message sending pipeline
public class MessageProviderImpl implements IMessageProvider
{
    @Resource
    private MessageChannel output; // Message sending pipeline

    @Override
    public String send()
    {
        String serial = UUID.randomUUID().toString();
        this.output.send(MessageBuilder.withPayload(serial).build()); // Create and send messages
        System.out.println("***serial: "+serial);

        return serial;
    }
}

        Controller

@RestController
public class SendMessageController
{
    @Resource
    private IMessageProvider messageProvider;

    @GetMapping(value = "/sendMessage")
    public String sendMessage()
    {
        return messageProvider.send();
    }
}

test

Start 7001eureka

Start rabbitmq      rabbitmq-plugins enable rabbitmq_management         http://localhost:15672/

Start 8801

visit        http://localhost:8801/sendMessage        

Message driven consumers

cloud-stream-rabbitmq-consumer8802

pom

<dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <!--Basic configuration-->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

yml

server:
  port: 8802

spring:
  application:
    name: cloud-stream-consumer
  cloud:
      stream:
        binders: # Configure the service information of rabbitmq to be bound here;
          defaultRabbit: # Represents the name of the definition, which is used for binding integration
            type: rabbit # Message component type
            environment: # Set the environment configuration related to rabbitmq
              spring:
                rabbitmq:
                  host: localhost
                  port: 5672
                  username: guest
                  password: guest
        bindings: # Integration of services
          input: # This name is the name of a channel
            destination: studyExchange # Represents the Exchange name definition to use
            content-type: application/json # Set the message type. This time, it is an object json. If it is text, set "text/plain"
            binder: defaultRabbit # Set the specific settings of the message service to be bound

eureka:
  client: # Configure Eureka registration on the client
    service-url:
      defaultZone: http://localhost:7001/eureka
  instance:
    lease-renewal-interval-in-seconds: 2 # Set the heartbeat interval (30 seconds by default)
    lease-expiration-duration-in-seconds: 5 # If the interval of 5 seconds is exceeded now (the default is 90 seconds)
    instance-id: receive-8802.com  # Displays the host name in the information list
    prefer-ip-address: true     # The access path becomes an IP address
 
 
 
 

Main startup class streamqmain8802

@SpringBootApplication
public class StreamMQMain8802
{
    public static void main(String[] args)
    {
        SpringApplication.run(StreamMQMain8802.class,args);
    }
}

Business class

@Component
@EnableBinding(Sink.class)
public class ReceiveMessageListener
{
    @Value("${server.port}")
    private String serverPort;

    @StreamListener(Sink.INPUT)
    public void input(Message<String> message)
    {
        System.out.println("Consumer 1,------->Received message:" + message.getPayload()+"\t port: "+serverPort);
    }
}

Test 8801 sends 8802 receives messages

http://localhost:8801/sendMessage

  Group consumption and persistence

According to 8802, clone will run 8803

start-up   

RabbitMQ

seven thousand and one   Service registration

eight thousand eight hundred and one    Message production

eight thousand eight hundred and two    Message consumption

eight thousand eight hundred and three    Message consumption

There are two problems after running   1. There is the problem of repeated consumption   2. Message persistence

At present, 8802 / 8803 have been received at the same time, and there is a problem of repeated consumption
8802

8803

  How to solve    Grouping and persistence Properties group

Actual production cases

For example, in the following scenario, when we deploy the order system in a cluster, we will get the order information from RabbitMQ,
If an order is obtained by two services at the same time, it will cause data errors. We have to avoid this situation.
At this time, we can use the message grouping in the Stream to solve the problem.


 

Note that multiple consumers in the same group in the Stream are competitive, which can ensure that messages will only be consumed once by one of the applications.
Different groups can be fully consumed (repeated consumption),
Competition will occur in the same group, and only one of them can consume.  

  grouping

Principle: when microservice applications are placed in the same group, it can ensure that messages will only be consumed once by one of them.
Different groups can be consumed. There will be competition in the same group, and only one of them can be consumed.

8802 / 8803 become different groups, and the two groups are different     group: A,B

8802 modify YML  

server:
  port: 8802

spring:
  application:
    name: cloud-stream-consumer
  cloud:
      stream:
        binders: # configure the service information of rabbitmq to be bound here;
          defaultRabbit: # indicates the name of the definition, which is used for binding integration
            type: rabbit # message component type
            Environment: # set the related environment configuration of rabbitmq
              spring:
                rabbitmq:
                  host: localhost
                  port: 5672
                  username: guest
                  password: guest
        bindings: # service integration processing
          input: # this name is the name of a channel, which will be explained when analyzing the specific source code
            destination: studyExchange # indicates the Exchange name definition to use
            Content type: application / json # sets the message type. This time, it is the object json. If it is text, set "text/plain"
        binder: defaultRabbit # sets the specific settings of the message service to be bound
             group: A
eureka:
  Client: # client configures Eureka registration
    service-url:
      defaultZone: http://localhost:7001/eureka
  instance:
    Lease renewal interval in seconds: 2 # set the heartbeat interval (30 seconds by default)
    Lease expiration duration in seconds: 5 # if the interval exceeds 5 seconds (90 seconds by default)
    instance-id: receive-8802.com  # Displays the host name in the information list
    prefer-ip-address: true     # The access path becomes an IP address
 

  8803 modify YML

 

 
server:
  port: 8803

spring:
  application:
    name: cloud-stream-consumer
  cloud:
      stream:
        binders: # configure the service information of rabbitmq to be bound here;
          defaultRabbit: # indicates the name of the definition, which is used for binding integration
            type: rabbit # message component type
            Environment: # set the related environment configuration of rabbitmq
              spring:
                rabbitmq:
                  host: localhost
                  port: 5672
                  username: guest
                  password: guest
        bindings: # service integration processing
          input: # this name is the name of a channel, which will be explained when analyzing the specific source code
            destination: studyExchange # indicates the Exchange name definition to use
            Content type: application / json # sets the message type. This time, it is the object json. If it is text, set "text/plain"
        binder: defaultRabbit # sets the specific settings of the message service to be bound
             group: B
eureka:
  Client: # client configures Eureka registration
    service-url:
      defaultZone: http://localhost:7001/eureka
  instance:
    Lease renewal interval in seconds: 2 # set the heartbeat interval (30 seconds by default)
    Lease expiration duration in seconds: 5 # if the interval exceeds 5 seconds (90 seconds by default)
    instance-id: receive-8803.com  # Displays the host name in the information list
    prefer-ip-address: true     # The access path becomes an IP address
 

  8802 / 8803 implements polling packets, with only one consumer at a time
Messages sent by 8801 module can only be received by one of 8802 or 8803, which avoids repeated consumption. 8802 / 8803 become the same group. Two groups are the same. Multiple micro service instances in the same group will only get one at a time

yml is changed to: group: A

Persistence

Through the above, the problem of repeated consumption is solved. Look at persistence, stop 8802 / 8803 and remove the group group:A of 8802. Group A of 8803 is not removed. 8801 sends 4 messages to rabbitmq first. Start 8802 first, there is no grouping attribute configuration, and there is no message printed in the background.

When 8803 is started again, there is a grouping attribute configuration, and messages on MQ are printed in the background  

7001 module  

Eureka Foundation_ Blog of unknown passers-by - CSDN blog

Topics: Java Back-end Spring Cloud stream