SpringCloud Day08 -- message driven (SpringCloud Stream)

Posted by VinzC on Thu, 03 Feb 2022 13:54:37 +0100

11. Spring cloud stream message driven

11.1 message driven overview

11.1.1 what is it?

Officially, Spring Cloud Stream is a framework for building message driven microservices.

The application interacts with the binder object in Spring Cloud Stream through inputs or outputs.
We configure binding, and the binder object of Spring Cloud Stream is responsible for interacting with the message middleware.
Therefore, we only need to figure out how to interact with Spring Cloud Stream to facilitate the use of message driven methods.

By using Spring Integration to connect the message broker middleware to realize message event driven.
Spring Cloud Stream provides personalized automatic configuration implementation for some vendors' message middleware products, citing the three core concepts of publish subscribe, consumption group and partition.

Currently, only RabbitMQ and Kafka are supported.

In a word: shield the differences between the underlying message middleware, reduce the switching cost, and unify the programming model of message

Official website reference documents:

  • https://spring.io/projects/spring-cloud-stream#overview

Spring Cloud Stream is a highly scalable event driven microservice framework for building connections with shared messaging systems. The framework provides a flexible programming model based on established and familiar spring idioms and best practices, including three core concepts: publish / subscribe, consumer group and message partition

  • https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/3.0.1.RELEASE/reference/html/
  • Spring Cloud Stream Chinese Instruction Manual: https://m.wang1314.com/doc/webapp/topic/20971999.html

11.1.2 standard MQ

  • Producers / consumers transmit information through message media. Content = = > message
  • Messages must go through a specific channel = = > message channelmessagechannel
  • How are messages in the message channel consumed? Who is responsible for receiving and processing = = > SubscribableChannel, the sub interface of the message channel MessageChannel, is subscribed by the MessageHandler message processor

11.1.3 why use Cloud Stream?

For example, we use RabbitMQ and Kafka. Due to the different architectures of the two message oriented middleware,
For example, RabbitMQ has exchange and kafka has Topic and Partitions partitions

The differences of these middleware lead to some troubles in our actual project development. If we use one of the two message queues and the business requirements behind it, I want to migrate to another message queue. At this time, it is undoubtedly disastrous. A lot of things have to be pushed down and redone because it is coupled with our system, At this time, springcloud Stream provides us with a way to decouple.

1. Why can stream unify the underlying differences?

Without the concept of binder, when our SpringBoot application needs to directly interact with message oriented middleware,
Because the original intention of each message middleware is different, their implementation details will be quite different
By defining the binder as the middle layer, the isolation between application and message middleware details is perfectly realized.
By exposing the unified Channel channel to the application, the application does not need to consider a variety of different message middleware implementations.

By defining Binder as the middle layer, the isolation between application and message middleware details is realized.

2.Binder

Without the concept of binder, when our SpringBoot application wants to directly interact with message middleware, due to the different original intention of each message middleware, there will be great differences in their implementation details. By defining binder as the middle layer, it perfectly realizes the isolation between application and message middleware details. The further encapsulation of message oriented middleware by Stream can make the code layer insensitive to the middleware, and even dynamically switch the middleware (rabbitmq to kafka), which makes the development of micro services highly decoupled, and services can pay more attention to their own business processes

By defining Binder as the intermediate

Binder can generate binding. Binding is used to bind producers and consumers of message containers. It has two types: INPUT and OUTPUT. INPUT corresponds to consumers and OUTPUT corresponds to producers.

11.1.4 the message communication mode in stream follows the publish subscribe mode

Broadcast on Topic

  • RabbitMQ is Exchange
  • In Kakfa, it is Topic

11.1.5 Spring Cloud Stream standard process routine

  • Binder: it is very convenient to connect middleware and shield differences
  • Channel: channel is an abstraction of Queue. In the message communication system, it is the medium for storage and forwarding. The Queue is configured through channel
  • Source and Sink: simply understood as the reference object is the Spring Cloud Stream itself. Publishing messages from the Stream is output, and receiving messages is input.

11.1.6 coding API and common notes

11.2 case description

RabbitMQ environment is OK

Three new sub modules are built in the project

  • Cloud stream rabbitmq provider8801 is a module for sending messages as a producer
  • Cloud stream rabbitmq consumer 8802 as a message receiving module
  • Cloud stream rabbitmq consumer 8803 as message receiving module

11.3 message driven producers

  • Build Module - cloud stream rabbitmq provider8801
  • POM
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
    </dependency>
    <!--Basic configuration-->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-devtools</artifactId>
        <scope>runtime</scope>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>
  • YML
server:
  port: 8801

spring:
  application:
    name: cloud-stream-provider
  cloud:
    stream:
      binders: # Configure the service information of rabbitmq to be bound here;
        defaultRabbit: # Represents the name of the definition, which is used for binding integration
          type: rabbit # Message component type
          environment: # Set the related environment configuration of rabbitmq
            spring:
              rabbitmq:
                host: 192.168.174.128
                port: 5672
                username: admin
                password: admin
      bindings: # Integration of services
        output: # This name is the name of a channel
          destination: studyExchange # Indicates the Exchange name definition to use
          content-type: application/json # Set the message type, json this time, and "text/plain" for text
          binder: defaultRabbit #Set the specific settings of the message service to be bound

eureka:
  client: # Configure Eureka registration on the client
    service-url:
      defaultZone: http://eureka7001.com:7001/eureka,http://eureka7002.com:7002/eureka,http://eureka7004.com:7004/eureka #Cluster version
  instance:
    lease-renewal-interval-in-seconds: 2 # Set the heartbeat interval (the default is 30 seconds)
    lease-expiration-duration-in-seconds: 5 # If the interval of 5 seconds is exceeded now (the default is 90 seconds)
    instance-id: send-8801.com  # Displays the host name in the information list
    prefer-ip-address: true     # The access path becomes an IP address
  • Main startup class
@SpringBootApplication
public class StreamMQMain8801
{
    public static void main(String[] args)
    {
        SpringApplication.run(StreamMQMain8801.class,args);
    }
}
  • Business class

Send message interface:

public interface IMessageProvider
{
    public String send() ;
}

Implementation class of sending message interface:

@EnableBinding(Source.class)//It can be understood as the definition of a message sending pipeline
@Slf4j
public class MessageProviderImpl implements IMessageProvider {

    @Resource
    private MessageChannel output;//Message sending pipeline
    @Override
    public String send() {
        String serial = UUID.randomUUID().toString();
        this.output.send(MessageBuilder.withPayload(serial).build());//Create and send messages
        log.info("*****serial:{}",serial);
        return serial;
    }
}

controller

@RestController
public class SendMessageController
{
    @Resource
    private IMessageProvider messageProvider;

    @GetMapping(value = "/sendMessage")
    public String sendMessage()
    {
        return messageProvider.send();
    }
}
  • test

1. Start Eureka 700170027004;

2. Start RabbitMQ

3. Start 8801

You can see the switches defined in exchange

4. Visit: http://localhost:8801/sendMessage

11.4 message driven consumers

  • Build Module - cloud stream rabbitmq consumer 8802
  • POM
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <!--Basic configuration-->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-devtools</artifactId>
        <scope>runtime</scope>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>
  • YML
server:
  port: 8802

spring:
  application:
    name: cloud-stream-consumer
  cloud:
    stream:
      binders: # Configure the service information of rabbitmq to be bound here;
        defaultRabbit: # Represents the name of the definition, which is used for binding integration
          type: rabbit # Message component type
          environment: # Set the related environment configuration of rabbitmq
            spring:
              rabbitmq:
                host: 192.168.174.128
                port: 5672
                username: admin
                password: admin
      bindings: # Integration of services
        input: # This name is the name of a channel
          destination: studyExchange # Indicates the Exchange name definition to use
          content-type: application/json # Set the message type, json this time, and "text/plain" for text
          binder: defaultRabbit #Set the specific settings of the message service to be bound


eureka:
  client: # Configure Eureka registration on the client
    service-url:
      defaultZone: http://eureka7001.com:7001/eureka,http://eureka7002.com:7002/eureka,http://eureka7004.com:7004/eureka #Cluster version
  instance:
    lease-renewal-interval-in-seconds: 2 # Set the heartbeat interval (the default is 30 seconds)
    lease-expiration-duration-in-seconds: 5 # If the interval of 5 seconds is exceeded now (the default is 90 seconds)
    instance-id: receive-8802.com  # Displays the host name in the information list
    prefer-ip-address: true     # The access path becomes an IP address
  • Main startup class:
@SpringBootApplication
public class StreamMQMain8802
{
    public static void main(String[] args)
    {
        SpringApplication.run(StreamMQMain8802.class,args);
    }
}
  • Business class
@Component
@EnableBinding(Sink.class)
public class ReceiveMessageListener {

    @Value("${server.port}")
    private String serverPort;

    @StreamListener(Sink.INPUT)
    public void input(Message <String> message){
        System.out.println("Consumer 1,------------>Received message: "+message.getPayload()+"\t port: "+serverPort);
    }
}
  • Test 8801 sends 8802 receives messages

visit: http://localhost:8801/sendMessage

11.5 group consumption and persistence

11.5.1 raising questions

  • According to 8802, clone will produce a copy of 8803

  • Start Eureka 700170027004, provider 8001 and consumer 80028003

  • Problems after operation

1. There is the problem of repeated consumption

2. Message persistence

11.5.2 analysis of problems

At present, 8802 / 8803 have been received at the same time, and there is a problem of repeated consumption

How to solve it Grouping and persistence Properties group

Actual production cases

Note that multiple consumers in the same group in the Stream are competitive, which can ensure that messages will only be consumed once by one of the applications.
Different groups can be fully consumed (repeated consumption). There will be competition in the same group, and only one of them can be consumed.

11.5.3 grouped cases

1. Principle

When microservice applications are placed in the same group, it can ensure that messages will only be consumed by one of them once.
Different groups can be fully consumed (repeated consumption). There will be competition in the same group, and only one of them can be consumed.

2. Code demonstration

  • 8802 / 8803 become different groups, and the two groups are different

8802 modify YML

8803 modify YML

Our own configuration:

In order to achieve high availability and load balancing, distributed microservice applications actually deploy multiple instances. This example starts two consumer microservices (8802 / 8803)

In most cases, the producer only wants to be consumed once when sending a message to a specific microservice. According to the above example of starting two applications, although they belong to the same application, the message is consumed twice. To solve this problem, the concept of consumption group is provided in Spring Cloud Stream.

Conclusion: or repeated consumption

  • 8802 / 8803 become the same group, and the two groups are the same

8802 / 8803 implements polling packets, with only one consumer at a time. Messages sent by 8801 module can only be received by one of 8802 or 8803, so as to avoid repeated consumption

8802 modify YML and modify the group attribute as: group: atrjxyA

8803 modify YML and modify the group attribute as: group: atrjxyA

Conclusion: for multiple micro service instances in the same group, only one will be obtained at a time

11.5.4 persistence cases

  • Stop 8802 / 8803 and remove the group: atrjxya of 8802. The group: atguiguA of 8803 is not removed
  • 8801 sends 4 messages to rabbitmq first
  • Start 8802 first, there is no grouping attribute configuration, and there is no message printed in the background

  • Start 8803 again, there is grouping attribute configuration, and messages on MQ are printed in the background

Topics: Java Back-end Spring Cloud

Hot Topics

©2024 Programmer Think