rabbitmq series message idempotency processing

Posted by Sephirangel on Fri, 17 Jan 2020 02:02:21 +0100

1. springboot Integrating Rabbit MQ

  1. We need to create two new projects, one as a producer and the other as a consumer.Add an amqp dependency to pom.xml:
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
  1. Add rabbitmq information to the application.yml file:
spring:
  rabbitmq:
    # Connection Address
    host: 127.0.0.1
    # port
    port: 5672
    # Login Account
    username: guest
    # Logon Password
    password: guest
    # Virtual Host
    virtual-host: /
  1. Create a new configuration item, rabbitmqConfig.java, in the producer project, declaring that it is named "byte-zb" Direct Switch and Queue, and bind the queue to the switch using routing-key of "byte-zb" with the following code:
@Configuration
public class RabbitConfig {

    public static final String QUEUE_NAME = "byte-zb";

    public static final String EXCHANGE_NAME = "byte-zb";

    public static final String ROUTING_KEY = "byte-zb";

    // Queue declaration
    @Bean
    public Queue queue(){
        return new Queue(QUEUE_NAME);
    }

    // Statement Switch
    @Bean
    public DirectExchange directExchange(){

        return new DirectExchange(EXCHANGE_NAME);
    }

    // Data binding declaration
    @Bean
    public Binding directBinding(){

        return BindingBuilder.bind(queue()).to(directExchange()).with(ROUTING_KEY);
    }
}
  1. The creator sends a message with the following code:
@RestController
public class Producer {
    public static final String QUEUE_NAME = "byte-zb";

    public static final String EXCHANGE_NAME = "byte-zb";

    @Autowired
    private AmqpTemplate amqpTemplate;

    @RequestMapping("/send")
    public void sendMessage(){

        JSONObject jsonObject = new JSONObject();
        jsonObject.put("email","11111111111");
        jsonObject.put("timestamp",System.currentTimeMillis());
        String json = jsonObject.toJSONString();
        System.out.println(json);
        amqpTemplate.convertAndSend(EXCHANGE_NAME,QUEUE_NAME,json);
    }

}
  1. Create a consumer consumption message in Consumer Engineering with the following code:
@Component
public class Consumer throws Exception{

    public static final String QUEUE_NAME = "byte-zb";

    @RabbitListener(queues = QUEUE_NAME)
    public void receiveMessage(String message){

        System.out.println("The message received is"+message);
    }
}

We start the producer, request the send interface, then open the rabbitmq console to find that there is an extra switch and queue named "byte-zb", and there is an unexpended message in the queue. Then start the consumer, we will find a message printed on the console and there is no message in the queue named "byte-zb" in the rabbitmq console.

2. Automatic Compensation Mechanism

What happens if consumer message consumption is unsuccessful?Let's modify the consumer code and have a look.

@Component
public class Consumer {

    public static final String QUEUE_NAME = "byte-zb";

    @RabbitListener(queues = QUEUE_NAME)
    public void receiveMessage(String message) throws Exception {

        System.out.println("The message received is"+message);
        int i = 1 / 0;
    }
}

We will see that the Consumer Engineering console keeps refreshing the error, and when the consumer matches an exception, that is, when the message consumption is unsuccessful, it is stored on the server of rabbitmq and retried until no exception is thrown.

If we keep throwing exceptions and our services hang up easily, is there any way to control whether we don't try again after a few unsuccessful attempts?The answer is yes.We added a section of configuration to the consumer application.yml.

spring:
  rabbitmq:
    # Connection Address
    host: 127.0.0.1
    # port
    port: 5672
    # Login Account
    username: guest
    # Logon Password
    password: guest
    # Virtual Host
    virtual-host: /
    listener:
      simple:
        retry:
          enabled: true # Turn on the consumer and try again
          max-attempts: 5 # max retries
          initial-interval: 3000 # Retry interval

The above configuration means that after an abnormal consumption, try again five times, three seconds apart.Continue to launch the consumer to see the effect, and we find that after five retries, we will not try again.

3. Use Message Compensation Mechanism in the light of actual cases

Exceptions like the one above will not actually succeed in any retry. In fact, the only way to use message compensation is to invoke a third-party interface.

Case study: The survivor throws a message to the queue, including a mailbox and sending content.The consumer receives the message and then calls the mail interface to send the message.Sometimes the mail interface may not work because of network and other reasons, then you need to try again.

In a tool class that calls an interface, if an exception occurs and we return null directly, the tool class specific code will not be pasted. What should we do after returning null?We just need to throw an exception, and rabbitListener will automatically retry when it catches it.

Let's transform the consumer code:

@Component
public class Consumer {

    public static final String QUEUE_NAME = "byte-zb";

    @RabbitListener(queues = QUEUE_NAME)
    public void receiveMessage(String message) throws Exception {

        System.out.println("The message received is"+message);
        JSONObject jsonObject = JSONObject.parseObject(message);
        String email = jsonObject.getString("email");
        String content = jsonObject.getString("timestamp");

        String httpUrl = "http://127.0.0.1:8080/email?email"+email+"&content="+content;
        // Return null if an exception occurs
        String body = HttpUtils.httpGet(httpUrl, "utf-8");
        //
        if(body == null){
            throw new Exception();
        }
    }
}

Of course we can customize exception throws.How to experiment? The first step is to start producers and consumers. We find consumers are trying again. The second step is to start the mail service. Then we will find that the mail is sent successfully and consumers will not try again.

4. Solving the Idempotency of Messages

Some students who have just come into contact with java may not be clear about idempotency.Idempotency means that repeated consumption results in inconsistent results.In order to guarantee idempotency, consumer consumption messages can only consume messages once.We can use the global message id to control idempotency.When a message has been consumed, we can choose to save the message id in the cache, and then when consumed again, we can query the cache, if this message id exists, we can handle the direct return well.Modify the producer code first, adding message id to the message:

@RequestMapping("/send")
    public void sendMessage(){

        JSONObject jsonObject = new JSONObject();
        jsonObject.put("email","11111111111");
        jsonObject.put("timestamp",System.currentTimeMillis());
        String json = jsonObject.toJSONString();
        System.out.println(json);

        	Message message = MessageBuilder.withBody(json.getBytes()).setContentType(MessageProperties.CONTENT_TYPE_JSON)
                .setContentEncoding("UTF-8").setMessageId(UUID.randomUUID()+"").build();
        amqpTemplate.convertAndSend(EXCHANGE_NAME,QUEUE_NAME,message);
    }

Consumer Code Modification:

@Component
public class Consumer {

    public static final String QUEUE_NAME = "byte-zb";

    @RabbitListener(queues = QUEUE_NAME)
    public void receiveMessage(Message message) throws Exception {

        Jedis jedis = new Jedis("localhost", 6379);

        String messageId = message.getMessageProperties().getMessageId();
        String msg = new String(message.getBody(),"UTF-8");
        System.out.println("The message received is:"+msg+"==news id For:"+messageId);

        String messageIdRedis = jedis.get("messageId");

        if(messageId == messageIdRedis){
            return;
        }
        JSONObject jsonObject = JSONObject.parseObject(msg);
        String email = jsonObject.getString("email");
        String content = jsonObject.getString("timestamp");

        String httpUrl = "http://127.0.0.1:8080/email?email"+email+"&content="+content;
        // Return null if an exception occurs
        String body = HttpUtils.httpGet(httpUrl, "utf-8");
        //
        if(body == null){
            throw new Exception();
        }
        jedis.set("messageId",messageId);
    }
}

We use redis to store message IDs on the consumer s id e for demonstration purposes only. For specific items, choose the appropriate tool to store them.

If the article is helpful, please remember to pay attention to it.
Welcome to my public number: byte legend, daily push technical articles for your reference.

Topics: Programming RabbitMQ JSON Jedis Spring