Excerpt from: https://www.cnblogs.com/zhengzhaoxiang/p/13976517.html
1, Reliable message final consistency transaction overview
The final consistency scheme of reliable message means that when the transaction initiator executes and completes the local transaction and sends a message, the transaction participants (message consumers) will be able to receive the message and process the transaction successfully. This scheme emphasizes that the final transaction must be consistent as long as the message is sent to the transaction participants. This scheme is completed by using message oriented middleware, as shown in the following figure:
The transaction initiator (message producer) sends the message to the message middleware, and the transaction participant receives the message from the message middleware. The transaction participant (message consumer) and the message middleware communicate through the network. The uncertainty of network communication will lead to distributed transaction problems. Therefore, the final consistency scheme of reliable messages should solve the following problems:
[1] Atomicity of local transaction and message sending: after the local transaction is successfully executed, the transaction initiator must send the message, otherwise it will discard the message. That is, the atomicity of local transactions and message sending is either successful or failed. The atomicity of local transactions and message sending is the key problem to realize the final consistency scheme of reliable messages. Try this operation first, send the message first, and then operate the database: in this case, the consistency between the database operation and the sent message cannot be guaranteed, because the message may be sent successfully and the database operation may fail.
1 begin transaction； 2 //1. Send MQ 3 //2. Database operation 4 commit transation;
The second scheme is to operate the database first and then send the message: there seems to be no problem in this case. If the sending of MQ message fails, an exception will be thrown, resulting in the rollback of database transactions. However, if it is a timeout exception and the database is rolled back, but MQ has actually been sent normally, which will also lead to inconsistency.
1 begin transaction； 2 //1. Database operation 3 //2. Send MQ 4 commit transation;
[2] Reliability of receiving messages by transaction participants: transaction participants must be able to receive messages from the message queue. If receiving messages fails, they can receive messages repeatedly.
[3] Problem of repeated consumption of messages: due to the existence of step 2, if a consumption node times out but the consumption is successful, the message middleware will deliver the message repeatedly, resulting in repeated consumption of messages. To solve the problem of repeated consumption of messages, it is necessary to realize the method idempotency of transaction participants.
2, Solution [local message table]
The scheme of local message table was originally proposed by eBay. The core of this scheme is to ensure the consistency of data business operations and messages through local transactions, and then send the message to the message middleware through a scheduled task. After confirming that the message is sent to the consumer successfully, delete the message. Let's take the registration of points as an example: in the following example, there are two micro service interactions, user service and point service. User service is responsible for adding users, and point service is responsible for increasing points.
[interaction process is as follows]: [1] user registration: user services add users and "integral message log" in local transactions. (the consistency between the user table and the message table is guaranteed through the local transaction) the following is the pseudo code. In this case, the local database operation and the stored integral message log are in the same transaction, and the local database operation and the recorded message log operation are atomic.
1 begin transaction； 2 //1. New users 3 //2. Store integral message log 4 commit transation;
[2] Scheduled task scanning log: how to ensure that messages are sent to the message queue? After the first step, the message has been written to the message log table. You can start an independent thread to scan the message in the message log table regularly and send it to the message middleware. After the message middleware feeds back that the message log is sent successfully, delete the message log, otherwise wait for the next cycle of the scheduled task to retry.
[3] Consumer news: how to ensure that consumers can consume news? Here, the ACK (message confirmation) mechanism of MQ can be used. The consumer listens to MQ. If the consumer receives the message and sends an ACK (message confirmation) to MQ after the business processing is completed, it indicates that the consumer's normal consumption message is completed, and MQ will no longer push the message to the consumer, otherwise the consumer will continue to retry sending the message to the consumer. The point service receives the message of "increasing points" and starts to increase points. After the point is increased successfully, it responds to ack to the message middleware, otherwise the message middleware will deliver the message repeatedly. Since the message will be delivered repeatedly, the "increasing points" function of the point service needs to realize idempotency.
3, Solution [RocketMQ transaction message solution]
RocketMQ is a distributed messaging middleware from Alibaba. It was open source in 2012 and officially became the top project of Apache in 2017. It is understood that, including Alibaba cloud's messaging products and acquired subsidiaries, Alibaba Group's messaging products run on RocketMQ, and RocketMQ has made eye-catching performance in the promotion of the double 11 National Congress in recent years. Apache RocketMQ version after 4.3 officially supports transaction messages, which provides convenient support for the implementation of distributed transactions. The design of RocketMQ transaction message is mainly to solve the atomicity problem between the message sending on the Producer side and the execution of local transactions. In the design of RocketMQ, the two-way communication ability between the broker and the Producer side makes the broker naturally exist as a transaction coordinator; The storage mechanism provided by RocketMQ itself provides the persistence ability for transaction messages; RocketMQ's high availability mechanism and reliable message design are that transaction messages can still ensure the final consistency of transactions in case of system exceptions. After RocketMQ 4.3, the complete transaction message is implemented. In fact, it is an encapsulation of the local message table. The local message table is moved to the interior of MQ to solve the atomicity problem between message sending on the Producer side and local transaction execution.
[execution process is as follows]: to facilitate understanding, we also describe the whole process with the example of registering and sending points. Producer is the MQ sender. In this case, it is a user service, which is responsible for adding users. The MQ subscriber is the message consumer. In this case, it is the point service, which is responsible for adding points.
[1] Producer sends transaction message: Producer (MQ sender) sends transaction message to MQ Server, and MQ Server marks the message status as Prepared. Note that this message cannot be consumed by consumers (MQ subscribers) at this time. In this example, producer sends the "add points message" to MQ Server.
[2] MQ Server response message sent successfully: if MQ Server receives the message sent to by Producer, the response is sent successfully. Indicates that MQ has received a message.
[3] Producer executes local transactions: the producer side executes business code logic and is controlled through local database transactions. In this example, producer performs the add user operation.
[4] Message delivery: if the} Producer local transaction is executed successfully, it will automatically send a commit message to the MQServer. After receiving the commit message, the MQ Server will mark the "add points message" status as consumable. At this time, the MQ subscriber (points service) will consume the message normally. If the Producer # local transaction fails to execute, it will automatically send a Rollback message to MQServer. After receiving the Rollback message, MQ Server will delete the "add points message". The MQ subscriber (point service) consumes the message. If the consumption is successful, it will respond with ack to MQ, otherwise it will receive the message repeatedly. Here, ACK will respond automatically by default, that is, if the program runs normally, ACK will be responded automatically.
[5] Transaction check back: if the execution end hangs up or times out during the local transaction of the Producer, MQ Server will keep asking other producers in the same group to obtain the transaction execution status. This process is called transaction check back. MQ Server will decide whether to post the message according to the transaction backcheck results. The above backbone processes have been implemented by RocketMQ. For the user side, the user needs to implement local transaction execution and local transaction backcheck methods respectively. Therefore, only pay attention to the execution status of local transactions (maintain the local transaction status table). RoacketMQ provides rocketmqllocaltransactionlistener interface:
1 public interface RocketMQLocalTransactionListener {
2 /**The prepare message is sent successfully. This method is called back. This method is used to execute local transactions
3 * @param msg The unique Id of the returned message can be obtained by using transactionId
4 * @param arg Parameters passed when calling the send method. If additional parameters can be passed to the send method when sending, you can get them here
5 * @return Return transaction status, COMMIT: COMMIT ROLLBACK: ROLLBACK unknown: callback
6 */
7 RocketMQLocalTransactionState executeLocalTransaction(Message msg, Object arg);
8
9 /**@param msg Determine the local transaction execution status of this message by obtaining the transactionId
10 * @return Return transaction status, COMMIT: COMMIT ROLLBACK: ROLLBACK unknown: callback
11 */
12 RocketMQLocalTransactionState checkLocalTransaction(Message msg);
13 }
[6] Send transaction message: the following is the API provided by RocketMQ for sending transaction message:
1 TransactionMQProducer producer = new TransactionMQProducer("ProducerGroup");
2 producer.setNamesrvAddr("127.0.0.1:9876");
3 producer.start();
4 //Set TransactionListener implementation
5 producer.setTransactionListener(transactionListener)；
6 //Send transaction message
7 SendResult sendResult = producer.sendMessageInTransaction(msg, null);
4, RocketMQ implements reliable message final consistency transaction
[business description] through RocketMQ middleware, reliable messages and ultimately consistent distributed transactions are realized to simulate the transfer transaction process of two accounts. The two accounts are in different banks (Zhang San in bank1 and Li Si in bank2). Bank1 and bank2 are two micro services. The transaction process is that Zhang San transfers the specified amount to Li Si. In the above transaction steps, Zhang San deducts the amount and sends a transfer message to bank2. The two operations must be an integrated transaction.
[core code]: the technical framework of the program is as follows:
[the interaction process is as follows]: [1] Bank1 sends a transfer message to MQ Server;
[2] Bank1 executes local affairs and deducts the amount;
[3] Bank2 receives messages, executes local transactions and adds amounts;
[database]: add {De to bank1 and bank2 databases_ Duplication, transaction record table (de duplication table), used for transaction idempotent control.
1 DROP TABLE IF EXISTS `de_duplication`; 2 CREATE TABLE `de_duplication` ( 3 `tx_no` varchar(64) COLLATE utf8_bin NOT NULL, 4 `create_time` datetime(0) NULL DEFAULT NULL, 5 PRIMARY KEY (`tx_no`) USING BTREE 6 ) ENGINE = InnoDB CHARACTER SET = utf8 COLLATE = utf8_bin ROW_FORMAT = Dynamic;
[version dependency]: Specifies the version of rocketmq spring boot starter in the parent project
1 <dependency> 2 <groupId>org.apache.rocketmq</groupId> 3 <artifactId>rocketmq-spring-boot-starter</artifactId> 4 <version>2.0.2</version> 5 </dependency>
[configure rocketMQ]: in application local Configure rocketMQ nameServer address and production group in propertis.
1 rocketmq.producer.group = producer_bank2 2 rocketmq.name-server = 127.0.0.1:9876
[Zhang San service layer code]:
1 import com.alibaba.fastjson.JSONObject;
2 import org.apache.rocketmq.spring.core.RocketMQTemplate;
3 import org.springframework.beans.factory.annotation.Autowired;
4 import org.springframework.messaging.Message;
5 import org.springframework.messaging.support.MessageBuilder;
6 import org.springframework.stereotype.Service;
7 import org.springframework.transaction.annotation.Transactional;
8
9 /**
10 * @author Administrator
11 * @version 1.0
12 **/
13 @Service
14 @Slf4j
15 public class AccountInfoServiceImpl implements AccountInfoService {
16
17 @Autowired
18 AccountInfoDao accountInfoDao;
19
20 @Autowired
21 RocketMQTemplate rocketMQTemplate;
22
23
24 //Send transfer message to mq
25 @Override
26 public void sendUpdateAccountBalance(AccountChangeEvent accountChangeEvent) {
27
28 //Convert accountChangeEvent to json
29 JSONObject jsonObject =new JSONObject();
30 jsonObject.put("accountChange",accountChangeEvent);
31 String jsonString = jsonObject.toJSONString();
32 //Generate message type
33 Message<String> message = MessageBuilder.withPayload(jsonString).build();
34 //Send a transaction message
35 /**
36 * String txProducerGroup production team
37 * String destination topic，
38 * Message<?> message, Message content
39 * Object arg parameter
40 */
41 rocketMQTemplate.sendMessageInTransaction("producer_group_txmsg_bank1","topic_txmsg",message,null);
42
43 }
44
45 //Update the account and deduct the amount
46 @Override
47 @Transactional
48 public void doUpdateAccountBalance(AccountChangeEvent accountChangeEvent) {
49 //Idempotent judgment: txNo is the UUID generated in the Ctroller, which is globally unique
50 if(accountInfoDao.isExistTx(accountChangeEvent.getTxNo())>0){
51 return ;
52 }
53 //Deduction amount
54 accountInfoDao.updateAccountBalance(accountChangeEvent.getAccountNo(),accountChangeEvent.getAmount() * -1);
55 //Add transaction log
56 accountInfoDao.addTx(accountChangeEvent.getTxNo());
57 if(accountChangeEvent.getAmount() == 3){
58 throw new RuntimeException("Artificial anomaly");
59 }
60 }
61 }
[Zhang San rocketmqllocaltransactionlistener]: write the interface implementation class of rocketmqllocaltransactionlistener to implement the two methods of executing local transactions and transaction backcheck.
1 import com.alibaba.fastjson.JSONObject;
2 import org.apache.rocketmq.spring.annotation.RocketMQTransactionListener;
3 import org.apache.rocketmq.spring.core.RocketMQLocalTransactionListener;
4 import org.apache.rocketmq.spring.core.RocketMQLocalTransactionState;
5 import org.springframework.messaging.Message;
6 import org.springframework.transaction.annotation.Transactional;
7
8 /**
9 * @author Administrator
10 * @version 1.0
11 **/
12 @Component
13 @Slf4j
14 //The producer group is the same as the group defined when sending the message
15 @RocketMQTransactionListener(txProducerGroup = "producer_group_txmsg_bank1")
16 public class ProducerTxmsgListener implements RocketMQLocalTransactionListener {
17
18 @Autowired
19 AccountInfoService accountInfoService;
20
21 @Autowired
22 AccountInfoDao accountInfoDao;
23
24 //The callback method after the transaction message is sent. When the message is sent to mq successfully, this method is called back
25 @Override
26 @Transactional
27 public RocketMQLocalTransactionState executeLocalTransaction(Message message, Object o) {
28
29 try {
30 //Parse the message and convert it to AccountChangeEvent
31 String messageString = new String((byte[]) message.getPayload());
32 JSONObject jsonObject = JSONObject.parseObject(messageString);
33 String accountChangeString = jsonObject.getString("accountChange");
34 //Convert accountChange (json) to AccountChangeEvent
35 AccountChangeEvent accountChangeEvent = JSONObject.parseObject(accountChangeString, AccountChangeEvent.class);
36 //Execute local transactions and deduct amount
37 accountInfoService.doUpdateAccountBalance(accountChangeEvent);
38 //When returning rocketmqllocaltransactionstate Commit, automatically send a commit message to mq, and mq changes the status of the message to consumable
39 return RocketMQLocalTransactionState.COMMIT;
40 } catch (Exception e) {
41 e.printStackTrace();
42 return RocketMQLocalTransactionState.ROLLBACK;
43 }
46 }
47
48 //Check back the transaction status to query whether to deduct the amount
49 @Override
50 public RocketMQLocalTransactionState checkLocalTransaction(Message message) {
51 //Parse the message and convert it to AccountChangeEvent
52 String messageString = new String((byte[]) message.getPayload());
53 JSONObject jsonObject = JSONObject.parseObject(messageString);
54 String accountChangeString = jsonObject.getString("accountChange");
55 //Convert accountChange (json) to AccountChangeEvent
56 AccountChangeEvent accountChangeEvent = JSONObject.parseObject(accountChangeString, AccountChangeEvent.class);
57 //Transaction id
58 String txNo = accountChangeEvent.getTxNo();
59 int existTx = accountInfoDao.isExistTx(txNo);
60 if(existTx>0){
61 return RocketMQLocalTransactionState.COMMIT;
62 }else{
63 return RocketMQLocalTransactionState.UNKNOWN;
64 }
65 }
66 }
[Li Si service layer code]:
1 import org.springframework.stereotype.Service;
2 import org.springframework.transaction.annotation.Transactional;
3
4 /**
5 * @author Administrator
6 * @version 1.0
7 **/
8 @Service
9 @Slf4j
10 public class AccountInfoServiceImpl implements AccountInfoService {
11
12 @Autowired
13 AccountInfoDao accountInfoDao;
14
15 //Update the account and increase the amount
16 @Override
17 @Transactional
18 public void addAccountInfoBalance(AccountChangeEvent accountChangeEvent) {
19 log.info("bank2 Update local account, account:{},amount of money:{}",accountChangeEvent.getAccountNo(),accountChangeEvent.getAmount());
20 if(accountInfoDao.isExistTx(accountChangeEvent.getTxNo())>0){
21 return ;
22 }
23 //Increase amount
24 accountInfoDao.updateAccountBalance(accountChangeEvent.getAccountNo(),accountChangeEvent.getAmount());
25 //Add transaction record for idempotent
26 accountInfoDao.addTx(accountChangeEvent.getTxNo());
27 if(accountChangeEvent.getAmount() == 4){
28 throw new RuntimeException("Artificial anomaly");
29 }
30 }
31 }
[MQ listening class]: listen to the target Topic by implementing the RocketMQListener interface
1 import com.alibaba.fastjson.JSONObject;
2 import org.apache.rocketmq.spring.annotation.RocketMQMessageListener;
3 import org.apache.rocketmq.spring.core.RocketMQListener;
4
5 /**
6 * @author Administrator
7 * @version 1.0
8 **/
9 @Component
10 @Slf4j
11 @RocketMQMessageListener(consumerGroup = "consumer_group_txmsg_bank2",topic = "topic_txmsg")
12 public class TxmsgConsumer implements RocketMQListener<String> {
13
14 @Autowired
15 AccountInfoService accountInfoService;
16
17 //receive messages
18 @Override
19 public void onMessage(String message) {
20 log.info("Start consumption message:{}",message);
21 //Parse message
22 JSONObject jsonObject = JSONObject.parseObject(message);
23 String accountChangeString = jsonObject.getString("accountChange");
24 //Convert to AccountChangeEvent
25 AccountChangeEvent accountChangeEvent = JSONObject.parseObject(accountChangeString, AccountChangeEvent.class);
26 //Set the account number as Li Si
27 accountChangeEvent.setAccountNo("2");
28 //Update local account and increase amount
29 accountInfoService.addAccountInfoBalance(accountChangeEvent);
31 }
32 }
5, Summary
The final consistency of reliable messages is to ensure the consistency of messages transmitted from the manufacturer to the consumer through message middleware. In this case, RocketMQ is used as message middleware. RocketMQ mainly solves two functions:
[1] Atomicity of local transactions and message sending;
[2] The reliability of the message received by the transaction participants;
Reliable message final consistency transaction is suitable for scenarios with long execution cycle and low real-time requirements. After the introduction of message mechanism, the synchronous transaction operation becomes asynchronous operation based on message execution, which avoids the influence of synchronous blocking operation in distributed transactions, and realizes the decoupling of the two services.