1, Overview
Distributed transaction messages are a unique feature of RocketMQ. In many scenarios, the strong consistency of transactions is not required, but only the final consistency of transactions. At this point, transaction messages can well meet the requirements.
By putting the local transaction and message sending into one local transaction, it is ensured that when the local transaction is successfully executed, the message will be successfully delivered to the message server. Finally, the high reliability of message middleware is used to ensure that the message will be consumed by downstream services.
Basic concepts
Distributed transaction
For distributed transactions, generally speaking, an operation consists of several branch operations, which belong to different applications and are distributed on different servers. Distributed transactions need to ensure that these branch operations either succeed or fail. Distributed transactions are the same as ordinary transactions in order to ensure the consistency of operation results.
Transaction message
RocketMQ provides a distributed transaction function similar to X/Open XA. The final consistency of distributed transactions can be achieved through transaction messages. XA is a distributed transaction solution and a step-by-step transaction processing mode.
Semi transaction message
For a message that cannot be delivered temporarily, the sender has successfully sent the message to the Broker, but the Broker has not received the final confirmation instruction. At this time, the message is marked as "temporarily undeliverable", that is, it cannot be seen by the consumer. Messages in this state are semi transaction messages.
Local transaction status
The result of the Producer callback operation is the local transaction status, which will be sent to TC, and TC will send it to TM. TM will determine the global transaction confirmation instruction according to the local transaction status sent by TC.
package org.apache.rocketmq.client.producer
/* Describes the local transaction execution status */
public enum LocalTransactionState {
COMMIT_MESSAGE, // Local transaction executed successfully
ROLLBACK_MESSAGE, // Local transaction execution failed
UNKNOW, // Uncertain indicates that a backcheck is required to determine the execution result of the local transaction
}
Message check back
Message back query, that is, re query the execution status of local transactions. Generally, you can go back to the DB to check whether the preprocessing operation is successful.
Note that message lookback is not a callback operation. Callback is a preprocessing operation, while message query is to view the execution results of preprocessing operations.
There are two most common reasons for message backtracking:
1) Callback operation returns unknown
2) TC did not receive the final global transaction confirmation instruction from TM (TM interacts with TC through the network, and there is a possibility of timeout as long as there is network jitter)
Message callback settings in RocketMQ
There are three common property settings for message lookback. They are all set in the configuration file loaded by the broker, for example:
transactionTimeout=20, specifies that TM should send the final confirmation status to TC within 20 seconds, otherwise a message query will be triggered. The default is 60 seconds.
transactionCheckMax=5. Specify that you can check back up to 5 times. After that, the message will be discarded and the error log will be recorded. The default is 15 times.
transactionCheckInterval=10, specifies that the set time interval for multiple message lookback is 10 seconds. The default is 60 seconds.
be careful
- Transaction messages do not support deferred messages
- Check the idempotency of transaction messages, because transaction messages may be consumed more than once (because there is a case of committing after rollback)
2, Example
scene
A demand scenario here is: ICBC user a transfers 10000 yuan to CCB user B.
We can use synchronization messages to handle this requirement scenario:
- ICBC system sends a synchronization message of RMB 10000 to B and M to Broker
- After the message is successfully received by the Broker, a successful ACK is sent to the ICBC system
- After receiving the successful ACK, ICBC system will deduct 10000 yuan from user A
- CCB system obtains message M from Broker
- CCB system consumption message M, that is, add 10000 yuan to users
There is a problem: if the deduction operation in step 3 fails, but the message has been successfully sent to the Broker. For MQ, as long as the message is written successfully, the message can be consumed. At this time, user B in CCB system increased by 10000 yuan. Data inconsistency occurred.
Solution: make steps 1, 2 and 3 atomic. Either all succeed or all fail. That is, after the message is sent successfully, the deduction must be successful. If the deduction fails, the rollback sends a successful message. The idea is to use transaction messages. A distributed transaction solution is used here.
Here, transaction messages are used to process business:
Message query in this example:
structure
Introduce dependency
<!-- https://mvnrepository.com/artifact/org.apache.rocketmq/rocketmq-spring-boot-starter -->
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-spring-boot-starter</artifactId>
<version>2.2.1</version>
</dependency>
The message header is used by the RocketMQHeaders constant
RocketMQHeaders class provides us with common system constants. We can use them to replace the KEYS of some headers, such as RocketMQHeaders.KEYS = "KEYS". We can directly use RocketMQHeaders.KEYS.
Common Header parameters:
| parameter | explain |
|---|---|
| KEYS | KEYS |
| TRANSACTION_ID | Transaction ID |
| MESSAGE_ID | Message ID |
| QUEUE_ID | Message Queue ID |
| TAGS | Message Tag tag |
| TOPIC | Message Topic topic |
code
Producer: simulate a user to initiate a transfer request
/*
* [[producer] simulate a user to initiate a transfer request
*/
@Slf4j
@Service
public class TransactionProducerService {
// TOPIC name
private static final String TOPIC = "transTopic";
// TAG information
private static final String TAG = "transTag";
@Autowired
private RocketMQTemplate rocketMQTemplate;
public TransactionSendResult sendHalfMsg(String msg){
// Generate transaction ID
String transactionId = UUID.randomUUID().toString().replace("-","");
log.info("[Send half message] transactionId={}", transactionId);
String transKeys = "transKey";
// Send transaction message
TransactionSendResult sendResult = rocketMQTemplate.sendMessageInTransaction(
TOPIC + ":" + TAG,
MessageBuilder.withPayload(msg)
.setHeader(RocketMQHeaders.TRANSACTION_ID, transactionId)
.setHeader(RocketMQHeaders.KEYS,transKeys) // Using encapsulated constants is less error prone than using "KEYS"
.build(),
msg
);
log.info("[Send half message] sendResult={}",msg);
return sendResult;
}
}
Listener (local transaction): simulate ICBC to conduct deduction activities
/*
* [Transaction listener (local transaction)] simulates ICBC to conduct deduction activities
*/
@Slf4j
@RocketMQTransactionListener()
public class ICBCTransactionListener implements RocketMQLocalTransactionListener {
@Override
public RocketMQLocalTransactionState executeLocalTransaction(Message msg, Object arg) {
MessageHeaders messageHeaders = msg.getHeaders();
String transactionId = (String) messageHeaders.get(RocketMQHeaders.TRANSACTION_ID);
log.info("Pre delivery message succeeded:{}",msg);
log.info("[Execute local transaction] message body parameters: transactionId={}", transactionId);
try {
StringBuilder money = new StringBuilder();
byte[] bytes = ((byte[])msg.getPayload());
for (int i = 0; i < bytes.length; i++) {
money.append(bytes[i] - '0');
}
log.info("[[successful execution of local transaction] ICBC account deduction" + money +"element!");
return RocketMQLocalTransactionState.COMMIT;
} catch (Exception e) {
return RocketMQLocalTransactionState.ROLLBACK;
}
}
@Override
public RocketMQLocalTransactionState checkLocalTransaction(Message msg) {
MessageHeaders headers = msg.getHeaders();
String transactionId = headers.get(RocketMQHeaders.TRANSACTION_ID, String.class);
log.info("Execute message backcheck:{}",msg);
log.info("[[back check local transactions] transactionId={}",transactionId);
// Execute relevant business
// if(...) {
// return RocketMQLocalTransactionState.ROLLBACK;
// else {
return RocketMQLocalTransactionState.COMMIT;
// }
// return RocketMQLocalTransactionState.UNKNOW;
}
}
Consumer: simulate CCB's deposit activities
/*
* [Consumer] simulate CCB's deposit activities
*/
@Slf4j
@Service
@RocketMQMessageListener(topic = "transTopic", selectorExpression = "transTag", consumerGroup = "cg")
public class CCBTransactionConsumerService implements RocketMQListener<String> {
@Override
public void onMessage(String message) {
// Idempotent judgment
// 1. Use unique fields for judgment, such as order number
// 2. Create a new table with unique fields to assist judgment
// Execute specific business
// if(...) {/ / execution failed
// log.error("[execution failed] transfer failed!");
// }else / / execution succeeds
log.info("[[executed successfully] transfer succeeded! Increase in CCB account" + message + "Yuan!");
// }
}
}
result
Swagger tests:
Consume messages and process business (actually different upstream and downstream services, where producers and consumers operate under the same project):
Rocketmq console display message:
The transaction message processing is completed and the business is successfully executed!
// }
}
}
## result Swagger Test: [External chain picture transfer...(img-qQ6NqBSv-1638109562163)] Processing message services (actually different upstream and downstream services, where producers and consumers operate under the same project): [External chain picture transfer...(img-qCVKdkEm-1638109562164)] RocketMQ-Console Display message: [External chain picture transfer...(img-J51LAB2k-1638109562167)] The transaction message processing is completed and the business is successfully executed!