Distributed transaction

Posted by Harsh on Wed, 02 Feb 2022 11:46:10 +0100

1. TCC of distributed transaction solution

1.1 what is a TCC transaction

TCC is the abbreviation of try, Confirm and Cancel. TCC requires each branch transaction to implement three operations: preprocess try, Confirm and Cancel. Try performs business check and resource reservation, Confirm performs business confirmation, and Cancel implements an operation opposite to try, that is, rollback. TM first initiates the try operation of all branch transactions. If the try operation of any branch transaction fails, TM will initiate the Cancel operation of all branch transactions. If all the try operations are successful, TM will initiate the Confirm operation of all branch transactions. If the Confirm/Cancel operation fails, TM will retry.

Branch transaction failure:

TCC is divided into three stages:

The Try stage is for business checking (consistency) and resource reservation (isolation). This stage is a preliminary operation. It can really form a complete business logic together with the subsequent Confirm.
The Confirm phase is to Confirm the submission. In the Try phase, Confirm is executed after all branch transactions are executed successfully. Generally, if TCC is adopted, it is considered that there will be no error in the Confirm phase. That is: as long as Try succeeds, Confirm will succeed. If there is an error in the Confirm phase, the retry mechanism or manual processing need to be introduced.
In the Cancel phase, when the business execution error needs to be rolled back, the business executing the branch transaction is cancelled and the reserved resources are released. Generally, if TCC is adopted, it is considered that the Cancel stage is also certain to be successful. If there is an error in the Cancel phase, the retry mechanism or manual processing need to be introduced.
TM transaction manager
The TM transaction manager can implement independent services or let the global transaction initiator play the role of TM. TM is independent to be called a common component and to consider the system structure and software reuse.
When TM initiates a global transaction again, it generates a global transaction record. The global transaction ID runs through the call link of the whole distributed transaction and is used to record the transaction context, track and record the status. Because the Confirm and Cancel failures need to be retried, idempotence needs to be realized. Idempotency means that the result of the same operation is the same no matter how many times it is requested.

1.2 TCC solution

As mentioned in the previous section, Seata also supports TCC, but the TCC mode of Seata does not support Spring Cloud. Our goal is to understand the principle of TCC and the process of transaction coordination. Therefore, we prefer a lightweight and easy to understand framework, so we finally determined Hmily.

Hmily is a high-performance distributed transaction TCC open source framework. It is developed based on Java language (JDK1.8) and supports RPC frameworks such as Dubbo and Spring Cloud for distributed transactions. It currently supports the following features:

Nested transaction support is supported
The asynchronous reading and writing of transaction log using the disruptor framework is no different from the performance of RPC framework.
Support springboot starter project startup, easy to use.
RPC framework support: dubbo,motan,springcloud.
Local transaction storage support: redis,mongodb,zookeeper,file,mysql.
Transaction log serialization support: java, hessian, kryo, protostuff.
It adopts the Aspect AOP perspective idea to seamlessly integrate with Spring and naturally support clusters.
RPC transaction recovery, timeout exception recovery, etc.

Hmily uses AOP to intercept local and remote methods participating in distributed transactions. Through multi-party interception, transaction participants can call the Try, Confirm and Cancel methods of the other party transparently; Pass transaction context; And record the transaction log, make compensation and retry as appropriate.

Hmily does not need transaction coordination services, but needs to provide a database (mysql/mongodb/zookeeper/redis/file) for log storage.

Like ordinary services, the TCC service implemented by Hmily only needs to expose one interface, that is, its Try business. The Confirm/Cancel business logic is only provided for the need of global transaction commit / rollback. Therefore, the Confirm/Cancel business only needs to be discovered by the Hmily TCC transaction framework and does not need to be perceived by other business services calling it.

Introduction to the official website: https://dromara.org/zh/projects/hmily/user-springcloud/

TCC should pay attention to three kinds of exception handling: null rollback, idempotent and suspension:

Empty rollback:

When the TCC resource Try method is not called, the two-stage Cancel method is called. The Cancel method needs to recognize that this is an empty rollback, and then directly return success.

The reason is that when a branch transaction is in service downtime or network abnormality, the branch transaction call is recorded as failure. At this time, the Try phase is not executed. When the fault is recovered, the distributed transaction rollback will call the Cancel method of the second phase, resulting in an empty rollback.

The key to the solution is to identify the empty rollback. The idea is very simple. You need to know whether a phase is executed. If it is executed, it is normal rollback; If it is not executed, it is an empty rollback. As mentioned earlier, TM generates global transaction records when initiating global transactions, and the global transaction ID runs through the whole distributed transaction call chain. Add an additional branch transaction record table, including global transaction ID and branch transaction ID. in the first stage, a record will be inserted in the Try method, indicating that the first stage has been executed. Read the record in the Cancel interface. If the record exists, it will be rolled back normally; If the record does not exist, it is an empty rollback.

Idempotent:

It has been learned from the previous introduction that in order to ensure that the two-stage submission retry mechanism of TCC will not cause data inconsistency, the two-stage Try, Confirm and Cancel interfaces of TCC are required to ensure idempotence, so as not to reuse or release resources. If idempotent control is not done well, it is likely to lead to serious problems such as data inconsistency.

Solution: add the execution status in the above "branch transaction record", and query the status before each execution.

Suspension:

Suspension means that for a distributed transaction, the two-stage Cancel interface executes before the Try interface.

The reason is that when the RPC calls the branch transaction Try, the branch transaction is registered first and then the RPC call is executed. If the RPC call network is congested at this time, the RPC call usually has a timeout time. After the RPC timeout, TM will notify RM to roll back the distributed transaction. It may be that after the rollback is completed, the RPC request will arrive at the participant for real execution, The business resources reserved by a Try method can only be used by the distributed transaction, and the business resources reserved in the first stage of the distributed transaction can no longer be processed. In this case, we call it suspension, that is, after the business resources are reserved, they cannot be processed.

The solution is that if the implementation of the second stage is completed, the implementation of that stage can no longer be continued. When executing a phase I transaction, judge whether there is a phase II transaction record in the "branch transaction record" table under the global transaction. If so, Try will not be executed.

For example, the scenario is that A transfers 30 yuan to B, and accounts A and B have different services.

Option 1:

Account A

try: 
	Check whether the balance is enough for 30 yuan
	Deduct 30 yuan
confirm: 
	empty
cancel: 
	Increase by 30 yuan

Account B

try: 
	Increase by 30 yuan
confirm: 
	empty
cancel: 
	30 yuan less

Description of scheme I:

Account A, the balance here is the so-called business resources. According to the principles mentioned above, business resources need to be checked and reserved in the first stage. Therefore, we first check whether the balance of account A is enough in the Try interface of money deduction TCC resources. If it is enough, deduct 30 yuan. The Confirm interface indicates the formal submission. Since the business resources have been deducted from the Try interface, nothing can be done in the Confirm interface in the second stage. The execution of the Cancel interface means that the whole transaction is rolled back. If account A rolls back, it needs to return the 30 yuan deducted from the Try interface to the account.
2) Account B adds money to account B in the Try interface in the first stage. The execution of the Cancel interface means that the whole transaction is rolled back. For account B to roll back, the 30 yuan added in the Try interface needs to be subtracted.

Problem analysis of scheme 1:

1) If the try of account A is not cancel led, an additional 30 yuan will be added.
2) Since try, cancel and confirm are all called by separate threads, and there will be repeated calls, idempotent needs to be implemented.
3) Account B adds 30 yuan to the try. When the try is completed, it may be consumed by other threads.
4) If the try of account B is not cancel led, it will be reduced by 30 yuan.

Problem solving:

1) The cancel method of account A needs to judge whether the try method is executed. Cancel can be executed only after the try is executed normally.
2) try, cancel and confirm methods implement idempotent.
3) Account number B is not allowed to update the account amount in the try method. Update the account amount in confirm.
4) The cancel method of account B needs to judge whether the try method is executed. Cancel can be executed only after the try is executed normally.

Optimization scheme:

Account A

try: 
	try Idempotent check
	try Hanging treatment
	Check whether the balance is enough for 30 yuan
	Deduct 30 yuan
confirm: 
	empty
cancel: 
	cancel Idempotent check
	cancel Empty rollback processing
	Increase by 30 yuan

Account B

try: 
	empty
confirm: 
	confirm Idempotent check
	Official increase of 30 yuan
cancel: 
	empty

1.3 Hmily implements TCC transactions

1.3.1 business description

This example realizes TCC distributed transaction through Hmily and simulates the transfer transaction process of two accounts.

The two accounts are in different banks (Zhang San in bank1 and Li Si in bank2). Bank1 and bank2 are two micro services.

The transaction process is that Zhang San transfers the specified amount to Li Si.

The above transaction steps, either succeed or fail together, must be an integral transaction.

1.3.2 procedure components

Database: MySQL-5.7.25

JDK: 64 bit jdk1 8.0_ two hundred and one

Microservices: spring-boot-2.1.3, spring cloud Greenwich RELEASE

Hmily: hmily-springcloud.2.0.4-RELEASE

Relationship between microservices and databases:

tcc-bank1-server bank 1, operate Zhang San's account and connect to the database bank1

tcc-bank2-server bank 2, operate the Li Si account and connect to the database bank2

Service registry: nacos registry

1.3.3 creating database

Create an hmily database to store the data recorded by the hmily framework.

CREATE DATABASE `hmily` 
CHARACTER SET 'utf8' 
COLLATE 'utf8_general_ci';

Create bank1 library and import the following table structure and data (including Zhang San account)

Create bank2 library and import the following table structure and data (including Li Si account)

It is consistent with the database in seata above.

Each database creates three log tables: try, confirm and cancel:

CREATE TABLE `local_try_log` (  
`tx_no` varchar(64) NOT NULL COMMENT 'affair id',  
`create_time` datetime DEFAULT NULL,  
PRIMARY KEY (`tx_no`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

CREATE TABLE `local_confirm_log` (  
`tx_no` varchar(64) NOT NULL COMMENT 'affair id',  
`create_time` datetime DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8

CREATE TABLE `local_cancel_log` (  
`tx_no` varchar(64) NOT NULL COMMENT 'affair id',  
`create_time` datetime DEFAULT NULL,  
PRIMARY KEY (`tx_no`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

1.3.4 create tcc-bank1-server

application.yml

server:
  port: 2006
spring:
  application:
    name: hmily-bank1-server
  datasource:
    username: root
    password: 123456
    url: jdbc:mysql://localhost:3306/bank1?useSSL=false
    driver-class-name: org.gjt.mm.mysql.Driver
    type: com.alibaba.druid.pool.DruidDataSource
  cloud:
    nacos:
      discovery:
        server-addr: localhost:8848


org:
  dromara:
    hmily :
      serializer : kryo
      recoverDelayTime : 30
      retryMax : 30
      scheduledDelay : 30
      scheduledThreadMax :  10
      repositorySupport : db
      started: true
      hmilyDbConfig :
        driverClassName: com.mysql.jdbc.Driver
        url :  jdbc:mysql://localhost:3306/hmily?useUnicode=true&useSSL=false
        username : root
        password : 123456

feign:
  hystrix:
    enabled: true


mybatis:
  mapper-locations: classpath:/mapper/*.xml
  configuration:
    log-impl: org.apache.ibatis.logging.stdout.StdOutImpl
logging:
  level:
    io:
      seata: info

config

package com.yao.hmily.config;


import org.dromara.hmily.common.config.HmilyDbConfig;
import org.dromara.hmily.core.bootstrap.HmilyTransactionBootstrap;
import org.dromara.hmily.core.service.HmilyInitService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.EnableAspectJAutoProxy;
import org.springframework.core.env.Environment;


/**
 * Data source agent
 * @author yao
 */
@Configuration
@EnableAspectJAutoProxy(proxyTargetClass=true)
public class HmilyConfiguration {

    @Autowired
    private Environment env;


    @Bean
    public HmilyTransactionBootstrap hmilyTransactionBootstrap(HmilyInitService hmilyInitService){
        HmilyTransactionBootstrap hmilyTransactionBootstrap = new HmilyTransactionBootstrap(hmilyInitService);
        hmilyTransactionBootstrap.setSerializer(env.getProperty("org.dromara.hmily.serializer"));
        hmilyTransactionBootstrap.setRecoverDelayTime(Integer.parseInt(env.getProperty("org.dromara.hmily.recoverDelayTime")));
        hmilyTransactionBootstrap.setRetryMax(Integer.parseInt(env.getProperty("org.dromara.hmily.retryMax")));
        hmilyTransactionBootstrap.setScheduledDelay(Integer.parseInt(env.getProperty("org.dromara.hmily.scheduledDelay")));
        hmilyTransactionBootstrap.setScheduledThreadMax(Integer.parseInt(env.getProperty("org.dromara.hmily.scheduledThreadMax")));
        hmilyTransactionBootstrap.setRepositorySupport(env.getProperty("org.dromara.hmily.repositorySupport"));
        hmilyTransactionBootstrap.setStarted(Boolean.parseBoolean(env.getProperty("org.dromara.hmily.started")));
        HmilyDbConfig hmilyDbConfig = new HmilyDbConfig();
        hmilyDbConfig.setDriverClassName(env.getProperty("org.dromara.hmily.hmilyDbConfig.driverClassName"));
        hmilyDbConfig.setUrl(env.getProperty("org.dromara.hmily.hmilyDbConfig.url"));
        hmilyDbConfig.setUsername(env.getProperty("org.dromara.hmily.hmilyDbConfig.username"));
        hmilyDbConfig.setPassword(env.getProperty("org.dromara.hmily.hmilyDbConfig.password"));
        hmilyTransactionBootstrap.setHmilyDbConfig(hmilyDbConfig);
        return hmilyTransactionBootstrap;
    }
}

dao

package com.yao.hmily.dao;

import org.apache.ibatis.annotations.Mapper;
import org.apache.ibatis.annotations.Param;

@Mapper
public interface AccountInfoDao {

    //Deduction amount of Zhang San
    /**
     *
     * @param accountNo  Bank card number
     * @param amount  Transfer amount
     * @return
     */
    int reduceAccountBalance(@Param("accountNo") String accountNo,
                             @Param("amount") Double amount);


    //Zhang San increased amount
    /**
     *
     * @param accountNo  Bank card number
     * @param amount  Transfer amount
     * @return
     */
    int addAccountBalance(@Param("accountNo") String accountNo,
                          @Param("amount") Double amount);



    /**
     * Add a branch transaction try execution record
     * @param localTradeNo Local transaction number
     * @return
     */
    int addTry(@Param("txNo") String localTradeNo);

    /**
     * Add a branch transaction confirm execution record
     * @param localTradeNo Local transaction number
     * @return
     */
    int addConfirm(@Param("txNo") String localTradeNo);

    /**
     * Add a branch transaction cancel execution record
     * @param localTradeNo Local transaction number
     * @return
     */
    int addCancel(@Param("txNo") String localTradeNo);

    /**
     * Query whether the branch transaction try has been executed
     * @param localTradeNo Local transaction number
     * @return
     */
    int isExistTry(@Param("txNo") String localTradeNo);
    /**
     * Query whether the branch transaction confirm has been executed
     * @param localTradeNo Local transaction number
     * @return
     */
    int isExistConfirm(@Param("txNo") String localTradeNo);

    /**
     * Query whether branch transaction cancel has been executed
     * @param localTradeNo Local transaction number
     * @return
     */
    int isExistCancel(@Param("txNo") String localTradeNo);

}

mapper.xml

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd" >
<mapper namespace="com.yao.hmily.dao.AccountInfoDao" >


    <update id="reduceAccountBalance">
        update account_info
        set
                account_balance = account_balance - #{amount}
        WHERE
                account_balance >= #{amount}
        and
                account_no = #{accountNo}
    </update>

    <update id="addAccountBalance">
        update account_info
        set
                account_balance = account_balance + #{amount}
        WHERE
                account_no = #{accountNo}
    </update>

    <insert id="addTry">
        insert into
                local_try_log
        values (#{txNo},now())
    </insert>

    <insert id="addConfirm">
        insert into
                local_confirm_log
        values (#{txNo},now());
    </insert>

    <insert id="addCancel" >
        insert into
                local_cancel_log
        values (#{txNo},now());
    </insert>

    <select id="isExistTry" resultType="int">
        select
                count(1)
        from
                local_try_log
        where
                tx_no = #{txNo}
    </select>

    <select id="isExistConfirm" resultType="int">
        select
                count(1)
        from
                local_confirm_log
        where
                tx_no = #{txNo}
    </select>

    <select id="isExistCancel" resultType="int">
        select
                count(1)
        from
                local_cancel_log
        where
                tx_no = #{txNo}
    </select>
</mapper>

feign remote call

package com.yao.hmily.service;

import com.yao.hmily.domain.CommonResult;
import org.dromara.hmily.annotation.Hmily;
import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;

@FeignClient(value = "hmily-bank2-server")
public interface Bank2Service {

    //Zhang San transfer
    @GetMapping("/transfer/{amount}")
    @Hmily
    CommonResult transfer(@PathVariable("amount") Double amount);
}

Business class

package com.yao.hmily.service.impl;


import com.yao.hmily.dao.AccountInfoDao;
import com.yao.hmily.domain.CommonResult;
import com.yao.hmily.service.Bank2Service;
import lombok.extern.slf4j.Slf4j;

import org.dromara.hmily.annotation.Hmily;
import org.dromara.hmily.core.concurrent.threadlocal.HmilyTransactionContextLocal;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import com.yao.hmily.service.AccountInfoService;
import org.springframework.transaction.annotation.Transactional;

import java.util.concurrent.TimeUnit;

@Service
@Slf4j
public class AccountInfoServiceImpl implements AccountInfoService {

    @Autowired
    private AccountInfoDao accountInfoDao;

    @Autowired
    private Bank2Service bank2Service;

    /**
     * Account deduction is the try method of tcc
     * try Idempotent check
     * try Hanging treatment
     * Check whether the balance is enough for 30 yuan
     * Deduct 30 yuan
     * @param accountNo
     * @param amount
     * @return
     */
    @Override
    @Hmily(confirmMethod = "commit",cancelMethod = "rollback") //As long as the annotation is marked, it is a try method. Specify the names of confirm and cancel in the annotation
    @Transactional
    public void updateAccountBalance(String accountNo, Double amount) {

        //Get global transaction ID
        String transId = HmilyTransactionContextLocal.getInstance().get().getTransId();
        log.info("bank1 try begin It's starting xid():"+transId);
        //Idempotent judgment: judge local_ try_ Is there a try log record in the log table? If so, it will not be executed again
        if (accountInfoDao.isExistTry(transId) > 0){
            log.info("bank1 Idempotent processing try It has been executed, and there is no need to repeat it, xid():"+transId);
            return ;
        }

        //Suspension processing: if cancel and confirm are executed, try is not executed
        if (accountInfoDao.isExistConfirm(transId) > 0 || accountInfoDao.isExistCancel(transId) > 0) {
            log.info("bank1 try Suspension treatment, cancel or confirm Executed, not allowed try，xid():"+transId);
            return ;
        }

        //Deduction balance
        if (accountInfoDao.reduceAccountBalance(accountNo, amount) <= 0){
            //Deduction failed
            throw new RuntimeException("bank1 try Deduction failed, xid():"+transId);
        }

        //Insert try execution record for idempotent judgment
        accountInfoDao.addTry(transId);


        //Remote call bank2
        CommonResult transfer = bank2Service.transfer(amount);

        if(transfer.getCode() != 200){
            //Call failed
            throw new RuntimeException("Remote service call failed, xid():"+transId);
        }

        //Artificial anomaly
        if (amount == 200){
            throw new RuntimeException("Man made anomalies, xid():"+transId);
        }
    }

    //confirm method
    public void commit(String accountNo, Double amount){
        //Get global transaction ID
        String transId = HmilyTransactionContextLocal.getInstance().get().getTransId();
        log.info("bank1 confirm begin It's starting xid():"+transId);
        //No action required
    }

    /** cancel method
     * cancel Idempotent check
     * cancel Empty rollback processing
     * Increase by 30 yuan
     * @param accountNo
     * @param amount
     */
    @Transactional
    public void rollback(String accountNo, Double amount){

        //Get global transaction ID
        String transId = HmilyTransactionContextLocal.getInstance().get().getTransId();
        log.info("bank1 cancel begin It's starting xid():"+transId);
        //Idempotent judgment: judge local_ try_ Is there a try log record in the log table? If so, it will not be executed again
        if (accountInfoDao.isExistCancel(transId) > 0){
            log.info("bank1 cancel Idempotent processing cancel It has been executed, and there is no need to repeat it, xid():"+transId);
            return ;
        }
        //Empty rollback: if try is not executed, cancel is not allowed
        if (accountInfoDao.isExistTry(transId) <= 0){
            log.info("bank1 Empty rollback processing try Not implemented, not allowed cancel Execution, xid():"+transId);
            return ;
        }
        //Increase available balance
        accountInfoDao.addAccountBalance(accountNo, amount);

        //Insert cancel execution record
        accountInfoDao.addCancel(transId);
    }
}

controller

package com.yao.hmily.controller;

import com.yao.hmily.domain.CommonResult;
import com.yao.hmily.service.AccountInfoService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class AccountInfoController {

    @Autowired
    private AccountInfoService accountInfoService;

    //Zhang San transfer
    @GetMapping("/transfer/{amount}")
    public CommonResult transfer(@PathVariable("amount") Double amount){
        accountInfoService.updateAccountBalance("1",amount);
        return new CommonResult(200,"Transfer succeeded");
    }
}

Startup class

package com.yao.hmily;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.openfeign.EnableFeignClients;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.EnableAspectJAutoProxy;

@SpringBootApplication
@EnableDiscoveryClient
@EnableFeignClients
@EnableAspectJAutoProxy
@ComponentScan(value = {"org.dromara.hmily","com.yao.hmily"})
public class Bank1Application2006 {
    public static void main(String[] args) {
        SpringApplication.run(Bank1Application2006.class,args);
    }
}

1.3.5 create tcc-bank2-server