High availability is another major feature of TiDB. All three components of TiDB/TiKV/PD can tolerate partial instance failure without affecting the availability of the whole cluster. The following describes the availability of these three components, the consequences of a single instance failure, and how to recover.

TiDB
TiDB is stateless. It is recommended to deploy at least two instances. The front end provides services through load balancing components. When a single instance fails, the Session in progress on this instance will be affected. From the perspective of application, a single request fails. After reconnection, you can continue to get services. After a single instance fails, you can restart the instance or deploy a new instance.

PD
PD is a cluster. It maintains data consistency through the Raft protocol. When a single instance fails, if the instance is not the leader of Raft, the service will not be affected at all. If the instance is the leader of Raft, a new Raft leader will be selected and the service will be restored automatically. PD is unable to provide external services during the election process, which is about 3 seconds. It is recommended to deploy at least three PD instances. After a single instance fails, restart the instance or add a new instance.

TiKV
TiKV is a cluster, which maintains data consistency through Raft protocol (the number of copies can be configured, and three copies are saved by default), and makes load balancing scheduling through PD. When a single node fails, all regions stored on this node will be affected. For the Leader node in the Region, the service will be interrupted and waiting for re election; for the Follower node in the Region, the service will not be affected. When a TiKV node fails and cannot be recovered within a period of time (default 30 minutes), PD will migrate the data on it to other TiKV nodes.

wtidb28.add.shbt.qihoo.net  192.168.1.1  TiDB/PD/pump/prometheus/grafana/CCS
wtidb27.add.shbt.qihoo.net  192.168.1.2  TiDB
wtidb26.add.shbt.qihoo.net  192.168.1.3  TiDB
wtidb22.add.shbt.qihoo.net  192.168.1.4  TiKV
wtidb21.add.shbt.qihoo.net  192.168.1.5  TiKV
wtidb20.add.shbt.qihoo.net  192.168.1.6  TiKV
wtidb19.add.shbt.qihoo.net  192.168.1.7  TiKV
wtidb18.add.shbt.qihoo.net  192.168.1.8  TiKV
wtidb17.add.shbt.qihoo.net  192.168.1.9  TiFlash
wtidb16.add.shbt.qihoo.net  192.168.1.10  TiFlash

The cluster uses 3TiDB node, 5TiKV and 2TiFlash architecture to test disaster recovery. The way of TiFlash is to deploy the cluster first, and then deploy TiFlash, version 3.1.0GA

There are three default replicas in the cluster, any two of the five machines are down. In theory, there are three situations. One is that two of the three replicas happen to be on the down two machines. One is that only one region in the three replicas is on the down two machines. The other is that there is no replica of some content in the down two machines. This time, we test two TiKV, wtidb21 and wtidb22 Node down.

Let's take a look at the status of the pre outage test table

mysql> select count(*) from rpt_qdas_show_shoujizhushou_channelver_mix_daily;
+----------+
| count(*) |
+----------+
|  1653394 |
+----------+
1 row in set (0.91 sec)

mysql> select count(*) from rpt_qdas_show_shoujizhushou_channelver_mix_daily force index (idx_day_ver_ch);
+----------+
| count(*) |
+----------+
|  1653394 |
+----------+
1 row in set (0.98 sec)

After two downtime at the same time:

mysql> select count(*) from rpt_qdas_show_shoujizhushou_channelver_mix_daily;               
ERROR 9002 (HY000): TiKV server timeout 
mysql> select count(*) from rpt_qdas_show_shoujizhushou_channelver_mix_daily force index (idx_day_ver_ch);
ERROR 9005 (HY000): Region is unavailable

Take a look at the two down stores_ ID

/data1/tidb-ansible-3.1.0/resources/bin/pd-ctl -i -u http://192.168.1.1:2379
» store

Know it's 1 and 4

Check the region of more than half copies on the failed node

[tidb@wtidb28 bin]$ /data1/tidb-ansible-3.1.0/resources/bin/pd-ctl  -u http://192.168.1.1:2379  -d region --jq='.regions[] | {id: .id, peer_stores: [.peers[].store_id] | select(length as $total | map(if .==(1,4) then . else empty end) | length>=$total-length)}'
{"id":18,"peer_stores":[4,6,1]}
{"id":405,"peer_stores":[7,4,1]}
{"id":120,"peer_stores":[4,1,6]}
{"id":337,"peer_stores":[4,5,1]}
{"id":128,"peer_stores":[4,1,6]}
{"id":112,"peer_stores":[1,4,6]}
{"id":22,"peer_stores":[4,6,1]}
{"id":222,"peer_stores":[7,4,1]}
{"id":571,"peer_stores":[4,6,1]}

Execute the script to stop kv on the remaining normal kv nodes:

ps -ef|grep tikv
sh /data1/tidb/deploy/scripts/stop_tikv.sh
ps -ef|grep tikv

Change its owner and copy it to the tidb directory
chown -R tidb. /home/helei/tikv-ctl

In this case, If there are too many regions, it will be slow and cumbersome to repair according to the region. It's convenient to use all, but it's possible to kill two copies of peer by mistake, that is to say, the machine you broke may have only one region on this machine, but it will also keep only one copy of region in the cluster
The following operation is to perform stop kv operation (kv is required to be closed) on all surviving nodes, and then execute

[tidb@wtidb20 tidb]$ ./tikv-ctl --db /data1/tidb/deploy/data/db unsafe-recover remove-fail-stores -s 1,4 --all-regions                                                                    
removing stores [1, 4] from configrations...
success

Restart pd node

ansible-playbook stop.yml --tags=pd
If the pd is off here, you can't board the library
[helei@db-admin01 ~]$ /usr/local/mysql56/bin/mysql -u xxxx -h xxxx -P xxx -pxxxx
xxxx...
...
...
ansible-playbook start.yml --tags=pd

Restart the surviving kv node
sh /data1/tidb/deploy/scripts/start_tikv.sh

Check no region in leader state
[tidb@wtidb28 bin]$ /data1/tidb-ansible-3.1.0/resources/bin/pd-ctl -u http://192.168.1.1:2379 -d region --jq '.regions[]|select(has("leader")|not)|{id: .id,peer_stores: [.peers[].store_id]}'
I haven't found any non leader region s here, only the number of copies is 3, and there are more than 3 machines hanging at the same time, and it happens that all regions of some regions are on these 3 machines. The previous step is unsafe All region and pd are not displayed in the region step in the leader state. Only when the data corresponding to the table query is lost, can empty regions be created. In my case, as long as I keep a copy, no matter whether unsafe executes all regions or specifies a specific region number, the following steps are not needed

After the cluster is started normally, PD CTL can be used to view the previous regions. Theoretically, after using unsafe -- all regions, only one of the remaining regions becomes the leader, and the remaining kv nodes copy two follower s to other store s through the raft protocol
For example, in this case
{"id":18,"peer_stores":[4,6,1]}
Through PD CTL, we can see that he is now hesitating about the damage of 1,4 kV nodes. After executing unsafe recover remove fail stores -- all regions, he will remove 1,4, and the remaining 6 will become leader s. By using the raft protocol, he will replicate a new follower in 5,7 nodes, and achieve 3 replicas to start the cluster successfully

» region 18
{
  "id": 18,
  "start_key": "7480000000000000FF0700000000000000F8",
  "end_key": "7480000000000000FF0900000000000000F8",
  "epoch": {
    "conf_ver": 60,
    "version": 4
  },
  "peers": [
    {
      "id": 717,
      "store_id": 6
    },
    {
      "id": 59803,
      "store_id": 7
    },
    {
      "id": 62001,
      "store_id": 5
    }
  ],
  "leader": {
    "id": 717,
    "store_id": 6
  },
  "written_bytes": 0,
  "read_bytes": 0,
  "written_keys": 0,
  "read_keys": 0,
  "approximate_size": 1,
  "approximate_keys": 0
}

If only 2 machines are hung at the same time, it will be over here. If only 1 machine is hung, there is no need to deal with it

First look at the data. Now it's OK. The previous steps are recovering smoothly

mysql> select count(*) from rpt_qdas_show_shoujizhushou_channelver_mix_daily;
+----------+
| count(*) |
+----------+
|  1653394 |
+----------+
1 row in set (0.86 sec)

mysql> select count(*) from rpt_qdas_show_shoujizhushou_channelver_mix_daily force index (idx_day_ver_ch);
+----------+
| count(*) |
+----------+
|  1653394 |
+----------+
1 row in set (0.98 sec)

Here's an episode
When I restore the nodes that are down in 1 and 4, no new data has been written to the cluster during this period. Originally, 6 was used as the leader and the newly generated 5 and 7 were used as the follower. After the recovery, 5 and 7 were removed and 1 and 4 were used as the follower again. region 18 is still the store of 1, 4 and 6_ id.

If three or more devices are hung at the same time, the above non leader step check will have content
We let the following three down this time:

wtidb22.add.shbt.qihoo.net  192.168.1.4  TiKV
wtidb21.add.shbt.qihoo.net  192.168.1.5  TiKV
wtidb20.add.shbt.qihoo.net  192.168.1.6  TiKV

First, stop all normal tikv. This case is wtidb19,wtidb18

Take a look at the two down stores_ ID

/data1/tidb-ansible-3.1.0/resources/bin/pd-ctl -i -u http://192.168.1.1:2379
» store

Know it's 1, 4, 5

Check the region of more than half copies on the failed node

[tidb@wtidb28 bin]$  /data1/tidb-ansible-3.1.0/resources/bin/pd-ctl  -u http://192.168.1.1:2379  -d region --jq='.regions[] | {id: .id, peer_stores: [.peers[].store_id] | select(length as $total | map(if .==(1,4,5) then . else empty end) | length>=$total-length)}'
{"id":156,"peer_stores":[1,4,6]}
{"id":14,"peer_stores":[6,1,4]}
{"id":89,"peer_stores":[5,4,1]}
{"id":144,"peer_stores":[1,4,6]}
{"id":148,"peer_stores":[6,1,4]}
{"id":152,"peer_stores":[7,1,4]}
{"id":260,"peer_stores":[6,1,4]}
{"id":480,"peer_stores":[7,1,4]}
{"id":132,"peer_stores":[5,4,6]}
{"id":22,"peer_stores":[6,1,4]}
{"id":27,"peer_stores":[4,1,6]}
{"id":37,"peer_stores":[1,4,6]}
{"id":42,"peer_stores":[5,4,6]}
{"id":77,"peer_stores":[5,4,6]}
{"id":116,"peer_stores":[5,4,6]}
{"id":222,"peer_stores":[6,1,4]}
{"id":69,"peer_stores":[5,4,6]}
{"id":73,"peer_stores":[7,4,1]}
{"id":81,"peer_stores":[5,4,1]}
{"id":128,"peer_stores":[6,1,4]}
{"id":2,"peer_stores":[5,6,4]}
{"id":10,"peer_stores":[7,4,1]}
{"id":18,"peer_stores":[6,1,4]}
{"id":571,"peer_stores":[6,5,4]}
{"id":618,"peer_stores":[7,1,4]}
{"id":218,"peer_stores":[6,5,1]}
{"id":47,"peer_stores":[1,4,6]}
{"id":52,"peer_stores":[6,1,4]}
{"id":57,"peer_stores":[4,7,1]}
{"id":120,"peer_stores":[6,1,4]}
{"id":179,"peer_stores":[5,1,4]}
{"id":460,"peer_stores":[5,7,1]}
{"id":93,"peer_stores":[6,1,4]}
{"id":112,"peer_stores":[6,1,4]}
{"id":337,"peer_stores":[5,6,4]}
{"id":400,"peer_stores":[5,7,1]}

Now there are two left

wtidb19.add.shbt.qihoo.net  192.168.1.7  TiKV
wtidb18.add.shbt.qihoo.net  192.168.1.8  TiKV

The following operation needs to be performed at all surviving sections (wtidb19 and wtidb18 in this case) first (kv is required to be closed), and then

[tidb@wtidb19 tidb]$ ./tikv-ctl --db /data1/tidb/deploy/data/db unsafe-recover remove-fail-stores -s 1,4,5 --all-regions                                                                 
removing stores [1, 4, 5] from configrations...
success

Restart pd node

ansible-playbook stop.yml --tags=pd
ansible-playbook start.yml --tags=pd

Restart the surviving kv node
sh /data1/tidb/deploy/scripts/start_tikv.sh

Check the regions that are not in the leader state. As you can see here, 1, 4 and 5 because all regions are on the damaged three machines, the data cannot be recovered after these regions are discarded

[tidb@wtidb28 tidb-ansible-3.1.0]$ /data1/tidb-ansible-3.1.0/resources/bin/pd-ctl  -u http://192.168.1.1:2379 -d region --jq '.regions[]|select(has("leader")|not)|{id: .id,peer_stores: [.peers[].store_id]}'
{"id":179,"peer_stores":[5,1,4]}
{"id":81,"peer_stores":[5,4,1]}
{"id":89,"peer_stores":[5,4,1]}

Confirm which table region belongs to according to region ID

[tidb@wtidb28 tidb-ansible-3.1.0]$ curl http://192.168.1.1:10080/regions/179
{
 "region_id": 179,
 "start_key": "dIAAAAAAAAA7X2mAAAAAAAAAAwOAAAAAATQXJwE4LjQuMAAAAPwBYWxsAAAAAAD6A4AAAAAAAqs0",
 "end_key": "dIAAAAAAAAA7X3KAAAAAAAODBA==",
 "frames": [
  {
   "db_name": "hl",
   "table_name": "rpt_qdas_show_shoujizhushou_channelver_mix_daily(p201910)",
   "table_id": 59,
   "is_record": false,
   "index_name": "key2",
   "index_id": 3,
   "index_values": [
    "20191015",
    "8.4.0",
    "all",
    "174900"
   ]
  },
  {
   "db_name": "hl",
   "table_name": "rpt_qdas_show_shoujizhushou_channelver_mix_daily(p201910)",
   "table_id": 59,
   "is_record": true,
   "record_id": 230148
  }
 ]
}

If you want to see the cluster status at this time,

» store
{
  "count": 5,
  "stores": [
    {
      "store": {
        "id": 1,
        "address": "192.168.1.4:20160",
        "version": "3.1.0",
        "state_name": "Down"
      },
      "status": {
        "leader_weight": 1,
        "region_count": 3,
        "region_weight": 1,
        "start_ts": "1970-01-01T08:00:00+08:00"
      }
    },
    {
      "store": {
        "id": 4,
        "address": "192.168.1.5:20160",
        "version": "3.1.0",
        "state_name": "Down"
      },
      "status": {
        "leader_weight": 1,
        "region_count": 3,
        "region_weight": 1,
        "start_ts": "1970-01-01T08:00:00+08:00"
      }
    },
    {
      "store": {
        "id": 5,
        "address": "192.168.1.6:20160",
        "version": "3.1.0",
        "state_name": "Down"

No data for monitoring

Library queries are still blocked

mysql> use hl
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select count(*) from rpt_qdas_show_shoujizhushou_channelver_mix_daily;

Create an empty region to solve the unavailable state. This command requires pd and kv to be in the closed state
Here, you must write - r one by one, or report an error:

[tidb@wtidb19 tidb]$ ./tikv-ctl --db /data1/tidb/deploy/data/db recreate-region -p 192.168.1.1:2379 -r 89,179,81
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ParseIntError { kind: InvalidDigit }', src/libcore/result.rs:1188:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
Aborted

./tikv-ctl --db /data1/tidb/deploy/data/db recreate-region -p '192.168.1.1:2379' -r 89
./tikv-ctl --db /data1/tidb/deploy/data/db recreate-region -p '192.168.1.1:2379' -r 81
./tikv-ctl --db /data1/tidb/deploy/data/db recreate-region -p '192.168.1.1:2379' -r 179

After starting pd and tikv, run again
[tidb@wtidb28 tidb-ansible-3.1.0]$ /data1/tidb-ansible-3.1.0/resources/bin/pd-ctl -u http://192.168.1.1:2379 -d region --jq '.regions[]|select(has("leader")|not)|{id: .id,peer_stores: [.peers[].store_id]}'
No results were expected. Zhengzhou test tube baby hospital: http://yyk.39.net/hospital/fc964_detail.html

You can see that the data is lost by querying again, because we have lost several region s (81, 89179)

mysql> select count(*) from rpt_qdas_show_shoujizhushou_channelver_mix_daily;
+----------+
| count(*) |
+----------+
|  1262523 |
+----------+
1 row in set (0.92 sec)

You can see that the index data is no longer in region (81, 89179), so it is the same as before

mysql> select count(*) from rpt_qdas_show_shoujizhushou_channelver_mix_daily force index (idx_day_ver_ch);
+----------+
| count(*) |
+----------+
|  1653394 |
+----------+
1 row in set (1.01 sec)

At this point, the test is complete

After reading this article, I'm sure you won't recover the data of virtual TiDB's multi-point power failure. Under normal circumstances, very few of them have multiple machines down at the same time in the cluster. If only one machine is down, it will not affect the operation of the cluster, and it will automatically handle it. When a TiKV node fails and cannot be recovered within a period of time (default 30 minutes), PD The data on it will be migrated to other TiKV nodes. But if there are two or even three or more down at the same time, then you who read this article believe that you will not be confused! ~

Zhengzhou gynecological hospital: http://jbk.39.net/yiyuanfengcai/yyjs_sysdfkyy/