|
| 1 | +--- |
| 2 | +title: Upgrade TiDB Using TiDB-Ansible |
| 3 | +category: operations |
| 4 | +--- |
| 5 | + |
| 6 | +# Upgrade TiDB Using TiDB-Ansible |
| 7 | + |
| 8 | +When you perform a rolling update for a TiDB cluster, the service is shut down serially and is started after you update the service binary and the configuration file. If the load balancing is configured in the front-end, the rolling update of TiDB does not impact the running applications. Minimum requirements: `pd*3, tidb*2, tikv*3`. |
| 9 | + |
| 10 | +> **Note:** If the binlog is enabled, and Pump and Drainer services are deployed in the TiDB cluster, stop the Drainer service before the rolling update. The Pump service is automatically updated in the rolling update of TiDB. |
| 11 | +
|
| 12 | +## Upgrade the component version |
| 13 | + |
| 14 | +To upgrade between large versions, you need to upgrade [`tidb-ansible`](https://github.com/pingcap/tidb-ansible). If you want to upgrade the version of TiDB from 1.0 to 2.0, see [TiDB 2.0 Upgrade Guide](tidb-v2-upgrade-guide.md). |
| 15 | + |
| 16 | +### Download the binary automatically |
| 17 | + |
| 18 | +1. Edit the value of the `tidb_version` parameter in the `/home/tidb/tidb-ansible/inventory.ini` file, and specify the version number you need to upgrade to. |
| 19 | + |
| 20 | + For example, to upgrade from `v2.0.2` to `v2.0.3`: |
| 21 | + |
| 22 | + ``` |
| 23 | + tidb_version = v2.0.3 |
| 24 | + ``` |
| 25 | +
|
| 26 | +2. Delete the existing `downloads` directory `/home/tidb/tidb-ansible/downloads/`. |
| 27 | +
|
| 28 | + ``` |
| 29 | + $ cd /home/tidb/tidb-ansible |
| 30 | + $ rm -rf downloads |
| 31 | + ``` |
| 32 | +
|
| 33 | +3. Use `playbook` to download the TiDB `v2.0.3` binary and replace the existing binary in `/home/tidb/tidb-ansible/resource/bin/` with it automatically. |
| 34 | +
|
| 35 | + ``` |
| 36 | + $ ansible-playbook local_prepare.yml |
| 37 | + ``` |
| 38 | +
|
| 39 | +### Download the binary manually |
| 40 | +
|
| 41 | +You can also download the binary manually. Use `wget` to download the binary and replace the existing binary in `/home/tidb/tidb-ansible/resource/bin/` with it manually. |
| 42 | +
|
| 43 | +``` |
| 44 | +wget http://download.pingcap.org/tidb-v2.0.3-linux-amd64-unportable.tar.gz |
| 45 | +``` |
| 46 | +
|
| 47 | +> **Note:** Remember to replace the version number in the download link with the one you need. |
| 48 | +
|
| 49 | +### Perform a rolling update using Ansible |
| 50 | +
|
| 51 | +- Apply a rolling update to the PD node (only upgrade the PD service) |
| 52 | +
|
| 53 | + ``` |
| 54 | + $ ansible-playbook rolling_update.yml --tags=pd |
| 55 | + ``` |
| 56 | +
|
| 57 | + When you apply a rolling update to the PD leader instance, if the number of PD instances is not less than 3, Ansible migrates the PD leader to another node before stopping this instance. |
| 58 | +
|
| 59 | +- Apply a rolling update to the TiKV node (only upgrade the TiKV service) |
| 60 | +
|
| 61 | + ``` |
| 62 | + $ ansible-playbook rolling_update.yml --tags=tikv |
| 63 | + ``` |
| 64 | +
|
| 65 | + When you apply a rolling update to the TiKV instance, Ansible migrates the Region leader to other nodes. The concrete logic is as follows: Call the PD API to add the `evict leader scheduler` -> Inspect the `leader_count` of this TiKV instance every 10 seconds -> Wait the `leader_count` to reduce to below 10, or until the times of inspecting the `leader_count` is more than 12 -> Start closing the rolling update of TiKV after two minutes of timeout -> Delete the `evict leader scheduler` after successful start. The operations are executed serially. |
| 66 | +
|
| 67 | + If the rolling update fails in the process, log in to `pd-ctl` to execute `scheduler show` and check whether `evict-leader-scheduler` exists. If it does exist, delete it manually. Replace `{PD_IP}` and `{STORE_ID}` with your PD IP and the `store_id` of the TiKV instance: |
| 68 | +
|
| 69 | + ``` |
| 70 | + $ /home/tidb/tidb-ansible/resources/bin/pd-ctl -u "http://{PD_IP}:2379" -d scheduler show |
| 71 | + $ curl -X DELETE "http://{PD_IP}:2379/pd/api/v1/schedulers/evict-leader-scheduler-{STORE_ID}" |
| 72 | + ``` |
| 73 | +
|
| 74 | +- Apply a rolling update to the TiDB node (only upgrade the TiDB service) |
| 75 | +
|
| 76 | + If the binlog is enabled in the TiDB cluster, the Pump service is automatically upgraded in the rolling update of the TiDB service. |
| 77 | +
|
| 78 | + ``` |
| 79 | + $ ansible-playbook rolling_update.yml --tags=tidb |
| 80 | + ``` |
| 81 | +
|
| 82 | +- Apply a rolling update to all services (upgrade PD, TiKV, and TiDB in sequence) |
| 83 | +
|
| 84 | + If the binlog is enabled in the TiDB cluster, the Pump service is automatically upgraded in the rolling update of the TiDB service. |
| 85 | +
|
| 86 | + ``` |
| 87 | + $ ansible-playbook rolling_update.yml |
| 88 | + ``` |
| 89 | +
|
| 90 | +- Apply a rolling update to the monitoring component |
| 91 | +
|
| 92 | + ``` |
| 93 | + $ ansible-playbook rolling_update_monitor.yml |
| 94 | + ``` |
| 95 | +
|
| 96 | +## Modify component configuration |
| 97 | +
|
| 98 | +This section describes how to modify component configuration using Ansible. |
| 99 | +
|
| 100 | +1. Update the component configuration template. |
| 101 | +
|
| 102 | + The component configuration template of the TiDB cluster is in the `/home/tidb/tidb-ansible/conf` folder. |
| 103 | +
|
| 104 | + | Component | Template Name of the Configuration File | |
| 105 | + | :-------- | :----------: | |
| 106 | + | TiDB | tidb.yml | |
| 107 | + | TiKV | tikv.yml | |
| 108 | + | PD | pd.yml | |
| 109 | +
|
| 110 | + The comment status if the default configuration item, which uses the default value. To modify it, you need to cancel the comment by removing `#` and then modify the corresponding parameter value. |
| 111 | +
|
| 112 | + The configuration template uses the yaml format, so separate the parameter name and the parameter value using `:`, and indent two spaces. |
| 113 | +
|
| 114 | + For example, modify the value of the `high-concurrency`, `normal-concurrency` and `low-concurrency` parameters to 16 for the TiKV component: |
| 115 | +
|
| 116 | + ```bash |
| 117 | + readpool: |
| 118 | + coprocessor: |
| 119 | + # Notice: if CPU_NUM > 8, the default thread pool size for coprocessors |
| 120 | + # will be set to CPU_NUM * 0.8. |
| 121 | + high-concurrency: 16 |
| 122 | + normal-concurrency: 16 |
| 123 | + low-concurrency: 16 |
| 124 | + ``` |
| 125 | +
|
| 126 | +2. After modifying the component configuration, you need to perform a rolling update using Ansible. See [Perform a rolling update using Ansible](#perform-a-rolling-update-using-ansible). |
0 commit comments