`mysqlbinlog` not found during incremental backup in v19.x and v20.x

[frouioui]

Overview of the Issue

We deprecated the vitess/lite image for the mysqld container in v19.0.0. However, we later realised that the official mysql image lacks the mysqlbinlog binary, leading to failures when running incremental backup with the mysqld container on an official or unofficial MySQL image. After attempting to solve the issue, in v20.0.0, the decision was made to retain the mysqld binary in the vitess/lite image while we look for a stronger alternative, but the following steps were overlooked:

• Mentioning the mysqlbinlog issue in the v19.0.0 release notes.
• Updating the getting started guide (v19 and v20) to use vitess/lite for the mysqld container.

Warning

As of now, for users employing an official or unofficial MySQL images for the mysqld container and wanting to use incremental backups, it is recommended to use vitess/lite:v20.0.0 or another appropriate tag.

Reproduction Steps

Reproduce the error

1. Follow the guide https://vitess.io/docs/20.0/get-started/operator/
2. Follow the guide https://vitess.io/docs/20.0/user-guides/migration/move-tables/
3. Follow the guide https://vitess.io/docs/20.0/user-guides/configuration-advanced/resharding/
4. Change the 306_down_shard_0.yaml file to include:
   • A higher memory limit/reserved for vttablet (1024Mi is good)
   • The backup location as shown below:

spec:
  backup:
    engine: xtrabackup
    locations:
      - volume:
          hostPath:
            path: /tmp
            type: Directory

5. Take an initial backup: vtctldclient BackupShard commerce/-
6. Insert more rows:

$ mysql
Your MySQL connection id is 52
Server version: 8.0.30-Vitess Version: 20.0.0 (Git revision 8aeb274f6c8eeb1d0e80e17ab6b0ab10a01f60dc branch 'HEAD') built on Thu Jun 27 05:04:18 UTC 2024 by vitess@buildkitsandbox using go1.22.4 linux/amd64

Copyright (c) 2000, 2024, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> use commerce;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> insert into product(sku, description, price) values ("test", "toto", 10);
Query OK, 1 row affected (0.02 sec)

mysql> exit 

7. Take an incremental backup: vtctldclient BackupShard --incremental-from-pos=auto commerce/-
8. See the output:

commerce/- (zone1-2469782763): time:{seconds:1719505831 nanoseconds:910131961} file:"backup.go" line:152 value:"Starting backup 2024-06-27.163031.zone1-2469782763"
commerce/- (zone1-2469782763): time:{seconds:1719505831 nanoseconds:911110503} file:"builtinbackupengine.go" line:215 value:"Executing Backup at 2024-06-27 16:30:31.908396961 +0000 UTC m=+23.944406844 for keyspace/shard commerce/- on tablet zone1-2469782763, concurrency: 4, compress: true, incrementalFromPos: auto"
commerce/- (zone1-2469782763): time:{seconds:1719505831 nanoseconds:912518795} file:"builtinbackupengine.go" line:255 value:"auto evaluating incremental_from_pos"
commerce/- (zone1-2469782763): time:{seconds:1719505831 nanoseconds:915694545} file:"builtinbackupengine.go" line:262 value:"auto evaluated incremental_from_pos: MySQL56/9fa187c3-34a0-11ef-9a9f-6e3944149d19:1-41,a04f553d-34a0-11ef-9a7b-b201b4fa8fbc:1-31"
commerce/- (zone1-2469782763): time:{seconds:1719505831 nanoseconds:951089878} level:ERROR file:"backup.go" line:178 value:"backup is not usable, aborting it: [rpc error: code = Unknown desc = mysqlbinlog not found in any of /usr/{sbin,bin,libexec,scripts}\nreading timestamps from binlog files [vt-2469782763-bin.000004 vt-2469782763-bin.000005]]"
E0627 10:30:31.954715   27825 main.go:56] rpc error: code = Unknown desc = TabletManager.Backup on zone1-2469782763: reading timestamps from binlog files [vt-2469782763-bin.000004 vt-2469782763-bin.000005]: rpc error: code = Unknown desc = mysqlbinlog not found in any of /usr/{sbin,bin,libexec,scripts}

---

Reproduce the normal behavior

If you start again, and on step four you also modify the mysql80Compatible field to use the vitess/lite:v20.0.0 image, continue with the other steps, the last step will succeed.

Binary Version

Vitess `>= v19.0.0`, and `>= 19.0` documentation.

Operating System and Environment details

Docker

Log Fragments

No response

[frouioui]

The issue has been closed, a warning has been added to the v19.0.0 release notes and the K8S operator getting started guide examples have been edited to use the vitess/lite Docker image for the MySQL container.

[markharding]

It's possible to use the mysql:8.0.XX-debian images instead of vitess/lite.

(Found via docker-library/mysql#847)