MySQL binlog replication client for Rust. Is a clean, idomatic Rust implementation of a MySQL binlog parser, including support for the JSONB type introduced in MySQL 5.6/5.7/8.0.
Its primary purpose is handling row-based logging messages, but it has rudimentary support for older statement-based replication. It's been tested against (MySQL) 5.6 and 5.7 and 8.0.
This library seeks to be competitive with mysqlbinlog at time-to-parse a full binlog file. All interesting datatypes are serializable using Serde, so it's easy to hook into other data processing flows.
Please note the lib currently has the following limitations:
- Supports only standard auth plugins mysql_native_password and caching_sha2_password.
- Currently, the library doesn't support SSL encryption.
- Doesn't handle split packets (16MB and more).
+-- binlog： Implementation of the ability to parse binlog events
+-- binlog-Adapter： Implementation of converting binlog event data structure to neutral data output
-- log: Default binlog data log output
-- relay_log: Default relay log output for binlog data
+-- binlog_cli： CLI Client
+-- common: Basic Type Definition
+-- conf: Project default configuration file
+-- connection: Provide MySQL/PostgreSQL/MariaDB connectivity and binlog subscription capabilities
+-- doc: Documents
+-- memory: allocator
+-- raft: raft Protocol(Broker Impl)
+-- relay_log: relay logs
+-- replayer: Main
+-- rpc: rpc Protocol
+-- sink: Relay data push to broker service
+-- slave: Provide MySQL slave disguise and dump capabilities
+-- tests: test case
In order to reduce the probability of execution errors and improve the functional features, we unify the versions of the Rust toolchain and switch the following commands:
$ rustup install nightly
$ rustup toolchain list
stable-aarch64-apple-darwin (default)
nightly-aarch64-apple-darwin (override)
#$ rustup override set nightly
# Or
$ rustup default nightly
$ rustup toolchain list
stable-x86_64-pc-windows-msvc (default)
nightly-x86_64-pc-windows-msvc (active)
You can check it out in the rustup toolchain list. If not, it will be automatically downloaded.
$ cargo test
running 2 tests
test tests::bench_add_two ... ok
test tests::it_works ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Install gnuplots
Add
[dev-dependencies]
criterion = "0.3"
[[bench]]
name = "my_benchmark"
harness = false
编写测试文件($PROJECT/benches/my_benchmark.rs)
use criterion::{black_box, criterion_group, criterion_main, Criterion};
fn fibonacci(n: u64) -> u64 {
match n {
0 => 1,
1 => 1,
n => fibonacci(n-1) + fibonacci(n-2),
}
}
fn criterion_benchmark(c: &mut Criterion) {
c.bench_function("fib 20", |b| b.iter(|| fibonacci(black_box(20))));
}
criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);
$ cargo bench
test result: ok. 0 passed; 0 failed; 7 ignored; 0 measured; 0 filtered out; finished in 0.01s
Running benches\aes_bencher.rs (target\release\deps\aes_bencher-f9a9ea35aca2ab2d.exe)
Gnuplot not found, using plotters backend
add_two time: [604.03 ps 619.40 ps 636.72 ps]
change: [-2.3923% +0.4866% +3.6080%] (p = 0.76 > 0.05)
No change in performance detected.
# 显示详细日志
cargo bench --bench aes_benchmark -- --verbose
# 只运行特定测试
cargo bench -p cryptolib --bench sm4_benchmark
cargo bench --bench aes_benchmark -- -n "AES Parallel"
报告在目录 target\criterion 下查看
# cargo tree
cargo build
Windows environment compilation error。 error: failed to run custom build command for openssl-sys v0.9.102
Caused by:
process didn't exit successfully: `mysql-cdc-rs\target\debug\build\openssl-sys-94071a3d762a0669\build-script-main` (exit code: 101)
--- stdout
cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_OPENSSL_NO_VENDOR
X86_64_PC_WINDOWS_MSVC_OPENSSL_NO_VENDOR unset
cargo:rerun-if-env-changed=OPENSSL_NO_VENDOR
OPENSSL_NO_VENDOR unset
running "perl" "./Configure" "--prefix=/mysql-cdc-rs/target/debug/build/openssl-sys-5ad9f46fc53da764/out/openssl-build/install" "--openssldir=SYS$MANAGER:[OPENSSL]" "no-dso" "no-shared" "no-ssl3" "no-tests" "no-comp" "no-zlib" "no-zlib-dynamic" "--libdir=lib" "no-md2" "no-rc5" "no-weak-ssl-ciphers" "no-camellia" "no-idea" "no-seed" "no-capieng" "no-asm" "VC-WIN64A"
Error configuring OpenSSL build:
Command: "perl" "./Configure" "--prefix=/mysql-cdc-rs/target/debug/build/openssl-sys-5ad9f46fc53da764/out/openssl-build/install" "--openssldir=SYS$MANAGER:[OPENSSL]" "no-dso" "no-shared" "no-ssl3" "no-tests" "no-comp" "no-zlib" "no-zlib-dynamic" "--libdir=lib" "no-md2" "no-rc5" "no-weak-ssl-ciphers" "no-camellia" "no-idea" "no-seed" "no-capieng" "no-asm" "VC-WIN64A"
Failed to execute: program not found
Instructions for compiling and configuring using Perl programs. From https://strawberryperl.com/ Download the Windows installation package. And installation, restart the command-line terminal and build again.
strawberry-perl-5.38.2.2-64bit.msi
http://s.codealy.com/rust_us/mysql_cdc_rs/2024.02%20binlog%20cli%20view.webm
See Binlog README.md
It is a MySQL binlog file parsing library based on Rust implementation,
Pure Rust implementation, no need for MySQL server library, but can also subscribe to MySQL master server.
The goal of the project is to parse every field of the binlog event as much as possible.
Parsed events matrix:
| Hex | Event Name | Support | Tested | Noted |
|---|---|---|---|---|
| 0x00 | UNKNOWN_EVENT | support and | not tested | |
| 0x01 | START_EVENT_V3 | too old and support | not tested | |
| 0x02 | QUERY_EVENT | support | tested | |
| 0x03 | STOP_EVENT | support | not tested | |
| 0x04 | ROTATE_EVENT | support | tested | |
| 0x05 | INTVAR_EVENT | support | tested | |
| 0x06 | LOAD_EVENT | not fully support | not tested | |
| 0x07 | SLAVE_EVENT | not fully support | not tested | |
| 0x08 | CREATE_FILE_EVENT | not fully support | not tested | |
| 0x09 | APPEND_BLOCK_EVENT | not fully support | not tested | |
| 0x0a | EXEC_LOAD_EVENT | not fully support | not tested | |
| 0x0b | DELETE_FILE_EVENT | not fully support | not tested | |
| 0x0c | NEW_LOAD_EVENT | support | not tested | |
| 0x0d | RAND_EVENT | support | not tested | |
| 0x0e | USER_VAR_EVENT | support | not fully tested | |
| 0x0f | FORMAT_DESCRIPTION_EVENT | support | tested | |
| 0x10 | XID_EVENT | not fully support | tested | |
| 0x11 | BEGIN_LOAD_QUERY_EVENT | not fully support | tested | |
| 0x12 | EXECUTE_LOAD_QUERY_EVENT | not fully support | not tested | |
| 0x13 | TABLE_MAP_EVENT | support | tested | |
| 0x14 | PreGaWriteRowsEvent(v0) | not support | not tested | |
| 0x15 | PreGaUpdateRowsEvent(v0) | not support | not tested | |
| 0x16 | PreGaDeleteRowsEvent(v0) | not support | not tested | |
| 0x17 | WRITE_ROWS_EVENTv1 | support | tested | |
| 0x18 | UPDATE_ROWS_EVENTv1 | support | tested | |
| 0x19 | DELETE_ROWS_EVENTv1 | support | tested | |
| 0x1a | INCIDENT_EVENT | not fully support | not tested | |
| 0x1b | HEARTBEAT_EVENT | not fully support | not tested | |
| 0x1c | IGNORABLE_EVENT | support | not tested | |
| 0x1d | ROWS_QUERY_EVENT | not fully support | not fully tested | |
| 0x1e | WRITE_ROWS_EVENTv2 | support | not fully tested | |
| 0x1f | UPDATE_ROWS_EVENTv2 | support | not fully tested | |
| 0x20 | DELETE_ROWS_EVENTv2 | support | not fully tested | |
| 0x21 | GTID_EVENT | support | tested | |
| 0x22 | ANONYMOUS_GTID_EVENT | support | tested | |
| 0x23 | PREVIOUS_GTIDS_EVENT | support | tested | |
| 0x24 | TRANSACTION_CONTEXT_EVENT | not support | not tested | |
| 0x25 | VIEW_CHANGE_EVENT | not support | not tested | |
| 0x26 | XA_PREPARE_LOG_EVENT | not support | not tested | |
| 0x27 | PARTIAL_UPDATE_ROWS_EVENT | not support | not tested | |
| 0x28 | TRANSACTION_PAYLOAD_EVENT | not support | not tested | |
| 0x29 | HEARTBEAT_LOG_EVENT_V2 | not support | not tested |