Investigation: CI failures on antalya-25.8.16.20002 release MasterCI run

[CarlosFelipeOR]

CI Run: https://github.com/Altinity/ClickHouse/actions/runs/23344149633/attempts/1
Commit: 49bb3f7beb5e6e424a1e94c749478fd23a8e6196 (PR #1537)
Branch: antalya-25.8
Version: 25.8.16.20002.altinityantalya

Context

The MasterCI run for the antalya-25.8.16.20002 release had a high number of failures on the first execution across integration tests, stateless tests, and regression tests. However, all tests passed after rerunning, and these tests had been passing consistently in the previous MasterCI runs (merges of other PRs into this version).

The failing tests were all under sanitizer builds (TSAN, MSAN) — except for the regression tests which failed on aarch64 runners. Since the sanitizer-based tests (integration and stateless) were not executed during the PR CI run of the last merged PR (#1537), they had not been validated before merging.

Given that these tests were previously passing and only started failing on the commit that merged PR #1537, we investigated whether the changes in #1537 (S3 URI encoding) could have caused any of these failures. The conclusion is that none of the failures are related to PR #1537 — they are all caused by sanitizer overhead, infrastructure resource constraints, and known flaky test patterns. All passed on rerun.

---

Integration tests (amd_tsan, 4/6) — 14 failures

Job: https://github.com/Altinity/ClickHouse/actions/runs/23344149633/job/67908450476
Log: https://altinity-build-artifacts.s3.amazonaws.com/REFs/antalya-25.8/49bb3f7beb5e6e424a1e94c749478fd23a8e6196//integration_tests_amd_tsan_4_6/integration_run_parallel_0.log

Root Cause

test_concurrent_alter_modify[mt] (~3.3% fail rate under tsan) and [replicated] (~2.1%) are known flaky tests that failed early on worker gw2. The 8 subsequent test_multiple_disks tests running on the same worker inherited corrupted cluster state and failed as a consequence. The job also hit the overall timeout.

The remaining 4 tests:

Test	Altinity CI (1 year)	Upstream (1 year)	Notes
test_mysql57_database_engine::test_restart_server	1 fail / 5,453 runs (0.02%)	162 failures	Flaky upstream (162 failures in 1 year). Very rare in our CI — first occurrence in 1 year.
test_s3_cache_locality::test_cache_locality[30000]	5 fails / 1,827 runs (0.27%)	0 failures (different parametrization upstream)	Rare flaky, only in our CI. Code analysis confirmed PR #1537 does not affect the code path used by this test.
test_storage_rabbitmq::test_rabbitmq_restore_failed_connection_without_losses_2	12 fails / 3,916 runs (0.31%)	1,031 failures	Heavily flaky upstream (1,031 failures in 1 year). Timing-sensitive RabbitMQ reconnection test.
test_storage_s3_queue::test_failure_in_the_middle	28 fails / 4,501 runs (0.62%)	294 failures	Flaky upstream (294 failures in 1 year). Uses a failpoint mechanism that is timing-sensitive.

Integration tests (amd_tsan, 5/6) — 8 failures

Job: https://github.com/Altinity/ClickHouse/actions/runs/23344149633/job/67908450560

Root Cause

CPU scheduling and memory threshold tests that are inherently sensitive to resource contention under TSAN. All are known flaky in upstream ClickHouse:

Test	Fail rate (tsan, Altinity 90d)	Upstream (90d)
test_cpu_time_fairness[random_longer_prd]	5.29% (12/227)	34 failures
test_cpu_time_fairness[random_longer_dev]	4.89% (11/225)	35 failures
test_cpu_time_fairness[random_equal]	4.82% (11/228)	34 failures
test_independent_pools	3.10% (4/129)	22 failures
test_threads_oversubscription	3.03% (7/231)	30 failures
test_create_workload	1.73% (4/231)	31 failures
test_slot_allocation_fairness	0.43% (1/231)	25 failures
test_memory_usage	0.43% (1/231)	15 failures

The 6 test_scheduler_cpu_preemptive tests tend to fail as a group — when one fails, the others usually do too. This is not cascade from a server crash but instability of CPU scheduling assertions under TSAN overhead.

Stateless tests (amd_msan, parallel, 2/2) — ~44 failures

Job: https://github.com/Altinity/ClickHouse/actions/runs/23344149633/job/67909693935

Root Cause

The ClickHouse server crashed under MSAN (likely an intermittent race condition triggering uninitialized memory detection). All ~44 reported failures (16 BROKEN + 9 FAIL + 10 "Scraping *_log" + "Killed by signal" + "Fatal messages" + "Exception in test runner") are cascade from this single server crash.

"Killed by signal" + "Fatal messages" have occurred 16 times in the last 90 days (3.27% rate, 16/490 runs) in MSAN builds across multiple branches (antalya-25.8, releases/25.8.16, etc.). Upstream ClickHouse doesn't show MSAN crashes specifically, but has similar crashes under ASAN (11 occurrences in 90 days).

The crash passed on rerun because it depends on thread scheduling timing — the bug exists but is non-deterministic.

Regression tests — aggregate_functions_3 (aarch64)

Attempt 1: https://github.com/Altinity/ClickHouse/actions/runs/23344149633/job/67908397378
Attempt 2: https://github.com/Altinity/ClickHouse/actions/runs/23344149633/job/67961567262
Report (attempt 2): https://altinity-build-artifacts.s3.amazonaws.com/REFs/antalya-25.8/49bb3f7beb5e6e424a1e94c749478fd23a8e6196/regression/aarch64/with_analyzer/zookeeper/without_thread_fuzzer/aggregate_functions3/report.html

Root Cause

Failed on 2 consecutive attempts on different runners, always with Code: 745 SERVER_OVERLOADED — ClickHouse rejected queries because CPU wait-to-busy ratio exceeded the threshold. Only 3 merge tests failed (anyHeavyMerge, anyLast_respect_nullsMerge, covarPopMerge).

Attempt	Runner	Duration	Result
1	runner-23344149633-67908397373	~2h	FAILURE
2	runner-23350372893-67934081233	~2h27m	FAILURE

The x86_64 run of the same suite passed on first attempt. SERVER_OVERLOADED has occurred 586 times in the last 90 days across multiple regression suites and versions. Failing on 2 different runners suggests the aarch64 fleet is systematically underpowered for this suite.

Regression tests — alter_replace (aarch64)

Attempt 1 (failure): https://github.com/Altinity/ClickHouse/actions/runs/23344149633/job/67908397021
Attempt 2 (cancelled/timeout): https://github.com/Altinity/ClickHouse/actions/runs/23344149633/job/67961567329
Attempt 4 (success): https://github.com/Altinity/ClickHouse/actions/runs/23344149633/job/67990619311

Root Cause

Attempt 1 failed with Code: 999 KEEPER_EXCEPTION (Coordination::Exception: Transaction failed, No node) during ALTER TABLE ... REPLACE PARTITION. Attempt 2 ran on a different runner but got stuck for 3h30m until the job timeout killed it — no logs were saved since the "Create and upload logs" step was skipped on cancellation. Attempt 4 passed normally on a third runner.

Attempt	Runner	Duration	Result
1	standby-1774013626654157	~1h40m	FAILURE
2	standby-1774036772517454	~3h30m	CANCELLED (timeout)
4	runner-23350372893-67990096032	~1h39m	SUCCESS

Regression tests — settings (aarch64)

Job (attempt 4): https://github.com/Altinity/ClickHouse/actions/runs/23344149633/job/67996929752
Report: https://altinity-build-artifacts.s3.amazonaws.com/REFs/antalya-25.8/49bb3f7beb5e6e424a1e94c749478fd23a8e6196/regression/aarch64/with_analyzer/zookeeper/without_thread_fuzzer/settings/report.html

Root Cause

The settings/default values test compares every setting from system.settings against a stored snapshot (default_values.py.default values>=25.8_antalya.snapshot). 7 settings failed with SnapshotNotFoundError — they are new settings that exist in the binary but have no corresponding entry in the snapshot file.

These are not flaky failures — they fail deterministically on every attempt because the snapshot was not updated.

Setting	Default value	Added by PR	Merged	Notes
serialize_string_in_memory_with_zero_byte	1 (true)	PR #1298 (upstream backport #90880)	2026-01-19	Came with base update to 25.8.14
export_merge_tree_part_throw_on_pending_mutations	1 (true)	PR #1294	2026-01-26	Already present since 25.8.14
export_merge_tree_part_throw_on_pending_patch_parts	1 (true)	PR #1294	2026-01-26	Already present since 25.8.14
enable_alias_marker	1 (true)	PR #1272	2026-01-28	Already present since 25.8.14
iceberg_partition_timezone	"" (empty)	PR #1349	2026-03-17	New in 25.8.16
export_merge_tree_part_filename_pattern	"{part_name}_{checksum}"	PR #1512	2026-03-17	New in 25.8.16
allow_local_data_lakes	0 (false)	PR #1531	2026-03-18	New in 25.8.16

The first 4 settings were added in PRs merged in January (part of 25.8.14). The remaining 3 were added in March (new in 25.8.16). The snapshot has not been updated since before any of these were merged.

Fix

Update the snapshot file settings/tests/snapshots/default_values.py.default values>=25.8_antalya.snapshot to include all 7 new settings with their default values.

Proposed actions

1. Increase aarch64 runner capacity — More vCPUs/RAM. aggregate_functions_3 failed on 2 different runners with SERVER_OVERLOADED; x86_64 runners don't have this problem.

2. Reduce test parallelism on aarch64 — If increasing capacity isn't immediately possible, reduce concurrent test threads on aarch64 to lower CPU contention.

3. Automatic test retries — Add retry mechanism (e.g. --reruns 2 in pytest for integration tests) so transient failures don't block the pipeline. Prefer this over xfail — fail rates are low (0.4%-5.3%) and xfail would hide a real regression if a test starts failing consistently.

4. Investigate MSAN crash separately — The recurring MSAN server crash (3.3% rate) is likely a real bug exposed intermittently by thread scheduling. Should be investigated as a standalone issue.

5. Investigate ZooKeeper stability on aarch64 — The KEEPER_EXCEPTION and the 3h30m hang suggest ZooKeeper may struggle under load on aarch64 machines.

6. Update settings snapshots — Add the 7 missing settings to the >=25.8_antalya snapshot. The settings/default values suite has been outdated since 25.8.14 and needs to be kept in sync when new settings are added.