TPCC latency regression: stable-25-1 with EnableLocalDBBtreeIndex-false

[naspirato]

Regression in 2b1cfa7
@kunga, @kruall, what we shoud do next?

• revert and merge in main Revert "Make DEFAULT_EVENTS_PER_MAILBOX unreachable (#14843)" #15488
• revert and merge in stable-25-1-1 Revert "Make DEFAULT_EVENTS_PER_MAILBOX unreachable (#14843)" to stable 25-1-1 #15489
• revert and merge in stable 25-1 Revert "Make DEFAULT_EVENTS_PER_MAILBOX unreachable (#14843)" to stable 25-1 #15509

Result of TPCC in revert vs commit

2b1cfa7 => in my repo 2b1cfa7
🔴 perf 4 4h 90p=508 ms https://nda.ya.ru/t/wYxtayFC7CbYAy

revert of 2b1cfa7 in my repo 926459d
🟢 perf 4 4h 90p=112ms https://nda.ya.ru/t/tAN3Uj3r7CbYDr

---

branch	timestamp	git_commit_timestamp	git_sha	tpmC	newOrderLatency90	reason
stable-25-1	27.02.2025 21:52:14	27.02.2025 13:58:56	d0aa06b	🔴203449	🔴508	?
stable-25-1	27.02.2025 06:40:48	26.02.2025 14:12:05	79eee97	🟢204346	🔴220	?
stable-25-1	26.02.2025 23:22:23	26.02.2025 14:12:05	79eee97	🔴203304	🔴508	?
stable-25-1	26.02.2025 06:42:07	25.02.2025 12:56:37	abfc4e2	🔴203181	🔴750	?
stable-25-1	25.02.2025 14:22:19	25.02.2025 09:09:54	09bfb76	🔴202196	🔴1000	?
stable-25-1	24.02.2025 22:47:40	24.02.2025 12:55:29	68d3b5a	🟢204260	🔴208	?
stable-25-1	24.02.2025 08:01:55	24.02.2025 02:51:21	f510f6f	🟢204570	🟠136	?
stable-25-1	24.02.2025 00:13:12	23.02.2025 15:24:23	6e0cee5	🟢204057	🔴 460	?
stable-25-1	23.02.2025 09:31:09	21.02.2025 18:00:21	e9a49fa	🟢204650	🟢 112	🟢ok
stable-25-1	22.02.2025 02:02:56	21.02.2025 18:00:21	e9a49fa	🟢204883	🟢 100	🟢ok

[naspirato]

run_type	run_start_datetime	git_sha	git_commit_timestamp	tpmC	latency 90p
EnableLocalDBBtreeIndex-true	28.02.2025 17:59:13	34500f4	28.02.2025 12:08:06	🟢204603	🟢124
EnableLocalDBBtreeIndex-false	01.03.2025 15:58:54	14dd0b0	28.02.2025 09:37:34	🔴202899	🔴750

good EnableLocalDBBtreeIndex-true

monium https://nda.ya.ru/t/PV86NBsg7CNssb

bad EnableLocalDBBtreeIndex-false

monium https://nda.ya.ru/t/f4rv8W797CNstQ

[kunga]

Perhaps the bug was added somewhere around #12198 / #8447

[naspirato]

21.02.2025 12:58:40 4ca0181 - 🔴
https://nda.ya.ru/t/qi447Cqt7CTSH5

68ccd29 - Fri Feb 21 08:33:08 2025 +0100 changes for FSTEC certification (#14878) Alexey Efimov 🔴
https://nda.ya.ru/t/PXRCaUQB7CV8aN

---

perf4 на 4 часа

a8fbe60 - Fri Feb 21 10:03:48 2025 +0300 Identify node_type in actor system's auto config (#14799) kruall 🔴
🔴perf3 p90 = 508ms -> revert нельзя сделать
https://nda.ya.ru/t/HuNTN-Lj7CWYvH

3fb5075 - Fri Feb 21 07:49:03 2025 +0100 correctly handle inf values in streaming query (#14875) Alexey Efimov 🔴
🟠 perf3 p90 = 208ms https://nda.ya.ru/t/lXFKmk-Z7CYGzn
🔴perf4 4h p90= 508ms https://nda.ya.ru/t/kiV-w42B7CYGws
perf3 3h revert in progress

9659d35 - Fri Feb 21 00:51:46 2025 +0000 Import libraries 250221-0050 Alexander Smirnov 🔴
🔴perf4 4h p90= 508ms https://nda.ya.ru/t/G5WGrsv_7CYWUB

7c09b27 - Fri Feb 21 00:51:45 2025 +0000 Merge branch 'rightlib' into merge-libs-250221-0050 Alexander Smirnov

---

perf5 на 4 часа

555fbb8 - Thu Feb 20 23:01:41 2025 +0300 Fix distconf configuration fetch machinery (#14864) Alexander Rutkovsky 🟢
🟢 perf5 4h p90=90ms https://nda.ya.ru/t/NSeqv98d7CYHk6

23aa667 - Thu Feb 20 22:35:38 2025 +0300 BATCH UPDATE/DELETE queries for per row updates with NoTx (#13340) Daniil Timizhev 🟠
🟠 perf3 p90 = 184ms https://nda.ya.ru/t/bqRm-17q7CVcUv ;
🟠 perf5 p90=124ms https://nda.ya.ru/t/_HbiQuXo7CZ6om

2b1cfa7 - Thu Feb 20 21:07:12 2025 +0300 Make DEFAULT_EVENTS_PER_MAILBOX unreachable (#14843) kruall 🟢

• 1ab62a5 - Thu Feb 20 20:47:05 2025 +0300 Relax some DS proxy response integrity checks (Relax some DS proxy response integrity checks #14856) Alexander Rutkovsky
• bd6334d - Thu Feb 20 20:13:28 2025 +0300 Improve NW and distconf HTML monitoring (Improve NW and distconf HTML monitoring #14854) Alexander Rutkovsky
• 90a6e27 - Thu Feb 20 19:02:25 2025 +0200 Rename export_s3_buffer_raw.cpp to export_s3_buffer.cpp (Rename export_s3_buffer_raw.cpp to export_s3_buffer.cpp #14853) Vasily Gerasimov

---

91bc1bd - Thu Feb 20 19:49:35 2025 +0300 EvWrite: Fix status (UNAVAILABLE -> UNDETERMINED) (#14839) Nikita Vasilev perf5 🟢 perf5 p90 = 90ms dash
https://nda.ya.ru/t/AhKxGx7o7CX3MN

3ee5135 - Thu Feb 20 17:44:51 2025 +0500 YQ-3561 add features for FQ-run (#14826) Pisarenko Grigoriy 🟢
https://nda.ya.ru/t/3uhx8wDW7CUmxo

85359f5 - Wed Feb 19 10:29:14 2025 +0300 store all versions of TTL in memory (#14677) Semyon 🟢

31aa280 - Mon Feb 17 17:56:07 2025 +0300 Fix bug with permission in validate_pr_description (#14667) Natasha Pirogova

92e49da - Fri Feb 14 23:11:42 2025 +0300 take ownership for ut/olap (#14595) zverevgeny 🟢 https://nda.ya.ru/t/A9yAZzgs7CU6u2

10.02.2025 00:51:35 a6ef48e - 🟢
https://nda.ya.ru/t/wYikAf687CTSEv

git log --graph --oneline --pretty=format:'%H - %ad %s %an' --first-parent a6ef48e773c792ab7bb05bc137a014844d71728a\^..4ca01816efdd3d939ebf7e336ba0f6637d286216
git log --graph --oneline --pretty=format:'%H - %ad %s %an' --first-parent 555fbb8^..9659d35
https://nda.ya.ru/t/OsshCqF-7CTRCY

[naspirato]

https://datalens.yandex.cloud/2ohtijbghz3cq-tpc-c-debug?state=ebbe01ea708

git log --graph --oneline --pretty=format:'%H - %ad %s %an' --first-parent  555fbb84960c627348a33fe3d0594eec72d6216f\^..9659d35bdde68d6f5c9fda24f8249509a9ea2e60

• 9659d35 - Fri Feb 21 00:51:46 2025 +0000 Import libraries 250221-0050 Alexander Smirnov 🔴
  🔴 perf4 4h p90= 508ms https://nda.ya.ru/t/G5WGrsv_7CYWUB
• 7c09b27 - Fri Feb 21 00:51:45 2025 +0000 Merge branch 'rightlib' into merge-libs-250221-0050 Alexander Smirnov 🔴
  🔴 perf 4 4h p90=750ms https://nda.ya.ru/t/FNdSJjMH7CYoNU
  |\
  | * f497aea - Thu Feb 20 21:37:55 2025 +0300 YQL-18303: Introduce Shift* overloads commit_hash:e9d589d43101dde8ca86955627fca9dbdaba1f7b imunkin
  | * ab90987 - Thu Feb 20 21:24:48 2025 +0300 [CBO] Added interesting orderings framework commit_hash:9ee882d80e40a0290a6ff9680dc5e7e3b3572492 pudge1000-7
  | * 2514a84 - Thu Feb 20 21:03:03 2025 +0300 , YT-24294: Fix retry of 503 http code error without X-Yt-Error commit_hash:a79152834c521c5dfea79b8a0b1af221078fb108 hiddenpath
  | * 26aea1d - Thu Feb 20 20:58:13 2025 +0300 Fix reimport after rXXXXXX commit_hash:5831a7282e9076bcdaed69e18fd886e648edf8f7 thegeorg
  | * d8dc9ab - Thu Feb 20 19:43:33 2025 +0300 Intermediate changes commit_hash:b18bdc10dcaaa4389f85fd7ae00556230afb4c60 robot-piglet
  | * 9fbc31d - Thu Feb 20 19:26:53 2025 +0300 Intermediate changes commit_hash:fd4a4d89253902f12075985df26fcf884c107bcf robot-piglet
  | * f7be692 - Thu Feb 20 18:32:48 2025 +0300 Drop internal v0 deps commit_hash:d5e60311a1b50570d1c9eb4a7a1620e9881c1a4e vvvv
  | * 85a37e6 - Thu Feb 20 17:52:55 2025 +0300 YQL-19588 early scan for error types commit_hash:65fc8d62e0b7fa491747980a3a40e14b4742d1bc vvvv
  | * a9b3865 - Thu Feb 20 15:37:07 2025 +0300 YT-3129 fix PR issues commit_hash:dd4f0684a1d1d35ab687fbea4049be0f25ae412b lucius
  | * c66255b - Thu Feb 20 15:08:50 2025 +0300 Added bloom filter preallocation in GJ mfilitov
  | * eaa19c6 - Thu Feb 20 15:08:47 2025 +0300 Intermediate changes commit_hash:09e2f46a2edbdcbcbac13a14a6e47d96fc88f66e robot-piglet
  | * 6515853 - Thu Feb 20 14:16:58 2025 +0300 Intermediate changes commit_hash:dbf7495ed55cb43d01c1e3ced535d1569d76cd2a robot-piglet
  | * 29b8273 - Thu Feb 20 12:15:12 2025 +0300 Enable yt.PartitionByConstantKeysViaMap by default commit_hash:ba5d894edbb1656c03aa52db29fb68f056212bdc aneporada
  | * 4916a4f - Thu Feb 20 10:16:57 2025 +0300 refactor out sanitizer plugin commit_hash:d0bd11c053e40d5117172fd59c19b6d6caf6fb8e pg
  | * d5474ec - Thu Feb 20 09:54:53 2025 +0300 Add COPY CMD in TS_FILES only for TS_PACKAGE vturov
  | * 257473b - Thu Feb 20 08:47:17 2025 +0300 fix(conf): forbid TS_FILES go to parent dir commit_hash:00bb5c9796cc72f4cce9de385fccbba84758891d zaverden
  | * 26d53e5 - Thu Feb 20 06:12:15 2025 +0300 Cosmetics commit_hash:489c1cab1ddfc76c10738d6302a63da1b673817c dtorilov
• 555fbb8 - Thu Feb 20 23:01:41 2025 +0300 Fix distconf configuration fetch machinery (Fix distconf configuration fetch machinery #14864) Alexander Rutkovsky 🟢
  🟢 perf5 4h p90=90ms https://nda.ya.ru/t/NSeqv98d7CYHk6
  🔴 perf4 4h p90=508 https://nda.ya.ru/t/m9Xn4lBy7CZ8BC
  🔴 perf3 4h p90= 750 https://nda.ya.ru/t/Q9VLYe4d7CaD6h
• 23aa667 - Thu Feb 20 22:35:38 2025 +0300 BATCH UPDATE/DELETE queries for per row updates with NoTx (BATCH UPDATE/DELETE queries for per row updates with NoTx #13340) Daniil Timizhev
  🟠 perf3 p90 = 184ms https://nda.ya.ru/t/bqRm-17q7CVcUv ;
  🟠 perf5 p90=124ms https://nda.ya.ru/t/_HbiQuXo7CZ6om
  🔴 perf3 4h p90=750ms https://nda.ya.ru/t/QIltKkuE7CaD2X
• 2b1cfa7 - Thu Feb 20 21:07:12 2025 +0300 Make DEFAULT_EVENTS_PER_MAILBOX unreachable (Make DEFAULT_EVENTS_PER_MAILBOX unreachable #14843) kruall
  🔴 perf3 4h p90= 750 https://nda.ya.ru/t/hXbKqgNC7CZp6k
• 1ab62a5 - Thu Feb 20 20:47:05 2025 +0300 Relax some DS proxy response integrity checks (Relax some DS proxy response integrity checks #14856) Alexander Rutkovsky
  🟢 perf3 4h p90=136 https://nda.ya.ru/t/72dSaYWw7CcwPY
• bd6334d - Thu Feb 20 20:13:28 2025 +0300 Improve NW and distconf HTML monitoring (Improve NW and distconf HTML monitoring #14854) Alexander Rutkovsky
  ⏹ perf3 4h p90=
• 90a6e27 - Thu Feb 20 19:02:25 2025 +0200 Rename export_s3_buffer_raw.cpp to export_s3_buffer.cpp (Rename export_s3_buffer_raw.cpp to export_s3_buffer.cpp #14853) Vasily Gerasimov
  🟢 perf4 4h p90=100 https://nda.ya.ru/t/m3G9PBhg7CacRJ
• 91bc1bd - Thu Feb 20 19:49:35 2025 +0300 EvWrite: Fix status (UNAVAILABLE -> UNDETERMINED) (EvWrite: Fix status (UNAVAILABLE -> UNDETERMINED) #14839) Nikita Vasilev
  🟢 perf5 p90 = 90ms dash https://nda.ya.ru/t/AhKxGx7o7CX3MN
  🟢 perf4 4h p90=100ms https://nda.ya.ru/t/9vpJn-FQ7CaCsh

[naspirato]

git log --graph --oneline --pretty=format:'%H - %ad %s %an' fa3d243d1431ecae6b77b2de955396627d675293

• fa3d243 - Wed Dec 4 16:16:51 2024 +0100 Store sticky pages separately (Store sticky pages separately #12198) kungurtsev
• 4f34c1a - Wed Dec 4 17:17:54 2024 +0300 compaction. lc levels configuration (compaction. lc levels configuration #12273) ivanmorozov333

[naspirato]

2b1cfa7 => in my repo 2b1cfa7
🔴 perf 4 4h 90p=508 ms https://nda.ya.ru/t/wYxtayFC7CbYAy

revert of 2b1cfa7 in my repo 926459d
🟢 perf 4 4h 90p=112ms https://nda.ya.ru/t/tAN3Uj3r7CbYDr