Test TSharedPageCache::Compaction_BTreeIndex is failing

[snaury]

With this patch the test passes:

diff --git a/ydb/core/tablet_flat/test/libs/exec/runner.h b/ydb/core/tablet_flat/test/libs/exec/runner.h
index 79712ab05fc..a5bc48f9c07 100644
--- a/ydb/core/tablet_flat/test/libs/exec/runner.h
+++ b/ydb/core/tablet_flat/test/libs/exec/runner.h
@@ -34,6 +34,7 @@ namespace NFake {
             , Logger(new TLogEnv(Time, ELnLev::Info, Sink))
             , StorageGroupCount(4)
         {
+            Env.SetDispatcherRandomSeed(TInstant::ParseIso8601("2025-02-17T01:00"), 0);
             if (auto logl = Logger->Log(ELnLev::Info)) {
                 logl << "Born at "<< TInstant::Now();
             }

With this patch the test fails:

diff --git a/ydb/core/tablet_flat/test/libs/exec/runner.h b/ydb/core/tablet_flat/test/libs/exec/runner.h
index 79712ab05fc..a5bc48f9c07 100644
--- a/ydb/core/tablet_flat/test/libs/exec/runner.h
+++ b/ydb/core/tablet_flat/test/libs/exec/runner.h
@@ -34,6 +34,7 @@ namespace NFake {
             , Logger(new TLogEnv(Time, ELnLev::Info, Sink))
             , StorageGroupCount(4)
         {
+            Env.SetDispatcherRandomSeed(TInstant::ParseIso8601("2025-02-18T01:00"), 0);
             if (auto logl = Logger->Log(ELnLev::Info)) {
                 logl << "Born at "<< TInstant::Now();
             }

When the test passes the sequence of events is as follows:

00000.015 DD| TABLET_EXECUTOR: TCompactionLogic PrepareForceCompaction for 1 table 101, mode Full, forced state None, forced mode Full
00000.015 DD| TABLET_EXECUTOR: TGenCompactionStrategy PrepareCompaction for 1: task 1, edge 9223372036854775807/0, generation 0
00000.015 II| TABLET_EXECUTOR: Leader{1:2:103} starting compaction
00000.015 II| TABLET_EXECUTOR: Leader{1:2:104} starting Scan{1 on 101, Compact{1.2.103, eph 1}}
00000.015 II| TABLET_EXECUTOR: Leader{1:2:104} started compaction 1
00000.015 DD| TABLET_EXECUTOR: TGenCompactionStrategy PrepareCompaction for 1 started compaction 1 generation 0
00000.025 DD| TABLET_SAUSAGECACHE: Save page collection [1:2:103:1:12288:2976:0] compacted pages [ 118 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 115 116 117 ]
00000.032 DD| TABLET_EXECUTOR: Leader{1:2:104} commited cookie 8 for step 103
00000.032 DD| TABLET_SAUSAGECACHE: Unregister [1:340:2377]
00000.032 II| TABLET_EXECUTOR: Leader{1:2:104} Compact 1 on TGenCompactionParams{101: gen 0 epoch +inf, 0 parts} step 103, product {1 parts epoch 2} done
00000.032 DD| TABLET_EXECUTOR: TGenCompactionStrategy CompactionFinished for 1: compaction 1, generation 0
00000.032 DD| TABLET_SAUSAGECACHE: Attach page collection [1:2:103:1:12288:2976:0] owner [1:30:2062]
00000.032 TT| TABLET_SAUSAGECACHE: Touch page collection [1:2:103:1:12288:2976:0] pages [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 115 116 117 118 ]

When the test fails the sequence of events is as follows:

00000.015 DD| TABLET_EXECUTOR: TCompactionLogic PrepareForceCompaction for 1 table 101, mode Full, forced state None, forced mode Full
00000.015 DD| TABLET_EXECUTOR: TGenCompactionStrategy PrepareCompaction for 1: task 1, edge 9223372036854775807/0, generation 0
00000.015 II| TABLET_EXECUTOR: Leader{1:2:103} starting compaction
00000.015 II| TABLET_EXECUTOR: Leader{1:2:104} starting Scan{1 on 101, Compact{1.2.103, eph 1}}
00000.015 II| TABLET_EXECUTOR: Leader{1:2:104} started compaction 1
00000.015 DD| TABLET_EXECUTOR: TGenCompactionStrategy PrepareCompaction for 1 started compaction 1 generation 0
00000.025 DD| TABLET_EXECUTOR: Leader{1:2:104} commited cookie 8 for step 103
00000.025 DD| TABLET_SAUSAGECACHE: Save page collection [1:2:103:1:12288:2976:0] compacted pages [ 118 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 115 116 117 ]
00000.031 II| TABLET_EXECUTOR: Leader{1:2:104} Compact 1 on TGenCompactionParams{101: gen 0 epoch +inf, 0 parts} step 103, product {1 parts epoch 2} done
00000.031 DD| TABLET_EXECUTOR: TGenCompactionStrategy CompactionFinished for 1: compaction 1, generation 0
00000.032 DD| TABLET_SAUSAGECACHE: Unregister [1:340:2377]
00000.035 DD| TABLET_SAUSAGECACHE: Attach page collection [1:2:103:1:12288:2976:0] owner [1:30:2062]
00000.035 TT| TABLET_SAUSAGECACHE: Touch page collection [1:2:103:1:12288:2976:0] pages [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 115 116 117 118 ]

The difference is whether executor handles finished compaction before or after shared cache handles unregister from the compaction actor. When CompactionFinished is handled first it appears that GC at the end of the unregister drops all pages (no more references?), which doesn't happen on good days (there are still some references that prevent pages from being dropped?).

I'm not sure how to fix that, but it underlines a real issue where saved compacted pages may not actually survive in practice.

[kunga]

The bug (or not a bug at all) happens when the size of compacted pages exceeds Shared Cache limits.

In the tests, Shared Cache limit is 8MB, and compacted data pages size is 10MB.

Compacted pages are sent with a loaded state to both:

• Shared Cache in TEvSaveCompactedPages request
• Executor in TProdCompact

Then the following happens:

1. Shared Cache receives TEvSaveCompactedPages, puts 10MB into the cache, exceeds the limit, and rotate all the pages into a passive state.

In a good scenario:

2. Shared Cache executes ProcessGCList, which is triggered by TEvUnregister, but as compacted pages are still loaded in the in-fly TEvSaveCompactedPages request, it does nothing, and compacted pages remain passive.
3. Executor receives TProdCompact, unloads compacted pages, and sends TEvTouch to Shared Cache.
4. Shared Cache receives TEvTouch, makes 8MB of compacted pages active, and drops the rest 2MB of them.

In a bad scenario:

2. Executor receives TProdCompact, unloads compacted pages, and sends TEvTouch to Shared Cache.
3. Shared Cache executes ProcessGCList, which is triggered by TEvUnregister, but as compacted pages are not required anymore (unloaded by Executor), it drops all of them.
4. Shared Cache receives TEvTouch but doesn't have any of these pages already.

To sum up:

• Unregister call doesn't matter a lot because any other tablet may also communicate with Shared Cache and trigger ProcessGCList method.
• Shared Cache has a suspicious eviction policy: when stored data exceeds the limit and there is nothing to evict, all stored data becomes passive and is likely to be dropped later.

Next steps:

• Add test cases when compacted pages size doesn't exceed Shared Cache limit.
• Rethink Shared Cache evict policy: should we back passive pages to active when there is enough room for them? In the case above, I expect that only 2MB of pages will be dropped and the rest 8MB will stay.