Columnshards : Too many requests while establishing sessions

[naspirato]

repeated in version 1135440
cluster 3cd
workload_log column + row + nemesis

Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: VERIFY failed (2025-01-29T01:56:06.490324+0300): action_id=dc0388c-ddcb11ef-ac90f4da-2b3ddf7f;tablet_id=72075186224037951;verification=++it->second.RequestsCount < 10;fline=gc.cpp:62;event=build_gc_request;address=g=2181038083;c=29;;current_gen=145;gen=145:0;count=10;
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]:   ydb/library/actors/core/log.cpp:754
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]:   ~TVerifyFormattedRecordWriter(): requirement false failed
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.491828Z :BLOB_CACHE WARN: fline=blob_cache.cpp:442;fail=TEvGetResult {Status# ERROR ResponseSz# 2 {[72075186224037951:144:9317:39:1:4195200:0] ERROR Size# 0 RequestedSize# 4195200} ErrorReason# "Too many requests while establishing sessions (DSPE5)." {[72075186224037951:144:4453:39:8:5856:0] ERROR Size# 0 RequestedSize# 5856} ErrorReason# "Too many requests while establishing sessions (DSPE5)."};
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.491836Z :BLOB_CACHE WARN: Read failed for range: { Blob: DS:2181038083:[72075186224037951:144:9317:39:1:4195200:0] Offset: 0 Size: 4195200 } status: ERROR
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.491842Z :BLOB_CACHE WARN: Read failed for range: { Blob: DS:2181038083:[72075186224037951:144:4453:39:8:5856:0] Offset: 0 Size: 5856 } status: ERROR
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.491846Z :BLOB_CACHE WARN: fline=blob_cache.cpp:442;fail=TEvGetResult {Status# ERROR ResponseSz# 1 {[72075186224037951:144:9317:39:0:4195200:0] ERROR Size# 0 RequestedSize# 4195200} ErrorReason# "Too many requests while establishing sessions (DSPE5)."};
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.491853Z :BLOB_CACHE WARN: Read failed for range: { Blob: DS:2181038083:[72075186224037951:144:9317:39:0:4195200:0] Offset: 0 Size: 4195200 } status: ERROR
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.491853Z :TX_COLUMNSHARD ERROR: event_type=NKikimr::NBlobCache::TEvBlobCache::TEvReadBlobRangeResult;fline=task.cpp:16;event=NewError;storage_id=__DEFAULT;blob_range={ Blob: DS:2181038083:[72075186224037951:144:9317:39:1:4195200:0] Offset: 0 Size: 4195200 };message=cannot get blob: ;status=ERROR;external_task_id=e9b0c46-ddcb11ef-b9503249-3d905089;consumer=CS::GENERAL;
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.491856Z :BLOB_CACHE WARN: fline=blob_cache.cpp:442;fail=TEvGetResult {Status# ERROR ResponseSz# 1 {[72075186224037951:144:9317:39:2:4791288:0] ERROR Size# 0 RequestedSize# 4791288} ErrorReason# "Too many requests while establishing sessions (DSPE5)."};
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.491858Z :BLOB_CACHE WARN: Read failed for range: { Blob: DS:2181038083:[72075186224037951:144:9317:39:2:4791288:0] Offset: 0 Size: 4791288 } status: ERROR
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.491867Z :TX_COLUMNSHARD ERROR: event_type=NKikimr::NBlobCache::TEvBlobCache::TEvReadBlobRangeResult;fline=columnshard_impl.cpp:602;event=DoOnError;storage_id=__DEFAULT;blob_id={ Blob: DS:2181038083:[72075186224037951:144:9317:39:1:4195200:0] Offset: 0 Size: 4195200 };status=cannot get blob: ;status_code=ERROR;
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.491886Z :TX_COLUMNSHARD WARN: tablet_id=72075186224037951;self_id=[50019:7465103974753904903:2308];ev=NKikimr::NColumnShard::TEvPrivate::TEvWriteIndex;tablet_id=72075186224037951;external_task_id=e9b0c46-ddcb11ef-b9503249-3d905089;fline=abstract.cpp:69;event=Abort;reason=cannot write index blobs: ERROR;
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.491895Z :TX_COLUMNSHARD WARN: tablet_id=72075186224037951;self_id=[50019:7465103974753904903:2308];ev=NKikimr::NColumnShard::TEvPrivate::TEvWriteIndex;tablet_id=72075186224037951;external_task_id=e9b0c46-ddcb11ef-b9503249-3d905089;fline=granule.cpp:105;event=OnCompactionFailed;reason=cannot write index blobs: ERROR;info=(granule:2;path_id:2;size:891472560;portions_count:2464;);
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.491906Z :TX_COLUMNSHARD ERROR: TxWriteIndex[7]  (CS::GENERAL) cannot write index blobs at tablet 72075186224037951
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.492132Z :BLOB_CACHE WARN: fline=blob_cache.cpp:442;fail=TEvGetResult {Status# ERROR ResponseSz# 2 {[72075186224037951:144:9317:39:1:4195200:0] ERROR Size# 0 RequestedSize# 4195200} ErrorReason# "Too many requests while establishing sessions (DSPE5)." {[72075186224037951:144:4453:39:8:5856:0] ERROR Size# 0 RequestedSize# 5856} ErrorReason# "Too many requests while establishing sessions (DSPE5)."};
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.492137Z :BLOB_CACHE WARN: Read failed for range: { Blob: DS:2181038083:[72075186224037951:144:9317:39:1:4195200:0] Offset: 0 Size: 4195200 } status: ERROR
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.492141Z :BLOB_CACHE WARN: Read failed for range: { Blob: DS:2181038083:[72075186224037951:144:4453:39:8:5856:0] Offset: 0 Size: 5856 } status: ERROR
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.492145Z :BLOB_CACHE WARN: fline=blob_cache.cpp:442;fail=TEvGetResult {Status# ERROR ResponseSz# 1 {[72075186224037951:144:9317:39:0:4195200:0] ERROR Size# 0 RequestedSize# 4195200} ErrorReason# "Too many requests while establishing sessions (DSPE5)."};
Jan 29 01:56:06 ydb-qa-01-klg-028 kikimr_31033[3027663]: 2025-01-28T22:56:06.492150Z :TX_COLUMNSHARD ERROR: event_type=NKikimr::NBlobCache::TEvBlobCache::TEvReadBlobRangeResult;fline=task.cpp:16;event=NewError;storage_id=__DEFAULT;blob_range={ Blob: DS:2181038083:[72075186224037951:144:9317:39:1:4195200:0] Offset: 0 Size: 4195200 };message=cannot get blob: ;status=ERROR;external_task_id=e9b1948-ddcb11ef-ac4f5adc-ab8d6171;consumer=CS::GENERAL;