CI Failure (key symptom) in ShadowIndexingCacheSpaceLeakTest.test_si_cache

[vbotbuildovich]

https://buildkite.com/redpanda/redpanda/builds/50331

Module: rptest.tests.test_si_cache_space_leak
Class: ShadowIndexingCacheSpaceLeakTest
Method: test_si_cache
Arguments: {
    "concurrency": 2,
    "message_size": 10000,
    "num_messages": 100000
}

test_id:    ShadowIndexingCacheSpaceLeakTest.test_si_cache
status:     FAIL
run time:   239.763 seconds

<BadLogLines nodes=docker-rp-9(1) example="ERROR 2024-06-16 07:26:22,781 [shard 0:fetc] cloud_storage - cache_service.cc:1455 - Exceeded cache size limit!  (size=10655959/20 reserved=0/0 pending=0/0 max=10485760/100000)">
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 535, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 190, in wrapped
    redpanda.raise_on_bad_logs(
  File "/root/tests/rptest/services/redpanda.py", line 1565, in raise_on_bad_logs
    lsearcher.search_logs(_searchable_nodes)
  File "/root/tests/rptest/services/utils.py", line 197, in search_logs
    raise BadLogLines(bad_loglines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-9(1) example="ERROR 2024-06-16 07:26:22,781 [shard 0:fetc] cloud_storage - cache_service.cc:1455 - Exceeded cache size limit!  (size=10655959/20 reserved=0/0 pending=0/0 max=10485760/100000)">

JIRA Link: CORE-4240

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/50551
*https://buildkite.com/redpanda/redpanda/builds/50548
*https://buildkite.com/redpanda/redpanda/builds/50633
*https://buildkite.com/redpanda/redpanda/builds/50636
*https://buildkite.com/redpanda/redpanda/builds/50716
*https://buildkite.com/redpanda/redpanda/builds/50745
*https://buildkite.com/redpanda/redpanda/builds/50782

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/50967
*https://buildkite.com/redpanda/redpanda/builds/51037
*https://buildkite.com/redpanda/redpanda/builds/51041

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/50967
*https://buildkite.com/redpanda/redpanda/builds/51037
*https://buildkite.com/redpanda/redpanda/builds/51041

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/51136

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/51184

[vbotbuildovich]

*https://buildkite.com/redpanda/redpanda/builds/51203
*https://buildkite.com/redpanda/redpanda/builds/51209
*https://buildkite.com/redpanda/redpanda/builds/51210
*https://buildkite.com/redpanda/redpanda/builds/51209
*https://buildkite.com/redpanda/redpanda/builds/51211

[piyushredpanda]

Closing older-bot-filed CI issues as we transition to a more reliable system.