[v24.3.x] tx/group compaction fixes

[vbotbuildovich]

Backport #24637 to branch v24.3.x. This issue was created as a placeholder because the original PR's commit(s) could not be automatically cherry-picked.
Here are the commands to execute:

git checkout v24.3.x
git checkout -b manual-backport-24637-v24.3.x-53
git cherry-pick -x 89478480b4 cdd274a2a5 24c8e89f4f 8c5ecca7b5 3736f00b6a 835f3fce5d 9eee6326c5 f7191ad708 6ca81b456f 3d65c3933d 6fc62bb05b 2b79687b9a ac2204129a 7c8d6335a3 c7f953efd9 9958ca60a0 6efd325f85 23c8e295a7 70e36ebe57 c833f50286 0051463f23

git push origin manual-backport-24637-v24.3.x-53
gh pr create \
  --title "[v24.3.x] tx/group compaction fixes" \
  --base "v24.3.x" \
  --label "kind/backport" \
  --head "manual-backport-24637-v24.3.x-53" \
  --draft \
  --repo "redpanda-data/redpanda" \
  --milestone "v24.3.x-next" \
  --body "Backport of PR https://github.com/redpanda-data/redpanda/pull/24637 "

JIRA Link: [CORE-8703](https://redpandadata.atlassian.net/browse/CORE-8703)