ingest: fix flaky TestConcurrentFetchers

[dimitarvdimitrov]

There's a race between produceCtx being cancelled and producing the record with it. So we increment the number of produced records, but then producing the record fails. In the end we expect to receive that record because we counted it.

This is the error that I saw in CI

--- FAIL: TestConcurrentFetchers (1.22s)
    --- FAIL: TestConcurrentFetchers/update_concurrency_with_continuous_production (10.01s)
        logging.go:33: level info component kafka_client msg immediate metadata update triggered why from user ForceMetadataRefresh
        logging.go:33: level info component kafka_client msg producing to a new topic for the first time, fetching metadata to learn its partitions topic test-topic
        logging.go:33: level info component kafka_client msg immediate metadata update triggered why forced load because we are producing to a topic for the first time
        logging.go:33: level info component kafka_client msg done waiting for metadata for new topic topic test-topic
        logging.go:33: level info component kafka_client msg initializing producer id
        logging.go:33: level info component kafka_client msg producer id initialization success id 2916317827932012471 epoch 0
        reader_test.go:2339:
            	Error Trace:	/__w/mimir/mimir/pkg/storage/ingest/reader_test.go:2339
            	            				/__w/mimir/mimir/pkg/storage/ingest/fetcher_test.go:650
            	            				/usr/local/go/src/runtime/asm_amd64.s:1700
            	Error:      	Received unexpected error:
            	            	context canceled
            	Test:       	TestConcurrentFetchers/update_concurrency_with_continuous_production
        fetcher_test.go:692:
            	Error Trace:	/__w/mimir/mimir/pkg/storage/ingest/fetcher_test.go:692
            	Error:      	Not equal:
            	            	expected: 782
            	            	actual  : 783
            	Test:       	TestConcurrentFetchers/update_concurrency_with_continuous_production
            	Messages:   	Should not fetch more records than produced
        fetcher_test.go:706: Total produced: 783, Total fetched: 782
        fetcher_test.go:707: Fetched with initial concurrency: 201
        fetcher_test.go:708: Fetched with high concurrency: 290
        fetcher_test.go:709: Fetched with low concurrency: 291
FAIL

What this PR does

Which issue(s) this PR fixes or relates to

Fixes #9916

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.