Skip to content

Fix TestStoreGateway_InitialSyncWithWaitRingStability flakyness #4305

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

pracucci
Copy link
Contributor

@pracucci pracucci commented Jun 22, 2021

What this PR does:
The TestStoreGateway_InitialSyncWithWaitRingStability is flaky and fails with this error:

=== RUN   TestStoreGateway_InitialSyncWithWaitRingStability/shuffle_sharding_strategy,_20_gateways,_RF_=_3,_SS_=_3_(bucket_index_enabled_=_false)
    gateway_test.go:300: random generator seed: 1624370314782121000
    gateway_test.go:347:
        	Error Trace:	gateway_test.go:347
        	Error:      	Received unexpected error:
        	            	invalid service state: Failed, expected: Running, failure: unable to start store-gateway dependencies: not healthy, 0 terminated, 1 failed: [register instance in the ring: failed to CAS store-gateway]
        	Test:       	TestStoreGateway_InitialSyncWithWaitRingStability/shuffle_sharding_strategy,_20_gateways,_RF_=_3,_SS_=_3_(bucket_index_enabled_=_false)

We've experienced a similar issue CASing the in-memory consul in the test TestShuffleShardWithCaching and, at that time, we solved it increasing retries and adding a delay between retries. Getting back to TestStoreGateway_InitialSyncWithWaitRingStability, I'm able to reproduce the issue locally in master but I can't reproduce it after this PR's change 🤞

Which issue(s) this PR fixes:
Fixes #4290

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Marco Pracucci <marco@pracucci.com>
@pracucci pracucci mentioned this pull request Jun 22, 2021
3 tasks
@pracucci pracucci merged commit c1023f3 into cortexproject:master Jun 22, 2021
@pracucci pracucci deleted the fix-flaky-TestStoreGateway_InitialSyncWithWaitRingStability branch June 22, 2021 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Flaky Test: TestStoreGateway_InitialSyncWithWaitRingStability
2 participants