Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Membership] Preserve latest IAmAliveTime across updates #9303

Merged

Conversation

ReubenBond
Copy link
Member

@ReubenBond ReubenBond commented Jan 29, 2025

In subsequent PRs, we leverage the IAmAliveTime value for more behavior related to disaster recovery scenarios. Each silo's IAmAliveTime value is updated on the membership table directly by that silo and it does not bump the membership table's version. Therefore, we cannot use IAmAliveTime in any algorithm which requires consistent membership views (eg, monitoring graph construction, directory membership). IAmAliveTime updates can be occasionally ignored or regressed by membership snapshot updates depending on when the snapshot was captured. That is undesirable but incrementing the membership version for every IAmAliveTiem would greatly increase write contention on the table and cause unnecessary churn for algorithms based on membership version, such as directory membership.

This PR improves upon the situation by locally preserving the latest known IAmAliveTime values for each silo across membership snapshots. It also allows updating snapshots without incrementing the snapshot version by determining if a snapshot is a logical successor to a previous snapshot (higher version or equal version with at least one greater IAmAliveTime value).

The goal is to decrease the chance of incorrectly treating a silo as "stale" (having missed multiple IAmAliveTime updates).

Microsoft Reviewers: Open in CodeFlow

@ReubenBond ReubenBond merged commit e4e933c into dotnet:main Feb 5, 2025
16 checks passed
@ReubenBond ReubenBond deleted the fix/disaster-recovery/preserve-iamalive branch February 5, 2025 19:32
@github-actions github-actions bot locked and limited conversation to collaborators Mar 8, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant