[server] NotifyLeaderAndIsr may trigger a spurious ISR shrink

[fxbing]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

main (development)

Please describe the bug 🐞

A newly elected leader may remove healthy followers from ISR as soon as its first write advances the leader LEO, before those followers send their next fetch. This includes the initial leader activation of a newly created partition. A repeated NotifyLeaderAndIsr request with the same leader epoch may cause the same problem.

This can appear as an isrShrinksPerSecond spike when traffic starts on an otherwise healthy table, followed by an ISR expansion after the next follower fetch. This was observed when an existing primary-key table created a new partition and immediately received PutKv traffic.

How to reproduce

1. Create a new partition and make TabletServer 1 the leader of one of its buckets with replicas and ISR [1, 2, 3].
2. Issue a write that advances the leader LEO before followers 2 and 3 send their first fetch to the leader.
3. Trigger the ISR expiration check.

Expected: the ISR remains [1, 2, 3] during the normal log.replica.max-lag-time grace period.

Actual: the ISR shrinks to [1]. Followers 2 and 3 can rejoin after their next fetch.

Root cause

There are two related follower lifecycle problems:

1. When a local replica is not already the leader, its followerReplicasMap may be empty. This happens both during the initial leader activation of a newly created partition, where ReplicaManager.maybeCreateReplica() creates a fresh Replica, and during a follower-to-leader transition, where Replica.makeFollower() has cleared the map. Replica.makeLeader() then creates new FollowerReplica objects. These objects start with an unknown LEO and lastCaughtUpTimeMs = 0. Because resetFollowerReplicaState() is not called, followers in the committed ISR do not receive the normal leader-election lag grace period.
2. When an existing leader receives another NotifyLeaderAndIsr request, updateAssignmentAndIsr() unconditionally replaces every FollowerReplica. This discards the follower LEO and caught-up timestamps, even when the leader epoch and replica assignment have not changed.

FollowerReplica.resetFollowerReplicaState() already defines the intended initialization for leader election and same-broker re-election, but it currently has no production caller.

Solution

Follow the replica lifecycle used by Kafka:

• Keep existing FollowerReplica objects for unchanged assignments, for example with computeIfAbsent, so repeated notifications do not discard their fetch state.
• When the leader epoch advances, reset each follower through the existing resetFollowerReplicaState(...) method. Followers in the committed ISR should receive the normal lag grace period.
• Add regression tests for an immediate first write after initial leader activation, a repeated notification with the same leader epoch, and a same-broker leader re-election.

Kafka similarly resets remote replica state when the leader epoch changes and retains existing replica objects with computeIfAbsent.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[fxbing]

Besides causing false alerts, this issue can increase control-plane load. Each spurious shrink → expand cycle triggers two AdjustIsr operations and two LeaderAndIsr ZNode updates. When many buckets are affected simultaneously, this may significantly increase ZooKeeper write pressure and Coordinator event queue time.