[BUG] Lag in follower_stats APIs' follower checkpoint

[gbbafna]

What is the bug?

I have seen follower_stats gives a not-so-latest checkpoint details. There is a divergence when we compare with status API results. The divergence is taking few minutes to converge.

status API is a sync API , which hits up the leader cluster and follower cluster for checkpoints

follower_stats API is an async API , which updates the shards checkpoints whenever it gets the updates from either of the clusters.

Though, follower_stats may not be that accurate, but the divergence from status shouldn't be huge.

How can one reproduce the bug?

1. Ingest some docs on leader cluster
2. Use follower_stats and status API . Observer that there is some difference in follower_checkpoint in both the APIs responses .

What is the expected behavior?

The difference between follower_stats and status APIs outputs should be minimal and converge within few seconds .