kvserver,changefeeds,crosscluster: set per-consumer catchup scan limit

[stevendanna]

Currently, it is easily possible for a single slow rangefeed consumer to acquire the entire catchup scan quota for a given store, preventing any other consumers from advancing.

In the long run, we need a more sophisticated approach to solve this. This change is aimed to be a small improvement that solves the most egregious case: a single slow consumer consuming the entire quota.

It introduces the concept of a ConsumerID into the rangefeed request. The idea of a ConsumerID is that it represents a logical rangefeed consumer such as a changefeed or LDR stream. Such consumers may make multiple MuxRangeFeed requests to a given node despite sharing the same downstream consumer.

When per-consumer catchup scan limiting is enabled, no single consumer is allowed to consumer more than 75% of a given store's capacity. If no ConsumerID is specified, a random consumer ID is assigned to all rangefeeds originating from a given MuxRangeFeed call.

Epic: none
Release note: None

[cockroach-teamcity]

This change is 

[wenyihu6]

Edit: I will use GitHub review.

[wenyihu6]

mod our slack exchange about multiple stores, up to you how you want to handle it

[ajstorm]

One small request

[ajstorm]

In DR weekly with @dt, we are of the opinion that this should merge with default = false, and keep this in our back pockets in case we need it to bail us out of a production situation. @stevendanna does that make sense to you?

[stevendanna]

Yes, thanks for flagging, I meant this to be disabled by default. I've grown very skeptical of this change over the last few days of investigations.

[sumeerbhola]

I've grown very skeptical of this change over the last few days of investigations.

Could you elaborate?

[stevendanna]

Could you elaborate?

Overall, this change doesn't feel very risky to me and may be a useful mitigation for some users; but, it comes at the cost of adding another layer of confusion. There are currently 2 rate limiters and CPU pacing to consider when thinking about a backlog of catchup scans. Recently, it took multiple CRDB developers a non-negligible amount of time to fully understand that the joint behaviour of the existing rate limiters.

Further, we found that since the introduction of these rate limiters, the impact of catch-up-scans on foreground workloads has improved dramatically. We recently ran an experiment where we set both limits to 35k and saw only minor impacts on the foreground workload. This isn't an argument for removing the limits immediately (since there are other problems with having up to 35k iterators open), but it is perhaps just more motivation to re-visit the overall structure of the catch-up scan process and the limits around it.

[stevendanna]

@sumeerbhola Wondering if you have time for a second look here.

@ajstorm Let me know if the cluster setting updates work for you.

Comments addressed.

[stevendanna]

bors r+

[craig]

Build succeeded:

• local_roachtest_fips
• lint
• linux_amd64_build
• docker_image_amd64
• unit_tests
• linux_arm64_build
• macos_arm64_build
• check_generated_code
• local_roachtest
• linux_amd64_fips_build
• acceptance
• macos_amd64_build
• windows_build
• examples_orms

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from 47e6b28 to blathers/backport-release-24.2-133789: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 24.2.x failed. See errors above.

---

error creating merge commit from 47e6b28 to blathers/backport-release-24.3-133789: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 24.3.x failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}