Reintroduce in-flight state caching changes (backed out because of poor performance)

[reivilibre]

Not much information to speak of at current, but this graph seems bad:

CPU usage has been poor on client readers, as well.

(m.org is running the commit on matrix-org-hotfixes from 7 days ago)

#10870 and friends are my first suspect. (They strip out batching when fetching state groups), but it would be worth confirming that...

[reivilibre]

After backing it out for a while (since 15h40 yest'day), we can see:

DB scheduling looks healthier, though I note that since backing it out there has been a higher peak of DB txn time...

[reivilibre]

The changes were backed out by #12126 — will need to think about how to reintroduce a fix for the original issue (#10301) safely.

[reivilibre]

We still had some OOMs after using the updated branch. I'm not sure it's giving us all that much. I'm going to close this issue and see how #12522 does in addressing #10301 — in our most recent case it seemed like backfill was the main culprit.