Repository navigation
Fix read-only cache availability and read timeouts - #2782
Open
michaeljohnston-nunkiharmonia wants to merge 2 commits into
Open
michaeljohnston-nunkiharmonia wants to merge 2 commits into
michaeljohnston-nunkiharmonia wants to merge 2 commits into
Conversation
Accept local-only blobs when uploads cannot refill the slow tier. Query only local misses upstream, while retaining durable-tier existence for writable mirrors. Cover both read-only settings and preserve write limits.
Bound connection acquisition and the first ByteStream response for every read route, including direct identity and compressed store reads. Retain the existing retry loops and keep zero as the disabled timeout setting. Exercise unresponsive endpoints, delayed first chunks, local hits during upstream failure, and the disabled-timeout control.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A local cache backed by a shared cache that it may read but not update could report a file as missing even when the local copy was available. If the shared cache stopped replying, cache checks and reads could also wait indefinitely despite a configured
rpc_timeout_s. This came up while testing reuse across fresh cache clients and failure during an upstream outage.This change fixes the two problems:
read_onlyorget, existence checks accept files in either tier and only ask upstream about local misses. Writable caches still require presence in the slow tier, so clients continue to refill it when needed.ByteStream.Read, including connection acquisition and the first response. Forwarded reads and direct store reads, with or without compression, use the same deadline helper.How was this verified?
On upstream
45a3b1b4, six new regressions failed: local-only availability, local hits with an unresponsive upstream, ordinary cache requests, direct reads with and without compression, and a stream that opens but delays its first chunk. The control tests passed. With the fix, all 65 tests acrossfast_slow_store_test,grpc_store_test,grpc_read_batching_test, andgrpc_wire_compression_testpassed using Bazel 9.1.1 with the repository's formatting and Clippy checks enabled.The tests also cover duplicate keys, both supported read-only slow-tier settings, absence of upstream writes, writable-cache behavior, and a delayed read with the timeout disabled. The server build, web build, docs structure checks, Vale, and typos checks passed locally. The config reference and text exports were regenerated. Hosted CI, Windows, and sanitizer runs remain to be checked upstream; this has not been deployed upstream.
Risk
Existing deployments with a nonzero RPC timeout will now enforce it on reads that previously ignored it. Slow responses may therefore fail and retry sooner. The default remains
0(disabled), and retries can make the overall operation take longer than one timeout interval. The timeout covers opening a read through its first response; it does not bound later chunks.A read-only existence hit means the file is available in either tier, not that it will survive loss of the local tier. No write permissions, wire formats, or stored data formats change.
AI assistance
gpt-6-astra xhigh was used for authoring and adversarial review.