Concurrent cache fetch retries waste work

[hannes-ucsc]

(Posted by Claude Code)

During a relatively small reindex, many notifications failed due to ConcurrentCacheFetchError because they all tried to access the same DUOS URL. Because the DUOS request happens after significant metadata extraction from TDR, each SQS-based retry discards that work.

Add explicit retries with jittered back-off to the cache service so that callers can wait for a concurrent fetch to complete instead of failing immediately.

[hannes-ucsc]

For demo, show that the retry log message is present in the logs. Show that ConcurrentCacheFetchError never or rarely caused a full Lambda function retry. Show that the project description is present for all projects.

[hannes-ucsc]

(Posted by Claude Code)

Demo on anvildev after reindex on 2026-08-26.

Retry log messages: No Cache get for … failed (attempt messages appeared in the contribute Lambda logs. No ConcurrentCacheFetchError was encountered during the reindex, so the retry logic was not triggered — every invocation either populated the cache or got a hit.

No Lambda failures due to ConcurrentCacheFetchError: Zero Worker failed messages referencing ConcurrentCacheFetchError in the post-fix logs. Compare to the pre-fix reindex on 2026-08-16, where ConcurrentCacheFetchError caused Lambda failures and SQS-based retries. During today's reindex: 1 cache miss (initial population of key 2ae35c7…), 494 cache hits, zero failures.

Project description present for all projects with a DUOS ID: 2 of 3 datasets have descriptions sourced from DUOS (DUOS-000668, DUOS-000667). The 1000G dataset has no DUOS ID and therefore no description, which is expected.

[hannes-ucsc]

When we promoted this to production, the hammerbox reindex overloaded DUOS. The main reason was that many contribute lambda invocations attempted to get dataset descriptions from DUOS at the same time. This is to be expected, since that description is now needed by every bundle, because we eliminated the dedicated DUOS bundle. The cache was supposed to mitigate the load. However, because the contributions requested different dataset descriptions, all the cache misses were independent (the cache only serializes fetches per full URL, not per site) and so many concurrent requests to DUOS were made.

What made it worse is that the HTTP client gave up only after 20s (we use LimitedRetryHttpClient for Terra services). When the resulting LimitedTimeoutException caused the cache lock to be released, another fetcher was already waiting to take it over, repeating the same request after only 20s, not 2 min, the lock expiration time.