Area: ingest · dedupe — follow-up decided while reviewing #775
Expected: while a tenant's dedupe store is unhealthy (the DynamoDB table not answering or not yet past its check, the Pebble instance failed to open), ingest keeps accepting events: the ones that could not be deduped are put on NATS to be deduped and handled once the store is healthy again, in strict order, so a client never sees a 503 for a backend whose only job is dedupe.
Actual: ingest with dedupe.enabled on fails closed per request while the store is unavailable — 503 with Retry-After on dedupe.ErrUnavailable — until the table check passes or the store answers again. #775 deliberately leaves the dedupe stores out of /readyz, since a store that only dedupes should not take the whole process out of rotation, so the per-request 503 is the only behavior left for a dedupe outage.
Impact: a dedupe outage is an ingest outage for every tenant with dedupe on, for as long as it lasts, even though the events could be accepted now and deduped later.
Note: strict ordering is the constraint. The parked events must be deduped and committed in the order they arrived, relative to each other and to the events that keep arriving once the store is back; otherwise a retransmission that lands after the recovery could be committed before its original.
Related: #775.
Area: ingest · dedupe — follow-up decided while reviewing #775
Expected: while a tenant's dedupe store is unhealthy (the DynamoDB table not answering or not yet past its check, the Pebble instance failed to open), ingest keeps accepting events: the ones that could not be deduped are put on NATS to be deduped and handled once the store is healthy again, in strict order, so a client never sees a
503for a backend whose only job is dedupe.Actual: ingest with
dedupe.enabledon fails closed per request while the store is unavailable —503withRetry-Afterondedupe.ErrUnavailable— until the table check passes or the store answers again. #775 deliberately leaves the dedupe stores out of/readyz, since a store that only dedupes should not take the whole process out of rotation, so the per-request503is the only behavior left for a dedupe outage.Impact: a dedupe outage is an ingest outage for every tenant with dedupe on, for as long as it lasts, even though the events could be accepted now and deduped later.
Note: strict ordering is the constraint. The parked events must be deduped and committed in the order they arrived, relative to each other and to the events that keep arriving once the store is back; otherwise a retransmission that lands after the recovery could be committed before its original.
Related: #775.