Relay returning 503 for all data due to envelope buffer even though my max payload size is well under the configured limit

[Manvay-S]

Self-Hosted Version

24.7.0

CPU Architecture

x86_64

Docker Version

25.0.3

Docker Compose Version

2.29.0

Machine Specification

• My system meets the minimum system requirements of Sentry

Steps to Reproduce

i've been noticing my relay envelope buffer has been getting filled up recently causing me to get 503 from my sentry endpoints. My relay logs are full of debug logs which I attached below so I'm not too sure what the root cause is because it wasn't happening earlier. The instance has been running for about 5 months.

My current workaround is to restart relay and then flush the dns cache in nginx so it can find the new relay and it works for at max a day.

Expected Result

NA

Actual Result

2025-03-27T20:31:30.817061Z DEBUG relay_server::services::project_upstream: sending request of size 3
2025-03-27T20:31:30.941304Z DEBUG relay_server::services::project_upstream: updating project states for 3/3 projects (attempt 1)
2025-03-27T20:31:30.941331Z DEBUG relay_server::services::project_upstream: sending request of size 3
2025-03-27T20:31:31.061315Z DEBUG relay_server::services::project_upstream: updating project states for 3/3 projects (attempt 1)
2025-03-27T20:31:31.061341Z DEBUG relay_server::services::project_upstream: sending request of size 3
2025-03-27T20:31:31.181486Z DEBUG relay_server::services::project_upstream: updating project states for 3/3 projects (attempt 1)
2025-03-27T20:31:31.181510Z DEBUG relay_server::services::project_upstream: sending request of size 3
2025-03-27T20:31:31.302687Z DEBUG relay_server::services::project_upstream: updating project states for 3/3 projects (attempt 1)
2025-03-27T20:31:31.302716Z DEBUG relay_server::services::project_upstream: sending request of size 3

2025-03-27T20:26:05.049817Z ERROR request{method=POST uri=/api/6/envelope/ version=HTTP/1.1}: relay_server::utils::managed_envelope: dropped envelope: internal error tags.project_key="cee2554365cdf85cd2648f3ed22ea593" tags.has_attachments=false tags.has_sessions=false tags.has_profiles=false tags.has_transactions=false tags.has_span_metrics=false tags.has_replays=false tags.has_checkins=false tags.event_category=None cached_summary=EnvelopeSummary { event_category: None, attachment_quantity: 0, session_quantity: 0, profile_quantity: 0, replay_quantity: 0, checkin_quantity: 0, secondary_transaction_quantity: 0, secondary_span_quantity: 0, span_quantity: 0, has_plain_attachments: false, payload_size: 484, profile_chunk_quantity: 0 } recomputed_summary=EnvelopeSummary { event_category: None, attachment_quantity: 0, session_quantity: 0, profile_quantity: 0, replay_quantity: 0, checkin_quantity: 0, secondary_transaction_quantity: 0, secondary_span_quantity: 0, span_quantity: 0, has_plain_attachments: false, payload_size: 0, profile_chunk_quantity: 0 }
2025-03-27T20:26:05.049888Z ERROR request{method=POST uri=/api/6/envelope/ version=HTTP/1.1}: relay_server::endpoints::common: error handling request error=failed to queue envelope error.sources=[envelope buffer capacity exceeded]
2025-03-27T20:26:05.049925Z DEBUG request{method=POST uri=/api/6/envelope/ version=HTTP/1.1}: tower_http::trace::on_response: finished processing request latency=0 ms status=503
2025-03-27T20:26:05.049935Z DEBUG request{method=POST uri=/api/6/envelope/ version=HTTP/1.1}: tower_http::trace::on_failure: response failed classification=Status code: 503 Service Unavailable latency=0 ms

Event ID

No response