Local error: tls: bad record MAC after upgrade from 1.22.2 -> 1.22.4

[tudor9301]

Overview of the Issue

2026-02-20T16:13:14.802Z [WARN] (view) health.service(|passing): Get "https://<consul_ip>:<consul_port>/v1/health/service/?index=&passing=1&stale=&wait=60000ms": local error: tls: bad record MAC (retry attempt 1 after "250ms")

Reproduction Steps

When running consul-template trying to fetch data from the kv store the error about MAC record shows, even if it is marked as a Warning.

Did the upgrade, rolled it back to the previous version and it works as expected, so my guess it's that the issue is from this release

Operating system and Environment details

Log Fragments

[AlexeyDemidov]

I'm getting a similar error in Nomad after upgrading from 1.22.3 to 1.22.4:

[WARN]  agent: (view) kv.block(rtrigger): Get "https://127.0.0.1:8501/v1/kv/trigger?index=11205344&stale=&wait=300000ms": local error: tls: bad record MAC (retry attempt 1 after "250ms")
[ERROR] agent: (runner) sending server error back to caller

The errors are sporadic and I don't see any corresponding errors in consul logs and I don't see any corresponding load spikes.
Rolling back to 1.22.3 fixed the issue.

I guess it is related to the following line in the changelog:

security: Configure HTTP server timeouts to prevent Slowloris denial-of-service attacks on agent HTTP endpoints and pprof endpoints. [GH-22739]

[tudor9301]

Does Consul currently expose a server-side config to override the new HTTP timeouts introduced in 1.22.4? My 60s blocking queries are being severed and throwing TLS errors. If no configuration exists yet, should we expect a patch that introduces a parameter for this, or will blocking queries be handled differently?

[askovpen]

all services uses in range service "xxx" get same timeout. with check block and without check block.

WARN] agent: (view) health.service(xxxxxx): Get "http://localhost:8500/v1/health/service/xxxxxx-svc?index=10331283&stale=&wait=300000ms": EOF (retry attempt 1 after "250ms")

[jinnatar]

Even consul monitor gets the axe killing log monitoring, so a very indiscriminate fix that cuts into some of the core patterns of using consul. Ran into these when every single template rendering from Nomad just eventually dies:

[WARN] agent: (view) kv.block(templates/vector/vector.toml): Get "http://127.0.0.1:8500/v1/kv/templates/vector/vector.toml?index=55218966&stale=&wait=300000ms": EOF (retry attempt 4 after "2s")

Rolling back restores previous feature of long polling.

[MorphBonehunter]

@AlexeyDemidov / @askovpen I'm not convinced, that this has something to to with the http timeout settings.
Admittedly the timeouts defined at builder.go#L946 and following lines and are comparable to your 60000ms / 30000ms, but I have blocking queries in vmagent SD which near to the 10 minutes max of such queries (&wait=525s) and they are not closed after the agent timeout stuff.
After some manual curl tests the problem ist the different return value between 1.22.3 and 1.22.4.
The blocking query should return an JSON object regardless if there is an timeout occurred or there was an change in the data (4th paragraph of the blocking docs) and this does not happen anymore.
In 1.22.4 it returns "EOF" in case the wait= parameter triggers.
This is not only a problem for consul-template...

1.22.4:

time curl "http://127.0.0.1:8500/v1/catalog/services?dc=abc&stale&index=22449054&wait=300s"
curl: (52) Empty reply from server

real    5m13,235s
user    0m0,003s
sys     0m0,015s

vs. 1.22.3 (shortened 😄):

time curl "http://127.0.0.1:8500/v1/catalog/services?dc=abc&stale&index=22449054&wait=300s"
{"alertmanager":["external","monitoring=enabled"],"ara":["external"],...}
real    5m11,082s
user    0m0,023s
sys     0m0,018s

But maybe that the same problem OP reports...maybe EOF is an bad record MAC ;)

[sanikachavan5]

Hi folks, the timeouts are indeed configurable, however they were missed as part of the release notes (apologies for the same). Timeouts can be customized via read_timeout, read_header_timeout, write_timeout, and idle_timeout options in the http_config block if longer timeouts are needed for health checks or other operations. REF: https://developer.hashicorp.com/consul/docs/reference/agent/configuration-file/general

[jinnatar]

Regardless of documentation, I don't feel no longer returning data after long polls is a patch version kind of change. More like major version with a year of deprecation warnings.

[MorphBonehunter]

@sanikachavan5 do you have an idea why the return value changed? Could that be an unintended side effect / regression of the timeout features?