Repository navigation
Metrics service is firewall-open to any source and answers its query API unauthenticated #17635
Description
The metrics service is the one store-adjacent service with no control in front of it
Measured read-only on the deployment. Nothing modified, no credentials used, no writes attempted.
| Service | Demands credentials? | Firewall scope | Effective control |
|---|---|---|---|
| Relational store | yes — host-based auth refused an anonymous connection | not externally opened | host-based auth |
| Dashboards | yes — 401 on the identity endpoint |
open to any source | application auth |
| Vector store | no — heartbeat answers 200 |
not externally opened | loopback bind |
| Cache/bus | no — liveness command answers without auth | yes — allowed from a routable /24 | none — see the correction below |
| Metrics | no — the label/series query API answers 200 |
open to any source | none |
Four of five are protected, by three different mechanisms. The metrics service is the only one where
the firewall is open and the service itself asks for nothing.
Why this reads as an oversight rather than a decision
The dashboard service is opened to the same scope and does enforce auth, so the pairing
"externally reachable + authenticated" was clearly the intent for that tier. The metrics service got
the first half and not the second.
What is actually exposed
Reconnaissance-grade, not credentials: service and job names, instance labels, scrape target paths,
and whatever identifiers the exporters place in label values. That is enough to map the deployment's
internal topology from outside it. It is not enough to authenticate as anything.
So: worth fixing promptly, not an emergency — and it is a deployment-config change, with no code
change required.
Acceptance criteria
- The metrics port is no longer reachable from an arbitrary source address — scoped to the
monitoring origin, the same way the dashboard tier is fronted. - Verified by a refusal from off-origin, not by an absence of reports. A connection attempt
from outside the permitted scope is rejected, and that rejection is the evidence pasted here. - The permitted origin still works — the dashboard tier can still scrape/query, demonstrated by a
successful query after the change. - Whichever mechanism is chosen (firewall scope, bind address, or a reverse proxy requiring auth)
is recorded as config in the tracked tree, not applied by hand on the host, so a rebuild
reproduces it.
Deliberately NOT in scope
Service-level authentication for the vector store and the cache/bus. Both are currently unauthenticated
and both are safe today because of where they sit on the network. Adding auth to them is coupled to
making them multi-node reachable — see the ordering note below — and belongs with that work, not here.
Ordering constraint this shares with #17538 AC 3
The vector store and cache/bus are protected by network position, which is exactly what a move to a
shared multi-node topology removes. Auth for those two must land before their reachability widens,
proven by a refusal, never after. Deferring service auth to a later version is safe only if the
multi-node reachability work defers with it; the unsafe combination is shipping reachability in one
release and auth in the next.
This issue is the part with no such coupling: the metrics service is already externally scoped, so its
fix stands alone and does not wait for the topology decision.
Related: #17538 (shared store topology, AC 3), #9919 (closed deployment/service-auth umbrella).
Correction (filed by the same session that filed this issue)
The cache/bus row above was wrong when this issue was filed, and it was wrong in the
unsafe direction. It claimed the cache was protected by firewall default-deny. It is not.
The original measurement read the firewall table through head -8 and reported the first four
rules as the complete rule set. The table is ~50 rules long. Reading it in full shows the cache
port allowed from a routable /24, not loopback and not denied.
Re-measured directly:
| Question | Observation |
|---|---|
| Bind | all interfaces, not loopback |
| Password configured | none — the password parameter is empty |
| Server info without credentials | readable |
| Keyspace without credentials | readable, non-empty |
| Firewall scope | allowed from a routable /24 |
So the cache is an unauthenticated data store reachable from an entire subnet, which is a
strictly worse posture than the metrics service this issue is about — it holds data, not labels.
That is tracked separately rather than here, because the repair is different in kind: the metrics
service needs its firewall scope corrected, while the cache needs authentication through the
canonical secrets manager. Same class — reachable implies authenticated — two instances, two
mechanisms.
The method failure is worth recording alongside the finding: a truncated table read as a
complete one. head on a rule list produces a confident, wrong answer with no indication that
anything was cut, and every conclusion drawn from it inherits the truncation silently.
Activity
mrveiss commentedon Sep 27, 2026
Scoping only — nothing applied, nothing to apply without the owner
Read-only. No deployment change, no credential read, no write attempted. This is a proposal.
AC 4 answered first: yes, the firewall is tracked — in two places, and they disagree
autobot-slm-backend/ansible/roles/monitoring/tasks/firewall.yml:17-43community.general.ufw: rule: allow, port: {{ prometheus_port }}/{{ grafana_port }}/{{ node_exporter_port }}— nofrom_ip, so any sourceautobot-infrastructure/shared/config/ufw-defaults.shufw allow 9090/tcp comment 'Prometheus',ufw allow 9100/tcp comment 'Node Exporter'— 17allowrules total, none with a sourceSo a rebuild does reproduce the exposure, which means it can also reproduce the fix: the change belongs in the ansible role, as
from_ip, or as the rule's removal. Nothing here needs to be applied by hand.The second row is its own finding: two tracked descriptions of the same firewall, one of them apparently unapplied. An unapplied file that reads as configuration is worse than no file, because it answers the question "is this managed?" with a yes that nothing enforces.
The recommendation: neither 1 nor 2 as stated — remove the rule, per port, for different reasons
Port 9090 (Prometheus): remove the
allow, do not scope it. The only consumer in the tracked tree is the co-located dashboard service, and it does not come through the host's external interface at all —autobot-monitoring/grafana/provisioning/datasources/prometheus.yml:15readsurl: http://autobot-prometheus:9090withaccess: proxy, i.e. server-side over the container network by container name. Returning 9090 to the default-deny changes nothing for that path. Scoping it to a monitoring origin instead would encode a consumer that does not exist yet, and a rule written for a hypothetical client cannot be verified and will rot.Port 9100 (Node Exporter): scope it, do not remove or loopback-bind it. This one is genuinely consumed across an interface — Prometheus in a container scrapes the exporter on the host, which it reaches over the container bridge, not over loopback. So option 2 would break the scrape. The correct form is
from_ipset to the bridge subnet, in the same ansible task. 9090 and 9100 must not receive the same fix, which is the trap in treating "the metrics ports" as one thing.Option 3 (authentication in front of the query API) is the right answer only for the day something off-host must query it — and then it must land before the reachability, never with it. That is the same sequencing as #17538 AC 3, inverted: there a store is protected by network position and the plan removes that protection; here a service was given reachability without the auth half. The pairing rule is the same in both directions — reachable ⇒ authenticated — and the dashboard tier is the proof that the pairing was the intent, since it got both.
What I could not establish
ufw statusrequires privilege and I did not escalate;ufwisactiveandenabledper systemd, which says the service runs, not what it permits. So "the tracked config opens these ports" is established and "the running firewall opens these ports" is not — and one observation in this issue's own table now needs re-checking against that gap, below.Two things in the same file that are worth their own look
ufw-defaults.sh:108isufw allow 6379/tcp comment 'Redis Stack', unscoped, and the cache answers its liveness command with no auth. If the applied rules match that file, the cache is in the same quadrant as the metrics service and a worse one, because it holds data rather than labels. If they do not match, then the file is unapplied — which is the finding above. Either way the discriminator is the same privileged read of the effective rules, and it should happen before this issue is called scoped.3000/tcp comment 'Browser Worker'and3001/tcp comment 'Grafana', whilessot_config.py:219says the browser service is9001and annotates3000as Grafana, and the only thing listening on 3000 here is bound to loopback. A rule whose comment misidentifies its service cannot be audited by reading it — which is how a port stays open for years.mrveiss commentedon Sep 27, 2026
The cache/bus instance of this class is #17500, which already existed and which I should have found before filing this issue — it documents the same defect from the compose file. Host confirmation is posted there. This issue stays scoped to the metrics service; the repairs differ in kind.
mrveiss commentedon Sep 28, 2026
Classification (for work assignment — not a fix proposal)
Scope:
security(secondaryansible)Primary files/dirs:
autobot-slm-backend/ansible/roles/monitoring/tasks/firewall.yml(verified on main; ufwport: "{{ prometheus_port }}"at:20)autobot-infrastructure/shared/config/ufw-defaults.sh(verified on main; comment 1 says nothing invokes it)autobot-monitoring/grafana/provisioning/datasources/prometheus.yml(verified on main; per comment 1, the only in-tree consumer)Umbrella: no.
Blocked by: host evidence. Comment 1 says two things are not yet established:
AC2 and AC3 need a refusal and a successful query observed on the host. No owner ruling on removing the rule versus scoping it is recorded.
Read vs inferred: Read the issue body (including its in-body correction) and its 2 comments, and verified the three paths exist on main. The cache/bus instance is tracked in security(redis): the compose file publishes an unauthenticated Redis on all interfaces #17500 (comment 2) and is out of scope here.
Undetermined: whether an off-host scraper exists, and whether the running firewall matches the tracked role.
Generated by Claude Code
mrveiss commentedon Sep 28, 2026
Chunking triage — a proposal, not an assignment
priority: critical— not applied. Setting it is the milestone owner's call.origin/mainsince 2026-08-14 references this issue.Nothing was relabelled, moved or closed by this pass.
mrveiss commentedon Sep 28, 2026
B5 spec — this issue splits, and only one half can be built now
Batch B5 (#17500 → #17635 → #17642). Full spec at
~/b5-spec-2026-09-28.md.Your measurement stands as the framing: of five store-adjacent services, four are protected by three different mechanisms, and the metrics service is the only one where the firewall is open and the service demands nothing.
Half 1 — firewall scope — buildable now. Restrict the metrics port to the monitoring source instead of any source, in the same ansible ufw layer #17642 fixes. Do not write a second guard: extend the one #17642 introduces with this port as a case, because a validated
srccovers this by construction.Half 2 — service authentication — WAITS ON A RULING. Whether a scrape target is authenticated, token-allowlisted or network-restricted is now on the owner's table together with
api.prometheus_endpoint(posted on #16375). The two are one surface: deciding them apart risks gating the app route while the port stays open, or the reverse.For the PR body: say the halves were split deliberately and why. Otherwise the next reader sees a security issue closed on a firewall change and assumes the auth question was missed.