fix(security): close final code-scanning findings + fix fuzz job (#480)

CodeQL go/clear-text-logging (query_translation.go): stop logging
auth.principal entirely. The Basic-Auth username is credential material;
the earlier sha256 fingerprint still tripped CodeQL's dataflow. Now no
credential-derived value reaches the log in any path — only auth.source
(a constant mechanism label). Trusted-proxy enduser.* fields still carry
identity for audit.

Semgrep httpsconnection-detected (check-vl-ast-coverage.py): replace
http.client.HTTPSConnection with urllib.request using the default verified
TLS context and a fixed URL. The resulting dynamic-urllib-use-detected
(false positive for a constant URL) is added to the security-heavy
--exclude-rule allowlist.

Security Heavy / fuzz: anchor all -fuzz patterns. -fuzz=FuzzExtractLogPatterns
also matched FuzzExtractLogPatternsFromWindowEntries, and Go refuses to fuzz
when the pattern matches more than one target. Also adds a fuzz run for the
previously-uncovered window-entries target.

Verified: go build/test ok, anchored fuzz targets each run standalone,
semgrep CI config -> 0 findings on the script.

fix(website): patch npm security advisories via overrides (#474)

Every alert is transitive. Adds package.json "overrides" pinning safe
versions of js-yaml, ws, shell-quote, brace-expansion, webpack-dev-server,
http-proxy-middleware, joi, body-parser and @babel/core, and regenerates
the lockfile (also clearing newer dompurify/fast-uri/svgo advisories via
patch bumps).

js-yaml has no safe 3.x, and gray-matter (Docusaurus front matter) calls
the removed js-yaml v3 safeLoad. So docusaurus.config.ts drives gray-matter
with a js-yaml v4 load engine via markdown.parseFrontMatter.

Verified: npm audit -> 0 vulnerabilities, tsc typecheck ok, site build
succeeds. Docs-site only; no proxy runtime change.

fix(docker): correct builder tag to golang:1.26.5-alpine3.24 (#476)

golang:1.26.5-alpine3.22 does not exist on Docker Hub — Alpine moved to
3.23/3.24 for the 1.26.5 line. The Go 1.26.5 bump (#472) merged with the
nonexistent tag, breaking the docker build and every e2e job on main.
Verified: `docker build` succeeds and produces both binaries.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

build(deps): bump body-parser from 1.20.5 to 1.20.6 in /website (#471)

Bumps [body-parser](https://github.com/expressjs/body-parser) from 1.20.5 to 1.20.6.
- [Release notes](https://github.com/expressjs/body-parser/releases)
- [Changelog](https://github.com/expressjs/body-parser/blob/master/HISTORY.md)
- [Commits](expressjs/body-parser@1.20.5...1.20.6)

---
updated-dependencies:
- dependency-name: body-parser
  dependency-version: 1.20.6
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

fix: close review gaps in routing and chart defaults (#454)

* fix: close review gaps in routing and chart defaults

* docs: add review gap changelog entry

* test: fix CI flake + restore right-edge-spike coverage for all sources

The report quality gate failed ("test count regressed") because
TestLock_GrafanaMergedFrames_NoRightEdgeSpike dropped its grafana-ua and
grafana-hdr source cases when residual suppression was narrowed to Drilldown.
Those cases still PASS on this branch — per-chunk axis trimming keeps
Explore/dashboard metric ranges spike-free without residual blanking — so they
are restored: they now positively verify that narrowing suppression to
Drilldown did not reintroduce the right-edge spike for the other sources.

Also harden the flaky TestPeerCache_WriteThroughAndReadAheadBranches (the
unrelated CI flake in the test/report jobs): wait on the client-side WTPushes
counter, which pushToOwner increments only after the peer records the push,
instead of racing the server-side record against the counter.

* test(e2e): pin grafana-lokiexplore-app to 2.0.4 to fix drilldown e2e-ui CI

The Logs Drilldown plugin was preinstalled unpinned (grafana-lokiexplore-app
with no @Version), so CI built Grafana with whatever was latest — now 2.1.1,
whose landing-page UI moved the "Filter by labels" combobox out from under the
Playwright selectors. This deterministically broke e2e-ui (drilldown-core) and
(drilldown-multitenant) on main and every PR branched from it, while a cached
older 2.0.4 still passed locally. Pin to the known-good 2.0.4 (verified: 15/15
drilldown specs green against a freshly-pinned Grafana). Bump deliberately
alongside selector updates when tracking a newer plugin.

* docs(changelog): expand Unreleased with detailed per-change entries for the review-gap work

* docs(changelog): document the Helm metrics-scrape NetworkPolicy breaking change

---------

Co-authored-by: szibis <szibis@users.noreply.github.com>

fix(rules-migrate): validate rule LogQL with the typed AST before tra…

…nslation (#453)

* fix(rules-migrate): validate rule LogQL with the typed AST before translation

Malformed LogQL in rule files — e.g. a 'drop level!=~"debug"' matcher —
previously slipped through conversion with the broken stage silently
skipped. The proxy's query handlers already reject such queries with a
Loki-style HTTP 400 via the typed AST validator; rules-migrate now runs
the same validator and fails conversion with the same parse error.

Remove the unused translator.ValidateDropKeepSyntax helper and its
parseError plumbing: with both entry points validating via the AST,
malformed matchers can no longer reach the drop/keep walker. Add
regression tests locking the HTTP 400 contract for query and
query_range.

* fix(logql): prevent parser panic on unterminated opaque parens; fix !~ drop/keep matcher loss

Two correctness bugs surfaced by expanded fuzz/table tests on the parser,
translator, and rules-migrate components:

1. consumeBalancedParens returned offset 0 when an opaque function call's
   parentheses never closed (e.g. "(A00000("); the caller then sliced
   input[start:0] and panicked. ValidateLogQL runs on every query/query_range
   request (and now in rules-migrate), so a crafted query could panic the
   parser. It now returns a Loki-style parse error. Found by fuzzing.

2. The drop/keep matcher-vs-bare classifier in walkDropKeepStages gated on
   strings.Contains(item, "="). The !~ operator is the only one with no '=',
   so a 'field!~"re"' conditional drop/keep was misread as a bare field name
   and silently lost during proxy post-processing. It is now parsed as a
   matcher condition, matching the proven-correct splitDropSpec path.

Also: harden the flaky TestPeerCache_WriteThroughAndReadAheadBranches (wait on
the client-side write-through counter rather than racing the server-side push
record), and add broad table + fuzz coverage for drop/keep classification,
rule-expression validation, and parser paren robustness. Fuzz-found crashers
captured as regression seeds.

* test(e2e): pin grafana-lokiexplore-app to 2.0.4 to fix drilldown e2e-ui CI

The Logs Drilldown plugin was preinstalled unpinned, so CI built Grafana with
the latest (now 2.1.1), whose landing-page UI moved the "Filter by labels"
combobox out from under the Playwright selectors — deterministically breaking
e2e-ui (drilldown-core) and (drilldown-multitenant) on main and every PR, while
a cached 2.0.4 passed locally. Pin to the known-good 2.0.4.

---------

Co-authored-by: szibis <szibis@users.noreply.github.com>

chore(docs): cleanup (#450)

Co-authored-by: szibis <szibis@users.noreply.github.com>

fix: round-2 review findings + Drilldown residual-clustering & live-t…

…ail Explore refresh regressions (#446)

* fix: address round-2 review findings (error-body redaction, residual trim, gate scoping, helm, lifecycle, fanout cap)

F1 Security: backend VL error bodies can echo the LogsQL query (selectors,
filter values) and 13 handler sites passed raw bodies to writeError/
writeDrilldownPartialFromUpstream, which log them. New redactBackendError
strips selectors, long quoted literals and hex/id runs (no-op under
-debug-log-raw-queries), applied at every backend-error-body site.

F2 Correctness: replace the Grafana querySplitting residual BLANKING (which
false-positived on legitimate short-range/coarse-step Grafana queries) with
per-chunk axis trimming. The windowed /hits path now honors each chunk's
[start,end] (like the direct stats path), so the residual contributes only its
real data and no right-edge mergeFrames spike forms. isQuerySplitLeftoverChunk
and both suppression call sites removed. Live-verified: sub-step Drilldown query
served (not blanked); 24h windowed /hits emits zero out-of-window buckets.

F3 API contract: scope the windowed-/hits rewrite to isGrafanaDrilldownRequest
OR isLikelyHighCardinalityField. Normal Grafana dashboard/Explore low-card
panels and direct API clients now get exact stats; Drilldown (any field) and
high-card fields keep windowed sampling. Live-verified at 24h.

F4 Helm: validateMetrics now also fails fast on a loopback-bound metrics-listen
(127.x/localhost/::1) when service.metrics.enabled — it renders cleanly but is
unreachable from the pod IP the Service/ServiceMonitor target.

F5 Helm: add networkPolicy.monitoringNamespace convenience (namespace name →
namespaceSelector) alongside monitoringFrom; empty both stays open (zero-config,
documented). Non-breaking.

F6 Lifecycle: ColdRouter owns stop/done channels + Stop(); Proxy.Shutdown stops
the manifest-refresh loop instead of leaking it on context.Background().

F7 Performance: cap purgePeerCaches fanout with errgroup.SetLimit(16) so a large
ring can't burst to every peer at once; per-peer timeout + partial-failure
reporting preserved.

Tests (-race): redactBackendError, windowed-hits gate, leftover-not-blanked
lock, ColdRouter Stop lifecycle, purge concurrency cap (40 peers, max in-flight
<=16). Helm template verified across loopback/valid/monitoringNamespace combos.
Full proxy+translator suites, lint, gofmt, helm parity, asset-sync all pass.

* test: use context.Background() in ColdRouter Stop test (staticcheck SA1012)

* fix(drilldown): suppress querySplitting residual chunk to stop high-card chart clustering

Regression: Grafana Drilldown high-cardinality charts (pod label, *_id fields)
clustered all data at one edge (left/'beginning') at 24h+ instead of spreading.

Root cause: Grafana querySplitting emits a tiny trailing RESIDUAL chunk (range <
step). For a by() metric query it yields a single-bucket, multi-series frame;
Grafana's mergeFrames/closestIdx collapses all N single-point series onto one
edge of the merged chart. Per-chunk axis trimming can't fix it (the bucket is
legitimately within [start,end]) — the residual must be suppressed.

Suppress at the handleQueryRange entry (where r.URL still carries start/end/step
and the original query parses as a metric expr — downstream, withOrgID/
injectAuthFingerprint reset r.Form and rewrite the URL, so deeper guards see an
empty range), scoped to metric (matrix) exprs so log queries are never blanked.
Defensive guards remain in proxyStatsQueryRange + the /hits leaf. isQuerySplitResidual
reads r.URL.Query() (deterministic) with an r.FormValue fallback.

Lock test TestLock_LeftoverChunkSuppressedInHits pins Grafana sub-step → suppressed,
range>=step + non-Grafana → served.

* test(drilldown): lock residual suppression through the real handleQueryRange entry

Deterministic in-process test proving the querySplitting residual is blanked at
the query_range entry for pod-label, *_id-field, and Explore-UA Grafana sub-step
metric queries — and NOT for full-step, non-Grafana, or log queries. Immune to
the Docker build-cache flakiness that obscured the live fix.

* fix(cache): live-tail Explore refresh — clamp recent-tail max-staleness below TTL

Near-now query_range requests are meant to bypass the response cache and fetch
the latest logs (recent-tail-refresh), but the bypass never fired: default
max-staleness (15s) > query_range cache TTL (10s), so the entry expired before it
was 'stale enough' to bypass. Live-tail Explore refreshes served cached logs (no
new data) until the entry expired — unlike the VL datasource (no cache).

shouldBypassRecentTailCache now clamps the effective max-staleness to ttl/2 when
it exceeds the endpoint TTL, so near-now refreshes go fresh within one TTL.
Verified live: query 8s apart now advances the newest log timestamp (was stale).

Also includes the CHANGELOG entry for the Drilldown residual-clustering fix.

* fix(cache): live-tail Explore refresh — bypass compatCache for near-now requests

The earlier recent-tail clamp fixed the INNER query cache, but query_range goes
through compatCacheMiddleware (the outer cache) which is active instead — and the
inner cache is explicitly skipped when it is. compatCacheMiddleware caches
query_range/query for a fixed 5min (chart stability) and its hit path had NO
freshness check, so live-tail Explore refreshes re-served the cached page (no new
logs) for up to 5min. VL datasource has no such cache, hence the difference.

compatCacheMiddleware's hit path now consults compatCacheShouldBypassForFreshness:
near-now request (end within recent-tail-refresh-window) + cached entry older than
recent-tail-refresh-max-staleness => re-fetch fresh; historical queries keep the
full cache. Default max-staleness lowered 15s -> 2s for live-tail freshness.

Verified live: {env=production} refreshed 3s apart now advances the newest log on
every refresh (was stuck). Unit test TestCompatCacheFreshnessBypass.

---------

Co-authored-by: szibis <szibis@users.noreply.github.com>

fix: address code-review findings (query-leak, over-broad residual/hi…

…ts, helm metrics) (#444)

1. Security: sanitize upstream transport errors before logging/returning.
   Go surfaces dial/TLS/timeout failures as *url.Error embedding the full
   backend URL incl. the LogQL/LogsQL query, leaking it into error logs/responses
   despite the debug-log redaction. New (p *Proxy) sanitizeUpstreamError redacts
   the URL query, preserves the underlying cause, no-op under -debug-log-raw-queries.

2. Correctness: narrow the querySplitting residual suppression from range<=2*step
   to range<step (a true sub-step residual). The broad form blanked legitimate
   Grafana Explore/dashboard queries with a short range or coarse step. Extracted
   isQuerySplitLeftoverChunk; both the dispatch hoist and the /hits path use it;
   lock test updated to encode the <step boundary.

3. API contract: scope tryHighCardCountByWindowedHits (lossy per-window top-N
   sampling) to isGrafanaSourcedRequest. Direct API clients now get exact
   stats (direct path + max-stats-query-series cap), not sampled /hits semantics.

4. Helm: derive the deployment metrics containerPort from extraArgs.metrics-listen
   (Service/ServiceMonitor named-port targets stay in lock-step) and fail fast when
   service.metrics.enabled is combined with an empty metrics-listen or
   register-instrumentation=false (previously rendered then failed startup / scraped
   a dead port). New loki-vl-proxy.metricsPort / validateMetrics helpers.

5. Helm: open the metrics port in the default NetworkPolicy when serviceMonitor is
   enabled (was 3100-only → scrape timeouts). New networkPolicy.monitoringFrom to
   restrict the scrape source.

Tests: sanitize + Grafana-gating unit tests; lock-test boundary update; helm
template verified across default/svcmon-on/metrics-listen-empty/reg-false/custom-port.
Full proxy+translator suites, helm parity, lint, vet, gofmt, asset-sync all pass.

Co-authored-by: szibis <szibis@users.noreply.github.com>

feat(cache): ring-wide cache purge — POST /admin/cache/flush?peers=1 …

…fans out to all peers (#442)

* feat(cache): ring-wide cache purge fanout (POST /admin/cache/flush?peers=1)

Operators previously had to curl /admin/cache/flush on every pod to clear the
fleet's caches. Add an opt-in fanout: /admin/cache/flush?peers=1 purges the local
instance AND every peer in the L3 ring, authenticated with the shared X-Peer-Token
(same auth the peer cache uses for get/set/has).

- Extract purgeLocalCaches() (L0 hot index + L1 memory + L2 disk; PurgeAll covers
  all three — L3 is exactly what the fanout adds).
- handleCacheFlush: opt-in ?peers (1/true/yes/on or bare ?peers); concurrent
  fanout with a 5s per-peer timeout; down peers reported in the JSON response,
  not fatal; local-only behavior unchanged without the flag.
- New peer-side POST /_cache/purge, gated by peerCacheMiddleware (X-Peer-Token),
  served on the main listener alongside the other /_cache/* routes. Purges THIS
  node only — peers never re-fan-out, so no broadcast storm.

Tests (cache_purge_fanout_test.go, -race): peer-side endpoint auth (401 without
token, 405 on GET, 200 + purge with token); 2-proxy fanout (local+peer purged
with ?peers, peer untouched without); unreachable peer reported not fatal;
peer-cache-disabled node purges local + notes disabled.

* test(perf): make TestPerf_Labels_WarmupCoverage deterministic (CI -race flake)

This warmup test is flaky on main and blocked unrelated PRs (failed ~87% under
-count under -race). Three compounding non-determinisms: (1) async cache writes
on a miss, (2) the warmup instant vs request instant compute different bucketed
cache keys (the handler also caps/derives start from live now), and (3) the 10s
LabelCacheTTL let the background-refresh threshold (8s) be crossed under load,
adding a spurious async backend call.

Fix: warm synchronously (already), bump the test TTL to 5m so background refresh
never fires, and replace the brittle single-pass 'post-warmup ≤1 call' assertion
with a convergence poll over a FIXED nowNs — issue all windows until one full
pass triggers zero backend calls, which deterministically proves the metadata
responses are cacheable regardless of async-write timing or key drift. Verified
200x under -race and with the label group at -parallel 8.

---------

Co-authored-by: szibis <szibis@users.noreply.github.com>