v1.63.1

[1.63.1] - 2026-07-22

Security

• Closed the last two code-scanning findings.
  • CodeQL go/clear-text-logging (internal/proxy/query_translation.go): the
    request log no longer emits auth.principal at all. The Basic-Auth username
    (and anything derived from it) is credential material and is now kept out of
    the log entirely, in every code path; only the auth mechanism (auth.source,
    a constant) is recorded. Identity for audit is carried by the trusted-proxy
    enduser.* fields.
  • Semgrep httpsconnection-detected (scripts/check-vl-ast-coverage.py):
    replaced http.client.HTTPSConnection with urllib.request using the default
    verified TLS context and a fixed, non-user-controlled URL. The companion
    dynamic-urllib-use-detected rule (a false positive for this constant URL) is
    added to the Semgrep --exclude-rule allowlist in security-heavy.yaml.
• Fixed the Security Heavy / fuzz job. go test -fuzz=FuzzExtractLogPatterns
  refused to run because the unanchored pattern also matched the newer
  FuzzExtractLogPatternsFromWindowEntries target (-fuzz must match exactly one).
  All -fuzz patterns are now anchored (^...$), and the previously-uncovered
  window-entries target is now fuzzed too.

v1.63.0

[1.63.0] - 2026-07-22

Fixed

• Dockerfile builder image tag corrected to golang:1.26.5-alpine3.24. The
  Go 1.26.5 bump referenced golang:1.26.5-alpine3.22, which does not exist on
  Docker Hub (Alpine moved to 3.23/3.24 for the 1.26.5 line), breaking the
  docker build and all e2e jobs on main. Verified the image builds.

Security

• Go toolchain bumped to 1.26.5 across the Dockerfile builder image, go.mod
  (bench/go.mod), and every CI setup-go pin, to pick up the patched standard
  library for CVE-2026-42505 (crypto/tls Encrypted Client Hello information
  disclosure) and CVE-2026-39822 (os.Root symlink-following directory
  traversal). Rebuilding the loki-vl-proxy and healthcheck binaries against the
  fixed stdlib clears both findings from the Trivy filesystem scan that was failing
  the Security workflow on main. No behavioral change.
• Website (docs) npm dependencies patched — 9 Dependabot alerts cleared.
  Added overrides in website/package.json to force safe versions of the
  vulnerable transitive packages: js-yaml (→4.3.0), ws (→8.21.x),
  shell-quote (→1.10.0), brace-expansion (→1.1.16), webpack-dev-server
  (→5.2.6), http-proxy-middleware (→2.0.10), joi (→17.13.4), body-parser
  (→1.20.6), and @babel/core (→7.29.7). Because gray-matter pins js-yaml v3's
  removed safeLoad, website/docusaurus.config.ts now supplies a js-yaml v4
  load engine via markdown.parseFrontMatter. npm audit reports 0
  vulnerabilities (including newer transitive advisories in dompurify,
  fast-uri and svgo, cleared via lockfile patch bumps); the site build and
  typecheck pass. Docs-site only — no proxy runtime change.

v1.62.2

[1.62.2] - 2026-07-22

Fixed

• Dockerfile builder image tag corrected to golang:1.26.5-alpine3.24. The
  Go 1.26.5 bump referenced golang:1.26.5-alpine3.22, which does not exist on
  Docker Hub (Alpine moved to 3.23/3.24 for the 1.26.5 line), breaking the
  docker build and all e2e jobs on main. Verified the image builds.

Security

• Go toolchain bumped to 1.26.5 across the Dockerfile builder image, go.mod
  (bench/go.mod), and every CI setup-go pin, to pick up the patched standard
  library for CVE-2026-42505 (crypto/tls Encrypted Client Hello information
  disclosure) and CVE-2026-39822 (os.Root symlink-following directory
  traversal). Rebuilding the loki-vl-proxy and healthcheck binaries against the
  fixed stdlib clears both findings from the Trivy filesystem scan that was failing
  the Security workflow on main. No behavioral change.

v1.62.0

[1.62.0] - 2026-06-11

Breaking Changes

• Helm: enabling the ServiceMonitor now requires an explicit metrics-scrape scope.
  With serviceMonitor.enabled=true and networkPolicy.enabled=true, the chart used to
  render a NetworkPolicy that opened the /metrics port to all sources (zero-config
  allow-all). It now fails helm template/helm upgrade unless one of
  networkPolicy.monitoringNamespace, networkPolicy.monitoringFrom, or the new
  networkPolicy.monitoringAllowAll=true is set. Affected: chart users with the
  ServiceMonitor and NetworkPolicy both enabled and no scrape scope configured — their
  next helm upgrade will error until they pick one. To restore the previous
  open-to-all behavior, set networkPolicy.monitoringAllowAll=true; to scope it
  (recommended), set networkPolicy.monitoringNamespace (a namespace name) or
  networkPolicy.monitoringFrom (an explicit NetworkPolicyPeer list).

Security

• Backend error bodies are now redacted on every remaining client-visible error path.
  1.58.1 routed the main handlers through redactBackendError, but several deeper paths
  still returned raw VictoriaLogs 4xx/5xx bodies — derived-volume hits, the
  stats_query_range//hits fast paths, the metric-range and query-translation backends,
  and the detected-fields/label backends — and a VL parse/error message can echo the
  translated LogsQL query (stream selectors, filter values). All of these now go through
  redactedBackendErrorMessage/redactedBackendStatusError, which extract VL's message
  and strip query-like content. The redactor also now masks single-quoted and
  backtick-quoted literals (≥12 chars), not just double-quoted, so quoted values in VL
  errors can no longer leak. No-op under -debug-log-raw-queries=true.

Fixed

• Grafana Explore/dashboard metric ranges at 24h+ are no longer blanked by Drilldown
  residual suppression. The querySplitting residual-chunk suppression (empty matrix for
  a sub-step trailing chunk) is now scoped to Drilldown-tagged traffic
  (isGrafanaDrilldownRequest) instead of all Grafana-sourced requests
  (isGrafanaSourcedRequest). Explore and dashboard panels rely on per-chunk axis
  trimming alone, which keeps them spike-free without ever returning an empty chunk —
  verified across all source tags in TestLock_GrafanaMergedFrames_NoRightEdgeSpike.
• Direct, non-Grafana high-cardinality count() by(field) queries now get exact stats.
  Window-sampled /hits (per-window top-N, right for Drilldown's visual exploration) was
  applied to any high-cardinality field; it is now gated on Drilldown OR a Grafana-sourced
  high-card field, so direct API clients and non-Grafana callers fall through to exact
  stats_query_range aggregation (bounded by max-stats-query-series).
• Helm: enabling the ServiceMonitor no longer silently opens /metrics to all sources.
  With serviceMonitor.enabled=true and networkPolicy.enabled=true, the chart now
  fails template rendering unless one of networkPolicy.monitoringNamespace,
  networkPolicy.monitoringFrom, or the new explicit networkPolicy.monitoringAllowAll=true
  is set — replacing the previous zero-config allow-all default for the metrics scrape
  ingress rule.
• Helm: corrected stale chart documentation for the stats-series default and the
  peer-auth token wiring so the values reference matches the shipped behavior.
• rules-migrate now validates rule expressions with the typed LogQL AST validator
  before translation. Malformed LogQL — e.g. a | drop level!=~"debug" matcher, which
  the proxy's query handlers already reject with HTTP 400 — previously slipped through
  rule-file conversion with the broken stage silently skipped; it now fails conversion
  with the same Loki-style parse error.
• LogQL parser no longer panics on an unterminated opaque function call. A query
  such as (A00000( — an unknown/opaque metric function (label_replace, label_join,
  …) whose argument parentheses never close — made consumeBalancedParens return offset
  0, and the caller then sliced input[start:0] and panicked with "slice bounds out of
  range". Because ValidateLogQL runs on every query/query_range request (and now in
  rules-migrate), a crafted query could panic the parser; it now returns a Loki-style
  parse error. Found by fuzzing.
• | drop/| keep matchers using the !~ operator are no longer silently dropped.
  !~ is the only drop/keep operator with no =, and the matcher-vs-bare-field classifier
  in walkDropKeepStages gated on strings.Contains(item, "="), so a field!~"re"
  conditional drop/keep was misread as a bare field name and lost during proxy
  post-processing. It is now parsed as a matcher condition like the other three operators.
  Found by the expanded drop/keep tests.

Changed

• Helm: recent-tail-refresh-max-staleness default lowered from 15s to 2s to match
  the binary default, so live-tail Explore stays fresh out of the box on chart deployments
  (the 15s chart value previously exceeded the inner cache TTL and never bypassed).
• Expanded test coverage for the LogQL parser, translator, and rules-migrate: added
  table-driven and fuzz tests for drop/keep matcher classification (all four operators,
  malformed-form skipping), rule-expression validation (valid translation + malformed
  rejection, error identifies group/rule), and unterminated-paren parser robustness.
• Hardened the flaky TestPeerCache_WriteThroughAndReadAheadBranches test: it now waits
  on the client-side write-through counter (incremented after the peer records the push)
  instead of racing the server-side record against the counter, which intermittently
  failed under loaded CI.
• Removed the unused translator.ValidateDropKeepSyntax helper: malformed drop/keep
  matchers cannot reach the translator from any entry point now that both the query
  handlers and rules-migrate validate via the LogQL AST first. Added regression
  tests locking the HTTP 400 contract for query and query_range.

v1.61.0

[1.61.0] - 2026-06-11

Fixed

• rules-migrate now validates rule expressions with the typed LogQL AST validator
  before translation. Malformed LogQL — e.g. a | drop level!=~"debug" matcher, which
  the proxy's query handlers already reject with HTTP 400 — previously slipped through
  rule-file conversion with the broken stage silently skipped; it now fails conversion
  with the same Loki-style parse error.
• LogQL parser no longer panics on an unterminated opaque function call. A query
  such as (A00000( — an unknown/opaque metric function (label_replace, label_join,
  …) whose argument parentheses never close — made consumeBalancedParens return offset
  0, and the caller then sliced input[start:0] and panicked with "slice bounds out of
  range". Because ValidateLogQL runs on every query/query_range request (and now in
  rules-migrate), a crafted query could panic the parser; it now returns a Loki-style
  parse error. Found by fuzzing.
• | drop/| keep matchers using the !~ operator are no longer silently dropped.
  !~ is the only drop/keep operator with no =, and the matcher-vs-bare-field classifier
  in walkDropKeepStages gated on strings.Contains(item, "="), so a field!~"re"
  conditional drop/keep was misread as a bare field name and lost during proxy
  post-processing. It is now parsed as a matcher condition like the other three operators.
  Found by the expanded drop/keep tests.

Changed

• Expanded test coverage for the LogQL parser, translator, and rules-migrate: added
  table-driven and fuzz tests for drop/keep matcher classification (all four operators,
  malformed-form skipping), rule-expression validation (valid translation + malformed
  rejection, error identifies group/rule), and unterminated-paren parser robustness.
• Hardened the flaky TestPeerCache_WriteThroughAndReadAheadBranches test: it now waits
  on the client-side write-through counter (incremented after the peer records the push)
  instead of racing the server-side record against the counter, which intermittently
  failed under loaded CI.
• Removed the unused translator.ValidateDropKeepSyntax helper: malformed drop/keep
  matchers cannot reach the translator from any entry point now that both the query
  handlers and rules-migrate validate via the LogQL AST first. Added regression
  tests locking the HTTP 400 contract for query and query_range.

v1.60.0

[1.60.0] - 2026-06-11

Changed

• Relocated the CI-consumed VL AST coverage registry (vl-ast-coverage.json) from
  docs/ to scripts/, next to the checker script that reads it; the docs/ tree
  now carries only documentation pages.

v1.59.0

[1.59.0] - 2026-06-08

Fixed

• Drilldown high-cardinality charts no longer cluster all data at one edge at 24h+. Grafana's querySplitting emits a tiny trailing residual chunk (range < step); for a count() by(field) metric query that residual is a single-bucket, multi-series frame, and Grafana's mergeFrames/closestIdx collapses all N single-point series onto ONE edge of the merged chart — a right-edge spike, or a left-edge "all data at the beginning" cluster on the pod label / *_id field panels. The residual is now suppressed at the query_range entry (where the request URL still carries start/end/step and the original query still parses as a metric expression — downstream, withOrgID/injectAuthFingerprint reset r.Form and rewrite the URL, so deeper guards see an empty range), scoped to metric (matrix) expressions so log queries are never blanked. (Supersedes the 1.58.1 "axis-trim, blanking removed" change, which regressed this.)
• Live-tail Explore now picks up new logs on refresh. Near-now query_range/query responses are cached by compatCacheMiddleware for 5 minutes (for chart stability), and its cache-hit path had no freshness check — so refreshing a live-tail Explore (or a now-relative dashboard) re-served the cached page and showed no new logs, unlike the VictoriaLogs datasource which has no such cache. The cache-hit path now consults compatCacheShouldBypassForFreshness: for a request whose end is near now (within -recent-tail-refresh-window) and whose cached entry is older than -recent-tail-refresh-max-staleness, it re-fetches instead of serving stale; historical (not-near-now) queries keep the full cache. The -recent-tail-refresh-max-staleness default is lowered from 15s to 2s so live tail stays fresh (it was also larger than the inner cache TTL, which is now additionally clamped in shouldBypassRecentTailCache as a safety net).

v1.58.1

[1.58.1] - 2026-06-08

Security

• Upstream transport errors no longer leak query parameters. Go's http.Client surfaces dial / TLS / timeout failures as a *url.Error whose message embeds the full backend URL — including the LogQL/LogsQL query — so those errors could leak query content (potentially sensitive log selectors) into error logs and handler error responses, even though the debug request log already redacts the query. Transport errors are now passed through sanitizeUpstreamError, which redacts the URL query while preserving the underlying cause (status mapping and the circuit breaker still classify correctly). No-op under -debug-log-raw-queries=true.

Fixed

• Grafana querySplitting residual suppression no longer blanks legitimate short-range queries. The 24h+ split residual-chunk suppression triggered for any Grafana-sourced request with range <= 2*step, which could silently return an empty matrix for normal Explore/dashboard queries with a short range or a coarse step. It is now scoped to a true sub-step residual (range < step, i.e. less than one full bucket); a query spanning at least one step is served normally.
• High-cardinality count() by(field) window-sampling is now scoped to Grafana requests. The window-sampled /select/logsql/hits path returns per-window-sampled top-N series rather than exact stats_query_range results — correct for rendering a Grafana chart, but the wrong default for a direct API client. It is now gated on isGrafanaSourcedRequest; non-Grafana callers get the exact direct path (bounded to max-stats-query-series top-N-by-count, like Loki's max_query_series).
• Helm: metrics listener, Service, and ServiceMonitor can no longer render contradictory configs. The deployment's metrics containerPort is now derived from extraArgs.metrics-listen (so the named Service/ServiceMonitor target always matches where /metrics actually serves), and the chart fails fast when service.metrics.enabled=true is combined with an empty metrics-listen or server.register-instrumentation=false (configs that previously rendered cleanly then failed startup or scraped a dead port).
• Helm: default NetworkPolicy now allows ServiceMonitor metrics scraping. When serviceMonitor.enabled=true, the NetworkPolicy automatically opens the metrics port (derived from metrics-listen) for Prometheus — previously it only opened the proxy port 3100, so enabling the ServiceMonitor under the default NetworkPolicy silently timed out the /metrics scrape. Restrict the scrape source via the new networkPolicy.monitoringFrom.

v1.58.0

[1.58.0] - 2026-06-07

Added

• Ring-wide cache purge. POST /admin/cache/flush?peers=1 (or bare ?peers) now purges the local instance's caches (L0 hot index + L1 memory + L2 disk) AND fans the purge out to every peer in the L3 ring, authenticated with the shared X-Peer-Token — the same auth the peer cache already uses for get/set/has. An operator clears the entire fleet's caches by hitting one admin-token-protected instance instead of curling each pod. The fanout is concurrent (5s per-peer timeout); an unreachable peer is reported in the JSON response ("peers": {"addr": "error: …"}), not fatal. A new peer-side endpoint POST /_cache/purge (gated by the peer-token middleware, served on the main listener like the other /_cache/* routes) receives the purge and clears that node's caches only — peers never re-fan-out, so there is no broadcast storm. Without ?peers the endpoint is unchanged (local-only).

v1.57.0

[1.57.0] - 2026-06-07

Added

• L0 hot-key index now surfaces as a cache tier in metrics. New tier="l0" label on loki_vl_proxy_cache_tier_requests_total, loki_vl_proxy_cache_tier_hits_total, loki_vl_proxy_cache_tier_misses_total, and loki_vl_proxy_cache_objects. L0 is the bounded per-key hotness sidecar that drives peer hot-key read-ahead — it is NOT a lookup tier in the L1→L2→L3 chain. A "hit" means the key was already in the hot index (was hot before this request); a "miss" is the first observation of that key. cache_objects{tier="l0"} reports the current bounded population of the hot index. Both OTLP push and Prometheus /metrics paths emit the new series.
• e2e-compat compose now exercises all four cache tiers end-to-end. The main e2e-proxy is configured with -disk-cache-path (L2 bbolt on a named volume that survives docker compose restart) and joins a 3-node static peer ring (-peer-discovery=static) together with two new sibling services loki-vl-proxy-peer-a and loki-vl-proxy-peer-b exposed on host ports 13150 and 13151. vmagent scrapes all three with distinct instance= labels so dashboards can disaggregate per-proxy cache stats.
• scripts/bench-cache-tiers.sh — measures each tier against the compose stack: l1 (cold-vs-warm hit ratio), l2 (L1→L2 promotion across docker compose restart), l3 (cross-peer fetch from peer-a → peer-b), long (7d → 7d-1h windowed cache reuse). Prints before/after counter deltas per tier so the expected promotion path is verifiable.
• L0 series now appear alongside L1/L2/L3 in the bench-resources dashboard's hit-ratio, requests/sec, and objects panels.

Fixed

• High-cardinality Drilldown label/field panels (pod, *_id) now render correctly and fast at 24h+. A detected_level filter combined with a by(field) aggregation on a churning, high-cardinality field (pod names, trace_id, span_id) previously returned ~142k uncapped single-point series at short ranges (flooding Grafana) or an empty matrix at 24h+ (the VictoriaLogs stats response exceeded the 16 MB body cap), and the panel rendered as a single right-edge spike. Three root causes were addressed: (1) the detected_level filter is now evaluated against VictoriaLogs' column-indexed level field instead of forcing a | unpack_logfmt re-parse of every line — ~64× faster (24h pod query 26s → ~0.6s; level-filtered field breakdowns now match the unfiltered baseline); (2) single-field count() by(field) over ranges ≥ 2h routes to the window-sampled /select/logsql/hits path so the returned series span the whole timeline instead of clustering the busiest short-lived values into a few buckets; (3) Grafana's 24h+ querySplitting residual chunk (a tiny trailing range ≤ 2× step that mergeFrames glued onto the main series as a spike) is now suppressed on every stats path, not just the /hits path. Low-cardinality panels and non-Grafana callers are unaffected.