Add SQL over logs (kdb.logs / kdb.stats)

[George-Payne]

Exposes a node's own CLEF log files as two queryable tables, kdb.logs and kdb.stats, over the existing Arrow Flight SQL surface in the SecondaryIndexing plugin. Read-only and node-local: a query only sees the log files on the node it connects to.

• LogViews creates kdb.logs / kdb.stats as per-query TEMP views over the node's log directory via read_json_objects, with a typed zero-row fallback when no files match (read_json errors on an empty set / fresh node). The main-log predicate excludes log-* siblings so it holds for any RollingInterval, including the undated log.json.
• render_message scalar function renders CLEF @mt templates, taking format-specifier tokens from the @r renderings and falling back to the JSON properties. It degrades to the raw template rather than failing the whole query on a malformed line.
• QueryEngine rewriter maps the kdb schema's logs / stats tables to the views and creates them on the executing connection per query, alongside the existing records mapping.
• Docs add a "Querying node logs" section to the Flight SQL page (both tables, their columns, and example queries) and a cross-link from the Database logs page.
• Tests cover the view projection and rendering (LogViewsTests, unit) and the end-to-end Flight SQL surface (LogsFlightSqlTests, integration): every level, message rendering, cross-view UNION, time-bounded queries, and rejection of non-SELECT / unqualified-view / unknown-table access.

[cloudflare-workers-and-pages]

Deploying eventstore with    Cloudflare Pages

Latest commit:	e979de6
Status:	✅  Deploy successful!
Preview URL:	https://2c8afb79.eventstore.pages.dev
Branch Preview URL:	https://logs-query.eventstore.pages.dev

View logs

[qodo-code-review]

PR Summary by Qodo

Expose node CLEF logs as SQL tables (kdb.logs / kdb.stats)

✨ Enhancement 📝 Documentation 🧪 Tests 🕐 40+ Minutes

AI Description

• Add read-only, node-local SQL access to CLEF logs via kdb.logs and kdb.stats.
• Create per-query TEMP views over on-disk log files with safe empty-set fallback.
• Render Serilog message templates via render_message, plus docs and integration coverage.

Diagram

sequenceDiagram
  participant Client as Flight SQL Client
  participant QE as QueryEngine
  participant RW as Rewriter
  participant LV as LogViews
  participant DB as DuckDB
  participant FS as Log Files
  Client->>QE: SELECT ... FROM kdb.logs / kdb.stats
  QE->>RW: Parse + rewrite table refs
  RW-->>QE: Rewritten SQL + HasLogs/HasStats flags
  QE->>LV: Create TEMP views (if flags set)
  LV->>FS: Enumerate matching log*.json
  LV->>DB: CREATE OR REPLACE TEMP VIEW __logs/__stats
  QE->>DB: Execute rewritten SQL
  DB-->>Client: Arrow result set

Loading

High-Level Assessment

Per-query TEMP views over read_json_objects is the right fit here: it naturally tracks log rotation, avoids startup failures when no files exist, and keeps the surface read-only. A persistent ingested log table/materialization would add ingestion/retention complexity and lose the “reflect what’s on disk now” behavior; static views can’t handle the empty-file-set error case.

Files changed (11) +623 / -3

Enhancement (5) +214 / -3

LogViews.csCreate TEMP views over node CLEF log files +73/-0

Create TEMP views over node CLEF log files

• Implements 'LogViews' to discover main vs stats CLEF files under the node’s log directory and to create '__logs'/'__stats' TEMP views using 'read_json_objects'. Includes a typed zero-row fallback so fresh nodes / disabled logging don’t error.

src/KurrentDB.SecondaryIndexing/LogsQuery/LogViews.cs

RenderMessageFunction.csAdd render_message scalar function for CLEF @mt templates +111/-0

Add render_message scalar function for CLEF @mt templates

• Adds a DuckDB scalar function to render Serilog/CLEF message templates. Uses Serilog’s template parser, prefers '@r' renderings for formatted tokens, falls back to JSON properties, and degrades to the raw template on malformed input; registers once via 'RenderMessageSetup'.

src/KurrentDB.SecondaryIndexing/LogsQuery/RenderMessageFunction.cs

QueryEngine.PreparedQuery.csTrack log/stat table usage in prepared query flags +18/-2

Track log/stat table usage in prepared query flags

• Extends the prepared-query header to encode 'HasLogs' and 'HasStats' flags so execution-time setup can create the appropriate TEMP views.

src/KurrentDB.SecondaryIndexing/Query/QueryEngine.PreparedQuery.cs

QueryEngine.Rewriter.csRewrite kdb.logs/kdb.stats to internal view names +7/-0

Rewrite kdb.logs/kdb.stats to internal view names

• Maps 'kdb.logs' -> '__logs' and 'kdb.stats' -> '__stats' during AST rewriting and sets the corresponding prepared-query flags.

src/KurrentDB.SecondaryIndexing/Query/QueryEngine.Rewriter.cs

QueryEngine.csCreate log/stat views on the executing DuckDB connection +5/-1

Create log/stat views on the executing DuckDB connection

• Injects 'LogViews' and creates '__logs'/'__stats' on the per-query executing connection (both for execution and schema reflection), based on prepared-query flags.

src/KurrentDB.SecondaryIndexing/Query/QueryEngine.cs

Tests (2) +333 / -0

LogsFlightSqlTests.csIntegration tests for Flight SQL log querying +143/-0

Integration tests for Flight SQL log querying

• Adds end-to-end tests that seed CLEF files into a temp log directory, validate querying 'kdb.logs' and 'kdb.stats' (including UNION and time windows), and assert rejection of unknown tables, unqualified internal views, and non-SELECT statements.

src/KurrentDB.SecondaryIndexing.Tests/IntegrationTests/LogsFlightSqlTests.cs

LogViewsTests.csUnit tests for LogViews and message rendering behavior +190/-0

Unit tests for LogViews and message rendering behavior

• Validates view population, file-selection rules (including undated log.json), typed projections, empty-directory behavior, and 'render_message' template edge-cases (format tokens, alignment, escaping, missing properties, malformed renderings).

src/KurrentDB.SecondaryIndexing.Tests/LogsQuery/LogViewsTests.cs

Documentation (2) +67 / -0

logs.mdAdd link to querying logs via Flight SQL +6/-0

Add link to querying logs via Flight SQL

• Adds a short section pointing users to the Flight SQL documentation for 'kdb.logs'/'kdb.stats', emphasizing read-only and node-local semantics.

docs/server/diagnostics/logs.md

flightsql.mdDocument kdb.logs/kdb.stats columns and example queries +61/-0

Document kdb.logs/kdb.stats columns and example queries

• Introduces a 'Querying node logs' section describing both tables, their node-local behavior, projected columns, and example queries for errors, volume, text search, and stats JSON extraction.

docs/server/features/queries/flightsql.md

Other (2) +9 / -0

KurrentDB.SecondaryIndexing.csprojEnable unsafe scalar UDF implementation and add Serilog parser dependency +2/-0

Enable unsafe scalar UDF implementation and add Serilog parser dependency

• Turns on 'AllowUnsafeBlocks' (for DuckDB string marshaling) and adds a Serilog package reference used for message-template parsing.

src/KurrentDB.SecondaryIndexing/KurrentDB.SecondaryIndexing.csproj

SecondaryIndexingPlugin.csWire LogViews + render_message setup into SecondaryIndexingPlugin DI +7/-0

Wire LogViews + render_message setup into SecondaryIndexingPlugin DI

• Registers the one-time DuckDB setup for 'render_message' and adds a singleton 'LogViews' rooted at the node’s configured log directory (component-specific).

src/KurrentDB.SecondaryIndexing/SecondaryIndexingPlugin.cs

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Log file enumeration can throw ✓ Resolved 🐞 Bug ☼ Reliability

Description

LogViews.MatchingFiles() calls Directory.EnumerateFiles() without handling IO/permission
exceptions, so a transient filesystem/permission issue can fail the whole query instead of returning
empty kdb.logs/kdb.stats. This reduces reliability for a feature intended to be query-safe even
on fresh/misconfigured nodes.

Code

src/KurrentDB.SecondaryIndexing/LogsQuery/LogViews.cs[R34-43]

+	private IReadOnlyList<string> MatchingFiles(Func<string, bool> match) {
+		if (!Directory.Exists(logsDir))
+			return [];
+
+		var files = new List<string>();
+		foreach (var path in Directory.EnumerateFiles(logsDir, "log*.json"))
+			if (match(Path.GetFileName(path)))
+				files.Add(path);
+
+		return files;

Evidence

MatchingFiles() only checks Directory.Exists() and then immediately enumerates files; there is
no exception handling around the enumeration loop.

src/KurrentDB.SecondaryIndexing/LogsQuery/LogViews.cs[34-43]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`LogViews.MatchingFiles()` checks `Directory.Exists()` but does not guard `Directory.EnumerateFiles()`; enumeration can still throw (e.g., `UnauthorizedAccessException`, `IOException`) when the directory exists but becomes unreadable or is raced.

### Issue Context
A core design goal in `LogViews` is “empty views without error” for common edge cases; unexpected enumeration failures currently violate that.

### Fix Focus Areas
- src/KurrentDB.SecondaryIndexing/LogsQuery/LogViews.cs[34-44]

Suggested change:
- Wrap the enumeration in `try/catch` for `IOException`/`UnauthorizedAccessException` (and potentially `DirectoryNotFoundException`), and return an empty list on failure (or degrade to the typed empty view source). Optionally add a debug log if a logger is available.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Formatted token alignment skipped 🐞 Bug ≡ Correctness

Description

render_message() appends the pre-rendered @r value for formatted tokens without applying
token.Alignment, so templates like {x,5:N0} will render without expected padding. This makes the
message column disagree with what Serilog would render for aligned+formatted tokens.

Code

src/KurrentDB.SecondaryIndexing/LogsQuery/RenderMessageFunction.cs[R60-66]

+					case PropertyToken { Format: not null } formatted:
+						if (renderings is { } r && renderingIndex < r.GetArrayLength() && r[renderingIndex].ValueKind is JsonValueKind.String)
+							output.Append(r[renderingIndex].GetString());
+						else
+							AppendProperty(output, root, formatted);
+						renderingIndex++;
+						break;

Evidence

The formatted-token @r fast-path appends the rendered string directly, while the only alignment
handling exists in AppendProperty(), so alignment is skipped when @r is used.

src/KurrentDB.SecondaryIndexing/LogsQuery/RenderMessageFunction.cs[55-66]
src/KurrentDB.SecondaryIndexing/LogsQuery/RenderMessageFunction.cs[87-103]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`RenderMessageFunction.Render()` uses the `@r` array fast-path for formatted tokens, but it appends the string directly and never applies `PropertyToken.Alignment`. Alignment is only applied inside `AppendProperty()`, so aligned+formatted tokens render incorrectly.

### Issue Context
This affects `kdb.logs.message`/`kdb.stats.message` for any CLEF entry whose `@mt` includes both alignment and format specifier.

### Fix Focus Areas
- src/KurrentDB.SecondaryIndexing/LogsQuery/RenderMessageFunction.cs[55-103]

Suggested change:
- In the `PropertyToken { Format: not null }` branch, when taking the `@r` string path, apply the same alignment logic as `AppendProperty()` before appending (pad left/right based on `token.Alignment.Direction` and `token.Alignment.Width`).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. logViews.Create() positional booleans ✓ Resolved 📘 Rule violation ⚙ Maintainability

Description

logViews.Create(connection, parsedQuery.HasLogs, parsedQuery.HasStats) passes boolean arguments
positionally, reducing readability and increasing the risk of swapping arguments as the call
evolves. The compliance checklist requires named boolean arguments at call sites for clarity.

Code

src/KurrentDB.SecondaryIndexing/Query/QueryEngine.cs[58]

+			logViews.Create(connection, parsedQuery.HasLogs, parsedQuery.HasStats);

Evidence

PR Compliance ID 8 requires boolean arguments to be passed with named arguments at call sites for
clarity. In QueryEngine, logViews.Create(...) is invoked with two positional boolean arguments
in multiple places, rather than logs: ... / stats: ....

CLAUDE.md: Naming conventions: versioned class suffix, accurate parameter names, and named boolean arguments at call sites
src/KurrentDB.SecondaryIndexing/Query/QueryEngine.cs[58-58]
src/KurrentDB.SecondaryIndexing/Query/QueryEngine.cs[117-117]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`LogViews.Create(DuckDBAdvancedConnection connection, bool logs, bool stats)` is called with positional boolean arguments, which hurts readability.

## Issue Context
The compliance checklist requires named boolean arguments at call sites to make intent self-documenting.

## Fix Focus Areas
- src/KurrentDB.SecondaryIndexing/Query/QueryEngine.cs[58-58]
- src/KurrentDB.SecondaryIndexing/Query/QueryEngine.cs[117-117]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Temp log dir not cleaned ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

LogsQueryFixture creates a unique temp log directory but never deletes it, so integration test
runs can leak directories/files under the system temp path. Over time (especially in CI), this can
accumulate and waste disk space.

Code

src/KurrentDB.SecondaryIndexing.Tests/IntegrationTests/LogsFlightSqlTests.cs[R18-25]

+public sealed class LogsQueryFixture : SecondaryIndexingEnabledFixture {
+	public LogsQueryFixture() {
+		var logsRoot = Path.Combine(Path.GetTempPath(), $"logs-query-{Guid.NewGuid():N}");
+		Directory.CreateDirectory(logsRoot);
+		Configuration = new Dictionary<string, string?>(Configuration!) {
+			[$"{KurrentConfigurationKeys.Prefix}:Logging:Log"] = logsRoot,
+		};
+	}

Evidence

The fixture creates logsRoot and points Logging:Log at it, but no teardown removes it; base
fixture teardown only targets the DB directory.

src/KurrentDB.SecondaryIndexing.Tests/IntegrationTests/LogsFlightSqlTests.cs[18-25]
src/KurrentDB.SecondaryIndexing.Tests/Fixtures/SecondaryIndexingFixture.cs[41-60]
src/KurrentDB.SecondaryIndexing.Tests/Fixtures/SecondaryIndexingFixture.cs[147-156]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`LogsQueryFixture` creates a temp `logsRoot` directory for seeding CLEF files but does not register teardown cleanup.

### Issue Context
`SecondaryIndexingFixture` only cleans up the database directory via `OnTearDown`; the extra logs directory is outside that path.

### Fix Focus Areas
- src/KurrentDB.SecondaryIndexing.Tests/IntegrationTests/LogsFlightSqlTests.cs[18-26]
- src/KurrentDB.SecondaryIndexing.Tests/Fixtures/SecondaryIndexingFixture.cs[41-60]

Suggested change:
- Store `logsRoot` in a private field.
- In the `LogsQueryFixture` constructor, capture the existing `OnTearDown` delegate and set a new `OnTearDown` that `await`s the previous teardown and then deletes `logsRoot` (e.g., using `DirectoryDeleter.TryForceDeleteDirectoryAsync(logsRoot, retries: 10)`).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---