*: Add embedded DuckDB storage backend

[thorfour]

Summary

Adds DuckDB as a second storage backend behind a new --storage-backend={clickhouse,duckdb} flag (default clickhouse). DuckDB is embedded — single file or in-memory, no separate server. Useful for single-node deployments where running a ClickHouse server is overkill.

The architecture refactor in the FrostDB removal made this a clean drop-in: profilestore.Ingester and query.Querier are the only interfaces a backend must satisfy.

Stacks on #6356.

What's in the package

pkg/duckdb/ mirrors pkg/clickhouse/ (~2k LoC, comparable shape):

• client.go — database/sql connection. Empty path → in-memory; non-empty path → file (created if missing). Pinned to one connection (DuckDB is single-writer per process).

• schema.go — DDL using DuckDB-native composite types:

  • labels: MAP(VARCHAR, VARCHAR)
  • stacktrace: STRUCT(address, mapping_*, line/function info)[] (LIST)

  Differs from the ClickHouse layout (parallel arrays in Nested, JSON labels) but maps cleanly onto the same row shape the ingester produces.

• ingester.go — Arrow record → Appender API. Decodes the encoded location blobs into struct-shaped maps for the LIST<STRUCT> column; labels populate as duckdb.Map.

• filter.go — Prometheus label-matcher → DuckDB SQL. Uses element_at(labels, 'k')[1] for label access (NULL when key absent) and regexp_matches for regex matchers.

• querier.go — full Querier interface. SQL translations from ClickHouse to DuckDB:

  ClickHouse	DuckDB
  arrayJoin(...)	UNNEST(...) + scalar subquery
  JSONAllPaths(map)	map_keys(...) + UNNEST
  intDiv(a, b)	(a / b)::BIGINT
  groupArray(tuple(...))	LIST({...} ORDER BY)
  SUM(BIGINT)	SUM(...)::BIGINT (downcast HUGEINT)

  Stacktrace results scan into duckdb.Composite[[]stacktraceLoc] via field-name mapping — much cleaner than CH's parallel-array dance.

• duckdb_test.go — round-trip smoke test: builds a synthetic Arrow record, ingests, then calls every Querier method (HasProfileData, ProfileTypes, Labels, Values, QueryRange, QuerySingle, QueryMerge, GetProfileMetadataMappings, GetProfileMetadataLabels) and asserts.

Wiring

pkg/parca/parca.go branches on flags.StorageBackend to wire either the ClickHouse or DuckDB ingester + querier. Shutdown collapses to a single closeBackend func. README regenerated for the new flags.

go.mod

• + github.com/marcboeker/go-duckdb/v2 v2.4.3. The project moved to github.com/duckdb/duckdb-go at v2.5.0+; will migrate when that release stabilises.
• DuckDB requires CGo. Cross-compile needs CC=... CGO_ENABLED=1. Pre-built bindings are pulled in for darwin/linux × amd64/arm64 + windows/amd64 via github.com/duckdb/duckdb-go-bindings.

Test plan

• go build ./...
• go vet ./...
• go test -short ./... — green, including the new pkg/duckdb round-trip test
• CI green
• Manual smoke: parca --storage-backend=duckdb --duckdb-path=/tmp/parca.db and ingest a profile end-to-end
• Bench DuckDB ingest throughput against expected scrape rates

Follow-ups

• Migrate import to github.com/duckdb/duckdb-go once v2.5+ is the canonical
• testcontainers-based integration tests (DuckDB doesn't need a container, but the same harness shape applies)
• Goreleaser CGo cross-compile matrix verification

🤖 Generated with Claude Code

[alwaysmeticulous]

✅ Meticulous spotted 0 visual differences across 288 screens tested: view results.

Meticulous evaluated ~4 hours of user flows against your PR.

_{Expected differences? Click here. Last updated for commit 6310c95 *: Add embedded DuckDB storage backend. This comment will update as new commits are pushed.}

[guettlibot]

Hi @thorfour — I came at this from the other end: I opened polarsignals/frostdb#1006 (a benchmark showing the default FrostDB backend's small-window queries scale with total block count, because it opens every block's Parquet metadata regardless of the query window). On a self-hosted single-node Parca that's the whole problem — retained data, not window size, drives latency.

This PR is exactly the fix I was hoping for. I benchmarked the idea on my real data (181M rows over 13.7h; CPU profile = 24M rows): a time-pruned aggregation runs in well under 1–2s in DuckDB, where Parca's current path times out on a 6h window. Full write-up + numbers: guettli/parcareport#31. DuckDB reads Parca's existing block Parquet fine (arrow-rs/DataFusion won't, FWIW — a row-group-ordinal metadata quirk), and I like that #6357 stores the stacktrace as LIST<STRUCT> so the symbolized table works directly.

Two honest questions on state:

1. This PR (and the FrostDB-removal stack it sits on, *: Phase 1 of FrostDB removal — ClickHouse-only cmd path #6353–*: Phases 3+4 — drop FrostDB-backed tests, rip frostdb out of go.mod #6356) has been open and untouched since May. Is that series still the intended path for OSS Parca storage, or is it parked?
2. I notice the storage energy has since moved to Vortex — there's a polarsignals/vortex fork and that seems to be where your focus is now. Totally understandable. Does that change the plan for OSS Parca? i.e. is the OSS story still "FrostDB → ClickHouse, with this embedded-DuckDB option for single-node," or is OSS Parca heading toward a Vortex-based format too?

And the practical one, on behalf of self-hosters:

What should users of Parca run today for decent single-node query performance? The default FrostDB is slow at any real retention; the ClickHouse backend (#6194) means operating a separate server; iceberg is experimental; and this DuckDB PR isn't merged. A one-line steer — "use X today, Y is coming" — would help a lot of self-hosted operators.

Happy to test or rebase #6357 against real workloads if that's useful.