[Feature] Trace Pipeline Support in BanyanDB

[wu-sheng]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

🌟 Feature Request: Trace Pipeline Support in BanyanDB (Target: v0.11)

Summary

Introduce a trace pipeline framework into BanyanDB to enable trace-oriented processing, sampling, and enrichment directly within the database layer.
The pipeline will serve as the foundation for intelligent, context-aware trace handling—allowing dynamic update, analysis, and retention decisions while preserving schema and performance guarantees.

---

Background

BanyanDB currently supports efficient ingestion and querying of trace data, but processing is span-oriented and lacks capabilities for post-ingestion, trace-level operations.

Modern observability scenarios—such as tail-based sampling, cross-service trace enrichment, or behavioral trace analysis—require the database to treat a group of spans (sharing the same trace ID) as one logical unit.

Introducing a trace pipeline provides a structured mechanism to:

• Analyze and modify trace data after ingestion.
• Aggregate spans consistently, regardless of their reported order or source.
• Make smarter data retention decisions based on complete trace context.

---

Goal

Deliver a trace-level pipeline mechanism that:

1. Processes collected spans as unified trace entities in best-effort strategy(No 100% guaranteed).
2. Supports runtime enrichment and analysis of existing tag values, further considering to declear trace-level schema, e.g. trace latency.
3. Maintains persistent file-based processor context across ingestion batches for continuity and scale.
4. Enables tail-based sampling via a reserved deletion/retention flag.
5. Integrates cleanly with OAP for schema declaration, dynamic loading, and test validation.

Use case

This issue includes several key features and use cases, mostly around trace post-analysis, e.g., tail sampling.

Most things were discussed between @hanahmily and me at a high level. No detail is confirmed yet.

Related issues

No response

Are you willing to submit a pull request to implement this on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct