llm-trace

Span tracing for LLM pipelines: where did the time, tokens and money go? One header, llm_trace.hpp. Needs nothing beyond the C++17 standard library.

A single user request in an LLM app often fans out into retrieval, reranking, one or more model calls and post-processing. llm-trace records each step as a span with timing, parent and child links, model, token counts and cost, then prints a readable tree or exports JSON in the OpenTelemetry (OTLP) span layout.

Features

• RAII spans: Tracer::start(name) returns an ActiveSpan that ends when it leaves scope
• Child spans via span.child(name), linked by trace id and parent span id
• LLM attributes on each span: model, input and output tokens, cost, error, plus arbitrary key/value attributes
• LLM_TRACE("name") macro to trace a whole function in one line
• export_text() for humans, export_json() in an OTLP-style resourceSpans layout
• Global tracer or your own instances; enable() / disable() at runtime
• Standard library only

Quick start

Copy the header into your project:

curl -fsSLO https://raw.githubusercontent.com/Mattbusel/llm-trace/main/include/llm_trace.hpp

Define LLM_TRACE_IMPLEMENTATION in exactly one .cpp file before including it; every other file just includes the header. Save this as main.cpp next to the header:

#define LLM_TRACE_IMPLEMENTATION
#include "llm_trace.hpp"
#include <iostream>

void retrieve() {
    LLM_TRACE("retrieve");            // RAII span covering the whole function
    /* vector search ... */
}

int main() {
    auto& tracer = llm::Tracer::global();
    {
        auto root = tracer.start("answer_question");
        retrieve();

        auto gen = root.child("generate");
        gen.set_model("gpt-4o-mini");
        gen.set_tokens(812, 164);
        gen.set_cost(0.00022);
        gen.set_attribute("temperature", "0.2");
    }   // spans end when they go out of scope

    std::cout << tracer.export_text() << "\n";
    std::cout << tracer.export_json() << "\n";   // OTLP-style resourceSpans JSON
}

g++ -std=c++17 -O2 main.cpp -o demo

API at a glance

Call	Purpose
Tracer::global() / Tracer()	Shared or local tracer
start(name)	Begin a root span
ActiveSpan::child(name)	Begin a nested span
set_model / set_tokens / set_cost / set_error / set_attribute	Annotate a span
export_text() / export_json() / completed_spans() / clear()	Read results

Notes and limitations

• Spans are kept in memory until you export and clear() them; nothing is sent over the network. Post the JSON to your collector yourself if you use one.
• LLM_TRACE declares a fixed variable name, so use it at most once per scope.

Build the examples

The repo builds examples/basic_trace.cpp, examples/span_hierarchy.cpp, examples/macro_trace.cpp, examples/json_export.cpp with CMake:

cmake -B build
cmake --build build

Part of llm-cpp

llm-trace is one of 26 single-header C++ libraries in llm-cpp, a toolkit for building LLM features into native code. Each library stands alone; combine them by giving each *_IMPLEMENTATION define its own .cpp file. See the llm-cpp README for the full list and examples of using several together.

License

MIT. See LICENSE.