Skip to content

Conversation

@hds
Copy link
Contributor

@hds hds commented Oct 24, 2025

Adding some usdt proboes to tracing-subscriber to try and debug an
issue with filtering via the EnvFilter not working correctly.

Probes have been added to the FilterState and Filtered structs.

To use this, you need to patch various tracing crates to use the ones from this branch:

[patch.crates-io]
tracing-core = { git = "https://github.com/tokio-rs/tracing.git", branch = "hds/usdt" }
tracing = { git = "https://github.com/tokio-rs/tracing.git", branch = "hds/usdt" }
tracing-subscriber = { git = "https://github.com/tokio-rs/tracing.git", branch = "hds/usdt" }

The application must call tracing_subscriber::register_probes() at
some point before the interesting probes will be executed, otherwise
they won't show up. This should be done before initializing the
Registry.

tracing_subscriber::register_probes()

Then you need to run dtrace or bpftrace (as root / with sudo) and have it invoke your application at /path/to/your/executable.

dtrace -qZs tracing_subscriber.d -c /path/to/your/executable

This is tracing_subscriber.d:

tracing_subscriber*:::filterstate-set {
    printf("probe=%s tid=%d id=%s enabled=%d\n", probename, tid, copyinstr(arg0), arg1);
}

tracing_subscriber*:::filterstate-did-enable {
    printf("probe=%s tid=%d id=%s enabled=%d\n", probename, tid, copyinstr(arg0), arg1);
}

tracing_subscriber*:::filterstate-add-interest {
    printf("probe=%s tid=%d interest=%s\n", probename, tid, copyinstr(arg0));
}

tracing_subscriber*:::filtered-register-callsite {
    printf("probe=%s tid=%d id=%s callsite=%p target=%s loc=%s filter_interest=%s\n", probename, tid, copyinstr(arg0), arg1, copyinstr(arg2),
            copyinstr(arg3), copyinstr(arg4));
}

tracing_subscriber*:::filtered-enabled {
    printf("probe=%s tid=%d id=%s callsite=%p filter_enabled=%d layer_enabled=%d\n", probename, tid, copyinstr(arg0), arg1, arg2, arg3);
}

tracing_subscriber*:::filtered-event-enabled {
    printf("probe=%s tid=%d id=%s callsite=%p filter_enabled=%d layer_enabled=%d\n", probename, tid, copyinstr(arg0), arg1, arg2, arg3);
}

tracing_subscriber*:::filtered-on-event {
    printf("probe=%s tid=%d id=%s callsite=%p enabled=%d\n", probename, tid, copyinstr(arg0), arg1,
            arg2);
}

tracing_subscriber*:::filtered-on-new-span {
    printf("probe=%s tid=%d id=%s callsite=%p span_id=%d enabled=%d\n", probename, tid,
            copyinstr(arg0), arg1, arg2, arg3);
}

tracing_subscriber*:::filtered-on-enter,tracing_subscriber*:::filtered-on-exit,tracing_subscriber*:::filtered-on-close {
    printf("probe=%s tid=%d id=%s span_id=%d enabled=%d\n", probename, tid,
            copyinstr(arg0), arg1, arg2);
}

The following bpftrace one-liner shows a bit of how to use it, but I haven't converted the dtrace script to a bpftrace one yet.

bpftrace -e 'usdt:/path/to/your/executable:tracing_subscriber:filtered-register-callsite { printf("%s\n", str(arg0)); }'

Adding some [usdt] proboes to `tracing-subscriber` to try and debug an
issue with filtering via the EnvFilter not working correctly.

Probes have been added to the `FilterState` and `Filtered` structs.

The application must call `tracing_subscriber::register_probes()` at
some point before the interesting probes will be executed, otherwise
they won't show up.

[usdt]: https://docs.rs/usdt
@hds hds requested review from a team and hawkw as code owners October 24, 2025 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants