Custom Backend Tracing

[jishminor]

Is your feature request related to a problem? Please describe.
It seems like there is no prescribed way to add additional custom information to triton traces in a custom backend. Is this correct?

Describe the solution you'd like
Ability to fetch parent ID of trace in backend, append extra information (in my case performance counter data), and write results to the TraceManager object built into the tritonserver executable implementation of Tritonserver, such that the extra trace data is generated and processed with the existing trace analysis tools for Triton.

Describe alternatives you've considered
It is pretty straight forward to add custom metrics as described in the documentation, however the metric types being either a rolling counter or gauge is a bit limiting when looking at performance counter data. Perf counter data would make more sense in the context of the Triton tracing infrastructure (ie request id X had Y L1_DCACHE misses etc).

My understanding may be incomplete, but although the ability to create traces is exposed through the tritonserver C API, it is only the tritonserver implementation in this repository that defines the TraceManager class, which seems designed to handle the actual capture and reporting of traces in Triton or OpenTelemetry format, and there is no way to access this from a custom backend.

[rmccorm4]

CC @oandreeva-nv for viz