Add middleware to process semantic conventions

[ivomagi]

As discovered while analyzing the request from #1143, it became apparent that the current semantic conventions do lack the information about the middleware running on the runtime within the process. Adding this information makes the traces more valuable - the proposed change makes it possible to visualize that Tomcat 9.0.39 application server is sending messages to a Kafka message queue which in turn has a subscriber running on GlassFish 5.0 persisting the content to Druid 0.15.0. In the current form, it seems that the semantic conventions leave this information at the span level, which would be creating redundancy in outbound communication from the agent and/or not capture the information in unified manner, making it harder for Otel-compliant servers to visualize such traces.

Fixes #1143

Changes

Added middleware definition to the resource semantic conventions.

Related issues #

Related oteps #

[linux-foundation-easycla]

• ❌ ivomagi The commit (da4425d ,c5ef9a64e5659520f720763a95bf8d06d6a3198b ,c011cd53f638892a7aa4b117ebad7992c2a52e18 ,99a1ed799c72ee0f02cef27eb4e50b8646dc71da) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

[Oberon00]

Please do not add this to process. All resources are process-wide, the middleware is IMHO not really more closely related to the process than, say the faas.instance_id.

CC @tigrannajaryan (cf #882, #1137)

[kenfinnigan]

Is a database server middleware?

The means of communicating with a database server, such as JDBC, is middleware, but I'm not sure that the database server itself is.

[ivomagi]

Being new to Otel I might be on a wrong track here as the databases seem to be treated as downstream dependencies vs part of the trace itself. But there is no reason why there could not be agents monitoring the databases themselves and building spans out of the database as well. And for example with Druid and other JVM-based databases, it is rather straightforward as of now. Also the middleware definition lists databases as one of the members of middleware family.

[kenfinnigan]

Which "middleware definition" are you referring to?

[ivomagi]

I guess I was too optimistic when refering to it as "definition", as for example Wikipedia version of it does not count the database as middleware and maybe rightfully so. This however does not change the fact that many of the databases (Druid, Cassandra, Derby) do run on a Java runtime and could very well be part of the trace itself (vs downstream dependencies). That being said, maybe Otel does treat databases as not part of the trace at all or treats them special members within the trace?

[Oberon00]

I think midleware is not really what you mean anyway. It is used by many webframeworks to mean something different, e.g. https://docs.djangoproject.com/en/3.1/topics/http/middleware/ or https://aspnetmonsters.com/2016/03/2016-02-28-what-is-middleware-anyway/

[ivomagi]

There are numerous traces in the wild internet, trying to define middleware. And maybe I am off the best path here with the naming, but it does not seem to change the need to enhance the spans in the trace with the information about what kind of "middleware" they are running. This is also something that most existing (non-otel-compliant) vendors offer and having this clearly defined in Otel should greatly reduce the friction in migrating from an existing APM vendor to standards-based solutions.

[iNikem]

Please do not add this to process. All resources are process-wide, the middleware is IMHO not really more closely related to the process than, say the faas.instance_id.

CC @tigrannajaryan (cf #882, #1137)

faas.instance_id says something about outside the process, the environment in which this process runs. middleware says something about inside the process.

[Oberon00]

That is not the point. The resource namespaces should make the name unique, not describe some hierarchy. Also, many app web servers (is Apache a middleware?) use multiple processes.

[tigrannajaryan]

@ivomagi please sign the CLA.

[ivomagi]

• ❌ ivomagi The commit (da4425d ,c5ef9a64e5659520f720763a95bf8d06d6a3198b ,c011cd53f638892a7aa4b117ebad7992c2a52e18) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

I signed it

[tigrannajaryan]

That is not the point. The resource namespaces should make the name unique, not describe some hierarchy.

This is correct and has been our approach so far. Namespaces are not intended to describe a hierarchy of objects. Namespaces are intended to ensure global uniqueness of names.

There is no need to create nested namespaces just because objects also logically are nested, although it is of course not prohibited.

[anuraaga]

Conceptually these seem to live at the layer of InstrumentationLibrary - they aren't really constant to the app (one app can have multiple middlewares) but can still group some spans together. Does it make sense to add information about the instrumented library in that struct?

[Oberon00]

they aren't really constant to the app (one app can have multiple middlewares)

I had the impression this PR is about a different kind of "middleware" that is more or less an aditional runtime within the language runtime, such as an Java AppServer (you can have only one per JVM -- at least usually)

[ivomagi]

• ❌ ivomagi The commit (da4425d ,c5ef9a64e5659520f720763a95bf8d06d6a3198b ,c011cd53f638892a7aa4b117ebad7992c2a52e18) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

I signed it

I signed it

[ivomagi]

Conceptually these seem to live at the layer of InstrumentationLibrary - they aren't really constant to the app (one app can have multiple middlewares) but can still group some spans together. Does it make sense to add information about the instrumented library in that struct?

Apparently this is the path Steve Flanders also recommended to take in the conversation I was having on the exact time you provided this alternative. Will be closing the PR and will use the InstrumentationLibrary route to send the information.

[ivomagi]

Will be using the InstrumentationLibrary.name and InstrumentationLibrary.version attributes which would (in case the OTLP protocol is used) be converted in the collector to otel.library.name and otel.library.version and sent to server.

[Oberon00]

Conceptually these seem to live at the layer of InstrumentationLibrary

While this is true, the values are not suitable for use as InstrumentationLibrary. A new concept on the same layer would be needed, e.g. open-telemetry/oteps#78

[ivomagi]

Well, it seems the InstrumentationLibrary.name and InstrumentationLibrary.version is not at the moment used to send this information. It seems that at least Java and PHP agents use these properties to send the module instrumenting the span and the module version. Two examples:

• name: io.opentelemetry.auto.servlet version: 0.10.0-SNAPSHOT
• name: io.opentelemetry.auto.spring-webmvc 0.10.0-SNAPSHOT
  So in a sense it seems I am running in circles. For now giving up and sending what is needed as span attributes. Still think it makes a whole lot of sense to have means to send middleware information in standardized way, current approach will create a problems down the road (less valuable traces or incompatible vendors and redundancy)