Project Tracking: Instrumentation Stability + Semantic Conventions Working Group

[jsuereth]

Description

Solve the technical needs required for Semantic Conventions and Instrumentation Stability, as outlined in this proposal.

Project Board

Pending TC Approval & Sufficient staffing/momentum

Deliverables

At the highest level, the group will be delivering (or improving existing status-quo of) the following:

• Telemetry Definition: Ensuring a common machine-readable format for defining signals produced in instrumentation, and associated tooling for human usage and development.
• Telemetry Stability: Improving compatibility definition beyond just addition of signals and clarifying x-signal compatibility of attributes for Resource, Instrumentation Scope and
• Telemetry Evolution: Expanding existing telemetry schema features and unifying with Definition/Stability work.
• Semantic Conventions: Outlining a repeatable process for marking semantic conventions as stable that allows domain experts to drive key decision.

See the proposal for details

Staffing / Help Wanted

This project is currently in proposal mode. We'd like to leverage this issue to advertise and attract interested parties.

Required staffing

Projects cannot be started until they the following participants have been identified:

• @jsuereth Project Lead (+ TC member)
• @reyang Second TC sponsor
• @tigrannajaryan schema-file change reviewer / contributor
• Need: Signal experts (metrics/logs/traces) willing to work on data-model + signal compatibility
• Need: Engineers willing to write prototypes / work on tooling
  • Semconv buid-tools:
    • RST/MD docgen
    • Jinja / code-gen (x-language)
    • Go / OTel Collector code-gen
    • Integration w/ language build tooling
• @lmolkova signal expert + tooling contributor
• @Mpdreamz signal expert + tooling contributor
• @trask java codegen contributor
• Maintainers willing to approve prototypes
  • @cijothomas .NET Maintainer + Prototype approver
  • @lalitb C++ Maintainer + Prototype approver
  • @lzchen Python Maintainer + Prototype approver
  • @MSNev JavaScript Approver

Meeting Times

Once a project is started, the working group should meet regularly for discussion. These meeting times should be posted on the OpenTelemetry public calendar.

Timeline

This project is feature-driven vs. time driven. However, given the press of semantic convention CLs, an improvement on telemetry stability + definition are needed within 2022. Additionally, any timeline will need to reflect overall staffing + interest across the community. An aggressively optimistic road map:

• 2022 Q4
  • Improve specification around telemetry stability, focusing on x-signal issues.
  • A unified telemetry definition format with documentation + codegen tools.
• 2023 Q1
  • Unification of Telemetry Schema + Telemetry definition
  • First semantic convention marked stable
• 2023 Q2
  • Expansion of Telemetry Evolution
  • Repeatable process for semantic convention definitions
  • Incremental improvements to build tools

Labels

We propose to repurpose existing labels in addition to adding two more:

• area:semantic-conventions: This now denotes expert working group issues around defining semantic conventions (the process, not the tooling). Additionally, this is (and should continue) to be used in tandem with "semconv:expert-area" labels.
• area:telemetry-tooling: Issues and request around the semconv codegen or docgen and related tooling (go's own build-tool, e.g) belong here or directly in the appropriate repository.
• area:telemetry-stability: This denotes issues and concerns around defining telemetry-schema, available migrations, the metadata format for specifying telemetry, etc.

The later two labels would be the immediate focus of this project and working group.

Linked Issues and PRs

Blocked

• Client Instrumentation Project: Project Tracking: Client Instrumentation #2734 and related efforts.
• Stabilizing service.name #1497
• Support Elastic Common Schema in OpenTelemetry oteps#199
• Fixed Schema Telemetry Producers can't be declared stable #2606

Unresolved issues to tackle by this project

• Clarify unit annotation format for metric semantic conventions #2719
• Added Code links to spans as an approved convention #2687
• Semantic conventions for Instrumentation Scope Attributes #2682
• Metrics Semantic Conventions not specified as YAML files #2666
• Library-specific metric semantic conventions #2610
• Record-level attribute semantic conventions common for all telemetry signal types #2488
• Decide on how we organize and interpret semconv #2475
• Refactor semantic conventions structure #2397
• Are invalid SpanContexts allowed as Links? Needs spec clarification. #2176
• Optional Metric Attributes #2077
• Migrate metric semantic conventions into YAML #2001
• Establish guidelines for semantic conventions names #1504
• Move semantic conventions that are applicable to the Resource from "trace" spec to a common area #1367
• Update Metrics Semantic Conventions README #1084
• Indicate which semantic conventions generate errors. #1003
• Proposal: Dictionary of common Attribute/Label definitions #855
• Standardization of Units needed #705
• Define what required means for a semantic convention #653
• Semantic convention namespacing recommendations #585
• Investigate and submit an OTEP on how other semantic conventions can be adopted #583
• Support map values and nested values for attributes #376

TODO(jsuereth): Attach relevant tooling/semconv/stability PRs

[tigrannajaryan]

@open-telemetry/instr-wg FYI.

[tigrannajaryan]

@jsuereth An OTEP that is probably relevant to be in the Linked Issues list: open-telemetry/oteps#199

[lmolkova]

This is another relevant issue:

• Fixed Schema Telemetry Producers can't be declared stable #2606

[lmolkova]

I'd love to help with

• data-model + signal compatibility
• Semconv buid-tools

[jsuereth]

Thanks @lmolkova!!! I've added you to the staffing section.

[cijothomas]

I am a maintainer for .NET SIG, and can sign up for reviewing any prototypes in .NET.

[cijothomas]

I am a maintainer for .NET SIG, and can sign up for reviewing any prototypes in .NET.

Slight correcting : As maintainer, I am anyway expected to review all PRs. What I meant to say is - I can prioritize reviewing any PRs directly contributing towards Semantic Convention/Instrumentation stability

[jsuereth]

Thanks @cijothomas !!! added.

[tigrannajaryan]

I'd like to help/continue working on Schema File related stuff. I am not sure how much time I can allocate to this, but at least I want to review all relevant PRs, even if I cannot submit them myself.

[reyang]

I've put myself as the Second TC Sponsor and assigned this issue to myself.

[jsuereth]

@tigrannajaryan Understood. Added you as specifically interested in schema-files. Appreciate the caveat on time and definitely don't want you to deviate from your current responsibilities here.

@reyang Thanks much for signing up as sponsor!

[trask]

I'd love to be involved with this and can sign up to write any codegen targeting Java (whether the codegen itself is written in Java or not)

[lzchen]

I am a Python maintainer and would be interested in reviewing prototypes in Python.

[MSNev]

I can sign up as well, I'm a JS approver and a JS Web sandbox Maintainer and a member of the Client RUM Sig

[lalitb]

I am a C++ maintainer, and if it helps, I can be a member to review/approve prototypes in C++.

[Mpdreamz]

I can sign up as well for the following parts:

• Signal experts (metrics/logs/traces) willing to work on data-model + signal compatibility

Especially in the context of open-telemetry/oteps#199 can help with classification of which fields work well for different types signals and backends

• Engineers willing to write prototypes / work on tooling

I've lead similar specification work to drive codegen @ Elastic, can definitely help scope efforts.
Most likely won't have time to actually put my hands on the keyboard to spin out a prototype though.

[reyang]

@Mpdreamz thanks! I've updated the issue description by adding your name. Which programming language would you prefer for prototyping?

[jsuereth]

Thanks to all folks who are interested. Please fill out this doodle of your preferred meeting time, ideally before friday as I'll be scheduling our first WG call next week.

Note: We are only selecting the timeslot not the specific date. The goal is for this meeting to happen every other week until the task list is complete.

[hughesjj]

I'm currently working to develop some go receivers and have prior experience with jinja+python, would love to join the committee.

[jsuereth]

For folks working on semantic conventions. Here's a set of guidelines for avoiding what we expect are breaking changes:

• Attributes defined for traces/logs that may be shared w/ metrics will likely have cardinality restrictions applied to them going forward.
• Attributes/Resource definitions will likely be rejected until we have better refinement on Resource/Entity (see Refine which attributes of Resource contribute to Metric Identity. #2775)
• Metrics definitions that mirror existing structure a likely to be ok (i.e. the report against a similar resource as today, the have similar labels). There may be changes to what RESOURCE a metric reports against going forward.
• There may be some changes that occur to match (or have simple mapping to/from) elastic common schema. This is less likely to affect metrics, but may affect traces/logs/event semantics.
• At some point, uniformity / machine-readable format will be required for all semantic conventions for all signals. Any table for metric/logs should match existing structure or update ALL tables with new fields + justification of the new field. Be warned, this means additional scrutiny and may be worth discussing separately from the semconv change itself.

[lsl-2020]

Hi experts, could anyone share the latest updates on this project please? My team is waiting for the official version of instrumentation libraries to unblock roll out to production. Thanks!

[dmitryax]

I can join the effort as a code owner of the collector metrics metadata format and the metrics builder:

Go / OTel Collector code-gen

BTW I recently proposed guidelines and stability guarantees for that part of the collector, in case anyone is interested: open-telemetry/opentelemetry-collector#6358

[joaopgrassi]

@lsl-2020 you can check the meeting notes to see what was discussed in the last meeting: https://docs.google.com/document/d/10xG7DNKWRhxNmFGt3yYd3980a9uwS8lMl2LvQL3VNK8/edit

[austinlparker]

Closed as no longer relevant due to refactoring of semconv SIGs