Move README status table to separate machine-readable file

[mx-psi]

Component(s)

No response

Describe the issue you're reporting

All components must have a status table describing their stability, supported pipeline types, available distributions and warnings (see #19172 for the latter). This is useful for end-users to understand how to use a component.

Currently, this table is on each component's README file which makes it difficult to ingest this information. I propose moving this table to a separate machine readable file so that

• we can validate its contents (e.g. validate presence of required fields, validate their values, in the future try to match the metadata with the actual code)
• it's easier to build automated tooling around it, for example to
  • include status information on the component registry,
  • help the community understand what components are available on contrib and what their capabilities are
  • automatically validate OCB manifests (e.g. forbid components with a given warning or below a given stability level)
  • in the future, help validate that a component can be added to the registry even if not living on contrib

We can still generate the README table from the machine-readable file by placing a comment marking the position of the table.

To kickstart this we would need to:

• Define a schema for the component's status table
• Create a tool to generate README table from the machine-readable file.
• Add status metadata to all components #21213
• Consider adding requirement to use metadata.yaml in new components #20908

Future work may include

• Validation of code against schema (e.g. supported pipeline types or stability)
• Adding information other than status to this file (e.g. description or statefulness)

[djaglowski]

Taking this a step further, wouldn't it make sense to have such a machine readable file for all standardized parts of the readme?

For example:

class: processor
type: transform
description: modifies telemetry based on configuration using the [OpenTelemetry Transformation Language](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl)
stability:
  traces: beta
  metrics: alpha
  logs: alpha
distributions: [contrib]
stateful: true # See https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/18878
warnings: # See https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/19172
-  unsound_transformations
-  identity_conflict
-  orphaned_telemetry

[TylerHelmuth]

Something like this would also be very useful for open-telemetry/opentelemetry-collector#7274

[TylerHelmuth]

And I'll argue that Stateful fits into Warnings and doesn't need its own section to indicate Stateless. But I'm not willing to die on that hill.

[sh0rez]

happy to look into this if no one else already does!

[codeboten]

I started looking at adding this to metadata.yaml a little while ago. I'd suggest we start incrementally with just the status. It would be great to use this information in the code as well, rather than having to set this in multiple places.

[atoulme]

I support this.

[mx-psi]

I added a couple items to 'future work' on the original post including @djaglowski and @codeboten's suggestions. I agree with @codeboten that we can start with just the status and work from there.

I started looking at adding this to metadata.yaml a little while ago.

@codeboten one open question for me if we re-use metadata.yaml is what to do with components like the hostmetrics receiver that do not have a single metadata.yaml file but rather one per scraper. It sounds to me like we should modify the metadata.yaml schema for this to work. Do you have any thoughts on that?

happy to look into this if no one else already does!

@sh0rez Sure :) I think we can start with only the status table and open a PR to do this on a few components to see how things work out in practice. I suggest you pick some that don't already have the existing metadata.yaml file so that we can continue discussing how to unify them both here. I am happy to help if you need anything.

[codeboten]

@mx-psi @sh0rez here's sorta what i had in mind to get us started: #19178

I was planning on adding a generated_status.go file as part of that change. If folks are ok with this as a starting point, i'll finish the PR later today and put it up for review