Skip to content

[FEATURE] Add Observability Simple Schema Support #1404

@YANG-DB

Description

@YANG-DB

Is your feature request related to a problem?

similar to traces the Observability Logs mapping should be defined under the Observability plugin

Currently Observability has no notion or index to hold the Structured Log signals as defined by the OTEL standard and other similar structured log protocols.
In correlation with the schema definition for traces & metrics - logs will also be explicitly defined and supported by Observability plugin.

This includes:

  • Index structure (mapping)
  • Index naming convention
  • Json schema for enforcement and validation of the structure

What solution would you like?

  • Move the structure and naming from the ingestion to the core Observability

  • Define a well structured Log schema

  • Allow incremental support for different log types and categories
    -> Http category logs
    -> Communication category logs

  • Once Observability Plugin is loaded it will add the index template for the Simple Schema for Observability

    • Traces mapping
    • Metrics mapping
    • Logs mapping

And all indices that follow these index pattern will use its mapping accordingly

This will present the foundation for adding future log types in a common and well-defined manner.

This would simply next use cases:

  • allow 3rd party ingestion developers to have a consolidated location to retrieve and validate their Observability schema
    • Fluent-D
    • Jaeger
    • Otel exporter
  • Allow the development of a pre-defined Observability dashboards that know and project the metrics structure

Simplify any ingestion framework lifecycle by not having to maintain or care about the lifecycle of the target index to populate

What alternatives have you considered?
N/A

Do you have any additional context?

Metadata

Metadata

Assignees

Labels

designdocumentationImprovements or additions to documentationenhancementNew feature or requestintegrationIntegration project

Type

No type

Projects

Status

In Progress

Status

Done

Relationships

None yet

Development

No branches or pull requests

Issue actions