Skip to content

[RFC-0011] - OTEL integration based on alerts #1149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

adri1197
Copy link

@adri1197 adri1197 commented Jul 23, 2025

Summary

  • GitRepository > Kustomization > HelmRepository > HelmRelease
image

Open Questions

  • What should it be proper naming conventions? In terms of the following elements:
    • Tracer
    • Service
    • Span
  • The current implementation gets the host got by the provider and automatically adds the endpoint to traces based on the convention v1/traces (automatically managed by Otel GO SDK). In this way, it could allow that the notifications are sent to another endpoint, while the traces could be properly sent, as well (even though this endpoint convention must be met). Another alternative, could be to make it configurable by parsing the URL or some kind of convention?
  • Tracing management is completely independent of the notifications, meaning that in case of either failing, the other should be able to run. Also, the trace logging is shown mainly as DEBUG mode.
  • The code design adds a new notifier called OtelTracer, even though it has not been populated as a new provider type. The rationale is to be able to rolling it out to the current providers that may support OpenTelemetry. At this moment, just generic type.

Part of: #1097
Part of: fluxcd/flux2#5321

@adri1197 adri1197 changed the title Otel [RFC-0011] - OpenTelemetry integration based on alerts Jul 23, 2025
@adri1197 adri1197 force-pushed the otel branch 2 times, most recently from 6e2dbbd to 7a793ec Compare July 24, 2025 08:14
@adri1197 adri1197 marked this pull request as ready for review July 24, 2025 08:17
@adri1197 adri1197 marked this pull request as draft July 24, 2025 13:20
@adri1197 adri1197 changed the title [RFC-0011] - OpenTelemetry integration based on alerts [RFC-0011] - OTEL integration based on alerts Jul 24, 2025
@adri1197 adri1197 force-pushed the otel branch 4 times, most recently from 261f89d to 95a19f7 Compare July 30, 2025 08:51
@stefanprodan stefanprodan added enhancement New feature or request area/alerting Alerting related issues and PRs experimental Issues and pull requests related to experimental features labels Jul 30, 2025
Copy link
Member

@matheuscscp matheuscscp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass

Signed-off-by: Adrian Fernandez De La Torre <adri1197@gmail.com>
@@ -215,6 +216,7 @@ func (s *EventServer) dispatchNotification(ctx context.Context, event *eventv1.E

go func(n notifier.Interface, e eventv1.Event) {
pctx, cancel := context.WithTimeout(context.Background(), timeout)
pctx = context.WithValue(pctx, notifier.AlertMetadataContextKey{}, alert.ObjectMeta)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please swap this line with defer cancel()

apiv1beta3 "github.com/fluxcd/notification-controller/api/v1beta3"
)

type AlertMetadataContextKey struct{}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
type AlertMetadataContextKey struct{}
type alertMetadataContextKey struct{}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, please revert the changes in this file, as otel should be its own provider type

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/alerting Alerting related issues and PRs enhancement New feature or request experimental Issues and pull requests related to experimental features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants