Obtain pod's logs

[jpkrohling]

I might be missing something, but apparently, the OTel Demo does not capture the logs produced by the services yet. I'm trying to get this working, and looks like we can use the collector helm chart to scrape logs from the node by using this:

opentelemetry-collector:
  mode: daemonset
  presets:
    logsCollection:
      enabled: true
      includeCollectorLogs: false

This would break traces (see open-telemetry/opentelemetry-helm-charts#623), but I'm sure we can come up with a workaround if the linked PR isn't the right solution.

My question: once the linked PR is merged, or when a similar solution is achieved, would this be an acceptable change to the demo? Or is there some other blocker that I'm not seeing?

[puckpuck]

The demo's logs are console output from the various components and not actual OpenTelemetry logs emitted using the Otel SDKs. The use case you present to use the collector to scrape the logs from each pod was not a use case we discussed in our SIG meetings thus far. It's certainly a valid use case, and we should investigate how we would approach and enable this in the demo.

[austinlparker]

Since we have OpenSearch available now, this is still germane. Would love help on it!

[enesonus]

I would like to work on this issue if its still available

[puckpuck]

We have converted several services to emit OTLP logs. We should create a new issue with a list of remaining services that need to be moved and track it from there.

[puckpuck]

Closed this in favor of tracking via #1472