add k8s object receiver

[omrozowicz-splunk]

Replacing k8seventsreceiver with k8objectsreceiver.

As using k8s objects receiver requires creating new rules in clusterRole.yaml, the PR contains splunk-otel-collector.clusterRole.rules which is the template function that aggregates and renders all the apiGroups rules with "get", "list", "watch" verbs, so there is no duplications.

It requires splunk-otel-collector with signalfx/splunk-otel-collector#2270 to be released.

There are slight indent differences in rendered templates because previously it was not entirely consistent (sometimes with whitespaces after apiGroups, sometimes not). With the change from thsi PR it's handled by the function:

- apiGroups:
  - {{ $groupName }}

Also, eventsEnabled name was changed to objectsEnabled so the name reflects better what actually we pull.

[rmfitzpatrick]

Do you have any guidance and examples on the differences in reported telemetry between the two receivers? Presumably this is going to affect any user systems that depend on the events receiver.

[hvaghani221]

Do you have any guidance and examples on the differences in reported telemetry between the two receivers? Presumably this is going to affect any user systems that depend on the events receiver.

k8sobject stores payload into the log body, but k8s_events only stores event.Message field as body and only adds following fields as resource/attribue

• k8s.object.kind
• k8s.object.name
• k8s.object.uid
• k8s.object.fieldpath
• k8s.object.api_version
• k8s.object.resource_version
• k8s.event.reason
• k8s.event.action
• k8s.event.start_time
• k8s.event.name
• k8s.event.uid
• k8s.namespace.name

k8s_event payload

k8sobjects payload

[dmitryax]

k8sobject stores payload into the log body, but k8s_events only stores event.Message field as body and only adds following fields as resource/attribute

This needs to be well documented in https://github.com/signalfx/splunk-otel-collector-chart/blob/main/UPGRADING.md

[omrozowicz-splunk]

One more thing, now sourcetype of events generated by k8sobjectsreceiver is kube:events. Do we want to keep this convention? We could generate the sourcetype based on the resource we pull/watch, following kube:object:<resource>, for ex. kube:object:pods. What do you think?

[dmitryax]

We could generate the sourcetype based on the resource we pull/watch, following kube:object:, for ex. kube:object:pods. What do you think?

Sorry, I'm not an expert when it comes to Splunk Platform specific fields. What are the expected values for sourcetype? Do we have any conventions for that?

And how do you want to achieve that? I think we can use the transform processor to compose sourcetype from event payload

[hvaghani221]

Sorry, I'm not an expert when it comes to Splunk Platform specific fields. What are the expected values for sourcetype? Do we have any conventions for that?

The source type is one of the default fields that the Splunk platform assigns to all incoming data. It tells the platform what kind of data you have, so that it can format the data intelligently during indexing. Source types also let you categorize your data for easier searching.

In SCK, we are also following kube:object:<resourcename> convention.

And how do you want to achieve that? I think we can use the transform processor to compose sourcetype from event payload

K8sobjects adds the resource name as an event.name attribute. I was thinking we can use logstransformprocessor
and use stanza's add operator to add com.splunk.sourcetype attribute.

There was a bug in k8sobject receiver. event.name attribute is empty when watch mode is used. I have created open-telemetry/opentelemetry-collector-contrib#16543 to fix that. Can you please review that as well?

[dmitryax]

@harshit-splunk thanks for explaining

In SCK, we are also following kube:object: convention.

Sounds good to me. Do we have to stick with what was adopted in SCK? Using k8s in k8s:object:<resourcename> I think is less confusing than kube.

K8sobjects adds the resource name as an event.name attribute. I was thinking we can use logstransformprocessor
and use stanza's add operator to add com.splunk.sourcetype attribute.

Let's better use transform processor since that is what will be the standard in OTel collector going forward

[hvaghani221]

Do we have to stick with what was adopted in SCK? Using k8s in k8s:object:<resourcename> I think is less confusing than kube.

Agrred, but we are already using kube prefix for container logs (i.e. kube:container:<container name>. So, we should stick with same prefix.

Let's better use transform processor since that is what will be the standard in OTel collector going forward

Okay, makes sense.

[omrozowicz-splunk]

Do we have to stick with what was adopted in SCK? Using k8s in k8s:object:<resourcename> I think is less confusing than kube.

Agrred, but we are already using kube prefix for container logs (i.e. kube:container:<container name>. So, we should stick with same prefix.

Let's better use transform processor since that is what will be the standard in OTel collector going forward

Okay, makes sense.

I've added setting sourcetype using transform processor, and it looks correct:

But we have two dependencies here, first one is k8sobjectreceiver that doesn't set event.name attribute correctly for watch, so it currently look like this in splunk:

We need to wait until this open-telemetry/opentelemetry-collector-contrib#16543 is merged

Another thing is that in transform processor was a syntax change between 0.64 and 0.65, so the current implementation works for 0.64 (notation with logs instead of logs_statement), but it won't work for 0.66

[dmitryax]

Agrred, but we are already using kube prefix for container logs (i.e. kube:container:. So, we should stick with same prefix.

Ok let's keep kube

We need to wait until this open-telemetry/opentelemetry-collector-contrib#16543 is merged

Reviewing it

Another thing is that in transform processor was a syntax change between 0.64 and 0.65, so the current implementation works for 0.64 (notation with logs instead of logs_statement), but it won't work for 0.66

It's just a configuration option, right? We will just need to update here it with the upgrade to 0.66. Please add a note comment

[omrozowicz-splunk]

Another thing is that in transform processor was a syntax change between 0.64 and 0.65, so the current implementation works for 0.64 (notation with logs instead of logs_statement), but it won't work for 0.66

It's just a configuration option, right? We will just need to update here it with the upgrade to 0.66. Please add a note comment

Done

[rmfitzpatrick]

https://stackoverflow.com/a/9208164

It appears these changes remove clusterReceiver.eventsEnabled functionality in total, making it obsolete. Is this correct?

[omrozowicz-splunk]

Yes, right, k8s object receiver is not providing exactly the same functionality as k8s events receiver. Updated.

[rmfitzpatrick]

So #588 (comment) could also be expanded so that we provide an actual deprecation path by using the objects receiver w/ just the watched events for this value instead of obsoleting in one pass. The warning check also needs to be updated with a description of the behavior.

imo users should never have their config become "broken" during an upgrade and we should provide at least a release of warnings and ideally remediation logic in helpers (cc @dmitryax)

[omrozowicz-splunk]

Ok, so I updated it in the way that when eventsEnabled is set to true we return warning message and event objects are watched, so behaviour is similar (but not the same, as previous event formatting was different, and the sourcetype was kube:events and now it is kube:object<resource_id>. It is described in details in the upgrading doc.

[omrozowicz-splunk]

This expect fail is here because of open-telemetry/opentelemetry-collector-contrib#16543 which is not yet merged

[dmitryax]

Is this PR functional to be merged with the current sourcetype? If so, can we add test_k8s_objects_sourcetype later instead?

[omrozowicz-splunk]

Currently sourcetypes render correctly except watch case. So the only test case which is failing now is the one checking

    - name: events
      mode: watch

Yes, we can add it later. I deleted pytest expect and ("sourcetype", "kube:object:event", 1), from test_k8s_objects_sourcetype

[rmfitzpatrick]

k8s_event payload

k8sobjects payload

Looking at this more, as a user I would find this migration less than desirable and potentially fully breaking. The k8sobjects receiver could benefit from following the otel semantic conventions and maybe contribute new ones: https://github.com/open-telemetry/opentelemetry-specification/blob/main/semantic_conventions/resource/k8s.yaml.

Given the substance of the changes could it make sense to not alter the events receiver usage at all but have an opt-in for the new objects receiver?

cc @dmitryax @jvoravong @atoulme

[dmitryax]

Given the substance of the changes could it make sense to not alter the events receiver usage at all but have an opt-in for the new objects receiver?

We can leave it with opt-in for now, but I believe we eventually need to deprecate the k8sevents receiver in favor of the k8sobjects receiver. The reason is that I don't think we should parse all the k8s event fields and compose attributes from it that can go with unnecessary high cardinality. Sending events in the format defined by the emitting system actually fits better the new events API that is currently being defined in the OTel spec. We will work on raising an issue about that for discussion in contrib with @harshit-splunk

[dmitryax]

Overall, looks good to me. just couple nits

[dmitryax]

LGTM. Just a couple more nits, and I think we can merge it. Thanks for all the work!

[rmfitzpatrick]

nit, no change needed