Agent version
7.78.3
Bug Report
Events for Jobs generated by a CronJob are missing the kube_cronjob tag.
Actual tags on the Job event:
kube_kind:job
kube_name:my-cronjob-29701140
kube_job:my-cronjob-29701140
(The kube_cronjob tag is missing)
Pod events generated from the same CronJob failure do have the kube_cronjob tag. This indicates that the data exists in the Tagger but is not being reflected in Job events.
Upon reviewing the code, Pod events use KubernetesPodUID to look up the Tagger, which returns tags with the ownership chain (Pod → Job → CronJob) already resolved. However, Job events fall into the default branch and go through KubernetesMetadata, where ownerReferences are not resolved, resulting in the missing kube_cronjob tag.
|
func getInvolvedObjectTags(involvedObject v1.ObjectReference, taggerInstance tagger.Component) []string { |
|
// NOTE: we now standardized on using kube_* tags, instead of |
|
// non-namespaced ones, or kubernetes_*. The latter two are now |
|
// considered deprecated. |
|
tagList := []string{ |
|
"kube_kind:" + involvedObject.Kind, |
|
"kube_name:" + involvedObject.Name, |
|
|
|
// DEPRECATED: |
|
"kubernetes_kind:" + involvedObject.Kind, |
|
"name:" + involvedObject.Name, |
|
} |
|
|
|
if involvedObject.Namespace != "" { |
|
tagList = append(tagList, |
|
"kube_namespace:"+involvedObject.Namespace, |
|
|
|
// DEPRECATED: |
|
"namespace:"+involvedObject.Namespace, |
|
) |
|
|
|
namespaceEntityID := types.NewEntityID(types.KubernetesMetadata, string(util.GenerateKubeMetadataEntityID("", "namespaces", "", involvedObject.Namespace))) |
|
namespaceEntity, err := taggerInstance.GetEntity(namespaceEntityID) |
|
if err == nil { |
|
tagList = append(tagList, namespaceEntity.GetTags(types.HighCardinality)...) |
|
} |
|
} |
|
|
|
var entityID types.EntityID |
|
|
|
switch involvedObject.Kind { |
|
case podKind: |
|
entityID = types.NewEntityID(types.KubernetesPodUID, string(involvedObject.UID)) |
|
case deploymentKind: |
|
entityID = types.NewEntityID(types.KubernetesDeployment, involvedObject.Namespace+"/"+involvedObject.Name) |
|
default: |
|
apiGroup := apiserver.GetAPIGroup(involvedObject.APIVersion) |
|
resourceType, err := apiserver.GetResourceType(involvedObject.Kind, apiGroup) |
|
if err != nil { |
|
log.Debugf("error getting resource type for kind '%s' and group '%s', tags may be missing: %v", involvedObject.Kind, apiGroup, err) |
|
} |
|
entityID = types.NewEntityID(types.KubernetesMetadata, string(util.GenerateKubeMetadataEntityID(apiGroup, resourceType, involvedObject.Namespace, involvedObject.Name))) |
|
} |
|
|
|
entity, err := taggerInstance.GetEntity(entityID) |
|
if err == nil { |
|
tagList = append(tagList, entity.GetTags(types.HighCardinality)...) |
|
} else { |
|
log.Debugf("error getting entity for entity ID '%s': tags may be missing", entityID) |
|
} |
|
|
|
kindTag := getKindTag(involvedObject.Kind, involvedObject.Name) |
|
if kindTag != "" { |
|
tagList = append(tagList, kindTag) |
|
} |
|
|
|
return tagList |
|
} |
Reproduction Steps
- Create a CronJob that intentionally fails (e.g.,
exit 1).
- Wait for it to execute and fail.
- Check the generated Kubernetes events in the Datadog Events Explorer.
- Verify that the Job event (
kube_kind:job) does not have the kube_cronjob tag.
- Verify that the Pod event (
kube_kind:pod) generated from the same failure does have the kube_cronjob tag.
Operational Impact:
When a monitor is configured with grouping by env, kube_namespace, and name, the name tag for Job events becomes the instance name (e.g., my-cronjob-29701140), which changes with every execution. As a result, each CronJob failure triggers a new, separate alert and fails to be grouped as failures of the same CronJob.
Agent configuration
Cluster Agent enabled with Kubernetes event collection (Helm):
...
datadog:
collectEvents: true
...
clusterAgent:
enabled: true
rbac:
create: true
...
Operating System
No response
Other environment details
AWS EKS, Datadog Helm chart 3.215.0
Agent version
7.78.3
Bug Report
Events for Jobs generated by a CronJob are missing the
kube_cronjobtag.Actual tags on the Job event:
kube_kind:jobkube_name:my-cronjob-29701140kube_job:my-cronjob-29701140(The
kube_cronjobtag is missing)Pod events generated from the same CronJob failure do have the
kube_cronjobtag. This indicates that the data exists in the Tagger but is not being reflected in Job events.Upon reviewing the code, Pod events use
KubernetesPodUIDto look up the Tagger, which returns tags with the ownership chain (Pod → Job → CronJob) already resolved. However, Job events fall into thedefaultbranch and go throughKubernetesMetadata, whereownerReferencesare not resolved, resulting in the missingkube_cronjobtag.datadog-agent/pkg/collector/corechecks/cluster/kubernetesapiserver/events_common.go
Lines 243 to 300 in da94522
Reproduction Steps
exit 1).kube_kind:job) does not have thekube_cronjobtag.kube_kind:pod) generated from the same failure does have thekube_cronjobtag.Operational Impact:
When a monitor is configured with grouping by
env,kube_namespace, andname, thenametag for Job events becomes the instance name (e.g.,my-cronjob-29701140), which changes with every execution. As a result, each CronJob failure triggers a new, separate alert and fails to be grouped as failures of the same CronJob.Agent configuration
Cluster Agent enabled with Kubernetes event collection (Helm):
Operating System
No response
Other environment details
AWS EKS, Datadog Helm chart 3.215.0