Skip to content

[BUG] Job events are not tagged with kube_cronjob. #52611

Description

@rnakamine

Agent version

7.78.3

Bug Report

Events for Jobs generated by a CronJob are missing the kube_cronjob tag.

Actual tags on the Job event:

  • kube_kind:job
  • kube_name:my-cronjob-29701140
  • kube_job:my-cronjob-29701140

(The kube_cronjob tag is missing)

Pod events generated from the same CronJob failure do have the kube_cronjob tag. This indicates that the data exists in the Tagger but is not being reflected in Job events.

Upon reviewing the code, Pod events use KubernetesPodUID to look up the Tagger, which returns tags with the ownership chain (Pod → Job → CronJob) already resolved. However, Job events fall into the default branch and go through KubernetesMetadata, where ownerReferences are not resolved, resulting in the missing kube_cronjob tag.

func getInvolvedObjectTags(involvedObject v1.ObjectReference, taggerInstance tagger.Component) []string {
// NOTE: we now standardized on using kube_* tags, instead of
// non-namespaced ones, or kubernetes_*. The latter two are now
// considered deprecated.
tagList := []string{
"kube_kind:" + involvedObject.Kind,
"kube_name:" + involvedObject.Name,
// DEPRECATED:
"kubernetes_kind:" + involvedObject.Kind,
"name:" + involvedObject.Name,
}
if involvedObject.Namespace != "" {
tagList = append(tagList,
"kube_namespace:"+involvedObject.Namespace,
// DEPRECATED:
"namespace:"+involvedObject.Namespace,
)
namespaceEntityID := types.NewEntityID(types.KubernetesMetadata, string(util.GenerateKubeMetadataEntityID("", "namespaces", "", involvedObject.Namespace)))
namespaceEntity, err := taggerInstance.GetEntity(namespaceEntityID)
if err == nil {
tagList = append(tagList, namespaceEntity.GetTags(types.HighCardinality)...)
}
}
var entityID types.EntityID
switch involvedObject.Kind {
case podKind:
entityID = types.NewEntityID(types.KubernetesPodUID, string(involvedObject.UID))
case deploymentKind:
entityID = types.NewEntityID(types.KubernetesDeployment, involvedObject.Namespace+"/"+involvedObject.Name)
default:
apiGroup := apiserver.GetAPIGroup(involvedObject.APIVersion)
resourceType, err := apiserver.GetResourceType(involvedObject.Kind, apiGroup)
if err != nil {
log.Debugf("error getting resource type for kind '%s' and group '%s', tags may be missing: %v", involvedObject.Kind, apiGroup, err)
}
entityID = types.NewEntityID(types.KubernetesMetadata, string(util.GenerateKubeMetadataEntityID(apiGroup, resourceType, involvedObject.Namespace, involvedObject.Name)))
}
entity, err := taggerInstance.GetEntity(entityID)
if err == nil {
tagList = append(tagList, entity.GetTags(types.HighCardinality)...)
} else {
log.Debugf("error getting entity for entity ID '%s': tags may be missing", entityID)
}
kindTag := getKindTag(involvedObject.Kind, involvedObject.Name)
if kindTag != "" {
tagList = append(tagList, kindTag)
}
return tagList
}

Reproduction Steps

  1. Create a CronJob that intentionally fails (e.g., exit 1).
  2. Wait for it to execute and fail.
  3. Check the generated Kubernetes events in the Datadog Events Explorer.
  4. Verify that the Job event (kube_kind:job) does not have the kube_cronjob tag.
  5. Verify that the Pod event (kube_kind:pod) generated from the same failure does have the kube_cronjob tag.

Operational Impact:
When a monitor is configured with grouping by env, kube_namespace, and name, the name tag for Job events becomes the instance name (e.g., my-cronjob-29701140), which changes with every execution. As a result, each CronJob failure triggers a new, separate alert and fails to be grouped as failures of the same CronJob.

Agent configuration

Cluster Agent enabled with Kubernetes event collection (Helm):

...

datadog:
  collectEvents: true
...

clusterAgent:
  enabled: true
  rbac:
    create: true

...

Operating System

No response

Other environment details

AWS EKS, Datadog Helm chart 3.215.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    oss/0External contributions priority 0pendingLabel for issues waiting a Datadog member's response.team/container-integrations

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions