TAS: reduce friction by defaulting the PodSet annotations

[mimowo]

What would you like to be added:

Allow scheduling workloads without TAS annotations (kueue.x-k8s.io/podset-required-topology or kueue.x-k8s.io/podset-preferred-topology) on every PodSet.

The idea is to default the annotations, but we need to decide where is the configuration.

Why is this needed:

Currently, users of TAS need to set annotations on every PodTemplate. While this gives control, it also creates friction and room for error, because users may forget to set the annotation. Also, most users want to use the "preferred" mode, and they don't need to control the topology level.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc (KEP update)
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[mimowo]

cc @mwielgus @mwysokin @tenzen-y

[mwysokin]

Maybe ResourceFlavor? Since it's usually related to the machine type and it can benefit only some of the machine types? Maybe the default could be kubernetes.io/hostname since it seems to give the best compact placement?

[tenzen-y]

Allow scheduling workloads without TAS annotations (kueue.x-k8s.io/podset-required-topology or kueue.x-k8s.io/podset-preferred-topology) on every PodSet.

This sounds like making TAS a default scheduling policy.
IIUC, TAS currently tries to pack Pods into the same and high-used nodes as much as possible (mostAllocated).
However, the policy sometimes does not meet inference and other workloads.

What about adding validations? I mean, if Jobs have TAS annotations in .metadata.annotations, we reject the Job and return appropriate places where they should put TAS annotations.

[mimowo]

One lightweight idea is to make TAS default when all RFs in a given CQ have topologyName. Then, for every resource flavor the default is the lowest level - which would be kubernetes.io/hostname effectively as proposed in #3754 (comment).

The benefit of this approach is that it allows users to migrate CQ-by-CQ to TAS, and does not require setting the annotations by the user.

This sounds like making TAS a default scheduling policy.

Not really, we would still require a deliberate admin action to configure the ResourceFlavors to have topologyName. Otherwise, default Kueue is used.

IIUC, TAS currently tries to pack Pods into the same and high-used nodes as much as possible (mostAllocated).
However, the policy sometimes does not meet inference and other workloads.

That is true, but I would prefer to think about this as a separate improvement to TAS. For example, we may introduce another PodSet annotation. We have "required", and "preferred", maybe we also need "spread" or "relax", etc.

[mimowo]

/assign

[tenzen-y]

One lightweight idea is to make TAS default when all RFs in a given CQ have topologyName. Then, for every resource flavor the default is the lowest level - which would be kubernetes.io/hostname effectively as proposed in #3754 (comment).

The benefit of this approach is that it allows users to migrate CQ-by-CQ to TAS, and does not require setting the annotations by the user.

Basically, sgtm.
One question is which enforcement level (required vs preferred) should Kueue automatically insert?

[mimowo]

One question is which enforcement level (required vs preferred) should Kueue automatically insert?

Preferred - this is our go to annotation recommended to users, for most workloads. "required" could make some workloads unschedulable. Still, if some workload is ultra sensitive on networking the user can set the "required" annotation and it will have precedence over the default (preferred).

[tenzen-y]

One question is which enforcement level (required vs preferred) should Kueue automatically insert?

Preferred - this is our go to annotation recommended to users, for most workloads. "required" could make some workloads unschedulable. Still, if some workload is ultra sensitive on networking the user can set the "required" annotation and it will have precedence over the default (preferred).

That sounds reasonable.