Add support for feature gates

[danielvegamyhre]

What would you like to be added:

Add support for feature gates in JobSet, like are used in upstream k8s and Kueue here.

Why is this needed:

• Allow customer to quickly disable a feature if a bug is detected
• Allow us to have "opt-in" features that are off by default.

[googs1025]

/assign hi @danielvegamyhre i'll try it

[googs1025]

Do we need to discuss which feature gates need to be added?

[googs1025]

/assign

[googs1025]

I have been thinking, can we include these two items in the feature gate? They can be enabled by default, and if there are more features in the future, they can be added. @danielvegamyhre
Establishment of a headless service
TTL after finished

[danielvegamyhre]

I don't think we can (or should) put existing features behind feature gates, it would be weird for the feature gate to exist on some versions but not others. I think we should put new features (not released yet) which are somewhat risky behind a feature gate. Not sure off the top of my head if this aligns with upstream k8s standard practices, @kannon92 do you know?

[googs1025]

Understood. As far as I understand, the current features of JobSet are all existing ones, and there are no new unstable features yet. In that case, we can provide the feature gate functionality for other developers to use, but without any default configuration. How does that sound? @danielvegamyhre