KEP-2170: Implement validations for TrainingRuntime and ClusterTrainingRuntime

[tenzen-y]

Related: #2170

We should create validations for the TrainingRuntime and ClusterTrainingRuntime, using the following tools:

• Mutating Webhooks
• CEL validations
• controller-tools/kubebuilder annotations

/area webhook

[tenzen-y]

/area webhook

[andreyvelich]

/assign @akshaychitneni

[google-oss-prow]

@andreyvelich: GitHub didn't allow me to assign the following users: akshaychitneni.

Note that only kubeflow members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @akshaychitneni

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tenzen-y]

@akshaychitneni Could you say /assign?