(oidc): Add considerations for impacted admission plugin

Signed-off-by: Bryce Palmer <bpalmer@redhat.com>

enhancements/authentication/direct-external-oidc-provider.md

@@ -212,6 +212,93 @@ These changes have already been implemented, and the initial PR for them can be
 
 In order to use `oc` with an external OIDC provider, the tool has been [extended](https://github.com/openshift/oc/pull/1640) with the necessary functionality, including command-line arguments that enable the required configuration. In particular, [`oauth2cli`](https://github.com/int128/oauth2cli) has been vendored into the `oc` codebase. One important consideration here is that depending on the OIDC provider, further functionality might be required, in which case `oc` will have to be extended to support that too.
 
+#### authorization.openshift.io/RestrictSubjectBindings admission plugin considerations
+
+OpenShift's default admission plugin `authorization.openshift.io/RestrictSubjectBindings`
+utilizes the OAuth stack, specifically the `User` and `Group` APIs, to ensure `RoleBinding`
+subjects do not violate any restrictions in the namespace they are going to be created in.
+These restrictions are specified by `RoleBindingRestriction` resources that are served
+by the openshift-apiserver.
+
+Removing the OAuth stack entirely during external OIDC setup may result in the `authorization.openshift.io/RestrictSubjectBindings` admission plugin not functioning as expected.
+
+Anticipated impacts are:
+
+- Logs in the KAS reflecting that informers for OAuth APIs could not be started
+- Logs in the KAS reflecting that the oauth-apiserver is unreachable
+- Errors, that are not actionable from the perspective of users, occurring during validation of `RoleBinding` resources when a `RoleBindingRestriction` resource in the same namespace specifies user/group restrictions
+
+These impacts may cause concern for cluster administrators that their cluster is in an unhealthy state and may not be easily correlated to the enablement of the OIDC functionality.
+
+To ensure a holistic change in cluster behavior when enabling the OIDC functionality, and reduce potentially concerning behavior in the control plane components, it is proposed that changes are made to:
+
+- Disable the `authorization.openshift.io/RestrictSubjectBindings` and `authorization.openshift.io/ValidateRoleBindingRestriction` admission plugins
+- Remove the `rolebindingrestrictions.authorization.openshift.io` CustomResourceDefinition
+- The `Authentication` api to communicate when OIDC can't be enabled due to existing `RoleBindingRestriction` resources through a new conditions field
+
+##### Changes to the kube-apiserver
+
+The OpenShift-specific patch to the kube-apiserver that adds this admission plugin is found here: https://github.com/openshift/kubernetes/blob/master/openshift-kube-apiserver/openshiftkubeapiserver/patch.go
+
+In order to prevent misleading logs about informers that failed to start or failure to connect to the oauth-apiserver, the following changes to this patch are to be made:
+
+- Informer for the `Group` API are only configured and started as part of the first run of the `authorization.openshift.io/RestrictSubjectBindings` admission plugin validation loop. This makes it such that the informer will not be configured or attempt to start when the admission plugin is disabled.
+- The post-start hook that checks for oauth-apiserver connectivity will be skipped if the `Authentication` resource `.spec.type` is set to `OIDC`. This will prevent logs in the kube-apiserver associated with not being able to connect to the oauth-apiserver, which we know is not running when OIDC is enabled.
+
+##### Changes to the cluster-kube-apiserver-operator
+
+When authentication type is set to OIDC, the `authorization.openshift.io/RestrictSubjectBindings` and `authorization.openshift.io/ValidateRoleBindingRestriction` admission plugins will be disabled.
+
+This will be done through updates to the appropriate config observers to update the `KubeAPIServerConfig.apiServerArguments` map to:
+
+- Remove the `authorization.openshift.io/RestrictSubjectBindings` and `authorization.openshift.io/ValidateRoleBindingRestriction` admission plugins from the `--enable-admission-plugins` argument
+- Add the `authorization.openshift.io/RestrictSubjectBindings` and `authorization.openshift.io/ValidateRoleBindingRestriction` admission plugins to the `--disable-admission-plugins` argument
+
+##### Changes to the `Authentication` API
+
+The OIDC authentication mode on the cluster will not be allowed to be enabled if any `RoleBindingRestriction` resources exist.
+
+To communicate the reason for the enablement of the OIDC functionality being blocked, the `Authentication` API will be extended with a new status field to communicate the condition of the OIDC feature.
+
+```go
+type AuthenticationStatus struct {
+    ...
+
+    // oidcConfig holds information related to the status
+    // of the configuration of the OIDC authentication type
+    // ...
+    //
+    // +openshift:enable:FeatureGate=ExternalOIDC
+    // +optional
+    OIDCConfig *OIDCConfig `json:"oidcConfig,omitempty"`
+}
+
+type OIDCConfig struct {
+    // conditions convey the state of the OIDC configuration
+    // ...
+    //
+    // +patchMergeKey=type
+    // +patchStrategy=merge
+    // +listType=map
+    // +listMapKey=type
+    Conditions []metav1.Condition `json:"conditions"`
+}
+```
+
+##### Changes to the cluster-authentication-operator
+
+To support the need to remove the `rolebindingrestrictions.authorization.openshift.io` CustomResourceDefinition when configuring the authentication type as OIDC, the cluster-authentication-operator will be updated to manage this CRD.
+
+This will mean vendoring the generated CRD manifests as outlined in https://github.com/openshift/api/tree/master?tab=readme-ov-file#vendoring-generated-manifests-into-other-repositories and adding a new controller to manage the CRD.
+
+Managing the CRD will consist of ensuring that the CRD is present on the cluster, and matches the desired manifest, when the authentication type is _not_ OIDC, and ensuring the CRD is not present present on the cluster when the authentication type _is_ OIDC.
+
+##### Changes to openshift/api
+
+As the cluster-authentication-operator will now be responsible for the `rolebindingrestrictions.authorization.openshift.io` CRD, it should no longer be added to the openshift/api payload manifests that are included in a payload image and get managed by CVO.
+
+This will likely mean removing the associated files from the hack/update-payload-crd.sh script here: https://github.com/openshift/api/blob/dd0f68969241c0548906ec98c12bb208512cbbb4/hack/update-payload-crds.sh#L6
+
 #### Authentication disruptions
 
 In case something goes wrong with the external provider, authentication might stop working. In such cases, cluster admins will still be able to access the cluster using a `kubeconfig` file with client certificates for an admin user. It is the responsibility of the cluster admins to make sure that such users exist; deleting all admin users might result in losing access to the cluster should any issues with the external provider arise.
@@ -238,10 +325,12 @@ Overall, for this feature there must be e2e tests that cover the following:
   - authenticate users with bearer tokens issued by the OIDC provider
   - ensure tokens issued by the built-in oauth stack do not work
   - ensure user mapping capabilities work as expected
+  - ensure that existence of `RoleBindingRestriction` resources blocks external OIDC configuration from progressing
 - on a cluster that uses an external OIDC provider, test reverting configuration back to the built-in OAuth stack (good/bad configurations should be tested)
 - on a cluster that uses an external OIDC provider, test monitoring and cluster-authentication-operator status when the provider becomes unavailable
 - version skew between participating components; e.g. the cluster-authentication-operator has picked up the new configuration but the kube-apiserver-operator hasn't yet
 - cluster still accessible if OIDC provider becomes unavailable using a `kubeconfig` (break-glass scenario)
+- on a cluster that uses an external OIDC provider, test that the `rolebindingrestrictions.authorization.openshift.io` CRD does not exist
 
 Finally, in order to make sure that others can test their components in an external OIDC environment, a cluster with an external OIDC configuration must be created and made available to the CI.