Merge pull request #5327 from ritazh/kep-5018-beta

KEP-5018: move to beta in 1.34

Author: k8s-ci-robot

keps/prod-readiness/sig-auth/5018.yaml

@@ -4,3 +4,5 @@
 kep-number: 5018
 alpha:
   approver: "soltysh" 
+beta:
+  approver: "soltysh" 

keps/sig-auth/5018-dra-adminaccess/README.md

@@ -44,13 +44,13 @@
 Items marked with (R) are required _prior to targeting to a milestone /
 release_.
 
-- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in
+- [x] (R) Enhancement issue in release milestone, which links to KEP dir in
       [kubernetes/enhancements] (not the initial KEP PR)
-- [ ] (R) KEP approvers have approved the KEP status as `implementable`
+- [x] (R) KEP approvers have approved the KEP status as `implementable`
 - [x] (R) Design details are appropriately documented
-- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and
+- [x] (R) Test plan is in place, giving consideration to SIG Architecture and
       SIG Testing input (including test refactors)
-  - [ ] e2e Tests for all Beta API Operations (endpoints)
+  - [x] e2e Tests for all Beta API Operations (endpoints)
   - [ ] (R) Ensure GA e2e tests meet requirements for
         [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
   - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free
@@ -59,12 +59,12 @@ release_.
         [all GA Endpoints](https://github.com/kubernetes/community/pull/1806)
         must be hit by
         [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
-- [ ] (R) Production readiness review completed
-- [ ] (R) Production readiness review approved
+- [x] (R) Production readiness review completed
+- [x] (R) Production readiness review approved
 - [x] "Implementation History" section is up-to-date for milestone
-- [ ] User-facing documentation has been created in [kubernetes/website], for
+- [x] User-facing documentation has been created in [kubernetes/website], for
       publication to [kubernetes.io]
-- [ ] Supporting documentation—e.g., additional design documents, links to
+- [x] Supporting documentation—e.g., additional design documents, links to
       mailing list discussions/SIG meetings, relevant PRs/issues, release notes
 
 <!--
@@ -179,9 +179,11 @@ objects as privileged. This feature includes:
    ```yaml
    metadata:
      labels:
-       resource.k8s.io/admin-access: "true"
+       resource.kubernetes.io/admin-access: "true"
    ```
 
+   Note: This label has been updated from `resource.k8s.io/admin-access` while the feature was in alpha.
+
    Assumptions:
 
    - It is not important to subdivide admin access to different types of
@@ -194,7 +196,7 @@ objects as privileged. This feature includes:
 
    In the REST storage layer, validate requests to create and update
    `ResourceClaim` or `ResourceClaimTemplate` objects with `adminAccess: true`.
-   Only authorize if namespace has the `resource.k8s.io/admin-access: "true"` label.
+   Only authorize if namespace has the `resource.kubernetes.io/admin-access: "true"` label.
 
 1. Grants privileged access to the requested device:
 
@@ -212,7 +214,7 @@ objects as privileged. This feature includes:
 ### Workflow
 
 1. A cluster administrator labels an admin namespace with
-   `resource.k8s.io/admin-access: "true"`.
+   `resource.kubernetes.io/admin-access: "true"`.
 
 1. Users who are authorized to create `ResourceClaim` or `ResourceClaimTemplate`
    objects in this admin namespace can set `adminAccess: true` field if they
@@ -284,7 +286,7 @@ shouldn't have allowed unrestricted access.
 Starting in Kubernetes 1.33 (when this KEP was introduced), a validation has
 been added to the REST storage layer to only authorize `ResourceClaim` or
 `ResourceClaimTemplate` with `adminAccess: true` requests if their namespace has
-the `resource.k8s.io/admin-access: "true"` label to only allow it for users with
+the `resource.kubernetes.io/admin-access: "true"` label to only allow it for users with
 additional privileges.
 
 The below flowchart starts with `ResourceClaim` creation from
@@ -403,19 +405,15 @@ The scheduler plugin and resource claim controller are covered by the workloads
 in
 https://github.com/kubernetes/kubernetes/blob/master/test/integration/scheduler_perf/dra/performance-config.yaml
 
-Those tests run in:
+Additional test cases have been added to `test/integration/scheduler_perf` to
+ensure `ResourceClaim` or `ResourceClaimTemplate` with `adminAccess: true`
+requests are only authorized if their namespace has the
+`resource.kubernetes.io/admin-access: "true"` label as described in this KEP.
+
+These tests run as part of the following with the `DRAAdminAccess` feature gate enabled.
 
-- [pre-submit](https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-integration)
-  and
-  [periodic](https://testgrid.k8s.io/sig-release-master-blocking#integration-master)
-  integration testing under
-  `k8s.io/kubernetes/test/integration/scheduler_perf.scheduler_perf` and
-  `k8s.io/kubernetes/test/integration/scheduler_perf.dra.dra` and the
-  `DRAAdminAccess` feature gate is already enabled.
-- Additional test cases will be added to `test/integration/scheduler_perf` to
-  ensure `ResourceClaim` or `ResourceClaimTemplate` with `adminAccess: true`
-  requests are only authorized if their namespace has the
-  `resource.k8s.io/admin-access: "true"` label as described in this KEP.
+- `k8s.io/kubernetes/test/integration/scheduler_perf.scheduler_perf`: [pre-submit](https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-integration&include-filter-by-regex=scheduler_perf.scheduler_perf), [periodic](https://testgrid.k8s.io/sig-release-master-blocking#integration-master&include-filter-by-regex=scheduler_perf.scheduler_perf), [triage search](https://storage.googleapis.com/k8s-triage/index.html?test=scheduler_perf)
+- `k8s.io/kubernetes/test/integration/scheduler_perf.dra.dra`: [pre-submit](https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-integration&include-filter-by-regex=scheduler_perf.dra.dra),[periodic](https://testgrid.k8s.io/sig-release-master-blocking#integration-master&include-filter-by-regex=scheduler_perf.dra.dra), [triage search](https://storage.googleapis.com/k8s-triage/index.html?test=scheduler_perf)
 
 ##### e2e tests
 
@@ -436,7 +434,7 @@ was developed as part of the overall DRA development effort. We have extended
 this test driver to enable `DRAAdminAccess` feature gate and added tests to
 ensure `ResourceClaim` or `ResourceClaimTemplate` with `adminAccess: true`
 requests are only authorized if their namespace has the
-`resource.k8s.io/admin-access: "true"` label as described in this KEP.
+`resource.kubernetes.io/admin-access: "true"` label as described in this KEP.
 
 Test links:
 
@@ -449,11 +447,8 @@ ResourceClaimTemplate and ResourceClaim for admin access
 [Feature:DRAAdminAccess] [FeatureGate:DRAAdminAccess] [Alpha]
 [FeatureGate:DynamicResourceAllocation] [Beta]
 
-- AdminAccess related tests in
-  https://github.com/kubernetes/kubernetes/blob/69ab91a5c59617872c9f48737c64409a9dec2957/test/e2e/dra/dra.go#L976
-  and
-  https://github.com/kubernetes/kubernetes/blob/69ab91a5c59617872c9f48737c64409a9dec2957/test/e2e/dra/dra.go#L1095
-  will be updated.
+- `cluster validate ResourceClaimTemplate and ResourceClaim for admin access`, [SIG Node](https://testgrid.k8s.io/sig-node-dynamic-resource-allocation#pull-kubernetes-kind-dra-all), [triage search](https://storage.googleapis.com/k8s-triage/index.html?pr=1&test=admin%20access)
+- `cluster DaemonSet with admin access`, [SIG Node](https://testgrid.k8s.io/sig-node-dynamic-resource-allocation#pull-kubernetes-kind-dra-all), [triage search](https://storage.googleapis.com/k8s-triage/index.html?pr=1&test=admin%20access)
 
 ### Graduation Criteria
 
@@ -464,13 +459,23 @@ ResourceClaimTemplate and ResourceClaim for admin access
 
 #### Beta
 
-- Gather feedback
+- Gather feedback from developers and surveys via implementations in the kubernetes-sigs/dra-example-driver: https://github.com/kubernetes-sigs/dra-example-driver/issues/97 and potentially other drivers
+- Complete feature AdminAccess
 - Additional tests are in Testgrid and linked in KEP
-- Implementations in the kubernetes-sigs/dra-example-driver
+- More rigorous forms of testing—e.g., downgrade tests and scalability tests
+- All functionality completed
+- All security enforcement completed
+- All monitoring requirements completed
+- All testing requirements completed
+- All known pre-release issues and gaps resolved 
+**Note:** Beta criteria must include all functional, security, monitoring, and testing requirements along with resolving all issues and gaps identified
 
-#### GA
 
+#### GA
+- 1 example of real-world usage
 - Allowing time for feedback
+- All issues and gaps identified as feedback during beta are resolved
+**Note:** GA criteria must not include any functional, security, monitoring, or testing requirements.  Those must be beta requirements.
 
 ### Upgrade / Downgrade Strategy
 
@@ -541,7 +546,12 @@ rollout. Similarly, consider large clusters and how enablement/disablement
 will rollout across nodes.
 -->
 
-Will be considered for beta.
+- kube-controller-manager: If the kube-controller-manager fails to create `ResourceClaim` objects from `ResourceClaimTemplate` due to misconfigurations or permission issues relating to `adminAccess`, then the associated Pods will remain in a pending state and won't be scheduled.
+- kube-scheduler: Bugs in the scheduler might lead to Pods not being scheduled even when resources are available or, scheduling Pods that shouldn't be scheduled due to unmet `adminAccess` requirements, all this should be part of the generic scheduler backoff behavior. It will not affect running workloads.
+- Workloads Without `ResourceClaims` will remain unaffected as the adminAccess feature doesn't interact with them. The new code paths introduced for adminAccess only engage when `ResourceClaims` are present in the Pod specification.
+- New Pods requiring `ResourceClaims` with `adminAccess` might remain unscheduled if the control plane components fail to process the claims correctly.
+- Existing Pods continue to run unaffected since `ResourceClaim` and `ResourceClaimTemplate`'s spec is immutable, including the adminAccess field, cannot be altered.
+
 
 ###### What specific metrics should inform a rollback?
 
@@ -557,8 +567,6 @@ the `scheduler_pending_pods` metric in the kube-scheduler or an increase in the
 Further analysis by reviewing logs and pod events is needed to determine whether
 errors are related to this feature.
 
-Will provide more details for beta.
-
 ###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
 
 <!--
@@ -567,15 +575,19 @@ Longer term, we may want to require automated upgrade/rollback tests, but we
 are missing a bunch of machinery and tooling and can't do that now.
 -->
 
-Will be considered for beta.
+This will be done manually before transition to beta by bringing up a cluster with kubeadm and changing the feature gate for individual components.
+
+Manual upgrade of the control plane to a version with the feature enabled will be tested. Existing pods not using the feature remained running. Creation of new pods and ResourceClaims that do not use the feature should be unaffected.
+
+Manual downgrade of the control plane to a version with the feature disabled was tested. Existing pods using the feature remained running. Creation of new pods and ResourceClaims that use the feature should be blocked.
 
 ###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
 
 <!--
 Even if applying deprecation policies, they may still surprise some users.
 -->
 
-Will be considered for beta.
+No.
 
 ### Monitoring Requirements
 
@@ -586,7 +598,7 @@ For GA, this section is required: approvers should be able to confirm the
 previous answers based on experience in the field.
 -->
 
-Will be considered for beta.
+Metrics in kube-controller-manager about total (resourceclaim_controller_resource_claims_adminaccess) and allocated ResourceClaims with adminAccess (resourceclaim_controller_allocated_resource_claims_adminaccess).
 
 ###### How can an operator determine if the feature is in use by workloads?
 
@@ -596,7 +608,9 @@ checking if there are objects with field X set) may be a last resort. Avoid
 logs or events for this purpose.
 -->
 
-Will be considered for beta.
+".status.allocation.devices.results[*].adminAccess" will be set to true for a claim using adminAccess when needed by a pod.
+
+Metrics in kube-controller-manager about total (resourceclaim_controller_resource_claims_adminaccess) and allocated ResourceClaims with adminAccess (resourceclaim_controller_allocated_resource_claims_adminaccess).
 
 ###### How can someone using this feature know that it is working for their instance?
 
@@ -640,7 +654,7 @@ These goals will help you determine what you need to measure (SLIs) in the next
 question.
 -->
 
-Will be considered for beta.
+SLO: 100% of unauthorized access attempts are denied.
 
 ###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
 
@@ -673,14 +687,17 @@ metric in scheduler will identify pods that are currently unschedulable because
 of the `DynamicResources` plugin or a misconfiguration of the `AdminAccess`
 field.
 
+Audit Policy can be created to ensure all create operations on ResourceClaim, ResourceClaimTemplate, and Namespace resources are logged at the metadata level to review successful and denied attempts to set the `AdminAccess`
+field.
+
 ###### Are there any missing metrics that would be useful to have to improve observability of this feature?
 
 <!--
 Describe the metrics themselves and the reasons why they weren't added (e.g., cost,
 implementation difficulties, etc.).
 -->
 
-Will be considered for beta.
+No
 
 ### Dependencies
 
@@ -705,7 +722,8 @@ and creating new ones, as well as about cluster-level services (e.g. DNS):
       - Impact of its degraded performance or high-error rates on the feature:
 -->
 
-Will be considered for beta.
+- The DynamicResourceAllocation feature gate must be enabled to create ResourceClaim, ResourceClaimTemplate. More details at [KEP-4381 - DRA Structured Parameters](https://github.com/kubernetes/enhancements/issues/4381)
+- A third-party DRA driver is required for how the driver should interpret the AdminAccess field to get acess to device specific resources without allocating them.
 
 ### Scalability
 
@@ -755,7 +773,7 @@ details). For now, we leave it here.
 
 ###### How does this feature react if the API server and/or etcd is unavailable?
 
-Will be considered for beta.
+The Kubernetes control plane will be down, so no new ResourceClaim or ResourceClaimTemplate will be created.
 
 ###### What are other known failure modes?
 
@@ -772,15 +790,35 @@ For each of them, fill in the following information by copying the below templat
     - Testing: Are there any tests for failure mode? If not, describe why.
 -->
 
-Will be considered for beta.
+- kube-scheduler cannot allocate ResourceClaims with AdminAccess.
+
+  - Detection: When pods fail to get scheduled, kube-scheduler reports that
+    through events and pod status. For DRA, messages include "cannot allocate
+    all claims" (insufficient resources) and "ResourceClaim not created yet"
+    (user or kube-controller-manager haven't created the ResourceClaim yet).
+    The
+    ["unschedulable_pods"](https://github.com/kubernetes/kubernetes/blob/9fca4ec44afad4775c877971036b436eef1a1759/pkg/scheduler/metrics/metrics.go#L200-L206)
+    metric will have pods counted under the "dynamicresources" plugin label.
+
+    To troubleshoot, "kubectl describe" can be used on (in this order) Pod
+    and ResourceClaim.
+
+  - Mitigations: When ResourceClaims or ResourceClaimTemplates with the `AdminAccess`
+field don't get created, debugging should focus on the namespace labels. The kube-controller-manager logs should have more information.
+
+  - Diagnostics: Audit Policy can be created to ensure all create operations on ResourceClaim, ResourceClaimTemplate, and Namespace resources are logged at the metadata level to review successful and denied attempts to set the `AdminAccess`
+field.
+
+  - Testing: E2E testing covers scenarios that successfully created ResourceClaims and ResourceClaimTemplates with the `AdminAccess` field in admin namespace and denied attempts in non-admin namespace.
 
 ###### What steps should be taken if SLOs are not being met to determine the problem?
 
-Will be considered for beta.
+If SLOs are not being met, not all 100% of unauthorized access attempts are denied. Debugging to determine the problem should review the namespace labels to verify correctness.
 
 ## Implementation History
 
 - Kubernetes 1.33: Alpha version of the KEP.
+- Kubernetes 1.34: Beta version of the KEP.
 
 ## Drawbacks
 

keps/sig-auth/5018-dra-adminaccess/kep.yaml

@@ -17,12 +17,12 @@ see-also:
   - "/keps/sig-node/4381-dra-structured-parameters"
 
 # The target maturity stage in the current dev cycle for this KEP.
-stage: alpha
+stage: beta
 
 # The most recent milestone for which work toward delivery of this KEP has been
 # done. This can be the current (upcoming) milestone, if it is being actively
 # worked on.
-latest-milestone: "v1.33"
+latest-milestone: "v1.34"
 
 # The milestone at which this feature was, or is targeted to be, at each stage.
 milestone: