Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KEP 2590: Graduate kubectl subresource to GA #5079

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
+103 −75
Open

KEP 2590: Graduate kubectl subresource to GA #5079

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bd97a39
sig-cli: graduate kubectl subresource support to GA
MadhavJivrajani Feb 1, 2024
67b794d
KEP 2590: update document for stable promotion
soltysh Jan 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
KEP 2590: update document for stable promotion 
Signed-off-by: Maciej Szulik <soltysh@gmail.com>
  • Loading branch information
@soltysh
soltysh committed Jan 28, 2025
commit 67b794d572c30548714e1d22876eba925d3bf7f2
165 changes: 83 additions & 82 deletions keps/sig-cli/2590-kubectl-subresource/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@
- [Goals](#goals)
- [Non-Goals](#non-goals)
- [Proposal](#proposal)
- [Notes/Constraints/Caveats](#notesconstraintscaveats)
- [Examples](#examples)
- [get](#get)
- [patch](#patch)
- [User Stories (Optional)](#user-stories-optional)
- [Story 1](#story-1)
- [Story 2](#story-2)
- [Risks and Mitigations](#risks-and-mitigations)
- [Design Details](#design-details)
- [Subresource support](#subresource-support)
- [Table printer](#table-printer)
Expand Down Expand Up @@ -43,43 +43,44 @@ Items marked with (R) are required *prior to targeting to a milestone / release*
- [x] (R) KEP approvers have approved the KEP status as `implementable`
- [x] (R) Design details are appropriately documented
- [x] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
- [x] e2e Tests for all Beta API Operations (endpoints)
- [x] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
- [x] (R) Minimum Two Week Window for GA e2e tests to prove flake free
- [x] (R) Graduation criteria is in place
- [x] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
- [x] (R) Production readiness review completed
- [x] (R) Production readiness review approved
- [x] "Implementation History" section is up-to-date for milestone
- [x] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
- [x] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

<!--
**Note:** This checklist is iterative and should be reviewed and updated every time this enhancement is being considered for a milestone.
-->

[kubernetes.io]: https://kubernetes.io/
[kubernetes/enhancements]: https://git.k8s.io/enhancements
[kubernetes/kubernetes]: https://git.k8s.io/kubernetes
[kubernetes/website]: https://git.k8s.io/website

## Summary

This KEP proposes supporting a new flag `--subresource` to get, patch, edit and replace kubectl
commands to fetch and update `status` and `scale` subresources.
This document proposes adding a new `--subresource` flag to the following kubectl
subcommands: `get`, `patch`, `edit`, `apply` and `replace`. The goal of this flag
is to simplify the process of fetching and updating `status` and `scale` subresources.

## Motivation

Today while testing or debugging, fetching subresources (like status) of API objects via kubectl
involves using `kubectl --raw`. Patching subresources using kubectl is not possible at all and
requires using curl directly. This method is very cumbersome and not user-friendly.
Today while testing or debugging, fetching subresources (like `status`) of API objects via kubectl
involves using `kubectl --raw`. Patching subresources using kubectl is not possible at all and
requires using `curl` directly. This method is very cumbersome and not user-friendly.

This KEP adds subresources as a first class option in kubectl to allow users to work with the API
in a generic fashion.
This enhancement adds subresources as a first class option in kubectl to allow users
to work with the API in a generic fashion.

### Goals

- Add a new flag `--subresource=[subresource-name]` to get, patch, edit
and replace kubectl commands to allow fetching and updating `status` and `scale`
subresources for all resources (built-in and CRs) that support these subresources.
- Display pretty printed table columns for the status (uses same columns as the main resource)
and scale subresources.
- Add a new flag `--subresource=[subresource-name]` to `get`, `patch`, `edit`, `apply`
and `replace` kubectl commands to allow fetching and updating `status` and `scale`
subresources for all resources (built-in and custom resources) that support these.
- Display pretty printed table columns for the `status` (uses same columns as the main resource)
and `scale` subresources.

### Non-Goals

Expand All @@ -88,25 +89,21 @@ and scale subresources.

## Proposal

kubectl commands like get, patch, edit and replace will now contain a
kubectl commands like `get`, `patch`, `edit`, `apply` and `replace` will now contain a
new flag `--subresource=[subresource-name]` which will allow fetching and updating
`status` and `scale` subresources for all API resources.

Note that the API contract against the subresource is identical to a full resource. Therefore updating
the status subresource to hold new value which could potentially be reconciled by a controller
to a different value is *expected behavior*.
Note that the API contract against the subresource is identical to a full resource.
Therefore updating the status subresource to hold new value which could potentially
be reconciled by a controller to a different value is *expected behavior*.

If `--subresource` flag is used for a resource that doesn't support the subresource,
If `--subresource` flag is used for a resource that doesn't support the subresource,
a `NotFound` error will be returned.

### Notes/Constraints/Caveats

Due to additional complexity and the general purpose of `apply` being a
declarative command, `--subresource` will not be expanded to the `apply` command.
### User Stories (Optional)

### Examples

#### get
#### Story 1

```shell
# for built-in types
Expand Down Expand Up @@ -142,7 +139,7 @@ $ kubectl get pod nginx-deployment-66b6c48dd5-dv6gl --subresource=scale
Error from server (NotFound): the server could not find the requested resource
```

#### patch
#### Story 2

```shell
# For built-in types
Expand All @@ -164,6 +161,12 @@ NAME DESIREDREPLICAS AVAILABLEREPLICAS
cron 3 2
```

### Risks and Mitigations

This feature adds a new flag which will be validated like any other flag for a limited
set of inputs. The remaining flags passed to every command will be validated as
before.

## Design Details

### Subresource support
Expand All @@ -182,8 +185,8 @@ If the subresource does not exist for an API resource, a `NotFound` error is ret

### Table printer

To support table view for subresources using kubectl get, table convertor support is added to
the scale and status subresoruces for built-in and CRD types.
To support table view for subresources using `kubectl get`, table convertor support is added to
the scale and status subresources for built-in and custom resource types.

For built-in types, `StatusStore` and `ScaleStore` are updated to implement the `TableConvertor` interface.
`StatusStore` uses the same columns as the main resource object.
Expand All @@ -193,7 +196,7 @@ The following column definitions for the `Scale` object are added to [printers.g
- `Desired Replicas` uses the json path of `.spec.replicas` of autoscalingv1.Scale object

For custom resources:
- the status subresoruce uses the same columns as defined for the full resource, i.e., `additionalPrinterColumns` defined in the CRD.
- the status subresource uses the same columns as defined for the full resource, i.e., `additionalPrinterColumns` defined in the CRD.
- the scale subresource follows the same column definitions as the built-in types, and are defined in [helpers.go].

[printers.go]: https://github.com/kubernetes/kubernetes/blob/master/pkg/printers/internalversion/printers.go#L88
Expand All @@ -207,53 +210,57 @@ to implement this enhancement.

##### Unit tests

- `k8s.io/kubernetes/pkg/printers/internalversion`: `2023-01-12` - 71.2
- `k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource`: `2023-01-12` - 70.9
- `k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/edit`: `2023-01-12` - 100
- `k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/get`: `2023-01-12` - 80.7
- `k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/patch`: `2023-01-12` - 56.3
- `k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/replace`: `2023-01-12` - 63.6
- `k8s.io/kubernetes/pkg/printers/internalversion`: `2025-01-23` - 73.5
- `k8s.io/cli-runtime/pkg/resource`: `2025-01-23` - 71.8
- `k8s.io/kubectl/pkg/cmd/apply`: `2025-01-23` - 82
- `k8s.io/kubectl/pkg/cmd/edit`: `2025-01-23` - 100
- `k8s.io/kubectl/pkg/cmd/get`: `2025-01-23` - 80.8
- `k8s.io/kubectl/pkg/cmd/patch`: `2025-01-23` - 56.4
- `k8s.io/kubectl/pkg/cmd/replace`: `2025-01-23` - 63.8

##### Integration tests

- `kubectl get`: [link to test coverage](https://github.com/kubernetes/kubernetes/blob/4802d7bb62c2623be8e4f940f6b5c1fcddd6c744/test/cmd/get.sh#L178-L184)
- [kubectl get](https://github.com/kubernetes/kubernetes/blob/00fa8f119077da3c96090aa5efc5dfc9c5a78977/test/cmd/get.sh#L178-L184): https://storage.googleapis.com/k8s-triage/index.html?pr=1&job=pull-kubernetes-integration&test=test-cmd%3A%20run_kubectl_get_tests
- [kubectl apply](https://github.com/kubernetes/kubernetes/blob/00fa8f119077da3c96090aa5efc5dfc9c5a78977/test/cmd/apply.sh#L417): https://storage.googleapis.com/k8s-triage/index.html?pr=1&job=pull-kubernetes-integration&test=test-cmd%3A%20run_kubectl_server_side_apply_tests
- [TestGetSubresourcesAsTables](https://github.com/kubernetes/kubernetes/blob/00fa8f119077da3c96090aa5efc5dfc9c5a78977/test/integration/apiserver/apiserver_test.go#L1458-L1678): https://storage.googleapis.com/k8s-triage/index.html?pr=1&job=pull-kubernetes-integration&test=TestGetSubresourcesAsTables
- [TestGetScaleSubresourceAsTableForAllBuiltins](https://github.com/kubernetes/kubernetes/blob/ed9572d9c7733602de43979caf886fd4092a7b0f/test/integration/apiserver/apiserver_test.go#L1681-L1876): https://storage.googleapis.com/k8s-triage/index.html?pr=1&job=pull-kubernetes-integration&test=TestGetScaleSubresourceAsTableForAllBuiltins

##### e2e tests

- e2e test added:
- [Link to testgrid](https://testgrid.k8s.io/sig-testing-canaries#ci-kubernetes-coverage-e2e-gci-gce&include-filter-by-regex=kubectl%20subresource)
- [Link to test](https://github.com/kubernetes/kubernetes/blob/11b974043604f5ccbeb6e5e62e1d9edcf00bc336/test/e2e/kubectl/kubectl.go#L2101)
- [kubectl subresource flag](https://github.com/kubernetes/kubernetes/blob/00fa8f119077da3c96090aa5efc5dfc9c5a78977/test/e2e/kubectl/kubectl.go#L2090-L2118): https://testgrid.k8s.io/sig-testing-canaries#ci-kubernetes-coverage-e2e-gci-gce&include-filter-by-regex=kubectl%20subresource

### Graduation Criteria

#### Alpha

- Add the `--subresource` flag to get, patch, edit and replace commands.
- Add the `--subresource` flag to `get`, `patch`, `edit` and `replace` subcommands.
- Unit tests and integration tests are added.

#### Beta

- Gather feedback from users.
- e2e tests are added.
- Add the `--subresource` flag to `apply` subcommand.

#### GA

Since v1.27 when the feature moved to beta, there have been no reported bugs concerning this feature.
In fact, it is reassuring to see the community use this feature quite commonly such as in bug reports:
https://github.com/kubernetes/kubernetes/issues/116311

Seeing this and given our added unit, integration and e2e tests gives us the confidence to graduate to GA.
https://github.com/kubernetes/kubernetes/issues/116311

Seeing this and given our added unit, integration and e2e tests gives us the confidence to graduate to stable.

### Upgrade / Downgrade Strategy

This functionality is contained entirely within kubectl and shares its strategy.
No configuration changes are required.
See [Version Skew Strategy](#version-skew-strategy).

### Version Skew Strategy

Not applicable. There is nothing required of the API Server, so there
can be no version skew.
The [kube-apiserver functionality](https://github.com/kubernetes/kubernetes/pull/103516)
required for the `--subresource` flag to work correctly was introduced in Kubernetes v1.24.
The current release (v1.33) exceeds the [supported version skew policy](https://kubernetes.io/releases/version-skew-policy/).
Therefore, there are no requirements for planning the upgrade or downgrade process.
needs to be completed.

## Production Readiness Review Questionnaire

Expand All @@ -270,21 +277,21 @@ Pick one of these and delete the rest.
- Components depending on the feature gate:
- [x] Other
- Describe the mechanism: A new flag for kubectl commands.
For the alpha stage, description will be added to expicitly call
For the alpha stage, description will be added to explicitly call
out this flag as an alpha feature.
- Will enabling / disabling the feature require downtime of the control
plane? No, disabling the feature would be a client behaviour.
plane? No, disabling the feature would be a client behavior.
- Will enabling / disabling the feature require downtime or reprovisioning
of a node?
No, disabling the feature would be a client behaviour.
of a node?
No, disabling the feature would be a client behavior.

###### Does enabling the feature change any default behavior?

While the feature now updates kubectl's behavior to allow updating subresources,
it is gated by the `--subresource` flag so it does not change kubectl's default
behavior.

Subresources can also be updated using curl today so this feature only
Subresources can also be updated using `curl` today so this feature only
provides a consistent way to use the API via kubectl, but does not allow additional
changes to the API that are not possible today.

Expand All @@ -295,8 +302,8 @@ for the `--subresource` flag and the ability to update subresources via kubectl.

However, this does not "lock" us to any changes to the subresources that were made
when the feature was enabled i.e. it does not remove the ability to update subresources
for existing API resources completely. If a subresource of an exisiting API resource needs
to be updated, this can be done via curl.
for existing API resources completely. If a subresource of an existing API resource needs
to be updated, this can be done via `curl`.

###### What happens if we reenable the feature if it was previously rolled back?

Expand All @@ -308,54 +315,48 @@ No, because it cannot be disabled or enabled in a single release.

### Rollout, Upgrade and Rollback Planning

<!--
This section must be completed when targeting beta to a release.
-->

###### How can a rollout fail? Can it impact already running workloads?

The feature is encapsulated entirely within the kubectl binary, so rollout is
an atomic client binary update. Subresources can always be updated via curl,
an atomic client binary update. Subresources can always be updated via `curl`,
so there are no version dependencies.

For kube-apiserver changes see [Version Skew Strategy](#version-skew-strategy).

###### What specific metrics should inform a rollback?

N/A

###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

This feature is completely with in the client. The upgrades and rollback of cluster will not be affected by this change.
This feature is completely within the client. The upgrades and rollback of cluster will not be affected by this change.
The update and downgrade of the kubectl version will only limit the availability of the `--subresource` flag and will not
change any API behavior.

<!--
Describe manual testing that was done and the outcomes.
Longer term, we may want to require automated upgrade/rollback tests, but we
are missing a bunch of machinery and tooling and can't do that now.
-->
For kube-apiserver changes see [Version Skew Strategy](#version-skew-strategy).

###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

No.

### Monitoring Requirements

Considering that this feature is a flag contained fully within in the `kubectl` client, there exist no
monitoring requirements for the same.

###### How can an operator determine if the feature is in use by workloads?

N/A
Cluster administrator can verify [audit entries](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/)
looking for `kubectl` invocations targeting `scale` and `status` subresources.

###### How can someone using this feature know that it is working for their instance?

N/A

###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
###### What are the reasonable SLOs (Service Level Objectives) for the above SLIs?

N/A
Since this functionality doesn't heavily modify kube-apiserver I'd expected
the SLO defined [here](https://github.com/kubernetes/community/blob/master/sig-scalability/slos/slos.md)
to apply.

###### What are the reasonable SLOs (Service Level Objectives) for the above SLIs?
###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?

N/A

Expand All @@ -367,7 +368,7 @@ N/A

###### Does this feature depend on any specific services running in the cluster?

No
No.

### Scalability

Expand Down Expand Up @@ -423,17 +424,17 @@ N/A

## Implementation History

2021-03-01: Initial [POC PR] created
2021-03-01: Initial [POC PR] created
2021-04-06: KEP proposed
2021-04-07: [Demo] in SIG CLI meeting
2022-05-25: PR for alpha implementation merged
2023-01-12: KEP graduated to Beta
2023-03-15: e2e test added for KEP as part of beta graduation
2024-02-01: KEP graduated to Stable
2025-01-23: KEP graduated to Stable

[POC PR]: https://github.com/kubernetes/kubernetes/pull/99556
[Demo]: https://youtu.be/zUa7dudYCQM?t=299

## Alternatives

Alternatives would be to use curl commands directly to update subresources.
Alternatives would be to use `curl` commands directly to update subresources.
8 changes: 5 additions & 3 deletions keps/sig-cli/2590-kubectl-subresource/kep.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,20 @@ participating-sigs:
- sig-arch
reviewers:
- "@soltysh"
- "@MadhavJivrajani"
approvers:
- "@soltysh"
- "@ardaguclu"

creation-date: 2021-04-06
last-updated: 2024-02-01
last-updated: 2025-01-23

status: implemented
stage: stable

latest-milestone: "v1.30"
latest-milestone: "v1.33"

milestone:
alpha: "v1.24"
beta: "v1.27"
stable: "v1.30"
stable: "v1.33"