-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kustomize Resource Ordering Proposal #865
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,179 @@ | ||
--- | ||
title: Kustomize Resource Ordering | ||
authors: | ||
- "@pwittrock" | ||
owning-sig: sig-cli | ||
participating-sigs: | ||
- sig-apps | ||
- sig-api-machinery | ||
reviewers: | ||
- "@apelisse" | ||
- "@anguslees" | ||
approvers: | ||
- "@monopole" | ||
editors: | ||
- "@pwittrock" | ||
creation-date: 2019-03-1 | ||
last-updated: 2019-03-1 | ||
status: implementable | ||
see-also: | ||
replaces: | ||
superseded-by: | ||
- n/a | ||
--- | ||
|
||
# Kustomize Resource Ordering | ||
|
||
## Table of Contents | ||
* [Table of Contents](#table-of-contents) | ||
* [Summary](#summary) | ||
* [Motivation](#motivation) | ||
* [Goals](#goals) | ||
* [Non-Goals](#non-goals) | ||
* [Proposal](#proposal) | ||
* [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints) | ||
* [Risks and Mitigations](#risks-and-mitigations) | ||
* [Graduation Criteria](#graduation-criteria) | ||
* [Implementation History](#implementation-history) | ||
* [Alternatives](#alternatives) | ||
|
||
[Tools for generating]: https://github.com/ekalinin/github-markdown-toc | ||
|
||
## Summary | ||
|
||
Kustomize orders Resource creation by sorting the Resources it emits based off their type. While | ||
this works well for most types (e.g. create Namespaces before other things), it doesn't work | ||
in all cases and users will need the ability to break glass. | ||
|
||
See [kubernetes-sigs/kustomize#836] for an example. | ||
|
||
## Motivation | ||
|
||
Users may need direct control of the Resource create / update / delete ordering in cases were | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The ordering for each of those operations may not always be the same |
||
sorting by Resource Type is insufficient. | ||
|
||
### Goals | ||
|
||
- Provide the ability for users to override the order that Kustomize emits Resources | ||
- Used by `kubectl apply` or order Resource operations | ||
- Used to override creation order for meta-Resources such as Namespaces | ||
|
||
### Non-Goals | ||
|
||
- Ensure certain Resources are *Settled* or *Ready* before other Resources | ||
- Ensure dependencies between Workloads | ||
|
||
## Proposal | ||
|
||
Provide a simple mechanism allowing users to override the order that | ||
Resource operations are Applied. Add a new field `sortOrder` that is | ||
a list of `ResourceSelector`s which match Resources based | ||
off their annotations. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Within one kustomization target, I think this solves the ordering problem. I'd like to see explanations or examples covering bases and overlays. Given an overlay composed from multiple bases. Does the order from each base get inherited in overlays? What if I want resources from one base to be prior to resources from another base? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good point, what about conflicting orderings (cycles) |
||
|
||
```go | ||
|
||
type Kustomization struct { | ||
// ... | ||
|
||
// sortOrder defines a precedence for ordering Resources. Resources | ||
// will be sorted in the order the match a ResourecSelector before | ||
// they are emitted. | ||
SortOrder []ResourceSelector `json:"sortOrder"` | ||
|
||
// ... | ||
} | ||
|
||
// ResourceSelector selects Resources that it matches | ||
type ResourceSelector struct { | ||
// matchAnnotations is a map of {key,value} pairs. A Resource matches if it has | ||
// *all* of the annotations in matchAnnotations appear in the Resource metaData. | ||
// Null and empty values are not allowed. | ||
// +optional | ||
MatchAnnotations map[string]string `json:"matchAnnotations,omitempty"` | ||
|
||
// TODO: Consider adding field: MatchExpressions []AnnotationSelectorRequirement | ||
} | ||
|
||
``` | ||
Example: | ||
|
||
``` | ||
sortOrder: | ||
- matchAnnotations: | ||
some-annotation-name-1: some-annotation-value-1 | ||
some-annotation-name-a: some-label-value-a | ||
- matchAnnotations: | ||
some-annotation-name-1: some-annotation-value-1 | ||
- matchAnnotations: | ||
some-annotation-name-2: some-annotation-value-2 | ||
``` | ||
|
||
The explicit user defined ordering using annotations will take precedence over the type based | ||
orderings. Types would be used as a fallback sorting function amongst Resource with equal | ||
precedence in the explicit ordering. | ||
|
||
- Resources annotated with `some-annotation-name-1: some-annotation-value-1` *and* annotated | ||
with `some-annotation-name-a: some-annotation-value-a` are first | ||
- These Resources are sorted by type | ||
- Resources annotated with `some-annotation-name-1=some-annotation-value-1` | ||
(without `some-annotation-name-a: some-annotation-value-a`) appear second | ||
- These Resources are sorted by type | ||
- Resources annotated with `some-annotation-name-2=some-annotation-value-2` appear third | ||
- These Resources are sorted by type | ||
- Resources not matching any annotation are last | ||
- These Resources are sorted by type | ||
|
||
Resources matching multiple orderings (e.g. have multiple matching annotations) appear | ||
in the position matching the earliest label / annotation. | ||
|
||
**Note:** Throw an error if there is a selector that selects a superset of another selector | ||
and it appears first. e.g. this should throw an error because the first selector will match | ||
everything that the second selector does, and it will have no effect. | ||
|
||
``` | ||
sortOrder: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How about generating the output in the same order as they are listed in
|
||
- matchAnnotations: | ||
some-annotation-name-1: some-annotation-value-1 | ||
- matchAnnotations: | ||
some-annotation-name-1: some-annotation-value-1 | ||
some-annotation-name-a: some-label-value-a | ||
``` | ||
|
||
### Risks and Mitigations | ||
|
||
Risk: Users build complex orderings that are hard to reason about. | ||
Mitigation: Good documentation and recommendations about how to keep things simple. | ||
|
||
## Graduation Criteria | ||
|
||
Use customer feedback to determine if we should support: | ||
|
||
- `LabelSelectors` | ||
- `AnnotationSelectorRequirement` (like `LabelSelectorRequirement` but for annotations) | ||
- Explicitly order "default" (e.g. Resources that don't match any of the selectors) instead of | ||
them being last. | ||
|
||
|
||
## Docs | ||
|
||
Update Kubectl Book and Kustomize Documentation | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. +1 |
||
|
||
## Test plan | ||
|
||
Unit Tests for: | ||
|
||
- [ ] Resources with annotations follow the ordering override semantics | ||
- [ ] Resources matching multiple orderings land in the right spot | ||
- [ ] Throw an error if a superset selector appears after its subset (e.g. a more restrict selector appears later) | ||
- This will have no effect, as everything will have already been matched | ||
- [ ] Resources are sorted by type as a secondary factor | ||
- [ ] Having multiple annotations with the same key and different values works correctly | ||
- [ ] Having multiple annotations with different keys and the same values works correctly | ||
- [ ] Empty and Null `MatchAnnotations` throw an error | ||
- [ ] Resources that don't appear in the ordering overrides appear last and are sorted by type | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Well, that will be one of the most thoroughly tested features we have, thanks! The place to most of these is perhaps in pkg/target (see any test file in there). |
||
|
||
### Version Skew Tests | ||
|
||
## Implementation History | ||
|
||
## Alternatives | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please include a discussion of at least some alternatives, otherwise what's the point of the KEP? In particular for this doc: The obvious alternative that I think needs to be considered is just process the resources in original document order. This is presumably the order intended by the upstream manifest author (since this is kubectl's behaviour). We would need a bit of explanation for the kustomize corner cases like newly-inserted resources, and when multiple upstreams are merged (conceivably any order that preserves the original separtae document order(s) is fine here). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
To get other community members to come up with alternatives :) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Would this be mutually exclusive with the sorted order, so users would need to manually put CRDs first, etc? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I don't understand... Obviously you can't primary-sort by both document order and some other (eg: type-based) order.
Yes, "document order" assumes that the user (or some other tool) has appropriately sorted the resources in the input document. Again: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can guess why, but it's not obvious when this ordering becomes problematic and when you can't rely on type ordering anymore (or at least an example would be useful)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you look at the example in the motivation?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You mean in the body of the PR, not in that KEP right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you replace "break glass" with a more direct but short mention of using labels and annotaitons?
Also, can we just get by with one or the other at the outset? I.e. ship label-based ordering then add annos if someone asks for it (or vice versa)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like this idea, I think it's fine to be opinionated. You could almost enforce a specific annotation and have kustomize remove it. Giving people the chance to use any label/annotation is maybe not restrictive enough, as they will be able to build arbitrary complex things with a poor semantic (noted in "risk & mitigation", I know), but I think we could help them by making a reasonable decision on what annotations to use (e.g. I know you won't like it:
kustomize.io-order-priority: X
).Alternative solutions:
I don't know if the order should be in each object (more implicit, and less coupled) or in the kustomize file directly (stronger coupling, more explicit).
You could also build something like an "edge"/graph based solution, e.g. this object before that one, or this after that.