docs: add implementation ideas for gep 1709

kubernetes-sigs · Feb 24, 2023 · 8ab0643 · 8ab0643
1 parent e733481
commit 8ab0643
Show file tree

Hide file tree

Showing 2 changed files with 308 additions and 3 deletions.
diff --git a/geps/gep-1709.md b/geps/gep-1709.md
@@ -7,7 +7,9 @@
 
 Add selectable profiles for conformance tests which implementations can
 subscribe to. Also add the ability to report conformance results back to the
-Gateway API project and receive recognition.
+Gateway API project and receive recognition. Conformance test reports will
+add conformance data to the implementations page for the reporter which can
+be linked to with badges from projects and repositories.
 
 ## Goals
 
@@ -24,6 +26,12 @@ Gateway API project and receive recognition.
 
 - We want to avoid adding any infrastructure for the reporting mechanism if
   feasible.
+- For this iteration we don't want to add configuration files for conformance
+  tests, instead leaving that to future iterations and working on the raw
+  machinery here (see [alternatives considered](#alternatives-considered)).
+- For this iteration we don't want to add container images for conformance test
+  runs, instead leaving that to future iterations (see
+  [alternatives considered](#alternatives-considered).
 
 ## Introduction
 
@@ -37,8 +45,304 @@ which implementations can prove they satisfy and be recognized for.
 
 ## API
 
-TODO
+The API for conformance profiles will be a pipeline that implementations can
+opt into. The workflow is effectively:
+
+1. select a [profile](#profiles)
+2. [integrate](#integration) tests in the downstream project
+3. [report results and get certified](#certification)
+
+The goal is to make selecting a conformance profile as simple of a process as
+feasible and support both the existing command line integration approach (e.g. `go test`) as well
+as a [Golang][go] approach using the conformance suite as a library.
+
+[go]:https://go.dev
+
+### Profiles
+
+Initially there will be three _categories_ of profiles which can be coupled with
+two _support levels_. These will be identified by short [CamelCase][camelcase]
+names. The initial categories we'll support will be:
+
+- `Layer4`
+- `Layer7`
+- `Mesh`
+
+> **NOTE**: these are simply the initial categories we're going to start with,
+> it's entirely reasonable for there to be more in the future.
+
+Support levels will be limited to `Core` and `Extended`, since
+`ImplementationSpecific` isn't meaningful in conformance at the time of
+writing.
+
+The initial profiles we provide certification for will be:
+
+- `Layer4Core`
+- `Layer4Extended`
+- `Layer7Core`
+- `Layer7Extended`
+
+> **NOTE**: `MeshCore` and `MeshExtended` are omitted _for now_ as of the time
+> of writing we don't actually have any mesh conformance tests. However if those
+> start to come online during initial implementation, it should be quite easy
+> to add them in given the precedent with the others.
+
+The technical implementation of these profiles is very simple: they are static
+compilations of existing [SupportedFeatures][feat] which represent the named
+profile. Contributors will need to join any new features added in the future to
+the corresponding conformance profile.
+
+Unlike `Core` the `Extended` support level presents a slight problem with the
+underlying basis on `SupportedFeatures` as implementations might support some
+(but not all) of a the `Extended` feature set. For implementations that intend
+to support _partial_ `Extended` features they will be required to _exclude_ any
+of the `Extended` features they don't support with test arguments (which we
+will cover in the [integration section](#integration)) and these exemptions
+will be reflected during the [reporting process](#reporting-process).
+
+> **NOTE**: There are situations where an implementation might actually be able
+> to fulfill conformance tests for multiple profiles (e.g. `Layer4Core` and
+> `Layer7Extended`, or perhaps an implementation can literally do everything,
+> including `Mesh`!). Rather than bother with even higher level compilations at
+> this point, we will simply report these separately and the implementations
+> can indicate that they support multiples to their end-users in whatever way
+> best suits them. If over time we find that the community is really strongly
+> in favor of compilations at a higher level we are set up to add them in a
+> follow up iteration.
+
+[camelcase]:https://en.wikipedia.org/wiki/Camel_case
+[feat]:https://github.com/kubernetes-sigs/gateway-api/blob/c61097edaa3b1fad29721e787fee4b02c35e3103/conformance/utils/suite/suite.go#L33
+
+### Integration
+
+Integrating the test suite into your implementation can be done using one of
+the following methods:
+
+- The [go test][go-test] command line interface which enables projects of any
+  language to run the test suite on any platform [Golang][go] supports.
+- Using the conformance test suite as a [Golang library][lib] within an already
+  existing test suite.
+
+> **NOTE**: Usage as a library is already an established colloquialism in the
+> community, this effort simply intends to make that more official.
+
+Selected conformance profiles can be identified as arguments in either
+configuration. For instance using the command line interface method to run the
+`Layer7Core` profile to verify core feature conformance:
+
+```console
+$ go test ./conformance/... -args -gateway-class=acme -conformance-profile=Layer7Core
+```
+
+Or the equivalent configuration using the Golang library:
+
+```go
+cSuite, err := suite.New(suite.Options{
+    GatewayClassName: "acme",
+    Profile: Layer7CoreProfile,
+    // other options
+})
+require.NoError(t, err, "misconfigured conformance test suite")
+cSuite.Setup(t)
+
+for i := 0; i < len(tests.ConformanceTests); i++ {
+    test := tests.ConformanceTests[i]
+    test.Run(t, cSuite)
+}
+```
+
+> **NOTE**: In the `suite.Options` above it's still possible to add `SkipTests`
+> but when used in conjunction with `Profile` this will result in a report that
+> the profile is not valid for reporting. Implementations in this state may be
+> able to report themselves as "in progress", see the
+> [certification section](#certification) for details.
+
+Alternatively for an `Extended` conformance profile where not all of the
+features are implemented (as described in the [profiles](#profiles) section
+above):
+
+```console
+$ go test ./conformance/... -args \
+    -gateway-class=acme \
+    -conformance-profiles=Layer7Extended,Layer4Core \
+    -unsupported-features=HTTPResponseHeaderModification,HTTPRouteMethodMatching,HTTPRouteQueryParamMatching,
+```
+
+Or the equivalent configuration using the Golang library:
+
+```go
+cSuite, err := suite.New(suite.Options{
+    GatewayClassName: "acme",
+    Profiles: sets.New(
+        Layer7ExtendedProfile,
+        Layer4Core,
+    ),
+    UnsupportedFeatures: sets.New(
+        suite.SupportHTTPResponseHeaderModification,
+        suite.SupportHTTPRouteMethodMatching,
+        suite.SupportHTTPRouteQueryParamMatching,
+    ),
+    // other options
+})
+require.NoError(t, err, "misconfigured conformance test suite")
+cSuite.Setup(t)
+
+for i := 0; i < len(tests.ConformanceTests); i++ {
+    test := tests.ConformanceTests[i]
+    test.Run(t, cSuite)
+}
+```
+
+> **NOTE**: `UnsupportedFeatures` must match features that are extended. This
+> option is invalid when used on a `Core` type profile.
+
+> **NOTE**: In the future we may consider expanding the options to configure
+> and run the conformance test suite using a container image and/or via a
+> configuration file, however it was decided to hold off from doing any of
+> those in this iteration to avoid it getting to big (see the
+> [alternatives](#alternatives-considered) section for more thoughts).
+
+Once an implementation has integrated with the conformance test suite, they can
+move on to [certification](#certification) to report the results.
+
+[go-test]:https://go.dev/doc/tutorial/add-a-test
+[go]:https://go.dev
+[lib]:https://pkg.go.dev/sigs.k8s.io/gateway-api@v0.6.1/conformance/utils/suite
+
+### Certification
+
+Implementations will be able to report their conformance testing results using
+our [reporting process](#reporting-process). Implementations will be able to
+visibly demonstrate their conformance results on their downstream projects and
+repositories using our [certification process](#certification-process).
+
+#### Reporting Process
+
+When conformance tests complete a log message will be emitted:
+
+```console
+Success! Conformance profiles "Layer7Extended" and "Layer4Core" passed for version v0.7.0
+The following extended features were flagged as not supported for "Layer7Extended":
+- HTTPResponseHeaderModification
+- HTTPRouteQueryParamMatching
+```
+
+For the above output upstream Gateway API can manually generate or update a report
+in the following format:
+
+```json
+kind: ConformanceReport
+implementation: acmeorg-acme
+profiles:
+- name: Layer7Extended
+  successes:
+  - tag: v0.7.0
+    unsupportedFeatures:
+    - HTTPResponseHeaderModification
+    - HTTPRouteMethodMatching
+  - tag: v0.6.2
+    unsupportedFeatures:
+    - HTTPResponseHeaderModification
+    - HTTPRouteMethodMatching
+    - HTTPRouteQueryParamMatching
+  - tag: v0.6.1
+    unsupportedFeatures:
+    - HTTPResponseHeaderModification
+    - HTTPRouteMethodMatching
+    - HTTPRouteQueryParamMatching
+- name: Layer4Core
+  successes:
+  - tag: v0.7.0
+  - tag: v0.6.2
+  - tag: v0.6.1
+maintainers:
+- @kubernetes-sigs/gateway-api-maintainers
+```
+
+> **NOTE**: In the above the `implementation` field is a combination of
+> `<organization>-<project>`. Organizations can be an open source organization,
+> an individual, a company, e.t.c.. Organizations can theoretically have more
+> than one `<project>` name and submit separate reports for each of them.
+
+> **NOTE**: The above assumes that the `acmeorg-acme` project was already
+> passing for versions prior to `v0.7.0` and that the new passing version was
+> added to the top of the list. This allows implementations to report their
+> historical passes which will become relevant as we get into future major
+> releases. You can also see that there's been progression to add the
+> `HTTPRouteQueryParamMatching` feature this release.
+
+> **NOTE**: The `maintainers` field indicates the Github usernames or team
+> names of those who are responsible for maintaining this file, so they can be
+> easily contacted when needed (e.g. for relevant release announcements
+> regarding conformance, e.t.c.).
+
+The implementation can compile its `ConformanceReport` and upload it to the
+Gateway API project by creating a pull request. The file should be uploaded
+to the following sub-directory:
+
+```console
+conformance/results/<organization>-<project>.yaml
+```
+
+> **NOTE**: the first time an implementation adds their report, they need
+> to also add the Golang templating to compile that report into their section
+> of the implementations page if that's not already been handled.
+
+This will start the certification process which will result in the
+implementation receiving a badge they can use on their project pages and/or
+repositories which indicates their conformance levels and link to their
+conformance reports in the upstream Gateway API website.
+
+> **NOTE**: No verification process (to prevent intentionally incorrect
+> conformance results) will be implemented at this time. We expect that this wont
+> be an issue in our community and even if someone were to try and "cheat" on
+> the reporting the reputation loss for being caught would make them look very
+> bad and would not be worth it.
+
+#### Certification Process
+
+During the process of the pull request described in the reporting process the
+implementer will be expected to run a command to compile updates before the
+pull request will be allowed to merge:
+
+```console
+make conformance.reports
+```
+
+Any updates will be compiled (via Golang templates) and will then be submitted
+as a commit in the pull request.
+
+The conformance report will emit a heading that displays the latest release
+version which conformance has been reported for, along with a note about any
+newer releases which have yet to be reported.
+
+CI will provide [badges](https://shields.io) to contributors upon completion.
+
+> **NOTE**: Badges which cover `Extended` support (such as `Layer7Extended`)
+> will look the same regardless of whether the implementation only fulfills
+> the extended feature set partially. The badge links will direct to the full
+> report, which will include a list of any feature exceptions.
+
+## Alternatives Considered
+
+### Conformance Test Configuration File
+
+Conformance testing is currently done mainly through command line with
+`go test` or via use of the conformance test suite as a library in Golang
+projects. We considered whether adding the alternative to provide a
+configuration file to the test suite would be nice, but we haven't heard
+any specific demand for that from implementors yet so we're leaving the
+idea here for later iterations to consider.
+
+### Conformance Test Container Image
+
+Providing a container image which could be fed deployment instructions for an
+implementation was considered while working on this GET but it seemed like a
+scope all unto itself so we're avoiding it here and perhaps a later iteration
+could take a look at that if there are asks in the community.
 
 ## References
 
-TODO
+- https://github.com/kubernetes-sigs/gateway-api/issues/1709
+- https://github.com/kubernetes-sigs/gateway-api/issues/1329
+
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -75,6 +75,7 @@ nav:
         - geps/gep-1426.md
         - geps/gep-1324.md
         - geps/gep-1282.md
+        - geps/gep-1709.md
       - Experimental:
         - geps/gep-1323.md
         - geps/gep-1016.md