Conformance profiles prototype

[mlavacca]

What type of PR is this?

/kind feature
/area conformance

What this PR does / why we need it:

This PR introduces the conformance profile machinery to verify compliance with given conformance profiles. It should be considered an implementation of https://github.com/kubernetes-sigs/gateway-api/blob/main/geps/gep-1709.md. The conformance profile test suite has been integrated and tested with Kong Kubernetes Ingress Controller, and below it is possible to see the outcome of such a test:

{
  "implementation": {
    "organization": "Kong",
    "project": "Kubernetes Ingress Controller",
    "url": "https://github.com/Kong/kubernetes-ingress-controller",
    "version": "2.9.3",
    "contact": [
      "@mlavacca"
    ]
  },
  "date": "2023-04-24T13:24:05+02:00",
  "gatewayAPIVersion": "TODO",
  "profiles": [
    {
      "name": "HTTP",
      "core": {
        "result": "partial",
        "summary": "",
        "statistics": {
          "Passed": 26,
          "Skipped": 3,
          "Failed": 0
        },
        "skippedTests": [
          "GatewayModifyListeners",
          "HTTPRouteHeaderMatching",
          "GatewayWithAttachedRoutes"
        ]
      },
      "extended": {
        "status": {
          "result": "success",
          "summary": "",
          "statistics": {
            "Passed": 1,
            "Skipped": 0,
            "Failed": 0
          }
        },
        "supportedFeatures": [
          "HTTPRouteMethodMatching"
        ],
        "unsupportedFeatures": [
          "HTTPRouteSchemeRedirect",
          "HTTPRoutePathRedirect",
          "HTTPRouteHostRewrite",
          "HTTPRoutePathRewrite",
          "HTTPRouteQueryParamMatching",
          "HTTPResponseHeaderModification",
          "HTTPRoutePortRedirect"
        ]
      }
    },
    {
      "name": "TLS",
      "core": {
        "result": "partial",
        "summary": "",
        "statistics": {
          "Passed": 7,
          "Skipped": 3,
          "Failed": 0
        },
        "skippedTests": [
          "GatewayModifyListeners",
          "GatewayWithAttachedRoutes",
          "TLSRouteSimpleSameNamespace"
        ]
      }
    }
  ]
}

The result above has been produced by using the following settings:

cSuite, err := suite.NewExperimentalConformanceTestSuite(suite.ExperimentalConformanceOptions{
  Options: suite.Options{
	  Client:               client,
	  GatewayClassName:     gatewayClass.Name,
	  Debug:                showDebug,
	  CleanupBaseResources: shouldCleanup,
	  BaseManifests:        conformanceTestsBaseManifests,
	  SupportedFeatures: sets.New(
		  // core
		  suite.SupportGateway,
		  suite.SupportReferenceGrant,
		  suite.SupportHTTPRoute,
		  suite.SupportTLSRoute,
  
		  // extended
		  suite.SupportHTTPRouteMethodMatching,
		  suite.SupportRouteDestinationPortMatching,
	  ),
	  SkipTests: []string{
		  // standard conformance
		  tests.HTTPRouteHeaderMatching.ShortName,
		  tests.GatewayWithAttachedRoutes.ShortName,
		  tests.GatewayModifyListeners.ShortName,
  
		  // TLSRoute
		  // https://github.com/Kong/kubernetes-ingress-controller/issues/3678
		  tests.TLSRouteSimpleSameNamespace.ShortName,
	  },
  },
  ConformanceProfiles: sets.New(
	  suite.HTTPConformanceProfileName,
	  suite.TLSConformanceProfileName,
  ),
  Implementation: confv1alpha1.Implementation{
	  Organization: "Kong",
	  Project:      "Kubernetes Ingress Controller",
	  URL:          "https://github.com/Kong/kubernetes-ingress-controller",
	  Version:      "2.9.3",
	  Contact:      []string{"@mlavacca"},
  },
})
assert.NoError(t, err)

cSuite.Setup(t)
cSuite.Run(t, tests.ConformanceTests)
report, err := cSuite.Report()
assert.NoError(t, err)

Which issue(s) this PR fixes:

Fixes #1784

Does this PR introduce a user-facing change?:

Addition of the conformance profiles. This feature enables an implementation to check compliance against a set of predetermined supported features and to produce a standard result (in the form of the `ConformanceProfile` API).

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[arkodg]

looks good @mlavacca, added some minor comments

[sunjayBhatia]

interesting data point (run with all supported features enabled and a few skipped tests):

{
  "implementation": {
    "organization": "projectcontour",
    "project": "contour",
    "url": "https://github.com/projectcontour/contour",
    "version": "foo",
    "contact": [
      "@bar"
    ]
  },
  "date": "2023-05-01T21:55:53Z",
  "gatewayAPIVersion": "TODO",
  "profiles": [
    {
      "name": "HTTP",
      "core": {
        "result": "failure",
        "summary": "",
        "statistics": {
          "Passed": 27,
          "Skipped": 1,
          "Failed": 1
        },
        "skippedTests": [
          "HTTPRouteRedirectHostAndStatus"
        ],
        "failedTests": [
          "GatewayObservedGenerationBump"
        ]
      },
      "extended": {
        "status": {
          "result": "partial",
          "summary": "",
          "statistics": {
            "Passed": 6,
            "Skipped": 2,
            "Failed": 0
          },
          "skippedTests": [
            "HTTPRouteRedirectScheme",
            "HTTPRouteRedirectPath"
          ]
        }
      }
    },
    {
      "name": "TLS",
      "core": {
        "result": "failure",
        "summary": "",
        "statistics": {
          "Passed": 9,
          "Skipped": 0,
          "Failed": 1
        },
        "failedTests": [
          "GatewayObservedGenerationBump"
...

looks like GatewayObservedGenerationBump (or any other common test between multiple profiles) shows up in two separate profiles, which I guess makes sense but could be a bit confusing when we go over the summary stats

[Xunzhuo]

LGTM, thanks for working on it @mlavacca !

[mlavacca]

looks like GatewayObservedGenerationBump (or any other common test between multiple profiles) shows up in two separate profiles, which I guess makes sense but could be a bit confusing when we go over the summary stats

Yes, all the failing tests for a specific profile are enumerated in their own failingTests section. When multiple profiles share the same failing tests, we have the result you posted above. Where would you put such a piece of shared information to improve the interface readability?

[youngnick]

This LGTM, so I'll approve and hold, and wait for further maintainer review.

/approve
/hold

[mlavacca]

@robscott Can you please take another look? All your comments should be addressed.

[robscott]

Thanks @mlavacca! Sorry for the delay on getting back to this one. This LGTM, have one tiny non-blocking nit but that can be handled in a follow up.

/lgtm
/approve
/hold cancel

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: keithmattix, mlavacca, robscott, sunjayBhatia, Xunzhuo, youngnick

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• conformance/OWNERS [robscott,youngnick]
• geps/OWNERS [robscott,youngnick]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

New changes are detected. LGTM label has been removed.

[mlavacca]

@robscott there were conflicts, and the rebase removed the LGTM label. Can you please lgtm it again?