retry on webhook failures

[vaikas]

Addresses #3904
(and probably others)

Add retries that retry in case of webhook failures. We already do these for various create/update methods, but this handles the getgeneric objects.

Proposed Changes

• Add a retry for the webhook failures that we see occasionally.

Release Note

- 🐛 Fix bug
Tests sometimes flake when the webhook fails.

Docs

[vaikas]

Failed with not being able to dispatch to transformer pod, from broker-filter logs:

{"level":"error","ts":"2020-08-21T20:59:28.703Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:212","msg":"failed to send event","commit":"a7c8c76","error":"failed to dispatch message: Post \"http://transformer-pod.test-broker-v1-beta1-data-plane-consumer-kfk5c.svc.cluster.local/\": dial tcp 10.11.248.254:80: connect: connection refused","stacktrace":"knative.dev/eventing/pkg/mtbroker/filter.(*Handler).send\n\tknative.dev/eventing/pkg/mtbroker/filter/filter_handler.go:212\nknative.dev/eventing/pkg/mtbroker/filter.(*Handler).ServeHTTP\n\tknative.dev/eventing/pkg/mtbroker/filter/filter_handler.go:204\nknative.dev/eventing/pkg/health.WithReadinessCheck.func1\n\tknative.dev/eventing/pkg/health/probes.go:44\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2041\nknative.dev/eventing/pkg/health.WithLivenessCheck.func1\n\tknative.dev/eventing/pkg/health/probes.go:34\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2041\ngo.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP\n\tgo.opencensus.io@v0.22.4/plugin/ochttp/server.go:92\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2836\nnet/http.(*conn).serve\n\tnet/http/server.go:1924"}

and IMC got this:

{"level":"error","ts":"2020-08-21T20:59:28.705Z","logger":"inmemorychannel-dispatcher","caller":"fanout/fanout_message_handler.go:189","msg":"Fanout had an error","commit":"a7c8c76","error":"unable to complete request to http://broker-filter.knative-eventing-4i4m0jhs2s.svc.cluster.local/triggers/test-broker-v1-beta1-data-plane-consumer-kfk5c/transform-trigger/7f09a4ae-f23d-4578-b7c2-6bf536f29430: unexpected HTTP response, expected 2xx, got 500","stacktrace":"knative.dev/eventing/pkg/channel/fanout.(*MessageHandler).dispatch\n\tknative.dev/eventing/pkg/channel/fanout/fanout_message_handler.go:189\nknative.dev/eventing/pkg/channel/fanout.createMessageReceiverFunction.func1.1\n\tknative.dev/eventing/pkg/channel/fanout/fanout_message_handler.go:143"}

[vaikas]

/test pull-knative-eventing-conformance-tests

[knative-test-reporter-robot]

The following jobs failed:

Test name	Triggers	Retries
pull-knative-eventing-conformance-tests		0/3
pull-knative-eventing-integration-tests		0/3

Failed non-flaky tests preventing automatic retry of pull-knative-eventing-integration-tests:

test/e2e.TestBrokerChannelFlowTriggerV1Beta1BrokerV1Beta1/Channel-messaging.knative.dev/v1
test/e2e.TestBrokerChannelFlowTriggerV1Beta1BrokerV1Beta1

[vaikas]

Integration test failed with:

    broker_channel_flow_helper.go:157: Failed to get the url for the channel "e2e-brokerchannel-channel": addressable does not have an Address: &{TypeMeta:{Kind:Channel APIVersion:messaging.knative.dev/v1} ObjectMeta:{Name:e2e-brokerchannel-channel GenerateName: Namespace:test-broker-channel-flow-trigger-v1-beta1-broker-v1-beta1-d6pmk SelfLink:/apis/messaging.knative.dev/v1/namespaces/test-broker-channel-flow-trigger-v1-beta1-broker-v1-beta1-d6pmk/channels/e2e-brokerchannel-channel UID:8dad82ed-e8db-400c-ab04-e29de7b073d8 ResourceVersion:42248 Generation:1 CreationTimestamp:2020-08-21 21:08:32 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[messaging.knative.dev/creator:prow-job@knative-tests.iam.gserviceaccount.com messaging.knative.dev/lastModifier:prow-job@knative-tests.iam.gserviceaccount.com messaging.knative.dev/subscribable:v1] OwnerReferences:[] Finalizers:[] ClusterName: ManagedFields:[]} Status:{Address:<nil>}}

And indeed the channel is not ready.

    tracker.go:128: Cleaning resource: "e2e-brokerchannel-channel"
        {
          "apiVersion": "messaging.knative.dev/v1",
          "kind": "Channel",
          "metadata": {
            "annotations": {
              "messaging.knative.dev/creator": "prow-job@knative-tests.iam.gserviceaccount.com",
              "messaging.knative.dev/lastModifier": "prow-job@knative-tests.iam.gserviceaccount.com",
              "messaging.knative.dev/subscribable": "v1"
            },
            "creationTimestamp": "2020-08-21T21:08:32Z",
            "generation": 1,
            "name": "e2e-brokerchannel-channel",
            "namespace": "test-broker-channel-flow-trigger-v1-beta1-broker-v1-beta1-d6pmk",
            "resourceVersion": "42248",
            "selfLink": "/apis/messaging.knative.dev/v1/namespaces/test-broker-channel-flow-trigger-v1-beta1-broker-v1-beta1-d6pmk/channels/e2e-brokerchannel-channel",
            "uid": "8dad82ed-e8db-400c-ab04-e29de7b073d8"
          },
          "spec": {
            "channelTemplate": {
              "apiVersion": "messaging.knative.dev/v1",
              "kind": "InMemoryChannel"
            }
          },
          "status": {
            "channel": {
              "apiVersion": "messaging.knative.dev/v1",
              "kind": "InMemoryChannel",
              "name": "e2e-brokerchannel-channel",
              "namespace": "test-broker-channel-flow-trigger-v1-beta1-broker-v1-beta1-d6pmk"
            },
            "conditions": [
              {
                "lastTransitionTime": "2020-08-21T21:08:50Z",
                "message": "hostname is the empty string",
                "reason": "EmptyHostname",
                "status": "False",
                "type": "Addressable"
              },
              {
                "lastTransitionTime": "2020-08-21T21:08:50Z",
                "message": "unsuccessfully observed a new generation",
                "reason": "NewObservedGenFailure",
                "status": "Unknown",
                "type": "BackingChannelReady"
              },
              {
                "lastTransitionTime": "2020-08-21T21:08:50Z",
                "message": "hostname is the empty string",
                "reason": "EmptyHostname",
                "status": "False",
                "type": "Ready"
              }
            ],
            "observedGeneration": 1
          }
        }

Digging...

[lberk]

/retest

[matzew]

/lgtm
/approve

[knative-prow-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: matzew, vaikas

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• test/OWNERS [matzew,vaikas]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment