Runing CronJobSource Problem "53: no such host"

[richard2006]

Describe the bug
About Knative Eventing v0.9.0， creating the CronJobSource, but got the problem :

{"level":"error","ts":1569760380.0051033,"logger":"fallback","caller":"cronjobevents/adapter.go:113","msg":"failed to send cloudevent{error 25 0  Post http://event-display.default.svc.cluster.local: dial tcp: lookup event-display.default.svc.cluster.local on 172.21.0.10:53: no such host}","stacktrace":"knative.dev/eventing/pkg/adapter/cronjobevents.(*Adapter).cronTick\n\t/home/prow/go/src/knative.dev/eventing/pkg/adapter/cronjobevents/adapter.go:113\nknative.dev/eventing/vendor/github.com/robfig/cron.FuncJob.Run\n\t/home/prow/go/src/knative.dev/eventing/vendor/github.com/robfig/cron/cron.go:92\nknative.dev/eventing/vendor/github.com/robfig/cron.(*Cron).runWithRecovery\n\t/home/prow/go/src/knative.dev/eventing/vendor/github.com/robfig/cron/cron.go:165"}

Expected behavior
Send the message to the event-display

To Reproduce
Create the CronJobSource:

apiVersion: sources.eventing.knative.dev/v1alpha1
kind: CronJobSource
metadata:
  name: test-cronjob
spec:
  schedule: "*/1 * * * *"
  data: '{"message": "sync"}'
  sink:
    apiVersion: serving.knative.dev/v1alpha1
    kind: Service
    name: event-display

And the event-display:

apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: event-display
  namespace: default
spec:
  template:
    spec:
      containers:
      - # This corresponds to
        # https://github.com/knative/eventing-contrib/blob/release-0.5/cmd/event_display/main.go
        image: gcr.io/knative-releases/github.com/knative/eventing-sources/cmd/event_display@sha256:bf45b3eb1e7fc4cb63d6a5a6416cf696295484a7662e0cf9ccdf5c080542c21d

Error Output:

$ kubectl logs cronjobsource-test-cronjob-9314365a-e2b0-11e9-a78e-2ef5695pjjbx

{"level":"error","ts":1569760620.0037365,"logger":"fallback","caller":"cronjobevents/adapter.go:113","msg":"failed to send cloudevent{error 25 0  Post http://event-display.default.svc.cluster.local: dial tcp: lookup event-display.default.svc.cluster.local on 172.21.0.10:53: no such host}","stacktrace":"knative.dev/eventing/pkg/adapter/cronjobevents.(*Adapter).cronTick\n\t/home/prow/go/src/knative.dev/eventing/pkg/adapter/cronjobevents/adapter.go:113\nknative.dev/eventing/vendor/github.com/robfig/cron.FuncJob.Run\n\t/home/prow/go/src/knative.dev/eventing/vendor/github.com/robfig/cron/cron.go:92\nknative.dev/eventing/vendor/github.com/robfig/cron.(*Cron).runWithRecovery\n\t/home/prow/go/src/knative.dev/eventing/vendor/github.com/robfig/cron/cron.go:165"}
{"level":"error","ts":1569760680.0039418,"logger":"fallback","caller":"cronjobevents/adapter.go:113","msg":"failed to send cloudevent{error 25 0  Post http://event-display.default.svc.cluster.local: dial tcp: lookup event-display.default.svc.cluster.local on 172.21.0.10:53: no such host}","stacktrace":"knative.dev/eventing/pkg/adapter/cronjobevents.(*Adapter).cronTick\n\t/home/prow/go/src/knative.dev/eventing/pkg/adapter/cronjobevents/adapter.go:113\nknative.dev/eventing/vendor/github.com/robfig/cron.FuncJob.Run\n\t/home/prow/go/src/knative.dev/eventing/vendor/github.com/robfig/cron/cron.go:92\nknative.dev/eventing/vendor/github.com/robfig/cron.(*Cron).runWithRecovery\n\t/home/prow/go/src/knative.dev/eventing/vendor/github.com/robfig/cron/cron.go:165"}
{"level":"error","ts":1569760740.0035992,"logger":"fallback","caller":"cronjobevents/adapter.go:113","msg":"failed to send cloudevent{error 25 0  Post http://event-display.default.svc.cluster.local: dial tcp: lookup event-display.default.svc.cluster.local on 172.21.0.10:53: no such host}","stacktrace":"knative.dev/eventing/pkg/adapter/cronjobevents.(*Adapter).cronTick\n\t/home/prow/go/src/knative.dev/eventing/pkg/adapter/cronjobevents/adapter.go:113\nknative.dev/eventing/vendor/github.com/robfig/cron.FuncJob.Run\n\t/home/prow/go/src/knative.dev/eventing/vendor/github.com/robfig/cron/cron.go:92\nknative.dev/eventing/vendor/github.com/robfig/cron.(*Cron).runWithRecovery\n\t/home/prow/go/src/knative.dev/eventing/vendor/github.com/robfig/cron/cron.go:165"}
{"level":"error","ts":1569760800.0040033,"logger":"fallback","caller":"cronjobevents/adapter.go:113","msg":"failed to send cloudevent{error 25 0  Post http://event-display.default.svc.cluster.local: dial tcp: lookup event-display.default.svc.cluster.local on 172.21.0.10:53: no such host}","stacktrace":"knative.dev/eventing/pkg/adapter/cronjobevents.(*Adapter).cronTick\n\t/home/prow/go/src/knative.dev/eventing/pkg/adapter/cronjobevents/adapter.go:113\nknative.dev/eventing/vendor/github.com/robfig/cron.FuncJob.Run\n\t/home/prow/go/src/knative.dev/eventing/vendor/github.com/robfig/cron/cron.go:92\nknative.dev/eventing/vendor/github.com/robfig/cron.(*Cron).runWithRecovery\n\t/home/prow/go/src/knative.dev/eventing/vendor/github.com/robfig/cron/cron.go:165"}

Knative release version
v0.9.0

Additional context
event-display service is running normally:

[root@iZ8vb67td1x5m1dnct3ki3Z ~]# kubectl get ksvc
NAME            URL                                        LATESTCREATED         LATESTREADY           READY   REASON
event-display   http://event-display.default.example.com   event-display-qjxbr   event-display-qjxbr   True

[lionelvillard]

Have you tried invoking event-display directly, eg. by using curl? See https://knative.dev/development/serving/samples/hello-world/helloworld-go/index.html for instructions.

[jamesward]

Similar error for me:

kubectl logs cronjobsource-cronjob-spri-403c53cd-e460-11e9-81ad-42010a8nzfnc
{"level":"info","ts":"2019-10-01T15:29:30.295Z","caller":"logging/config.go:102","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}\n"}
{"level":"info","ts":"2019-10-01T15:29:30.295Z","caller":"logging/config.go:103","msg":"Logging level set to info"}
{"level":"info","ts":"2019-10-01T15:29:30.296Z","caller":"logging/config.go:71","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: no such file or directory"}
{"level":"info","ts":"2019-10-01T15:29:30.296Z","logger":"cronjobsource","caller":"metrics/config.go:235","msg":"Flushing the existing exporter before setting up the new exporter."}
{"level":"info","ts":"2019-10-01T15:29:30.296Z","logger":"cronjobsource","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/sources cronjobsource prometheus 5000000000 9090  false false  }. Start the server for Prometheus exporter."}
{"level":"info","ts":"2019-10-01T15:29:30.296Z","logger":"cronjobsource","caller":"metrics/config.go:244","msg":"Successfully updated the metrics exporter; old config: <nil>; new config &{knative.dev/sources cronjobsource prometheus 5000000000 9090  false false  }"}
{"level":"error","ts":"2019-10-01T15:29:30.412Z","logger":"cronjobsource","caller":"tracing/opencensus.go:146","msg":"error building zipkin endpoint","error":"lookup cronjobsource on 10.0.0.10:53: no such host","stacktrace":"knative.dev/eventing/vendor/knative.dev/pkg/tracing.WithExporter.func1\n\t/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/pkg/tracing/opencensus.go:146\nknative.dev/eventing/vendor/knative.dev/pkg/tracing.(*OpenCensusTracer).ApplyConfig\n\t/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/pkg/tracing/opencensus.go:58\nknative.dev/eventing/pkg/tracing.SetupStaticPublishing\n\t/home/prow/go/src/knative.dev/eventing/pkg/tracing/setup.go:62\nmain.main\n\t/home/prow/go/src/knative.dev/eventing/cmd/cronjob_receive_adapter/main.go:107\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
{"level":"error","ts":"2019-10-01T15:29:30.412Z","logger":"cronjobsource","caller":"cronjob_receive_adapter/main.go:110","msg":"Error setting up trace publishing","error":"unable to set OpenCensusTracing config: lookup cronjobsource on 10.0.0.10:53: no such host","stacktrace":"main.main\n\t/home/prow/go/src/knative.dev/eventing/cmd/cronjob_receive_adapter/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
{"level":"info","ts":"2019-10-01T15:29:30.412Z","logger":"cronjobsource","caller":"cronjob_receive_adapter/main.go:122","msg":"Starting Receive Adapter","adapter":{"Schedule":"* * * * *","Data":"{\"message\": \"hello, world\"}","SinkURI":"http://default-broker.default.svc.cluster.local","Name":"cronjob-spring-upper","Namespace":"default","Reporter":{}}}

[jamesward]

I'm getting similar errors with any event consumer that is a knative service. Switching to a deployment + k8s service gets things working, but not ideal.

[meteatamel]

I also ran into the same issue with my HelloWorld step in knative-tutorial. I was told that I need to update Istio use cluster local gateway as explained here. I haven't tried this yet.

[vaikas]

@tcnghia Are there any known issues? Is this something that should work with the 0.9 comprehensive install, or do you need to do something else to get this working? I'd expect that the comprehensive version of the install would support this out of the box?
Is there some step missing from our documentation? I'm just curious if this is indeed an eventing problem (looks like it might not be?) but seems to be an issue either with the:

1. installation
2. istio misconfigured
3. me?

[lionelvillard]

or maybe this: https://knative.dev/development/install/installing-istio/ section Updating your install to use cluster local gateway

[richard2006]

Have you tried invoking event-display directly, eg. by using curl? See https://knative.dev/development/serving/samples/hello-world/helloworld-go/index.html for instructions.

Yes, I have tried invoking event-display directly, but it seems no problem.

[root@iZ2zeae8wzyq0ypgjowzq2Z ~]# curl -H "host:event-display.default.example.com" http://39.106.232.122
{"error":"unknown encoding for message &{map[Accept:[*/*] Accept-Encoding:[gzip] Forwarded:[for=172.20.0.1;proto=http, for=127.0.0.1] K-Proxy-Request:[activator] User-Agent:[curl/7.29.0] X-Forwarded-For:[172.20.0.1, 127.0.0.1, 172.20.2.55] X-Forwarded-Proto:[http] X-Istio-Attributes:[Ck8KF2Rlc3RpbmF0aW9uLnNlcnZpY2UudWlkEjQSMmlzdGlvOi8vZGVmYXVsdC9zZXJ2aWNlcy9ldmVudC1kaXNwbGF5LWsyY21wLWY5ZndoClEKGGRlc3RpbmF0aW9uLnNlcnZpY2UuaG9zdBI1EjNldmVudC1kaXNwbGF5LWsyY21wLWY5ZndoLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwKNwoYZGVzdGluYXRpb24uc2VydmljZS5uYW1lEhsSGWV2ZW50LWRpc3BsYXktazJjbXAtZjlmd2gKKgodZGVzdGluYXRpb24uc2VydmljZS5uYW1lc3BhY2USCRIHZGVmYXVsdApHCgpzb3VyY2UudWlkEjkSN2t1YmVybmV0ZXM6Ly9hY3RpdmF0b3ItODZjY2M4NmM5Ny1odGxjcC5rbmF0aXZlLXNlcnZpbmc=] X-Request-Id:[e33e0cfa-f18b-48a0-8590-31254bbeaec0]] []}"}[root@iZ2zeae8wzyq0ypgjowzq2Z ~]# kubectl get pods
NAME                                                              READY   STATUS    RESTARTS   AGE
cronjobsource-test-cronjob-4ef4fd29-e299-11e9-aad4-569953f7gz46   1/1     Running   0          3d20h
event-display-k2cmp-deployment-7d654d4545-k7kk5                   2/2     Running   0          18s
kafka-default-0                                                   1/1     Running   0          4d4h
kafka-source-bxk22-85c7478db5-kh5lp                               1/1     Running   0          3d20h
nginx-577596968c-g5kxc                                            1/1     Running   0          15d
nginx-577596968c-x45hk                                            1/1     Running   0          15d
zookeeper-default-0                                               1/1     Running   0          4d4h

[richard2006]

@tcnghia Are there any known issues? Is this something that should work with the 0.9 comprehensive install, or do you need to do something else to get this working? I'd expect that the comprehensive version of the install would support this out of the box?
Is there some step missing from our documentation? I'm just curious if this is indeed an eventing problem (looks like it might not be?) but seems to be an issue either with the:

1. installation
2. istio misconfigured
3. me?

Hi @vaikas-google , I think you're right, it's not an eventing problem. I found the VirtualService have been changed between the v0.8.0 and the v0.9.0.
With the 0.9.0 of event-display:

  http:
  - match:
    - authority:
        regex: ^event-display\.default\.example\.com(?::\d{1,5})?$
      gateways:
      - knative-serving/knative-ingress-gateway
    - authority:
        regex: ^event-display\.default(\.svc(\.cluster\.local)?)?(?::\d{1,5})?$
      gateways:
      - knative-serving/cluster-local-gateway

With the 0.8.0 of event-display:

  http:
  - match:
    - authority:
        regex: ^event-display\.default\.example\.com(?::\d{1,5})?$
      gateways:
      - knative-serving/knative-ingress-gateway
    - authority:
        regex: ^event-display\.default(\.svc(\.cluster\.local)?)?(?::\d{1,5})?$
      gateways:
      - knative-serving/knative-ingress-gateway

[lionelvillard]

Have you upgraded to serving 0.9.0? In that case you also need to install the istio cluster-local-gateway.

[vaikas]

Thanks all for the pointers, yes, the current instructions that I had been following for the repro omitted the installation of the cluster-local-gateway.

[vaikas]

and little more, I had been using GKE Istio addon, and trying to install the cluster-local-gateway appeared to be problematic, so mayhaps we need to update the installation instructions for that.
@lionelvillard did you install Istio manually yourself? Or did the Istio come with your cluster? Just trying to figure out what all might need to get updated in our docs.
@richard2006 which platform are you running on btw?