knative-serving error applying manifests when knative-operator is not done installing

[NohaIhab]

Bug Description

knative-serving goes to error (unhandled exception) when it applies manifests if the KnativeServing CRD is not yet created. This CRD is created by knative-operator.
knative-serving should check on the CRD before applying the KnativeServing CR, and handle the exception if it is not there.

To Reproduce

this run of the CI in kserve integration tests:
https://github.com/canonical/kserve-operators/actions/runs/6227740976/job/16903015665

Environment

juju 3.1/stable
microk8s 1.25-strict/stable

Relevant log output

unit-knative-serving-0: 20:55:28 ERROR unit.knative-serving/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 190, in <module>
    main(KnativeServingCharm)
  File "/var/lib/juju/agents/unit-knative-serving-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-knative-serving-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-knative-serving-0/charm/venv/ops/framework.py", line 344, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-knative-serving-0/charm/venv/ops/framework.py", line 841, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-knative-serving-0/charm/venv/ops/framework.py", line 930, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 102, in _on_install
    self._apply_and_set_status()
  File "./src/charm.py", line 64, in _apply_and_set_status
    self.resource_handler.apply()
  File "/var/lib/juju/agents/unit-knative-serving-0/charm/venv/charmed_kubeflow_chisme/kubernetes/_kubernetes_resource_handler.py", line 340, in apply
    apply_many(
  File "/var/lib/juju/agents/unit-knative-serving-0/charm/venv/charmed_kubeflow_chisme/lightkube/batch/_many.py", line 72, in apply_many
    returns[i] = client.apply(
  File "/var/lib/juju/agents/unit-knative-serving-0/charm/venv/lightkube/core/client.py", line 457, in apply
    return self.patch(type(obj), name, obj, namespace=namespace,
  File "/var/lib/juju/agents/unit-knative-serving-0/charm/venv/lightkube/core/client.py", line 325, in patch
    return self._client.request("patch", res=res, name=name, namespace=namespace, obj=obj,
  File "/var/lib/juju/agents/unit-knative-serving-0/charm/venv/lightkube/core/generic_client.py", line 245, in request
    return self.handle_response(method, resp, br)
  File "/var/lib/juju/agents/unit-knative-serving-0/charm/venv/lightkube/core/generic_client.py", line 196, in handle_response
    self.raise_for_status(resp)
  File "/var/lib/juju/agents/unit-knative-serving-0/charm/venv/lightkube/core/generic_client.py", line 190, in raise_for_status
    raise transform_exception(e)
  File "/var/lib/juju/agents/unit-knative-serving-0/charm/venv/lightkube/core/generic_client.py", line 188, in raise_for_status
    resp.raise_for_status()
  File "/var/lib/juju/agents/unit-knative-serving-0/charm/venv/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://10.152.183.1/apis/operator.knative.dev/v1beta1/namespaces/knative-serving/knativeservings/knative-serving?force=true&fieldManager=kubeflow'

Additional context

No response

[orfeas-k]

This happens also to knative-eventing with the following logs

unit-knative-eventing-0: 17:25:02.808 INFO juju.cmd.containeragent.unit start "unit"
unit-knative-eventing-0: 17:25:02.808 INFO juju.worker.upgradesteps upgrade steps for 3.4.5 have already been run.
unit-knative-eventing-0: 17:25:02.809 INFO juju.worker.probehttpserver starting http server on 127.0.0.1:65301
unit-knative-eventing-0: 17:25:02.828 INFO juju.api cannot resolve "controller-service.controller-microk8s.svc.cluster.local": lookup controller-service.controller-microk8s.svc.cluster.local: operation was canceled
unit-knative-eventing-0: 17:25:02.828 INFO juju.api connection established to "wss://10.152.183.179:17070/model/09b94b46-eaeb-4dfc-8162-72f40b6cbbee/api"
unit-knative-eventing-0: 17:25:02.830 INFO juju.worker.apicaller [09b94b] "unit-knative-eventing-0" successfully connected to "10.152.183.179:17070"
unit-knative-eventing-0: 17:25:02.862 INFO juju.api connection established to "wss://controller-service.controller-microk8s.svc.cluster.local:17070/model/09b94b46-eaeb-4dfc-8162-72f40b6cbbee/api"
unit-knative-eventing-0: 17:25:02.864 INFO juju.worker.apicaller [09b94b] "unit-knative-eventing-0" successfully connected to "controller-service.controller-microk8s.svc.cluster.local:17070"
unit-knative-eventing-0: 17:25:02.886 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-knative-eventing-0: 17:25:02.886 INFO juju.worker.logger logger worker started
unit-knative-eventing-0: 17:25:02.890 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-knative-eventing-0: 17:25:02.895 INFO juju.worker.leadership knative-eventing/0 promoted to leadership of knative-eventing
unit-knative-eventing-0: 17:25:02.902 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-knative-eventing-0: 17:25:02.902 INFO juju.worker.caasupgrader unblocking abort check
unit-knative-eventing-0: 17:25:02.910 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-knative-eventing-0
unit-knative-eventing-0: 17:25:02.916 INFO juju.worker.uniter unit "knative-eventing/0" started
unit-knative-eventing-0: 17:25:02.919 INFO juju.worker.uniter resuming charm install
unit-knative-eventing-0: 17:25:02.919 INFO juju.worker.uniter.charm downloading ch:amd64/focal/knative-eventing-497 from API server
unit-knative-eventing-0: 17:25:02.920 INFO juju.downloader downloading from ch:amd64/focal/knative-eventing-497
unit-knative-eventing-0: 17:25:02.952 INFO juju.downloader download complete ("ch:amd64/focal/knative-eventing-497")
unit-knative-eventing-0: 17:25:02.957 INFO juju.downloader download verified ("ch:amd64/focal/knative-eventing-497")






unit-knative-eventing-0: 17:25:06.455 INFO juju.worker.uniter hooks are retried true
unit-knative-eventing-0: 17:25:06.507 INFO juju.worker.uniter found queued "install" hook
unit-knative-eventing-0: 17:25:06.934 INFO unit.knative-eventing/0.juju-log Running legacy hooks/install.
unit-knative-eventing-0: 17:25:07.204 INFO unit.knative-eventing/0.juju-log No otel-collector relation detected, observability won't be enabled for knative-eventing
unit-knative-eventing-0: 17:25:07.207 INFO unit.knative-eventing/0.juju-log Rendering manifests
unit-knative-eventing-0: 17:25:07.280 INFO unit.knative-eventing/0.juju-log HTTP Request: PATCH https://10.152.183.1/api/v1/namespaces/knative-eventing?force=true&fieldManager=kubeflow "HTTP/1.1 201 Created"
unit-knative-eventing-0: 17:25:07.314 INFO unit.knative-eventing/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/operator.knative.dev/v1beta1/namespaces/knative-eventing/knativeeventings/knative-eventing?force=true&fieldManager=kubeflow "HTTP/1.1 404 Not Found"
unit-knative-eventing-0: 17:25:07.330 ERROR unit.knative-eventing/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 153, in <module>
    main(KnativeEventingCharm)
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/ops/main.py", line 548, in main
    manager.run()
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/ops/main.py", line 527, in run
    self._emit()
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/ops/main.py", line 516, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/ops/main.py", line 147, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 91, in _on_install
    self._apply_and_set_status()
  File "./src/charm.py", line 53, in _apply_and_set_status
    self.resource_handler.apply()
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/charmed_kubeflow_chisme/kubernetes/_kubernetes_resource_handler.py", line 340, in apply
    apply_many(
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/charmed_kubeflow_chisme/lightkube/batch/_many.py", line 72, in apply_many
    returns[i] = client.apply(
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/lightkube/core/client.py", line 457, in apply
    return self.patch(type(obj), name, obj, namespace=namespace,
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/lightkube/core/client.py", line 325, in patch
    return self._client.request("patch", res=res, name=name, namespace=namespace, obj=obj,
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/lightkube/core/generic_client.py", line 245, in request
    return self.handle_response(method, resp, br)
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/lightkube/core/generic_client.py", line 196, in handle_response
    self.raise_for_status(resp)
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/lightkube/core/generic_client.py", line 190, in raise_for_status
    raise transform_exception(e)
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/lightkube/core/generic_client.py", line 188, in raise_for_status
    resp.raise_for_status()
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://10.152.183.1/apis/operator.knative.dev/v1beta1/namespaces/knative-eventing/knativeeventings/knative-eventing?force=true&fieldManager=kubeflow'
For more information check: https://httpstatuses.com/404
unit-knative-eventing-0: 17:25:07.559 ERROR juju.worker.uniter.operation hook "install" (via hook dispatching script: dispatch) failed: exit status 1
unit-knative-eventing-0: 17:25:07.560 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-knative-eventing-0: 17:25:12.567 INFO juju.worker.uniter awaiting error resolution for "install" hook
unit-knative-eventing-0: 17:25:13.093 INFO unit.knative-eventing/0.juju-log Running legacy hooks/install.
unit-knative-eventing-0: 17:25:13.719 INFO unit.knative-eventing/0.juju-log No otel-collector relation detected, observability won't be enabled for knative-eventing
unit-knative-eventing-0: 17:25:13.725 INFO unit.knative-eventing/0.juju-log Rendering manifests
unit-knative-eventing-0: 17:25:13.875 INFO unit.knative-eventing/0.juju-log HTTP Request: PATCH https://10.152.183.1/api/v1/namespaces/knative-eventing?force=true&fieldManager=kubeflow "HTTP/1.1 200 OK"
unit-knative-eventing-0: 17:25:13.951 INFO unit.knative-eventing/0.juju-log HTTP Request: PATCH https://10.152.183.1/apis/operator.knative.dev/v1beta1/namespaces/knative-eventing/knativeeventings/knative-eventing?force=true&fieldManager=kubeflow "HTTP/1.1 404 Not Found"
unit-knative-eventing-0: 17:25:13.984 ERROR unit.knative-eventing/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 153, in <module>
    main(KnativeEventingCharm)
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/ops/main.py", line 548, in main
    manager.run()
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/ops/main.py", line 527, in run
    self._emit()
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/ops/main.py", line 513, in _emit
    self.framework.reemit()
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/ops/framework.py", line 870, in reemit
    self._reemit()
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 91, in _on_install
    self._apply_and_set_status()
  File "./src/charm.py", line 53, in _apply_and_set_status
    self.resource_handler.apply()
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/charmed_kubeflow_chisme/kubernetes/_kubernetes_resource_handler.py", line 340, in apply
    apply_many(
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/charmed_kubeflow_chisme/lightkube/batch/_many.py", line 72, in apply_many
    returns[i] = client.apply(
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/lightkube/core/client.py", line 457, in apply
    return self.patch(type(obj), name, obj, namespace=namespace,
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/lightkube/core/client.py", line 325, in patch
    return self._client.request("patch", res=res, name=name, namespace=namespace, obj=obj,
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/lightkube/core/generic_client.py", line 245, in request
    return self.handle_response(method, resp, br)
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/lightkube/core/generic_client.py", line 196, in handle_response
    self.raise_for_status(resp)
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/lightkube/core/generic_client.py", line 190, in raise_for_status
    raise transform_exception(e)
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/lightkube/core/generic_client.py", line 188, in raise_for_status
    resp.raise_for_status()
  File "/var/lib/juju/agents/unit-knative-eventing-0/charm/venv/httpx/_models.py", line 749, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://10.152.183.1/apis/operator.knative.dev/v1beta1/namespaces/knative-eventing/knativeeventings/knative-eventing?force=true&fieldManager=kubeflow'
For more information check: https://httpstatuses.com/404
unit-knative-eventing-0: 17:25:14.241 ERROR juju.worker.uniter.operation hook "install" (via hook dispatching script: dispatch) failed: exit status 1
unit-knative-eventing-0: 17:25:14.241 INFO juju.worker.uniter awaiting error resolution for "install" hook

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6270.

This message was autogenerated

[NohaIhab]

The issue here is that the knative-serving charm manifests rely on the CRDs created by knative-operator charm, thus the error logs we see above when knative-serving charm is calling self.resource_handler.apply() and the lightkube client returns 404 for knativeserving resource:

httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://10.152.183.1/apis/operator.knative.dev/v1beta1/namespaces/knative-serving/knativeservings/knative-serving?force=true&fieldManager=kubeflow'

and similarly for knative-eventing charm and knativeeventing resource.

To fix this, there needs to be a relation between knative-operator and:

1. knative-serving charm
2. knative-eventing charm

the relation will depict this dependency between the charms, and both of the above charms will be blocked from applying manifests until the knative-operator charm has applied the corresponding CRD.

[NohaIhab]

This behavior is also happening with admission-webhook and resource-dispatcher. When we implement this, let's try to make it generic so we can re-use the same solution there.