Deploying via Helm is unsuccessful

[paul-snively]

Setup:

Minikube 1.27.0
Kubernetes 1.23.
Strimzi 0.13.1 installed via OperatorHub

KafkaCluster "kafka" created in namespace "moonraker"
registry-schemas KafkaTopic in namespace "moonraker" ready
confluent-schema-registry KafkaUser in namespace "moonraker" ready

Install operator via:

helm install -n operators schema-registry lsstsqre/strimzi-registry-operator --set clusterName="kafka",clusterNamespace="moonraker"

Logs from the operator pod:

[2022-10-09 14:49:36,310] kubernetes.client.re [DEBUG   ] response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"strimzischemaregistries.roundtable.lsst.codes is forbidden: User \"system:serviceaccount:operators:strimzi-registry-operator\" cannot list resource \"strimzischemaregistries\" in API group \"roundtable.lsst.codes\" in the namespace \"moonraker\"","reason":"Forbidden","details":{"group":"roundtable.lsst.codes","kind":"strimzischemaregistries"},"code":403}

[2022-10-09 14:49:36,313] kopf._core.reactor.r [DEBUG   ] Starting Kopf 1.35.6.
[2022-10-09 14:49:36,313] kopf._core.engines.a [INFO    ] Initial authentication has been initiated.
[2022-10-09 14:49:36,313] kopf.activities.auth [DEBUG   ] Activity 'login_via_client' is invoked.
[2022-10-09 14:49:36,314] kopf.activities.auth [DEBUG   ] Client is configured in cluster with service account.
[2022-10-09 14:49:36,315] kopf.activities.auth [INFO    ] Activity 'login_via_client' succeeded.
[2022-10-09 14:49:36,315] kopf._core.engines.a [INFO    ] Initial authentication has finished.
[2022-10-09 14:49:36,330] kopf._cogs.clients.w [DEBUG   ] Starting the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide.
[2022-10-09 14:49:36,331] kopf._cogs.clients.w [DEBUG   ] Stopping the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide.
[2022-10-09 14:49:36,332] kopf._core.reactor.o [WARNING ] Not enough permissions to list namespaces. Falling back to a list of namespaces which are assumed to exist: {'moonraker'}
[2022-10-09 14:49:36,332] kopf._cogs.clients.w [DEBUG   ] Starting the watch-stream for namespaces.v1 cluster-wide.
[2022-10-09 14:49:36,332] kopf._cogs.clients.w [DEBUG   ] Starting the watch-stream for strimzischemaregistries.v1beta1.roundtable.lsst.codes in 'moonraker'.
[2022-10-09 14:49:36,333] kopf._cogs.clients.w [DEBUG   ] Starting the watch-stream for secrets.v1 in 'moonraker'.
[2022-10-09 14:49:36,333] kopf._cogs.clients.w [DEBUG   ] Stopping the watch-stream for namespaces.v1 cluster-wide.
[2022-10-09 14:49:36,334] kopf._cogs.clients.w [DEBUG   ] Stopping the watch-stream for strimzischemaregistries.v1beta1.roundtable.lsst.codes in 'moonraker'.
[2022-10-09 14:49:36,334] kopf._cogs.clients.w [DEBUG   ] Stopping the watch-stream for secrets.v1 in 'moonraker'.
[2022-10-09 14:49:36,334] kopf._core.reactor.o [WARNING ] Not enough permissions to watch for resources: changes (creation/deletion/updates) will not be noticed; the resources are only refreshed on operator restarts.
[2022-10-09 14:49:36,335] kopf._core.reactor.o [WARNING ] Not enough permissions to watch for namespaces: changes (deletion/creation) will not be noticed; the namespaces are only refreshed on operator restarts.
[2022-10-09 14:49:36,335] kopf._core.reactor.o [ERROR   ] Watcher for strimzischemaregistries.v1beta1.roundtable.lsst.codes@moonraker has failed: ('strimzischemaregistries.roundtable.lsst.codes is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "strimzischemaregistries" in API group "roundtable.lsst.codes" in the namespace "moonraker"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'strimzischemaregistries.roundtable.lsst.codes is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "strimzischemaregistries" in API group "roundtable.lsst.codes" in the namespace "moonraker"', 'reason': 'Forbidden', 'details': {'group': 'roundtable.lsst.codes', 'kind': 'strimzischemaregistries'}, 'code': 403})
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.96.0.1:443/apis/roundtable.lsst.codes/v1beta1/namespaces/moonraker/strimzischemaregistries')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
    await coro
  File "/opt/venv/lib/python3.10/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
    async for raw_event in stream:
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/watching.py", line 82, in infinite_watch
    async for raw_event in stream:
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/watching.py", line 159, in continuous_watch
    objs, resource_version = await fetching.list_objs(
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/fetching.py", line 28, in list_objs
    rsp = await api.get(
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 111, in get
    response = await request(
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 85, in request
    await errors.check_response(response)  # but do not parse it!
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
    raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIForbiddenError: ('strimzischemaregistries.roundtable.lsst.codes is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "strimzischemaregistries" in API group "roundtable.lsst.codes" in the namespace "moonraker"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'strimzischemaregistries.roundtable.lsst.codes is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "strimzischemaregistries" in API group "roundtable.lsst.codes" in the namespace "moonraker"', 'reason': 'Forbidden', 'details': {'group': 'roundtable.lsst.codes', 'kind': 'strimzischemaregistries'}, 'code': 403})
[2022-10-09 14:49:36,336] kopf._core.reactor.o [ERROR   ] Watcher for secrets.v1@moonraker has failed: ('secrets is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "secrets" in API group "" in the namespace "moonraker"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'secrets is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "secrets" in API group "" in the namespace "moonraker"', 'reason': 'Forbidden', 'details': {'kind': 'secrets'}, 'code': 403})
Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
    response.raise_for_status()
  File "/opt/venv/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.96.0.1:443/api/v1/namespaces/moonraker/secrets')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
    await coro
  File "/opt/venv/lib/python3.10/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
    async for raw_event in stream:
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/watching.py", line 82, in infinite_watch
    async for raw_event in stream:
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/watching.py", line 159, in continuous_watch
    objs, resource_version = await fetching.list_objs(
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/fetching.py", line 28, in list_objs
    rsp = await api.get(
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 111, in get
    response = await request(
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 85, in request
    await errors.check_response(response)  # but do not parse it!
  File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
    raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIForbiddenError: ('secrets is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "secrets" in API group "" in the namespace "moonraker"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'secrets is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "secrets" in API group "" in the namespace "moonraker"', 'reason': 'Forbidden', 'details': {'kind': 'secrets'}, 'code': 403})

It seems the created ServiceAccount doesn't have the permissions it needs to do what it needs to do.

[harksodje]

@paul-snively, I am currently experiencing the same issue.

[harksodje]

@paul-snively, I was able to resolve this issue. The clusterrolebinding of the schema-registry is referencing wrong namespace.
This is default:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    meta.helm.sh/release-name: kafka
    meta.helm.sh/release-namespace: kafka
  creationTimestamp: "2022-10-25T14:49:15Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: strimzi-registry-operator
  resourceVersion: "12193"
  uid: 60fa9e7a-d0bb-4811-8a94-e65a31378456
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: strimzi-registry-operator
subjects:
- kind: ServiceAccount
  name: strimzi-registry-operator
  namespace: strimzi-registry-operator

I just changed the and the pod start working
Also, you have to specify while installing the helm chart

[karanalang]

faced similar issue, resolved by creating the ServiceAccount in namespace - 'strimzi-registry-operator'

[husker-du]

The property operatorNamespace of the strimzi-registry-operator sets the namespace where the operator is going to be installed, by default this value is set to the namespace strimzi-registry-operator. Therefore, the helm release has to be deployed in this namespace by setting the option -n strimzi-registry-operator. If this namespace does not exists, it can be created by the helm command using the option --create-namespace.

In summary, the following command deploys the strimzi-registry-operator in the strimzi-registry-operator namespace for a strimzi Kafka cluster named kafka-cluster deployed in the strimzi namespace:

$ helm upgrade --install --create-namespace strimzi-registry-operator lsstsqre/strimzi-registry-operator --set clusterNamespace=strimzi,clusterName=kafka-cluster,operatorNamespace=strimzi-registry-operator -n strimzi-registry-operator

The property operatorNamespace=strimzi-registry-operator can be omitted because this is the default value.

[karanalang]

Hi Carlos, Can you pls check issue # 84 as well ? #84 This is in pending state, and not assigned to anyone yet. thanks, Karan Alang

…

On Mon, Dec 26, 2022 at 7:59 AM Carlos Tomás ***@***.***> wrote: The property operatorNamespace of the strimzi-registry-operator sets the namespace where the operator is going to be installed, by default this value is set to the namespace "strimzi-registry-operator". Therefore, the helm release has to be deployed in this namespace by setting the option -n strimzi-registry-operator. If this namespace does not exists, it can be created by the helm command using the option --create-namespace. In summary, the following command deploys the strimzi-registry-operator in the strimzi-registry-operator namespace for a strimzi Kafka cluster named kafka-cluster deployed in the strimzi namespace: $ helm upgrade --install --create-namespace strimzi-registry-operator lsstsqre/strimzi-registry-operator --set clusterNamespace=strimzi,clusterName=kafka-cluster,operatorNamespace=strimzi-registry-operator -n strimzi-registry-operator The property operatorNamespace=strimzi-registry-operator can be omitted because this is the default value. — Reply to this email directly, view it on GitHub <#79 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADKEGTHHUSOZ62V5VC3SL33WPG6FVANCNFSM6AAAAAARAXHES4> . You are receiving this because you commented.Message ID: ***@***.***>

[strowi]

Hi,

ran into a similar issue where the ServiceAccount seems to be missing permissions.
With version 2.1.0 i had to patch the ClusterRole to get rid of the following 2:

kopf._cogs.clients.w [DEBUG   ] Stopping the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide
kopf._cogs.clients.w [DEBUG   ] Stopping the watch-stream for namespaces.v1 cluster-wide.

I just added them to an appropriate rule instead of creating an extra one:

rules:
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - list
  - get
  **- watch**
- apiGroups:
  - ""
  resources:
  - secrets
  - configmaps
  - services
  **- namespaces**
  verbs:
  - get
  - list
  - watch
  - patch
  - create

Maybe this can help someone else stumbling over this.