Operator could not create Keeper Pods

[yildizozgur]

Hi,
I have deployed altinity-clickhouse-operator:0.25.6 and deployed altinity clickhouse helm-chart with enabled clickhouse installation(CHI) and keeper(CHK) to manage repliacation between clickhouse pods. Operator and CR deployments are in same k8s cluster but in different namespaces. Operator deploys ClickhouseInstallation successfully. However it does not deploy Keeper pods.I have checked operator logs, events, and states of resources. There is no significant information or issue to point out issue. Keeper states InProgress. It seems Operator couldnot or does not try to deploy. Do you know whether there is some configuration is missing or is this an operator/helmchart bug.

All details are in this issue link(I have already created an issue on helm chart repo - > Altinity/helm-charts#117)

[Slach]

could you share
kubectl get pods --all-namespaces | grep clickhouse-operator

and
kubectl get chk --all-namespaces
kubectl get sts --all-namespaces | grep keeper

[yildizozgur]

could you share kubectl get pods --all-namespaces | grep clickhouse-operator

and kubectl get chk --all-namespaces kubectl get sts --all-namespaces | grep keeper

Thank you for the quick reply. Here is the query results

`❯ kubectl get pods --all-namespaces | grep clickhouse-operator
obe-clickhouse-operator-qa clickhouse-operator-5bbfbd6c65-b7sh6 2/2 Running 0 151m

~ took 3s
❯ kubectl get chk --all-namespaces
error: the server doesn't have a resource type "chk"

~
❯ kubectl get sts --all-namespaces | grep keeper | grep -v zookeeper

~
❯ `

[Slach]

❯ kubectl get chk --all-namespaces
error: the server doesn't have a resource type "chk"

did you delete ClickHouseKeeperInstallation manually?

in message
Altinity/helm-charts#117 (comment)

You provided contradictory information

❯ k get clickhousekeeperinstallations.clickhouse-keeper.altinity.com -w NAME
CLUSTERS   HOSTS   STATUS       HOSTS-COMPLETED   AGE     SUSPEND
sentry-dev-clickhouse-keeper   1          1       InProgress                     2m32s   

[Slach]

could you share
kubectl get crds | grep altinity

[yildizozgur]

no I did not delete it. The interesting thing is when queried with "chk" I could not get any result. But when type "k get clickhousekeeperinstallations.clickhouse-keeper.altinity.com" I see keeper cluster. But it is always in Inprogress state.

❯ kubectl get crds | grep altinity
clickhouseinstallations.clickhouse.altinity.com 2025-12-12T08:39:01Z
clickhouseinstallationtemplates.clickhouse.altinity.com 2025-12-12T08:39:01Z
clickhousekeeperinstallations.clickhouse-keeper.altinity.com 2025-12-12T08:39:01Z
clickhouseoperatorconfigurations.clickhouse.altinity.com 2025-12-12T08:39:01Z

~
❯ k get clickhouseinstallation
NAME CLUSTERS HOSTS STATUS HOSTS-COMPLETED AGE SUSPEND
sentry-dev-clickhouse 1 1 Completed 15h

~
❯ k get clickhousekeeperinstallations.clickhouse-keeper.altinity.com
NAME CLUSTERS HOSTS STATUS HOSTS-COMPLETED AGE SUSPEND
sentry-dev-clickhouse-keeper 1 1 InProgress 15h

~
❯

[Slach]

this is really weird, chk this is just alias for clickhousekeeperinstallation.clickhouse-keeper.altinity.com

could you share
kubectl get crds -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.names.shortNames[*]}{@}{","}{end}{"\n"}{end}' | column -t -s $'\t'

[yildizozgur]

deployed crds don't have any shortnames or alias

❯ kubectl get crds -o jsonpath='{range .items[]}{.metadata.name}{"\t"}{range .spec.names.shortNames[]}{@}{","}{end}{"\n"}{end}' | column -t -s $'\t' | grep -i "clickhouse"
clickhouseinstallations.clickhouse.altinity.com
clickhouseinstallationtemplates.clickhouse.altinity.com
clickhousekeeperinstallations.clickhouse-keeper.altinity.com
clickhouseoperatorconfigurations.clickhouse.altinity.com

[Slach]

this is really weird

did you use only helm when install operator?
or install operator separatelly?

could you share
helm list --all-namespaces | grep -E "click|altinity"

[Slach]

according to https://github.com/Altinity/clickhouse-operator/blob/0.25.6/deploy/operator/parts/crd.yaml#L7997-L8013

CRD definitely shall contains chk alias

[yildizozgur]

In our platform, crds are deployed by cluster deployment - in another pipeline. So, in order to mange it safely we don't use shortname, otherwise it is possible to conflicts between tenants.
We are running on multi tenant env in a cluster. So, Operator deployed in one namespace(obe-clickhouse-operator-qa), CRs are deployed in another tenant namespace(obe-sentry-dev). Operator should manage CRs in tenant namespace. And I have configured Operator to watch obe-sentry-dev namespace.

In order to confirm whether something missed during crds deployment on cluster pipeline, we deployed all crds manually for testing. But unfortunately nothing has changed. The operator still could not create statefulset on watched namespace. On the other side it could create clickhouseinstallation(chi). It also creates clickhousekeeperinstallations, but it could not or doesnot create sts on tenant namespace.

❯ kubectl get crds -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.names.shortNames[*]}{@}{","}{end}{"\n"}{end}' | column -t -s $'\t' | grep -i clickhouse clickhouseinstallations.clickhouse.altinity.com chi, clickhouseinstallationtemplates.clickhouse.altinity.com chit, clickhousekeeperinstallations.clickhouse-keeper.altinity.com chk, clickhouseoperatorconfigurations.clickhouse.altinity.com chopconf,

[Slach]

ok. thanks for clarify your installation details
could you share
kubectl logs -l app=clickhouse-operator --since=48h | grep -i keeper

[Slach]

and
kubectl get clickhousekeeperinstallations.clickhouse-keeper.altinity.com --all-namespaces

[yildizozgur]

Hi,
Thanks a lot. I have added logs as attachment.

❯ kubectl get clickhousekeeperinstallations.clickhouse-keeper.altinity.com --all-namespaces
NAMESPACE NAME CLUSTERS HOSTS STATUS HOSTS-COMPLETED AGE SUSPEND
obe-sentry-dev sentry-dev-clickhouse-keeper 1 1 InProgress 52m

❯ kubectl logs -n obe-clickhouse-operato.yaml

[Slach]

kubectl get clickhousekeeperinstallations.clickhouse-keeper.altinity.com -n obe-sentry-dev sentry-dev-clickhouse-keeper -o yaml
kubectl logs -l app=clickhouse-operator --since=48h | grep -C 30 -i keeper

did you deploy sentry-dev-clickhouse-keeper 52 minutes ago from scratch?

[Slach]

try restart operator and share full logs from operator startup

is anything other infrastucture components like ArgoCD/Flux watched custom resources in obe-sentry-dev namespace?