Cluster gets recreated every sync interval?

[boris-savic]

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.6.1
• Where do you run it - cloud or metal? Kubernetes or OpenShift? Hetzner Cloud - Kubernetes
• Are you running Postgres Operator in production? YES
• Type of issue? Bug ?

Our operator seems to be recreating our cluster every sync interval - based on the log below and patronictl history output. I can't figure out what is wrong or how to prevent this. Does this happen to anyone else? Do you have any suggestions on what to do ?

Operator log output:

time="2021-03-08T19:17:45Z" level=info msg="SYNC event has been queued" cluster-name=database/olaii-db pkg=controller worker=0
time="2021-03-08T19:17:45Z" level=info msg="there are 1 clusters running" pkg=controller
time="2021-03-08T19:17:45Z" level=info msg="syncing of the cluster started" cluster-name=database/olaii-db pkg=controller worker=0
time="2021-03-08T19:17:45Z" level=debug msg="fetching possible additional team members for team \"olaii\"" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:45Z" level=debug msg="team API is disabled" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:45Z" level=info msg="syncing secrets" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:45Z" level=debug msg="secret database/olaii-pay.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:45Z" level=debug msg="secret database/standby.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:45Z" level=debug msg="secret database/olaii-guestlist.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:45Z" level=debug msg="secret database/olaii-pos.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:45Z" level=debug msg="secret database/olaii.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:46Z" level=debug msg="secret database/olaii-ticketing.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:46Z" level=debug msg="secret database/pooler.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:46Z" level=debug msg="secret database/plusone.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:47Z" level=debug msg="secret database/olaii-auth.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:48Z" level=debug msg="secret database/megasplet-bestado.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:48Z" level=debug msg="secret database/postgres.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:48Z" level=debug msg="syncing master service" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:48Z" level=debug msg="syncing replica service" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="No load balancer created for the replica service" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=info msg="volume claims do not require changes" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="syncing statefulsets" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is true" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="set statefulset's rolling update annotation to false: caller/reason from cache" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="set statefulset's rolling update annotation to true: caller/reason statefulset changes" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=info msg="statefulset database/olaii-db is not in the desired state and needs to be updated" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="-      restartPolicy: Always," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="-      dnsPolicy: ClusterFirst," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="-      serviceAccount: postgres-pod," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="-      schedulerName: default-scheduler," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="-      kind: PersistentVolumeClaim," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="-      apiVersion: v1," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="-      status: {" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="-        phase: Pending" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="-      }" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="+      status: {}" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="-  }," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="-  revisionHistoryLimit: 10" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="+  }" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="metadata.annotation are different" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="-  zalando-postgres-operator-rolling-update-required: false" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="+  zalando-postgres-operator-rolling-update-required: true" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=info msg="reason: new statefulset containers's postgres (index 0) security context does not match the current one" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="updating statefulset" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="patching statefulset annotations" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:49Z" level=debug msg="patching statefulset annotations" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:50Z" level=debug msg="calling Patroni API on a pod database/olaii-db-0 to set the following Postgres options: map[max_connections:250 max_worker_processes:8]" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:50Z" level=debug msg="making PATCH http request: http://10.42.8.208:8008/config" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:50Z" level=debug msg="performing rolling update" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:50Z" level=info msg="there are 3 pods in the cluster to recreate" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:50Z" level=debug msg="name=olaii-db-0 phase=Running ip=10.42.8.208" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:50Z" level=debug msg="name=olaii-db-1 phase=Running ip=10.42.6.209" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:50Z" level=debug msg="name=olaii-db-2 phase=Running ip=10.42.7.210" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:17:50Z" level=debug msg="subscribing to pod \"database/olaii-db-0\"" cluster-name=database/olaii-db pkg=cluster
E0308 19:17:50.494214       1 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"olaii-db.166934de252e3681", GenerateName:"", Namespace:"database", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"postgresql", Namespace:"database", Name:"olaii-db", UID:"7f45cc48-412b-4e91-909e-25b81ea2fe3d", APIVersion:"acid.zalan.do/v1", ResourceVersion:"186141734", FieldPath:""}, Reason:"Update", Message:"Performing rolling update", Source:v1.EventSource{Component:"postgres-operator", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc008653185f8fa81, ext:13008583460, loc:(*time.Location)(0x2e2f8a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc009bbf786ca4d8d, ext:351013022301784, loc:(*time.Location)(0x2e2f8a0)}}, Count:198, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "olaii-db.166934de252e3681" is forbidden: User "system:serviceaccount:postgres-operator:postgres-operator" cannot patch resource "events" in API group "" in the namespace "database"' (will not retry!)
E0308 19:17:58.092387       1 reflector.go:138] pkg/controller/controller.go:477: Failed to watch *v1.PostgresTeam: failed to list *v1.PostgresTeam: postgresteams.acid.zalan.do is forbidden: User "system:serviceaccount:postgres-operator:postgres-operator" cannot list resource "postgresteams" in API group "acid.zalan.do" at the cluster scope
time="2021-03-08T19:18:14Z" level=info msg="pod \"database/olaii-db-0\" has been recreated" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:18:14Z" level=debug msg="unsubscribing from pod \"database/olaii-db-0\" events" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:18:14Z" level=debug msg="subscribing to pod \"database/olaii-db-2\"" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:18:37Z" level=info msg="pod \"database/olaii-db-2\" has been recreated" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:18:37Z" level=debug msg="unsubscribing from pod \"database/olaii-db-2\" events" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:18:37Z" level=debug msg="switching over from \"olaii-db-1\" to \"database/olaii-db-0\"" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:18:37Z" level=debug msg="making POST http request: http://10.42.6.209:8008/failover" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:18:37Z" level=debug msg="subscribing to pod \"database/olaii-db-0\"" cluster-name=database/olaii-db pkg=cluster
E0308 19:18:37.433537       1 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"olaii-db.1669488cc6e2325b", GenerateName:"", Namespace:"database", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"postgresql", Namespace:"database", Name:"olaii-db", UID:"7f45cc48-412b-4e91-909e-25b81ea2fe3d", APIVersion:"acid.zalan.do/v1", ResourceVersion:"186141734", FieldPath:""}, Reason:"Switchover", Message:"Switching over from \"olaii-db-1\" to \"database/olaii-db-0\"", Source:v1.EventSource{Component:"postgres-operator", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc0087a53b329a65b, ext:21653766747888, loc:(*time.Location)(0x2e2f8a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc009bc0359948507, ext:351060337544061, loc:(*time.Location)(0x2e2f8a0)}}, Count:34, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "olaii-db.1669488cc6e2325b" is forbidden: User "system:serviceaccount:postgres-operator:postgres-operator" cannot patch resource "events" in API group "" in the namespace "database"' (will not retry!)
time="2021-03-08T19:18:38Z" level=debug msg="successfully switched over from \"olaii-db-1\" to \"database/olaii-db-0\"" cluster-name=database/olaii-db pkg=cluster
E0308 19:18:38.626516       1 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"olaii-db.1669488d485c2866", GenerateName:"", Namespace:"database", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"postgresql", Namespace:"database", Name:"olaii-db", UID:"7f45cc48-412b-4e91-909e-25b81ea2fe3d", APIVersion:"acid.zalan.do/v1", ResourceVersion:"186141734", FieldPath:""}, Reason:"Switchover", Message:"Successfully switched over from \"olaii-db-1\" to \"database/olaii-db-0\"", Source:v1.EventSource{Component:"postgres-operator", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc0087a5441d33e66, ext:21655939001593, loc:(*time.Location)(0x2e2f8a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc009bc03a382ff6c, ext:351061504167989, loc:(*time.Location)(0x2e2f8a0)}}, Count:34, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "olaii-db.1669488d485c2866" is forbidden: User "system:serviceaccount:postgres-operator:postgres-operator" cannot patch resource "events" in API group "" in the namespace "database"' (will not retry!)
time="2021-03-08T19:18:38Z" level=debug msg="unsubscribing from pod \"database/olaii-db-0\" events" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:18:38Z" level=info msg="recreating old master pod \"database/olaii-db-1\"" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:18:38Z" level=debug msg="subscribing to pod \"database/olaii-db-1\"" cluster-name=database/olaii-db pkg=cluster
E0308 19:18:38.832771       1 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"olaii-db.1669488da4b138c7", GenerateName:"", Namespace:"database", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"postgresql", Namespace:"database", Name:"olaii-db", UID:"7f45cc48-412b-4e91-909e-25b81ea2fe3d", APIVersion:"acid.zalan.do/v1", ResourceVersion:"186141734", FieldPath:""}, Reason:"Switchover", Message:"Switchover from \"olaii-db-1\" to \"database/olaii-db-0\" FAILED: <nil>", Source:v1.EventSource{Component:"postgres-operator", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc0087a54a28d84c7, ext:21657488080221, loc:(*time.Location)(0x2e2f8a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc009bc03b162a82e, ext:351061736929441, loc:(*time.Location)(0x2e2f8a0)}}, Count:34, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "olaii-db.1669488da4b138c7" is forbidden: User "system:serviceaccount:postgres-operator:postgres-operator" cannot patch resource "events" in API group "" in the namespace "database"' (will not retry!)
E0308 19:18:42.309682       1 reflector.go:138] pkg/controller/controller.go:477: Failed to watch *v1.PostgresTeam: failed to list *v1.PostgresTeam: postgresteams.acid.zalan.do is forbidden: User "system:serviceaccount:postgres-operator:postgres-operator" cannot list resource "postgresteams" in API group "acid.zalan.do" at the cluster scope
time="2021-03-08T19:19:02Z" level=info msg="pod \"database/olaii-db-1\" has been recreated" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:19:02Z" level=debug msg="unsubscribing from pod \"database/olaii-db-1\" events" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:19:02Z" level=info msg="pods have been recreated" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:19:02Z" level=debug msg="set statefulset's rolling update annotation to false: caller/reason applyRollingUpdateFlag" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:19:02Z" level=debug msg="patching statefulset annotations" cluster-name=database/olaii-db pkg=cluster
E0308 19:19:02.345779       1 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"olaii-db.166934eedbba9751", GenerateName:"", Namespace:"database", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"postgresql", Namespace:"database", Name:"olaii-db", UID:"7f45cc48-412b-4e91-909e-25b81ea2fe3d", APIVersion:"acid.zalan.do/v1", ResourceVersion:"186141734", FieldPath:""}, Reason:"Update", Message:"Rolling update done - pods have been recreated", Source:v1.EventSource{Component:"postgres-operator", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc008654374975551, ext:84790713254, loc:(*time.Location)(0x2e2f8a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc009bc0993ac7d0c, ext:351085238451622, loc:(*time.Location)(0x2e2f8a0)}}, Count:198, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "olaii-db.166934eedbba9751" is forbidden: User "system:serviceaccount:postgres-operator:postgres-operator" cannot patch resource "events" in API group "" in the namespace "database"' (will not retry!)
time="2021-03-08T19:19:02Z" level=debug msg="syncing pod disruption budgets" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:19:02Z" level=debug msg="syncing roles" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:19:02Z" level=debug msg="closing database connection" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:19:02Z" level=debug msg="syncing databases" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:19:02Z" level=debug msg="closing database connection" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:19:02Z" level=debug msg="syncing prepared databases with schemas" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:19:02Z" level=debug msg="syncing pooler is not required" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T19:19:02Z" level=info msg="cluster has been synced" cluster-name=database/olaii-db pkg=controller worker=0
E0308 19:19:25.718109       1 reflector.go:138] pkg/controller/controller.go:477: Failed to watch *v1.PostgresTeam: failed to list *v1.PostgresTeam: postgresteams.acid.zalan.do is forbidden: User "system:serviceaccount:postgres-operator:postgres-operator" cannot list resource "postgresteams" in API group "acid.zalan.do" at the cluster scope

[FxKu]

add an additional pod capability, like described in #1377. And update you cluster role to allow watching PostgresTeam objects.

[boris-savic]

Hi, thanks for the lighting speed reply at this late hour!

I need to add SYS_NICE to Postgres operator pod right?

Also how can I add the allow watching PostgresTeam object permission?

[boris-savic]

Update: I have added SYS_NICE to Postgres operator but it's still recreating the pods.

This only started happening after I updated from 1.4.x to latest 1.6.1

Logs:

time="2021-03-08T21:18:33Z" level=info msg="SYNC event has been queued" cluster-name=database/olaii-db pkg=controller worker=0
time="2021-03-08T21:18:33Z" level=info msg="there are 1 clusters running" pkg=controller
time="2021-03-08T21:18:33Z" level=info msg="syncing of the cluster started" cluster-name=database/olaii-db pkg=controller worker=0
time="2021-03-08T21:18:33Z" level=debug msg="fetching possible additional team members for team \"olaii\"" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:33Z" level=debug msg="team API is disabled" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:33Z" level=info msg="syncing secrets" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:33Z" level=debug msg="secret database/standby.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:33Z" level=debug msg="secret database/olaii-guestlist.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:33Z" level=debug msg="secret database/plusone.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:33Z" level=debug msg="secret database/olaii-ticketing.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:33Z" level=debug msg="secret database/olaii-auth.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:34Z" level=debug msg="secret database/olaii.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:34Z" level=debug msg="secret database/postgres.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:35Z" level=debug msg="secret database/pooler.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:35Z" level=debug msg="secret database/megasplet-bestado.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:35Z" level=debug msg="secret database/olaii-pay.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:36Z" level=debug msg="secret database/olaii-pos.olaii-db.credentials already exists, fetching its password" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:36Z" level=debug msg="syncing master service" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:36Z" level=debug msg="syncing replica service" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="No load balancer created for the replica service" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=info msg="volume claims do not require changes" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="syncing statefulsets" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="cached StatefulSet value exists, rollingUpdate flag is true" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="set statefulset's rolling update annotation to false: caller/reason from cache" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="set statefulset's rolling update annotation to true: caller/reason statefulset changes" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=info msg="statefulset database/olaii-db is not in the desired state and needs to be updated" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="-      restartPolicy: Always," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="-      dnsPolicy: ClusterFirst," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="-      serviceAccount: postgres-pod," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="-      schedulerName: default-scheduler," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="-      kind: PersistentVolumeClaim," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="-      apiVersion: v1," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="-      status: {" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="-        phase: Pending" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="-      }" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="+      status: {}" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="-  }," cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="-  revisionHistoryLimit: 10" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="+  }" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="metadata.annotation are different" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="-  zalando-postgres-operator-rolling-update-required: false" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="+  zalando-postgres-operator-rolling-update-required: true" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=info msg="reason: new statefulset containers's postgres (index 0) security context does not match the current one" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="updating statefulset" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="patching statefulset annotations" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="patching statefulset annotations" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="calling Patroni API on a pod database/olaii-db-0 to set the following Postgres options: map[max_connections:250]" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="making PATCH http request: http://10.42.8.215:8008/config" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:37Z" level=debug msg="performing rolling update" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:38Z" level=info msg="there are 3 pods in the cluster to recreate" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:38Z" level=debug msg="name=olaii-db-0 phase=Running ip=10.42.8.215" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:38Z" level=debug msg="name=olaii-db-1 phase=Running ip=10.42.6.215" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:38Z" level=debug msg="name=olaii-db-2 phase=Running ip=10.42.7.216" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:18:38Z" level=debug msg="subscribing to pod \"database/olaii-db-0\"" cluster-name=database/olaii-db pkg=cluster
E0308 21:18:38.338747       1 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"olaii-db.166a790e38e09b44", GenerateName:"", Namespace:"database", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"postgresql", Namespace:"database", Name:"olaii-db", UID:"7f45cc48-412b-4e91-909e-25b81ea2fe3d", APIVersion:"acid.zalan.do/v1", ResourceVersion:"186255234", FieldPath:""}, Reason:"Update", Message:"Performing rolling update", Source:v1.EventSource{Component:"postgres-operator", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc009c149953f9f44, ext:13326662771, loc:(*time.Location)(0x2e2f8a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc009c30b79078799, ext:1812926966484, loc:(*time.Location)(0x2e2f8a0)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "olaii-db.166a790e38e09b44" is forbidden: User "system:serviceaccount:postgres-operator:postgres-operator" cannot patch resource "events" in API group "" in the namespace "database"' (will not retry!)
time="2021-03-08T21:19:04Z" level=info msg="pod \"database/olaii-db-0\" has been recreated" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:04Z" level=debug msg="unsubscribing from pod \"database/olaii-db-0\" events" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:04Z" level=debug msg="subscribing to pod \"database/olaii-db-1\"" cluster-name=database/olaii-db pkg=cluster
E0308 21:19:10.274003       1 reflector.go:138] pkg/controller/controller.go:477: Failed to watch *v1.PostgresTeam: failed to list *v1.PostgresTeam: the server could not find the requested resource (get postgresteams.acid.zalan.do)
time="2021-03-08T21:19:31Z" level=info msg="pod \"database/olaii-db-1\" has been recreated" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:31Z" level=debug msg="unsubscribing from pod \"database/olaii-db-1\" events" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:31Z" level=debug msg="switching over from \"olaii-db-2\" to \"database/olaii-db-1\"" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:31Z" level=debug msg="making POST http request: http://10.42.7.216:8008/failover" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:31Z" level=debug msg="subscribing to pod \"database/olaii-db-1\"" cluster-name=database/olaii-db pkg=cluster
E0308 21:19:31.387586       1 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"olaii-db.166a7abda9e5a4d6", GenerateName:"", Namespace:"database", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"postgresql", Namespace:"database", Name:"olaii-db", UID:"7f45cc48-412b-4e91-909e-25b81ea2fe3d", APIVersion:"acid.zalan.do/v1", ResourceVersion:"186255234", FieldPath:""}, Reason:"Switchover", Message:"Switching over from \"olaii-db-2\" to \"database/olaii-db-1\"", Source:v1.EventSource{Component:"postgres-operator", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc009c318d6dc86d6, ext:1866353722895, loc:(*time.Location)(0x2e2f8a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc009c318d6dc86d6, ext:1866353722895, loc:(*time.Location)(0x2e2f8a0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:postgres-operator:postgres-operator" cannot create resource "events" in API group "" in the namespace "database"' (will not retry!)
time="2021-03-08T21:19:32Z" level=debug msg="successfully switched over from \"olaii-db-2\" to \"database/olaii-db-1\"" cluster-name=database/olaii-db pkg=cluster
E0308 21:19:32.556094       1 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"olaii-db.166a7abdef5fea57", GenerateName:"", Namespace:"database", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"postgresql", Namespace:"database", Name:"olaii-db", UID:"7f45cc48-412b-4e91-909e-25b81ea2fe3d", APIVersion:"acid.zalan.do/v1", ResourceVersion:"186255234", FieldPath:""}, Reason:"Switchover", Message:"Successfully switched over from \"olaii-db-2\" to \"database/olaii-db-1\"", Source:v1.EventSource{Component:"postgres-operator", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc009c31920bc0257, ext:1867519363982, loc:(*time.Location)(0x2e2f8a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc009c31920bc0257, ext:1867519363982, loc:(*time.Location)(0x2e2f8a0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:postgres-operator:postgres-operator" cannot create resource "events" in API group "" in the namespace "database"' (will not retry!)
time="2021-03-08T21:19:34Z" level=debug msg="unsubscribing from pod \"database/olaii-db-1\" events" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:34Z" level=info msg="recreating old master pod \"database/olaii-db-2\"" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:34Z" level=debug msg="subscribing to pod \"database/olaii-db-2\"" cluster-name=database/olaii-db pkg=cluster
E0308 21:19:34.896724       1 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"olaii-db.166a7abe7a8f46b3", GenerateName:"", Namespace:"database", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"postgresql", Namespace:"database", Name:"olaii-db", UID:"7f45cc48-412b-4e91-909e-25b81ea2fe3d", APIVersion:"acid.zalan.do/v1", ResourceVersion:"186255234", FieldPath:""}, Reason:"Switchover", Message:"Switchover from \"olaii-db-2\" to \"database/olaii-db-1\" FAILED: <nil>", Source:v1.EventSource{Component:"postgres-operator", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc009c319b4b5cab3, ext:1869854500837, loc:(*time.Location)(0x2e2f8a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc009c319b4b5cab3, ext:1869854500837, loc:(*time.Location)(0x2e2f8a0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:postgres-operator:postgres-operator" cannot create resource "events" in API group "" in the namespace "database"' (will not retry!)
time="2021-03-08T21:19:57Z" level=info msg="pod \"database/olaii-db-2\" has been recreated" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:57Z" level=debug msg="unsubscribing from pod \"database/olaii-db-2\" events" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:57Z" level=info msg="pods have been recreated" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:57Z" level=debug msg="set statefulset's rolling update annotation to false: caller/reason applyRollingUpdateFlag" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:57Z" level=debug msg="patching statefulset annotations" cluster-name=database/olaii-db pkg=cluster
E0308 21:19:57.205001       1 event.go:264] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"olaii-db.166a791f427aa19d", GenerateName:"", Namespace:"database", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"postgresql", Namespace:"database", Name:"olaii-db", UID:"7f45cc48-412b-4e91-909e-25b81ea2fe3d", APIVersion:"acid.zalan.do/v1", ResourceVersion:"186255234", FieldPath:""}, Reason:"Update", Message:"Rolling update done - pods have been recreated", Source:v1.EventSource{Component:"postgres-operator", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xc009c15bdfb60b9d, ext:86502195924, loc:(*time.Location)(0x2e2f8a0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xc009c31f4afaf8ec, ext:1892154391588, loc:(*time.Location)(0x2e2f8a0)}}, Count:2, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events "olaii-db.166a791f427aa19d" is forbidden: User "system:serviceaccount:postgres-operator:postgres-operator" cannot patch resource "events" in API group "" in the namespace "database"' (will not retry!)
time="2021-03-08T21:19:57Z" level=debug msg="syncing pod disruption budgets" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:57Z" level=debug msg="syncing roles" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:57Z" level=debug msg="closing database connection" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:57Z" level=debug msg="syncing databases" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:57Z" level=debug msg="closing database connection" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:57Z" level=debug msg="syncing prepared databases with schemas" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:57Z" level=debug msg="syncing pooler is not required" cluster-name=database/olaii-db pkg=cluster
time="2021-03-08T21:19:57Z" level=info msg="cluster has been synced" cluster-name=database/olaii-db pkg=controller worker=0

[PetterSa]

I am having the same issue. As boris-savic this is happening after upgrading the postgres-operator.
I saw this message in both our logs:

time="2021-03-08T21:18:37Z" level=info msg="reason: new statefulset containers's postgres (index 0) security context does not match the current one" cluster-name=database/olaii-db pkg=cluster

What does this mean?

[boris-savic]

Since this is happening on our production I've temporarily scaled the operator down to 0 replicas, so it's not updating every cycle.

Hope this gets fixed soon :)

[Jasstkn]

I've also found following lines in logs:

syncing connection pooler from (nil, nil) to (nil, nil)
--

[davidkarlsen]

Downgrading to the previous operator version worked for me.

[jabbors]

Also had the same problem with 1.6.1

12/03/2021 10:54:43 time="2021-03-12T08:54:43Z" level=info msg="reason: new statefulset containers's postgres (index 0) security context does not match the current one" cluster-name=metabase/acid-metabase pkg=cluster

Rolling back to version 1.6.0 worker for me as well.

[FxKu]

This is now fixed with the latest release.

[boris-savic]

Thanks.

I've updated but my cluster still keeps updating every sync interval.

This is from the logs:

time="2021-04-01T18:04:30Z" level=info msg="statefulset postgres/olaii-db is not in the desired state and needs to be updated" cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-      restartPolicy: Always," cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-      dnsPolicy: ClusterFirst," cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-      serviceAccount: postgres-pod," cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-      }," cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-      schedulerName: default-scheduler" cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="+      }" cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-      kind: PersistentVolumeClaim," cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-      apiVersion: v1," cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-      status: {" cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-        phase: Pending" cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-      }" cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="+      status: {}" cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-  }," cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-  revisionHistoryLimit: 10" cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="+  }" cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="metadata.annotation are different" cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-{" cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-  zalando-postgres-operator-rolling-update-required: false" cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg="-}" cluster-name=postgres/olaii-db pkg=cluster
time="2021-04-01T18:04:30Z" level=debug msg=+null cluster-name=postgres/olaii-db pkg=cluster

[FxKu]

Huh? Can you post the line logging the reason why it's doing the rolling update?

[boris-savic]

Huh? Can you post the line logging the reason why it's doing the rolling update?

Could it be this?

time="2021-04-02T10:34:30Z" level=info msg="reason: new statefulset's annotations do not match the current one" cluster-name=postgres/olaii-db pkg=cluster

I haven't changed any annotations though.