reflector is not recreating secrets anymore

[dberuben]

Hi,

I had OutOfMemory so i decided to increase the memory limits to 1024Mi but since nothing works has expected, even more secrets are created after a couples of minutes, or even not created. i have a couple of errors to share but nothing much.
Any idea?

Timetstamp problem!

2020-11-10 13:12:51.154 +00:00 [INF] (ES.Kubernetes.Reflector.Secrets.SecretMirror) Created reflection Secret for reflector/gcr-ls-docker-creds in namespace sbx-toto. 2020-11-10 13:12:55.414 +00:00 [INF] (ES.Kubernetes.Reflector.Secrets.SecretMirror) Created reflection Secret for reflector/google-sa in namespace sbx-toto. 2020-11-10 13:23:47.256 +00:00 [INF] (ES.Kubernetes.Reflector.Secrets.SecretMirror) Created reflection Secret for reflector/docker-artifactory-secret-tmp in namespace sbx-toto.

I also created another namespace where secrets should be replicated but nothing happend after a couple of minutes.

An error that a catch but really relevant

2020-11-10 13:03:40.449 +00:00 [ERR] (ES.Kubernetes.Reflector.Secrets.SecretMirror) Failed to process Added Secret sbx-annotatenamespace/docker-artifactory-secret due to exception Microsoft.Rest.HttpOperationException: Operation returned an invalid status code 'NotFound' at k8s.Kubernetes.PatchNamespacedSecretWithHttpMessagesAsync(V1Patch body, String name, String namespaceParameter, String dryRun, String fieldManager, Nullable1 force, String pretty, Dictionary2 customHeaders, CancellationToken cancellationToken) at ES.Kubernetes.Reflector.Secrets.SecretMirror.OnResourcePatch(IKubernetes client, V1Secret target, V1Secret source) in /src/ES.Kubernetes.Reflector.Secrets/SecretMirror.cs:line 105 at ES.Kubernetes.Reflector.Core.Mirroring.ResourceMirror2.Reflect(TResource source, TResource target) in /src/ES.Kubernetes.Reflector.Core/Mirroring/ResourceMirror.cs:line 439
at ES.Kubernetes.Reflector.Core.Mirroring.ResourceMirror2.ReflectSourceToSelf(TResource item) in /src/ES.Kubernetes.Reflector.Core/Mirroring/ResourceMirror.cs:line 345 at ES.Kubernetes.Reflector.Core.Mirroring.ResourceMirror2.OnResourceWatcherEvent(WatcherEvent1 request) in /src/ES.Kubernetes.Reflector.Core/Mirroring/ResourceMirror.cs:line 204 at ES.Kubernetes.Reflector.Core.Mirroring.ResourceMirror2.OnEvent(WatcherEvent e) in /src/ES.Kubernetes.Reflector.Core/Mirroring/ResourceMirror.cs:line 109
at ES.Kubernetes.Reflector.Core.Queuing.FeederQueue1.<>c__DisplayClass8_0.<<InitializeAndStart>g__ReadChannel|0>d.MoveNext() in /src/ES.Kubernetes.Reflector.Core/Queuing/FeederQueue.cs:line 50

Currently i'm using emberstack/kubernetes-reflector:5.4.1, i also tested with 5.4.7 and 5.4.17 but same.

[dberuben]

e.g when it takes a couple of minutes
all dockerconfigjson and Opaque should be reflected at the same time

NAME TYPE DATA AGE
default-token-sm2fs kubernetes.io/service-account-token 25m
docker-artifactory-secret kubernetes.io/dockerconfigjson 7m10s
docker-artifactory-secret-tmp kubernetes.io/dockerconfigjson 11m
docker-hub-secret kubernetes.io/dockerconfigjson 7m12s
gcr-ls-docker-creds kubernetes.io/dockerconfigjson 22m
google-sa Opaque 22m

[dberuben]

I found the issue ... something was blocking the controle plane :D

[razvvan]

@dberuben can you share some more details on what was blocking? I'm seeing the same issue