Set NSS_SDB_USE_CACHE=no to avoid memory growth

[jordansissel]

On affected systems[1], the readinessProbe costs 5mb of memory (file system cache) per invocation which is observed in some cases[2] to never be reclaimed. This is a bug in the kernel and/or kubernetes, but in the meantime, this env var can help prevent the readinessProbe from causing memory usage growth and eventually causing Kubernetes to evict pods.

[1] Certain versions of CentOS and GKE are known affected.
[2] It is observed in production that for pods without a strict memory limit, the container's file system cache (dentry) is not reclaimed appropriately, causing the kubelet to believe the pod is using far more memory than is actually being used by processes in the pod.

References: #1635

This PR is a proposal to work around a bug in the kernel/kubernetes. The reason I propose it is that this issue took 5 working days to investigate[3] and eventually find a workaround. I would spare others this same fate by having the workaround enabled by default ;)

[3] The investigation traveled the gradient from elasticsearch, to kubernetes, to the linux kernel, and all things in between. It was an interesting but ultimately unenjoyable adventure given I really didn't want to be spending a week digging into the kernel.

[jordansissel]

We have this deployed by setting env vars in one production cluster (ECK 0.9.0):

        containers: 
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: -Xms5g -Xmx5g
          # Curl in the readinessProbe triggers a kernel memory accounting bug(feature?)
          # So we disable it that part of curl.
          - name: NSS_SDB_USE_CACHE
            value: "no"

The impact of this change is observed in the following charts:

Memory usage in a one-week period where I was engaged with this exciting issue. Note the massive memory growth (caused by readinessProbe/curl) and high plateau (far above the 'memory request'). The right-side of the chart shows the beginning of my test where I set the NSS_SDB_USE_CACHE=no env var.

If we zoom into a 24 hour period just before I enabled this ENV var, we can see that memory usage is quite stable:

[jordansissel]

If there's 👍 on this proposed implementation, I can write tests. Let me know whatcha like :)

[sebgl]

Awesome ❤️
We probably need to update unit tests accordingly - something the team can do if you're busy @jordansissel.

[jordansissel]

need to update unit tests accordingly

Ahh, env vars need to be sorted (I see WithEnv sorts them, but the expected := ... in the test suite doesn't sort them.). I pushed a new commit which should pass this particular failure (does locally). Waiting on CI now.

[anyasabo]

👍 My initial idea was to use go-spew (which also might make the error output below easier to read), but that will only sort map keys

[anyasabo]

Thanks for adding and including the issue links in the comments

[jordansissel]

The elasticsearch-ci/docs check is stalled (or dead?) due to problems on elasticsearch-ci. The infra team is aware and working on the problem.

[jordansissel]

CI finally finished (<3 to our infra team who've been working on stabilizing Jenkins resources much of today)

[jordansissel]

Copying other references here for posterity:

• Allow ECK to specify a custom readiness check for Elasticsearch and Kibana #1581 (comment)
• https://issuetracker.google.com/issues/140577001
• kubelet counts active page cache against memory.available (maybe it shouldn't?) kubernetes/kubernetes#43916