Autocreate data folder in Kubernetes deployments

[exekias]

We need to persist beats data folder, so Pod restarts can still access
to the previous data.

This is particularly important in the case of Filebeat, as it needs the
registry file to avoid sending all logs again.

[ruflin]

First reading this I thought it should be filebeat/data as that is kind of a standard directory we use for our data files. How does k8s now about this directory? Is it configured somewhere or is this based on convention?

[exekias]

This is mapping a folder from the container to the host. Kubernetes doesn't know anything about it, we request it to be created here, and mount it in the container, so data folder is persisted to the host.

I wanted to avoid using the default folder, in case someone is messing around by deploying Filebeat both in the host and as a container, which could end up in 2 Filebeat instances sharing the data folder.

[ruflin]

It seems the travis tests suggest that the yaml is now invalid? https://travis-ci.org/elastic/beats/jobs/397307177 Related?

[exekias]

It is related indeed, it seems the path autocreate (DirectoryOrCreate) feature was introduced in k8s 1.8, so older versions would not work with this change. This is why the original emptydir + comment was there.

We should think about this twice, as it would break manifests compatibility with old versions, but is also a much better default for new deploys.

[exekias]

So our options are:

• Document the need to manually create the /var/lib/filebeat-data folder + update manifests for older Kubernetes versions.
• Keep things as they are, but stress more the need to move from emptyDir to hostPath in our docs, as the comment in the manifest is missed by most users.

[ruflin]

• Could we have 2 versions of the manifests?
• Do we have any numbers on what the most common versions of k8s are? Are users in general likely to update their k8s version frequently or are they stuck with certain versions because of breaking changes?

[moadqassem]

@exekias Just a small question, don't you need to update the filebeat config map too?

[exekias]

What kind of update? this change should be transparent to Filebeat, as we only change the mount point

[moadqassem]

@exekias filebeat.registry_file: /var/lib/filebeat-data

[exekias]

No, as this is mounted to the default path inside the container: https://github.com/exekias/beats/blob/43340ee3b983638bd7972623db5d6a056bf6cd05/deploy/kubernetes/filebeat-kubernetes.yaml#L101

[exekias]

I've updated tests to remove k8s 1.7 and earlier, and add the last 1.11 release. I also updated docs with manual settings to use the manifest in deprecated versions.

This should be ready to go

[exekias]

I think we should not backport this to 6.3, so it will go to 6.4 once merged

[ruflin]

I think the travis failures are related?

[exekias]

It seems new minikube is failing to deploy a local k8s, will have a look

[exekias]

Until kubernetes/minikube#2704 is fixed we are stuck to this version of minikube, I rolled back my changes to ensure tests pass.

[ruflin]

Would be good to get this tests on Jenkins where we could test newer versions because we control the environment.