Added auto-scale to the collector

[jpkrohling]

A Horizontal Pod Autoscaler (HPA) was added in this PR, along with a new MinReplicas and MaxReplicas. With that, the collector should now automatically scale up and down based on the CPU and/or memory consumption. When none of the new properties are specified, the minimum amount of replicas is 1, while the maximum number of replicas is 100. The HPA configuration is added only when the deployment strategy is either production or streaming.

Closes #848, even though the scaling of the storage isn't implemented by this one.

Signed-off-by: Juraci Paixão Kröhling juraci@kroehling.de

[jpkrohling]

This shows how the auto-scaling works in OpenShift. Note that this should also work in plain Kubernetes, but I'm not able to generate enough load on my local machine with minikube + ES (1
GiB) + tracegen.

[jpkrohling]

@kevinearls not sure we want to add a new e2e test for this, but perhaps you might have a good idea that wouldn't be too fragile?

[jpkrohling]

I just ran a longer test this morning, showing that it scales up and down. First, I deployed the simple-prod instance, along with tracegen (10 replicas). After about 20 minutes, 10 replicas of collector were available. Removing the tracegen deployment caused the number of replicas to go down back to 1 after about 20 minutes. Then, I changed simple-prod to add a maxReplicas set to 5. Then, I deployed tracegen again, and verified that the collector gets scaled only up to 5 replicas. Removing the tracegen causes the collector to eventually settle at 1 replica again.

$ kubectl get deployments
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
simple-prod-collector   1/1     1            1           2m2s
simple-prod-query       1/1     1            1           2m2s
tracegen                10/10   10           10          30s

$ kubectl get deployments
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
simple-prod-collector   10/10   10           10          19m
simple-prod-query       1/1     1            1           19m
tracegen                10/10   10           10          17m

$ kubectl get deployments
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
simple-prod-collector   1/1     1            1           44m
simple-prod-query       1/1     1            1           44m

$ kubectl get deployments
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
simple-prod-collector   5/5     5            5           59m
simple-prod-query       1/1     1            1           59m
tracegen                10/10   10           10          11m

$ kubectl get deployments
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
simple-prod-collector   5/5     5            5           71m
simple-prod-query       1/1     1            1           71m
tracegen                10/10   10           10          24m

$ kubectl get deployments
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
simple-prod-collector   5/5     5            5           71m
simple-prod-query       1/1     1            1           71m

$ kubectl get deployments
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
simple-prod-collector   1/1     1            1           88m
simple-prod-query       1/1     1            1           88m

And here's a set of screenshots from OpenShift (older events are at the bottom):

[objectiser]

Looks good. Just some minor comments.

[objectiser]

Shouldn't tracegen image be versioned inline with the other jaeger components?

[jpkrohling]

Yes, opened an issue (#866) to track this. The first image will probably be 1.17 (next release).

[objectiser]

What happened to the convention of naming instances after the test?

[jpkrohling]

We started naming them all "my-instance" some time ago, as individual names don't bring much value and we had a few copy/paste mistakes.

[objectiser]

@jpkrohling Approval subject to tests passing :)

[jpkrohling]

Local run has shown that a new permission is missing. I'm testing it locally and will update the PR once I confirm the tests are passing.