Add support for DeamonSet annotations, extraVolume and extraVolumeMounts

[decayofmind]

Is your feature request related to a problem?

Yes.

I know that it's possible to set deploymentSecret.create as false to use Vault sidecar which will inject a file with distribution-id in it.

However if CSI or External Secrets Operator is used to work with Secrets in a cluster, there's no way how to provide distribution-id to the Cortex agent Pod.

Describe the solution you'd like

• Add daemonset.annotations to daemonset.yaml
  • This will allow to use tools like Reloader where a special annotation on a DaemonSet will make it watch change to Secret created by the chart (or externally) and recreate Pods.
  • ExternalSecrets object will populate a proper distribution-id to a Secret created by the chart with deploymentSecret.create: true, however all Pods will need to be recreated to fetch that change.

Alternative (additional) solutions are:

• Add daemonset.extraVolumes and daemonset.extraVolumeMounts
  • This will allow to set deploymentSecret.create: false and use CSI drivers supplying custom Secret objects, created on a side
• Add support for deploymentSecret.name to work similar as dockerPullSecret.name
  • This will allow to set deploymentSecret.create: falseand provide name for any other Secret object containingdistribution-id` created externally.

[decayofmind]

It's a low effort to implement and I can help with PR if you are open for external contributions.