Support more granular Vault Agent and volume configs

[rollerd]

Just wanted to start a conversation and get some input regarding increasing the configure-ability of both the Vault agent and the volumes/paths that get mounted into the pods.

It seems like a large number of the open issues could be resolved if people had a way to modify more fields in the Vault agent config as well as more control over volumes and paths in both the sidecar/init pods and app pods.

My initial foray into this was attempting to add support for other auto-auth methods (specifically approle auth), which required modifying not only the Vault agent config, but also creating volumes from secrets which housed things like roleid and secretid.
From looking at the requirements for the other auto-auth methods, it would seem that a similar way to add either volumes or env vars mounted from kube secrets is needed.

So my question is, how should all that look? @smitthakkar96 I believe rightly pointed out that just throwing a ton more annotations into the mix might not be the best solution.
However, having annotations that contain a bunch of raw json configuration also doesn't seem to be ideal either.

A thought was to have an annotation that could specify an existing kube secret from which to mount a volume/file(s)/env vars into the sidecar/init pods. This handles the case of making credentials available to the vault agent config.
For actually modifying the vault agent config, would passing some fields as key-value pairs in an auto-auth config annotation work?

For example, the only difference in config to make approle auth vs kubernetes auth work is the following:

Approle:

{
  "auto_auth": {
    "method": {
      "type": "approle",
      "mount_path": "auth/approle",
      "config": {
        "remove_secret_id_file_after_reading": false,
# these paths require files mounted from a kube secret
        "role_id_file_path": "/vault/approle/roleid",
        "secret_id_file_path": "/vault/approle/secretid"
      }
    },

Kubernetes:

{
  "auto_auth": {
    "method": {
      "type": "kubernetes",
      "mount_path": "auth/kubernetes",
      "config": {
        "role": "some_role_name"
      }
    },

mount_path already has its own annotation, so could we just use a 'config' annotation with k/v pairs for that section?

I'm happy to put in some work on this, but I wanted to try and get as much input as possible instead of just throwing stuff at the PR wall and hoping something sticks since I'm sure there's a bunch of stuff I'm missing. @jasonodonnell any thoughts on what you'd like to see on this?

TL;DR
would the following annotations cover most of the use cases for configuring vault agent and volumes?

vault.hashicorp.com/agent-auto-auth-method: xxxxx
vault.hashicorp.com/agent-config: "role_id_file_path=/path/to/secret1, secret_id_file_path=/path/to/secret2"
vault.hashicorp.com/agent-injector-secret: "kube-secret-name=/path/to/secret1"
vault.hashicorp.com/app-volume-mount: "/some/path/in/app/pod"

[jasonodonnell]

Hi @rollerd,

We already offer a way of mounting a configuration file via the vault.hashicorp.com/agent-configmap annotation. So what is really missing here, as far as I can see, is a way of mounting custom secrets that have approle role_id and secret_id.

With this in mind we would probably only have a single annotation, vault.hashicorp.com/agent-extra-secret=<name of secret> which would mount a secret to a static directory (maybe /vault/custom) which makes the scope of this much smaller.

Something like:

vault.hashicorp.com/agent-inject: true
vault.hashicorp.com/agent-configmap: myconfigmap
vault.hashicorp.com/agent-extra-secret: mysecret

Thoughts?

[rollerd]

Hey @jasonodonnell ,

Thanks for the quick response! I started digging into this today and came across a few issues.
The code to add a new annotation and extra-secrets volume is pretty straightforward and I was able to get things working with the agent-configmap annotation that you suggested.

However, unless I'm missing something (like with the agent-configmap) it seems there are some issues:

• It looks like an agent-configmap with a different inline template would have to be created for each service/app you would want to deploy.
  In other words, I can create a configmap that has a template, reference that configmap in my pod annotation and deploy it, but then my secret path is tied to that configmap. If I want to deploy a new pod using the same agent configuration from that configmap but only change the secret path, I need to create a whole new configmap with a different template that references the new secret path.
  The solution seems to be to read the specified configmap first and then patch it with the annotations for vault and template settings if the user provides any.
  Does this seem reasonable or is the intention to have an explicit configmap for each service/pod?

• While the fields for the agent config for using approles can reference files (mounted from secrets), the other auto-auth methods might require a separate annotation for mounting those secrets as environment variables. I don't have a way to test this, but it seems like the AWS auth for example, either needs to have the secrets hard-coded into the configmap or it needs to be able to read them from the environment. Obviously, the environment is the lesser of two evils there.
  This seems like it should be pretty easy to do with a second annotation and new function addition in the patch.go file.
  What are your thoughts on one more annotation for taking care of the env vars case?
  The docs for AWS auth do allow for a file for credentials.

Thanks in advance for your feedback!

[rollerd]

Created PR #119 that adds an annotation for creating volumes from Kube secrets that get mounted to /vault/custom/<somesecret> in the sidecar/init containers.

This works and the only downside at this point is having to create a separate full configmap - complete with custom template - for each service/app that might run.

[rollerd]

Anything I can add/remove/change to get some feedback on the PR referenced above?

[rollerd]

The related PR for this is merged. Closing this out.