Secrets.yml configuration can be supplied as an environment variable

[izgeri]

Description

We support supplying the secrets.yml config for Summon as a literal string passed to the Summon command:
https://github.com/cyberark/summon/blob/434da01c1c7ecff7a58d614048ae167f91dd0425/internal/command/flags.go#L31-L34

In this issue, we would like to add support to the buildpack for specifying the secrets.yml config as a literal string in an environment variable. This allows developers to separate their Conjur configuration from their application artifacts, which will enable them to change their Conjur config without needing to rebuild the app. It was motivated in particular by Java applications, which absent this change require building the secrets.yml into the jar file.

The way this flow would work is that you'd define a SECRETS_YAML env var in your manifest, like:

---
applications:
- name: my-app
  services:
  - conjur
  buildpacks:
  - conjur_buildpack
  - ruby_buildpack
  env:
    SECRETS_YAML: |
      AWS_ACCESS_KEY_ID: !var aws/iam/user/robot/access_key_id
      AWS_SECRET_ACCESS_KEY: !var aws/iam/user/robot/secret_access_key

where the | character is used to allow multi-line SECRETS_YAML config with line breaks preserved, as per the yaml spec.

In practice, you could set this env var in the manifest as above or with the cf env command and run cf restage to restart the app with a new secrets config. This should be implemented so that if a secrets.yml file exists, it would override the env var; for example, our logic would be something like:

• check for SECRETS_YAML_PATH env var; use the file there if it exists. if not,
• check for secrets.yml file in standard locations; use the file if one exists. if not,
• check for SECRETS_YAML env var; use the content there, if it exists. if not,
• fail to apply buildpack.

Background

From #18 (comment) as reported by @mkkeffeler.

i think the issue we are having is related to this. #18

We have a manifest where we specify path: /path/to/jar.jar

this uploads the jar. the jar DOESNT contain secrets.yml. this fails, saying it can't find secrets.yml.

We update path to path:/path/to and this directory contains jar.jar, and secrets.yml (voila, conjur buildpack works, but PCF can't start the app for some reason, seemingly because it doesn't know what to do with the jar.

We don't want to have to put secrets.yml in our jar files because that means for any config change we have to rebuild our app, which is not consistent with the "build once deploy anywhere" strategy that we see people adopting across the industry.

Thoughts on this? reccomendations?

@mkkeffeler I left the bug template below in case you want to add any more info to this card using this format - it can be a helpful way to understand clearly how the product isn't meeting your expectations right now. We'll look into this and try to get back to you shortly. Please also let me know if the title doesn't accurately encapsulate your problem. Thanks!

[BradleyBoutcher]

Has the user attempted the instructions mentioned in the readme for using a non-standard secrets.yml path? Specifically adding the following to their manifest.

env:
    SECRETS_YAML_PATH: <path/to/yaml>

[izgeri]

I think the problem is that you build the jar file and push the app by calling

cf push YOUR-APP -p target/YOUR-APP-VERSION.jar

and it's not clear what the mechanism is to also push the secrets.yml file unless it's included in the jar, which:

means for any config change we have to rebuild our app, which is not consistent with the "build once deploy anywhere" strategy that we see people adopting across the industry

I'm looking into this to see what options there might be. other java teams that I know of using this buildpack are building the secrets.yml file into the jar as far as I know, but I'm not sure whether they have this same concern.

[izgeri]

I asked my contacts at VMWare and they let me know that there are buildpacks that will write environment vars to a file - so you could store your secrets.yml config in an env var, invoke such a buildpack, and our buildpack could read the contents of the file as usual.

But we also support supplying the secrets.yml config for Summon as a literal string passed to the Summon command:
https://github.com/cyberark/summon/blob/434da01c1c7ecff7a58d614048ae167f91dd0425/internal/command/flags.go#L31-L34

So we could add support in this buildpack for specifying the secrets.yml config as a literal string in an environment variable, if it would help avert the need for building the secrets.yml as a jar.

The way this flow might work is that you'd define a SECRETS_YAML env var in your manifest, like:

---
applications:
- name: my-app
  services:
  - conjur
  buildpacks:
  - conjur_buildpack
  - ruby_buildpack
  env:
    SECRETS_YAML: |
      AWS_ACCESS_KEY_ID: !var aws/iam/user/robot/access_key_id
      AWS_SECRET_ACCESS_KEY: !var aws/iam/user/robot/secret_access_key

where the | character is used to allow multi-line SECRETS_YAML values with line breaks preserved, as per the yaml spec.

You could also potentially set the var with the cf env command and run cf restage to restart the app with a new secrets config. I imagine this being implemented so that if a secrets.yml file exists, it would override the env var; the logic might be:

• check for SECRETS_YAML_PATH env var; use the file there if it exists. if not,
• check for secrets.yml file in standard locations; use the file if one exists. if not,
• check for SECRETS_YAML env var; use the content there, if it exists. if not,
• fail to apply buildpack.

@mkkeffeler I'd be interested in your feedback on this idea, and whether it meets your needs.

[TheSecMaven]

Has the user attempted the instructions mentioned in the readme for using a non-standard secrets.yml path? Specifically adding the following to their manifest.

env:
    SECRETS_YAML_PATH: <path/to/yaml>

yes we tried, again though, if its not packaged in the jar and you are directly pushing that jar this still fails

[TheSecMaven]

@izgeri this makes sense, im not aware of those buildpacks so will have to ask around, as that does seem like a workaround.

we found a workaround

jar uf jarfile.jar secrets.yml

copies secrets.yml into the root of the jar file. so this works, but is not preferred. best would be like you said, specify the text of the file as an env var. i think that is the cleanest of all 3 solutions discussed here now (jar uf, buildpack env var to file, and env var raw string)

[izgeri]

I agree. I'm going to change this issue to an enhancement request to support specifying the secrets.yml configuration in an env var. Thanks for your feedback @mkkeffeler!