Generating multiple files with custom values

[jlaberge-league]

Hi!

I am evaluating Pkl for a use-case that I have around generating YAML files for Prometheus.

I have a folder structure like this:

prometheus/
-- base/
---- app1.yaml
-- overlays/
---- env1/
------ app1.yaml
---- env2/

I have my pkl setup (simplified) like this:

templates/
-- spec.pkl
apps/
-- app1.pkl (amends spec.pkl)

What I am looking to achieve is something like the following:

/// app1.pkl

environments {
  base {
    myValue = 10
  }
  
  env1 {
    myValue = 20
  }
}

and the output should look something like:

# base/app1.yaml
myValue: 10

# overlays/env1/app1.yaml
myValue: 20

and then env2/app1.yaml shouldn't exist since it wasn't configured in the environments. However, app2 for instance might have it configured.

I could likely do something like:

for (env in environments) {
      ["overlays/\(env)/\(app).yaml"] {
        value = new Dynamic {
          example {
            value = env.myValue
            }
          }
        }
      }

however, I'm not sure if this is ideal as the block is quite large and dumping it all in the output seems wrong.

What I would expect to work is something like:

for (env in environments) {
  ["overlays/\(env)/\(app).yaml"] {
    value = object1
    myValue = env.myValue // overrides the myValue used in spec
  }
}

Additional context:

I generate 3 different files for each application, this is currently being done by having output.files configured for three different files with value = object1, object2, object3

each application might have different values per environment so the number of yaml files needed will always be 3 * (number of environments + 1), as base is always generated.

[HT154]

Would you be able to share (at least a reduced portion of) your spec.pkl? Understanding how you've laid out the types for the top-level environments property would help provide some better direction here.

[HT154]

Gotcha, I think I'm understanding your goals better now! Mixins are a nice way to achieve overrides/extensions like this.

Here's how I might set this up:

extends "slo.pkl"

hidden appName: String(!isEmpty)

hidden targets: AppSLO

slos: Mapping<String, SLO> // define multiple SLOs, each defined in terms of `targets`

environments: Mapping<String, Mixin<AppSLO>>

output {
  files {
    default {
      renderer = new YamlRenderer {}
    }
    for (sloName, sloGen in slos) {
      ["base/\(appName).yaml"] { // generate the base file for each slo for the app
        value = new Dynamic {
          slos = module.slos.toMap().values // produce the base output with the "defaults" defined by targets
        }
      }
      for (envName, envMixin in environments) {
        ["overlays/\(envName)/\(appName).yaml"] {
          value = new Dynamic {
            slos = ((module) { targets = module.targets |> envMixin }).slos.toMap().values // produce the per-env overlays by applying the per-env mixin to `targets` and extracting the resulting modified module's SLOs (now derived from the modified value of `targets`)
          }
        }
      }
    }
  }
}

And app1.pkl might look like this:

targets {
  availability = 99.9
}

slos {
  ["slo1"] {
    objective = targets.availability
  }
  ["slo2"] {
    // idk something else
  }
}

environments {
  ["xyz"] {
    availability = 99.5
    // note that if `targets` contains other properties this does not modify them, it just overrides `availability`
  }
}

Hopefully this gives you a good jumping-off point to fill in your other properties!

[jlaberge-league]

Perhaps I am doing something a bit wrong here with how I've structured my code. But with how things are currently, I have something like:

slos {
    slo1 { // "slo1" is not something I want to include in the rendered YAML
        service = "slo1"
        slos {
            new SLO {
                target = targets.availability
            }
        }
    }
}

// modified output now based on your design
output {
    renderer = new YamlRenderer {}
    files {
        default {
            renderer = new YamlRenderer {}
        }
        for (sloName, sloSpec in slos) {
            ["base/slos/\(appName)-\(sloName).yaml"] {
                value = sloSpec // I had to modify this otherwise the rendered YAML is incorrect
            }
            for (envName, envMixin in environments) {
                ["overlays/\(envName)/slos/\(appName)-\(sloName).yaml"] {
                    value = ((sloSpec) { targets = module.targets |> envMixin }) // this renders, but it just adds the targets at the end of the file, not replacing the values therein
                }
            }
        }
    }
}

Apologies for this rabbit hole and any confusion, hopefully this adds some additional clarity:

slo1 should be it's own YAML file, slo2 it's own file as well.

The above structure that I have, if rendered directly as YAML would look something like:

slos:
  slo1:
    service: slo1
    slos:
     - target: 99.95

Whereas it should look like this: (which is why I changed value = sloSpec which works for base, and am trying to achieve the same thing for the overlays/mixin approach

service: slo1
slos:
- target: 99.95

[jlaberge-league]

I got the mixin part working as well, doing this:

value = ((module) { targets = module.targets |> envMixin }).slos.toMap().getOrNull(sloName)

I'm not sure if there is a more elegant approach or if there is any bad smells to how I am having it rendered now. Part of me is worried about the getOrNull bit, but it's likely a non-issue for me at the moment.

[HT154]

Gotcha, iterating on this a little more:

// this should be a Mapping<String, SLO>, so you have keys instead of properties
// this is preferable to a Dynamic or even some Typed class as you can enforce Mapping values are SLO instances
slos {
    ["slo1"]  {
        objective = targets.availability
    }
}

output {
    files {
        default {
            renderer = new YamlRenderer {}
        }
        for (sloName, sloSpec in slos) {
            ["base/slos/\(appName)-\(sloName).yaml"] {
                value = new Dynamic { // push the "presentation logic" of how the file gets rendered down into the output config and out of the module schema
                    service = sloName
                    slos = new Listing { sloSpec }
                }
            }
            for (envName, envMixin in environments) {
                ["overlays/\(envName)/slos/\(appName)-\(sloName).yaml"] {
                    value = new Dynamic {
                        service = sloName
                        slos = new Listing { ((module) { targets = module.targets |> envMixin })[sloName] } // we know the slo with this key exists, so we can just subscript directly instead of using getOrNull and/or toMap
                    }
                }
            }
        }
    }
}

PS: thanks for the patience! This is a challenging modeling problem that I think Pkl can really help you with, but this sort of thing definitely needs some thorough brainstorming.

[jlaberge-league]

The feeling is mutual 😅 , appreciate your efforts here as well!

when using new Listing, my YAML renders like this:

service: slo1
slos:
- - name: availability # notice the extra dash in the first element
     target: 90
   - name: latency 

[HT154]

Hmm I think my accumulated mental model of your schema has drifted a bit, can you share the templates/slo.pkl and templates/slos.pkl as you've got them now? If I'm thinking about this correctly you can get what you want by removing the new Listing, but I'd rather be sure I'm not steering you wrong!

[jlaberge-league]

It works like this:

value = ((module) { targets = module.targets |> envMixin }).slos[sloName]

I think I am good to consider this question answered! Thanks for all the help