-
Notifications
You must be signed in to change notification settings - Fork 438
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
POC for #196 - Relax computed importstr #805
base: master
Are you sure you want to change the base?
Conversation
Adds a new builtin function (importstrDyn) that accept a computed string. I did not modified the importstr logic itself as I don't understand well enough how the parser is working.
Any feedbacks? |
This is one of my main blocker to not create external commandline tool in order to use std.native function and do the same. |
Thanks for the ping and sorry I didn't reply sooner. This is a long standing discussion. Basically the import mechanism was intended as a way to statically import libraries and split code into multiple packages, not for loading stuff dynamically. The matter is not settled, but I expect the impact on the language to be (slightly) net negative. Here's my summary of arguments for and against relaxing the restriction. Pros:
Cons:
The cons are not really deal breakers, but the pros seem like relatively minor convenience things and the alternatives are not so bad. The most important things if you want to move this forward would be to describe the use cases and why the alternatives are not sufficient (why not specify the paths directly and why not pass the data as TLA). FYI the implementation on the interpreter side is really not a problem. This is entirely an issue of language design.
I'm not sure what you mean. Can you elaborate? Do you mean using Jsonnet as a library and providing a native function there which emulates computed imports? Do you still have the same use case as you mentioned in #196? |
Yes, using Jsonnet as a library to implement additional methods such as |
Yeah, importYaml stuff is is the most convincing use case here. We're also looking into alternative solutions, e.g. first-class support for loading different types of files with And you can always do things like |
@sbarzowski also I like to 'hide' jsonnet from user when it's not needed and make it like seems like basic json/yaml. So instead to requires the end-user to know about jsonnet to write the following: build(
{
config: std.parseJson(importstr './config/myconfig.json')
}
) I prefer: build(
{
config_file: './config/myconfig.json'
}
) And process the user input in the |
My use case if the following: I have a 2 phases process. First phase uses a jsonnet program to generate a json file driving an imperative process to generate some files (generated file names comes from some info in the json file). In the second step I generate a json that aggregates some of the files generated during the imperative process. As you may imagine, in the first phase the filename/key of the target filenames are dynamic/configurable (users of the jsonnet program can even add supplemental entries). phase1.jsonnet
file1 and file2 are generated by an imperative process. phase2.jsonnet (we use -m to generate files in a different folder)
phase2_wannabe.jsonnet
An alternative to that would be to do higher order jsonnet (ie generate phase2.jsonnet during phase1), but I feel this would pushing it too much and think that computed importstr would be better / more readable. Does it help you understand the use case? PS: the real use case is related to kubernetes secrets generation. Phase1 define what secrets should be deployed and from which vault they should be retrieved. Imperative process reads this config and generates secrets files from there. Phase 2 generates kubernetes secret resources by importstr the content of the secret files. The kube secret resources could be generated directly in the imperative process, but all other kubernetes resources are generated during phase 2 and we try to do as little as possible during the imperative phase to ease debugging / testing / reproducibility. |
Thanks for providing more context. Re @ant31: Re @mbarbero:
Yeah, good call, that's kind of the point of Jsonnet – keep logic pure, sane and separate from any operational stuff.
Agreed. Generating Jsonnet source code with Jsonnet is going to be hard to follow.
How about loading the content of secret files using extVar/tlaVar? Is there some reason why that wouldn't be practical? |
@sbarzowski Thanks for your efforts in identifying our precise needs.
We're talking dozens of files here. It would create humongous command line or function parameters list. Also, secrets file don't just contain a single password or passphrase. Sometimes, it's complete files with dozens of passwords in XML... (and this is out of our control). That makes both extVar and tlaVar impracticable. |
Not that much, it actually joins @mbarbero case, because you could externalise this configuration/user input in a yaml : values.yaml: config_file: ./config/myconfig.json
username: me
password: bla manifest.jsonnet {
values: parseYaml(importstr 'values.yaml')
config: importDyn(values.config_file)
} |
You don't need to pass every single secret as a separate tla/extVar – that of course could get quickly out of hand. I meant passing all secrets at once in one structure file. E.g. you could have a secrets.json which contains all secrets (or it could contain XMLs which contains the actual if that's your thing). If you already have a directory structure with all these secrets, you can serialize them all to JSON with a tool such as
The main disadvantage of this approach is having to copy all of these secrets to one place ahead of time, whether or not they're going to actually be used. It also doesn't work well if your directory structure has symlinks etc. This is something you can use here and now, independently from what we decide here. |
What do you think about providing How useful do you think that would be for use cases like yours? |
@sparkprime Since we got back to this discussion, perhaps you have some thoughts? |
@sbarzowski Sorry for late reply. Yes, this solution would be workable. As long as it's supported natively by jsonnet (without having to call the intermediate dir2json binary), I think it's a decent alternative to computed |
Adds a new builtin function (importstrDyn) that accept a computed string.
I did not modified the importstr logic itself as I don't understand well
enough how the parser is working. Of course, it would better if importstr could be modified to accept any computed string.
With
test.jsonnet
and with
file-dyn.txt
Executing
$ jsonnet test.jsonnet
outputsHopefully, this PR will trigger new discussions and maybe the addition of such a feature. I really don't see how this would diminish the hermiticity (not more than extVar or tla IMO) nor how it would prevents static analysis.