Load variables from config files

[omarabid]

Does just load the cargo.toml file so that I can access parameters like the project name, version, etc...

I find it redundant to put these parameters in both cargo.toml and a .env file. It is better to have them in one location.

[casey]

Although written in Rust, Just is in general language agnostic, so I'm hesitant to add any language-specific features. Can you give some examples of parameters you'd like to have access to your justfiles?

[omarabid]

Like, for example, the project name/version. I was not aware that Just is language-agnostic. Another idea is to make it possible to load any .toml or config file through just and use the variables inside.

[casey]

I think that's a good way to do it, i.e. letting users load arbitrary TOML files. Indeed, Just's justfile parses Cargo.toml with sed to extract the version string so it would benefit from this feature ^_^

This feature requires a few things:

1. A way for users to indicate that they want to load a particular file, and what the format is
2. The code to load files and make the variables available in justfiles
3. Syntax to use those variables

For 1 and 3, it might look like this:

# the file extension, `.toml` makes Just interpret the file as TOML
# the file stem, `foo`, determines the name of object containing the loaded variables
load "foo.toml"

# equivalently but more explicitly
foo = load("foo.toml", "toml")
# (however, the above would present difficulties, because it implies that a function can return something other than a string)

# variables can be retrieved from the `foo` object using dot notation
bar:
  echo {{foo.bar}}
  echo {{foo.baz.buzz}}

Since TOML files can contain values that are not strings, e.g. floating point numbers, lists, and table, and the only kind of values in Just are strings, we'd need to figure out how to bridge the gap.

As an MVP, I think I might only allow accessing string values, since they require no conversion. Then, later, when we have more experience with the feature, we can figure out what the right thing to do with other types is.

[omarabid]

You just gave me a solution @casey. Haha, Thanks but I agree that parsing can be challenging and add more complexity to the program. Is there some parsers implemented in pure Bash?

[casey]

Actually, since there is already a good TOML parser in rust, the parsing part shouldn't be too hard.

I couldn't find any TOML parsers in pure Bash, only this one, which has a shell interface but is written in Go.

[dennyweiss]

Before going the toml only route, please consider that there is a realistic need for loading config values from multiple sources.

I think it would be clever to separate the config value usage into two areas:

1. accessing/reading from a config backend system (toml, yaml, .env, Environments Variables, Rest-Endpoints for secret storage services like vault or Azure Key Vault)

2. using values through a unified (function) interface like the actual env var function env_var_or_default() where the key could include a backend prefix e.g. vault::foo.bar or config-1.toml::foo.bar

What do you (@omarabid, @casey) think?

[casey]

@dennschu Those are totally fair comments!

Before going the toml only route, please consider that there is a realistic need for loading config values from multiple sources.

I wouldn't want to support TOML only, but want to limit the number of formats supported in the MVP, so doing TOML first seems reasonable.

1. accessing/reading from a config backend system (toml, yaml, .env, Environments Variables, Rest-Endpoints for secret storage services like vault or Azure Key Vault)

2. using values through a unified (function) interface like the actual env var function env_var_or_default() where the key could include a backend prefix e.g. vault::foo.bar or config-1.toml::foo.bar

Multiple sources by having declaration lines that bring a name into scope that represents that source, and a common notation for accessing properties:

cfg0 := load("foo.toml")
cfg1 := load("foo.yaml")
cfg2 := something_else()

# can all be accessed with dot notation
foo:
  echo {{cfg0.bar}}
  echo {{cfg1.baz}}
  echo {{cfg2.bob}}

Does that sound reasonable?

[omarabid]

The types supported could be limited to what the ecosystem has in term of stable libraries and Serde support. My understanding is that you need some kind of Serde adapter and then you can support easily any other type given that a library for that exists?

[casey]

The issue with types has more to do with Just's type system than available crates and Serde support.

Currently, just only has one type of value, strings. I would like to introduce additional types, so that the language can be expanded However, I dislike dynamic type systems, so the introduction of a second type will require adding a static type system to Just, and who knows when I'll get around to that 😅

[omarabid]

*I meant file type. I don't think there is a reason to implement types for bash instructions. It'd be helpful for quite advanced situation but that might not be the best focus for Just now?

[casey]

Right, for file types, anything with a high quality parser and serde support will be easy.

My concern is that if I implement a dynamic type system it will be very hard to move to a static type system in backwards compatible way. And, my prior is that static typing is useful even for small programs.