Add support for datasource directories

[rhuss]

When the argument given with --datasource points to a directory, every file in this directory is processed with the plain-file logic for datasource (e.g. alias == filebasename, type determined by file extension).

Relates to #117 and builds on top of #119 with rhuss@b5c2992 on top.

I think this is a simple and backwards compatible extension to file based datasources.

[hairyhenderson]

Thanks @rhuss for this PR!

Instead of inferring directories vs files, I think I'd still rather a separate and explicit dir: scheme. I think it'd be much simpler to code, and would avoid surprises. The support for relative file URLs was essentially a concession for the simplest of use-cases. When more complex use-cases are involved (such as this one), I think I'd much rather require users to be explicit:

$ gomplate --datasource mydir=dir:///tmp/datasources/ ...

Essentially what you'd need to do is call addSourceReader("dir", readDir) around line 44, and implement readDir - it should simply defer to readFile when files are encountered.

[hairyhenderson]

Also - I'm assuming #119 will get merged before this one, but for the record it's much easier to review PRs that are based on master 😉

[rhuss]

Also - I'm assuming #119 will get merged before this one, but for the record it's much easier to review PRs that are based on master

It's also easier to implement on top of something which I hope is very likely to be merged instead of doing quite some rebase conflict solving afterwards (e.g. because of moved functions).

I don't mind to have wait and rebase it until #119 is merged (or cherry pick the singles commit b5c2992 which contains the whole functionality as I mentioned).

[rhuss]

I think it'd be much simpler to code, and would avoid surprises. The support for relative file URLs was essentially a concession for the simplest of use-cases. When more complex use-cases are involved (such as this one), I think I'd much rather require users to be explicit:

Actually I disagree, since using the dir datasource scheme as suggested leads to an involved usage pattern, by having datasources with multiple parameters. I don't see any conflicts with the suggested approach and the approach file basename == datasource alias is a nice and intuitive convention. You also have less to specify on the command line and can generate general purpose Docker images which mount the datasource directory as volumes (you then can reuse this image directly without changing ENRTYPOINT or CMD).

Please have a look later at the implementation, its really straight forward. There is only one minor change that an existing check has need to be relaxed but there is no ambiguity

[hairyhenderson]

leads to an involved usage pattern, by having datasources with multiple parameters

I'm not sure what you mean by this - can you elaborate?

Please have a look later at the implementation, its really straight forward. There is only one minor change that an existing check has need to be relaxed but there is no ambiguity

Sure, I'm having a look - it might take few hours/days because I have some day-job priorities 😉

[rhuss]

I'm not sure what you mean by this - can you elaborate?

I refer to your initial usage example with having two parameters to datasource:

{{ (datasource "mydir" "foo").whatever }}
{{ (datasource "mydir" "bar").something }}

[hairyhenderson]

Right - my assumption was that the ds alias wouldn't be inferred from the filename. While it may seem convenient, it raises other issues. What happens when there's a file with the same name as a separately-defined ds alias? What happens when multiple directory datasources are provided and contain some files the same names?

I'd much rather err on the side of being explicit rather than convenience, especially for a new feature. If it turns out that this is more popular than expected this doesn't close the door to making this more convenient later.

[rhuss]

No problem, it's your pet project anyway.

But I have to move on (tbh the review process of #119 was a bit too much compared to the size of the feature, including some unnecessary but timely turns) and I'm quite happy with the current implementation. I use it as a base image for the examples in our new Kubernetes Pattern book, where it is very useful within a Kubernetes init-container for creating configuration files from templates dynamically during startup.

For this, I create a minimal Docker image based on scratch which is preconfigured to expect templates under /in, template datasources under /params and stores the processed files under /out.

Feel free to merge this PR if you like. I'll track your project and would switch to the url scheme approach when implemented.

References:

• Gomplate base image
• Example for the Configuration Template pattern

Thanks a lot for creating gomplate !

[hairyhenderson]

Thanks again for the PR - I'm going to close this in favour of #334.