`to-directory-tree` doesn't create subdirectories

[timbertson]

aside: thanks for for-directory-tree, I'm very excited to use it :D

$ echo '{ a = "hello" }' | dhall to-directory-tree --output issue
$ cat issue/a
hello

$ echo '{ `a/b` = "hello" }' | dhall to-directory-tree --output issue
dhall: issue/a/b: openFile: inappropriate type (Not a directory)

If I mkdir -p issue/a first it works, but that's obviously inconvenient.

While I'm at it, my initial assumption was that to-directory-tree would take a map, instead of a record literal. That seems like it could be more flexible (because the keys can be programmatically generated too), and would also remove the need to support Optional Text (you could just filter out the unwanted items from the map). Should I make a separate issue for that, or does it intentionally not support maps?

[sjakobi]

@timbertson Instead of

$ echo '{ `a/b` = "hello" }' | dhall to-directory-tree --output issue

you can use

$ echo '{ a = { b = "hello" } }' | dhall to-directory-tree --output issue

Does that work for you?

[Profpatsch]

We should disallow / in the keys for that and tell the user that they should use nested records instead.

[Gabriella439]

Yeah, I would be fine with changing the to-directory-tree command to disallow / in the keys

[timbertson]

Oh, I was kind of relying on that actually. I am combining a number of files from multiple places, and (for example) a couple of unrelated files both belong in `scripts/` (one for docker, one for Travis). Having to pluck those out from each place and put them into a record makes it hard to combine multiple file sets.

…

On Sat, 18 Jan 2020, 7:13 am Gabriel Gonzalez, ***@***.***> wrote: Yeah, I would be fine with changing the to-directory-tree command to disallow / in the keys — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1633?email_source=notifications&email_token=AAADOXHYSSVMXWQY7ZI5UCDQ6IGO7A5CNFSM4KIBRJM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJI22BI#issuecomment-575778053>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADOXE2VOPKTIFAQKB7JHDQ6IGO7ANCNFSM4KIBRJMQ> .

[sjakobi]

Hehe, that reminds me of https://xkcd.com/1172/. (No offense intended)

@timbertson Could you illustrate a bit what you're doing and how you want to manipulate these file sets in Dhall?

AFAIK, the idea of the to-directory-tree command was that you can use Dhall's record operators (/\, //, .{...}, etc.) to manipulate and combine directory trees / file sets.

[timbertson]

Sure. It may be that I can use /\ to do everything I need, it just may be fragile when multiple things come into play (recursive merges and overrides).

The basic idea is that I want to package up some common boilerplate, and reference it (via remote import) from many git repos. So (greatly) simplified I might have a Travis.dhall module which provides:

{
, `.travis.yaml` = "TODO"
, `script/lint.sh` = "TODO"
}

The script/lint.sh path would be referenced in .travis.yaml, so the user of this module shouldn't be responsible for choosing where that file goes.

Then say I've got a Docker.dhall module which provides:

{
, `Dockerfile` = "TODO"
, `script/compile.sh` = "TODO"
}

Again, the script path is used by the Dockerfile, so it should all be under control of this module.

To bring together the final set, I was doing ./Travis.dhall // ./Docker.dhall // { ... extra files as a record literal }

I believe I can use nested records and /\. But thinking ahead, I expect I might want some default provided files (for convenience) which some users may wish to override or opt out of. I could do that with, say:

{
, `Dockerfile` = "TODO"
, script = {
  , `compile.sh` = "TODO"
  , `optionalStuff.sh` = Some Text "TODO"
  }
}

Which then makes it quite difficult to combine and override:

let dockerStuff =
  let base = ./Docker.dhall
  in
  base // { script = base.script // { `optionalStuff.sh` = None Text } }

in
./Travis.dhall /\ dockerStuff /\ { ... extra files as a record literal }

So it all seems possible, but:

• Having to manually merge multiple levels with // to override is awkward
• Knowing when to use // vs /\ is not obvious (using /\ when you want to override will at least give you an error, but doing the opposite will result in silently omitting files in any overridden subdirectory (script/ in this case))

Flat records with directory-separated paths makes all this much simpler, because the data structure is flat. I can see how it's awkward to have two representations for the same thing though. So perhaps this is just another use case for dhall-lang/dhall-lang#448 or dhall-lang/dhall-lang#340

[sjakobi]

Thanks for the great explanation!

• Having to manually merge multiple levels with // to override is awkward

Indeed. Lenses should eventually make this easier but they're not here yet.

• Knowing when to use // vs /\ is not obvious (using /\ when you want to override will at least give you an error, but doing the opposite will result in silently omitting files in any overridden subdirectory (script/ in this case))

Fair point. IMHO it's generally good to start with /\, and to use // only when fields must be overridden. You could also give record completions a try.

I can see how it's awkward to have two representations for the same thing though.

Yes, I think that would make things more complicate down the road.

I think you should give nested records a try for now. If that turns out to be too inergonomic or cause other problems, please report back, and we'll try to find a solution.

[timbertson]

Yeah, I'll give that a try. I hopefully won't actually need overrides much or at all in the end anyway. Thanks for the discussion. Should I leave this open (and/or rename) to track "disallow keys with a slash in them"?

…

On Sat, 18 Jan 2020 at 20:35, Simon Jakobi ***@***.***> wrote: Thanks for the great explanation! - Having to manually merge multiple levels with // to override is awkward Indeed. Lenses <dhall-lang/dhall-lang#754 (comment)> should eventually make this easier but they're not here yet. - Knowing when to use // vs /\ is not obvious (using /\ when you want to override will at least give you an error, but doing the opposite will result in silently omitting files in any overridden subdirectory ( script/ in this case)) Fair point. IMHO it's generally good to start with /\, and to use // only when fields must be overridden. You could also give record completions <http://hackage.haskell.org/package/dhall-1.29.0/docs/Dhall-Tutorial.html#g:12> a try. I can see how it's awkward to have two representations for the same thing though. Yes, I think that would make things more complicate down the road. I think you should give nested records a try for now. If that turns out to be too inergonomic or cause other problems, please report back, and we'll try to find a solution. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1633?email_source=notifications&email_token=AAADOXHDTAIDP4FNQZHFMRLQ6LENRA5CNFSM4KIBRJM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJJUMAY#issuecomment-575882755>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADOXCWO2J4TIDA63LAP3TQ6LENRANCNFSM4KIBRJMQ> .

[Gabriella439]

@timbertson: For what it's worth, I think we might be able to support deep overrides after all and I have a comment related to that here: dhall-lang/dhall-lang#754 (comment)

[bfrk]

@timbertson wrote:

While I'm at it, my initial assumption was that to-directory-tree would take a map, instead of a record literal. That seems like it could be more flexible (because the keys can be programmatically generated too), and would also remove the need to support Optional Text (you could just filter out the unwanted items from the map). Should I make a separate issue for that, or does it intentionally not support maps?

+1 from me. I was just going to open an issue to request this feature.

[sjakobi]

@timbertson, @bfrk By "map" you mean this type, right?

List { mapKey : Text, mapValue : Text }

[bfrk]

Yup. Also known as https://prelude.dhall-lang.org/Map/Type

[harris-chris]

This is a very useful feature, but I seem to be hitting a bit of a limitation:

• if you're passing in a record, the keys are set at the type level, so you can't have an arbitrary or programmatically defined number of files/directories at any given level of your structure
• but if you're passing in a fileMap, then every item in the list of the fileMap has to have the same type, therefore every branch of the directory structure has to have the same depth
  I have a function which would ideally generate a config folder which looks a bit like this:

config
|
| -> sub-config-1
|         |
|         | -> _x many files_
|
| -> sub-config-2
          | 
          | -> sub-config-2-1
                    |
                    | -> _y many files_

where x and y vary across different calls of the function. I don't think this is possible from a single dhall to-directory-tree call because of the limitation above. Up until now I've been calling the function multiple times but this is quite slow (it takes a number of minutes to generate all the config).