Migrate from the .cabal format to a widely supported format

[Profpatsch]

In the wake of the exact-printer initiative, I proposed another approach: why not say good bye to the .cabal file format and switch to something that is widely supported.

There is a few alternatives, the most important attribute would be that they are widely supported in industry.

• JSON
• YAML
• TOML

Note that all of these are (mostly) isomorphic to JSON (scalars, lists, dicts), which is important for easy translation between them (e.g. for config generation purposes).

What would this give the Haskell ecosystem?

• Editor support: every modern editor like vscode has a way of assigning JSON schema to a file, which gives completion and inline documentation for free everywhere
• Also, syntax highlighting and auto formatting come for free
• Cabal doesn’t need to implement its own parser. If JSON is chosen, the parser is even context-free.

What would it give to users?

• Instant familiarity with the format: you don’t touch cabal files all too often as a user, so you don’t want to learn yet another syntax
• Templating cabal config with standard tooling (e.g. jq, yj), which is important e.g. in a monorepo context
• Inline documentation without setup

What are others doing?

Most modern package managers that don’t go the full turing-complete configuration route (e.g. Scala’s sbt, Erlang) usually converge their config on a widely supported syntax.

Examples:

• npm, yarn (package.json, package-lock.json)
• Stack (stack.yaml, stack.yaml.lock)
• hpack (project.yaml)
• cargo (Cargo.toml)
• Elm (elm-package.json)
• Maven (pom.xml)
• poetry (pyproject.toml, poetry.lock)

Counterexamples:

• go (go.mod), though flat shasums and go packages have no configuration file
• pip (requirements.txt), though see poetry above
• sbt, hex: both use their turing complete parent languages
• leiningen (project.clj), clojure is a lisp, and sexps are already a data format

---

I don’t expect cabal would drop support for the cabal file format very soon, rather it would start out by generating a .cabal file from the .json/.toml/.yaml for consumption by older version of cabal. Then after a multi-year grace period, the new format would become the standard and projects could drop their autogenerated .cabal files.

[Profpatsch]

Note that some people have mentioned dhall as a possible alternative, but using it would destroy most benefits, namely:

• Familiarity
• Editor support
• widely available tooling for ops integration
• simple parser
• isomorphic to json

[Profpatsch]

However, I expect that there would be a dhall library for generating cabal.json files, which can aid into integrating dhall-based (dev)ops setups with Haskell packages.

[philderbeast]

Is this cabal.json the plan.json from the cabal docs or is it a .cabal in JSON format?

plan.json (JSON)
A JSON serialization of the computed install plan intended for integrating cabal with external tooling. The cabal-plan package provides a library for parsing plan.json files into a Haskell data structure as well as an example tool showing possible applications.

[ocharles]

Note that some people have mentioned dhall as a possible alternative

Also note that this already exists as dhall-to-cabal. While its goal was to generate .cabal files, that's not the only solution. A more integrated solution would be a cabal-install that can actually consume these files. I'm not saying this is the solution, just mentioning this as prior art. I'll step out of the conversation for now and let others share there thoughts, but if any one wants to talk about Dhall in particular here, I do have thoughts

[Profpatsch]

Is this cabal.json the plan.json from the cabal docs or is it a .cabal in JSON format?

It is the current .cabal file in a not-home-grown syntax

[emilypi]

I proposed another approach: why not say good bye to the .cabal file format and switch to something that is widely supported.

When I think about it, don't think these are mutually exclusive tickets: an exact printer gets us a reasonable source representation. This is good for a few reasons: we can derive translational tools from the same representation - we only need change the parser and the printer. This frees up efforts to migrate between formats!

There is a few alternatives, the most important attribute would be that they are widely supported in industry.

Of the three suggestions, TOML is the most attractive. YAML is has too much variable syntax, and JSON is aesthetically (and mechanically) displeasing for me to write as a human. TOML's grammar is minimal and admits a small and easy to generate + verify parser and lexer (note: toml-parser is a little outdated) that eliminates the need for us to write it from scratch. In fact, maintaining this would be a dream, and would be a boon for tooling, since we can derive an ABNF for our specific flavor quite easily.

Then after a multi-year grace period, the new format would become the standard and projects could drop their autogenerated .cabal files.

👍

[TikhonJelvis]

As an experiment, I took a library I wrote a few years ago and manually converted its Cabal file to TOML. I like it! The conversion can be totally systematic.

TOML was noticeably nicer to edit—Emacs has a simple built-in TOML mode and I didn't have to worry about indentation/formatting. (I've done Haskell for over a decade now and I'm still not consistent in how I format Cabal files!) Structured commands for navigating and editing the TOML file would be nice; I don't know if something like this already exists, but if it doesn't, adding it to Emacs would be easy. I wouldn't even think of trying something like that for Cabal's custom syntax.

I've used YAML a lot more than TOML in the past. Compared to YAML, I found needing to quote all my strings a bit annoying; on the other hand, TOML was much nicer to pick up and doesn't have weird corner cases to worry about. At work I recently ran into some weird YAML files that used anchors in a way that didn't work in Python—not something that would happen with TOML.

In my dream world we would use an S-expression based syntax (like sexplib) but I know that is not to be :(.

I immediately found that multiline strings were useful. Multiline strings and comments seems like the bare minimum for a human-oriented format; YAML and TOML support that, JSON doesn't.

It's a bit long, but here's the whole file:

cabal-version = "2.2"

[package]
name = "modular-arithmetic"
version = "2.0.0.1"
synopsis = "A type for integers modulo some constant."
description = """
A convenient type for working with integers module some constant. It saves you from manually wrapping numeric operations all over the place and prevents a range of simple mistakes. @Integer `Mod` 7@ is the type of integers (mod 7) backed by @Integer@.

We also have some cute syntax for these types like @ℤ/7@ for integers modulo 7.
"""
homepage = "https://github.com/TikhonJelvis/modular-arithmetic"
bug-reports = "https://github.com/TikhonJelvis/modular-arithmetic/issues"
license = "BSD-3-Clause"
license-file = "LICENSE"
author = "Tikhon Jelvis <tikhon@jelv.is>"
maintainer = "Tikhon Jelvis <tikhon@jelv.is>"
category = "Math"
build-type = "Simple"
extra-source-files = ["README.md", "CHANGELOG.md"]

[source-repository.head]
type = "git"
location = "git://github.com/TikhonJelvis/modular-arithmetic.git"

[library]
hs-source-dirs = ["src"]
ghc-options = ["-Wall"]
default-language = "Haskell2010"
exposed-modules = [
  "Data.Modular"
]
build-depends = [
  "base >4.9 && <5",
  "typelits-witnesses <0.5"
]

[test-suite.examples]
hs-source-dirs = ["test-suite", "src"]
main-is = "DocTest.hs"
default-language = "Haskell2020"
type = "exitcode-stdio-1.0"
build-depends = [
  "base >4.9 && <5",
  "doctest >= 0.9",
  "typelits-witnesses <0.5"
]

[TikhonJelvis]

Another benefit: the format would be naturally extensible. Cabal could provide a section for plugin/tool/etc config, and tools would have no issues parsing values from there. I'm imagining something like this:

[plugin.liquid-haskell]
smt-solver = "z3mem"

My experience has been that providing "extension points" in formats is always useful. We can't figure out everything people want to do with their libraries ahead of time but we can make the format adaptable. If people need something Cabal doesn't support, they can add it while still keeping a single canonical file for library-specific settings.

[gbaz]

For yaml there's also of course hpack. So anyone who wants to write cabal files in either yaml or dhall is welcome to do so. Note that we don't have exactprinters for either of those formats either, as far as I know. As I recall, due to the semantics of yaml, conditional clauses are rather unpleasant there, among a few other issues (and pretty-printing reorders things in unpleasant ways as well). (And also as emily notes, the yaml grammar is rather complicated as is).

Toml does seem promising, but I worry that its support for conditionals or other more complex syntax wouldn't be particularly great either. Translations of some more complex files might be worthwhile, to experiment with this.

Btw, note that cabal is already extensible, via "x-" fields.

In any case, I think the right next step is to get the cabal grammar pinned down and to have an exactprinter for at least the format we already have and is widespread.

[philderbeast]

This is good for a few reasons: we can derive translational tools from the same representation - we only need change the parser and the printer. This frees up efforts to migrate between formats!

I like this and am the maintainer of the translational tool hpack-dhall that can translate:

dhall -> cabal
dhall -> json
dhall -> yaml # the package.yaml format of hpack
dhall -> dhall # with imports resolved

I am a bit wary about each format being capable of doing a faithful representation. For instance, hpack's conditionals can break dhall's typing. This is the trouble @gbaz just mentioned.

[Ericson2314]

I want this to work, but unfortunately I see some issues with all the proposed formats so far. I think TOML / YAML / JSON will never work, but Dhall, while it might not be a good fit today, can be made to work.

TOML / YAML / JSON

If Cabal files were merely data they would be fine, but unfortunately they are code, due to the conditionals, and parameters used in those conditions. This is true of Cargo packages too, and the solution there has been to stuff syntax into strings. Firstly, this largely defeats the point as we still need application-specific parsers (and pretty printers!) to handle those strings.

But more worryingly, I have reason to believe this has warped the design process of Cargo. See, for example, the back and for with @djc and me in rust-lang/rfcs#3143, where @djc agrees Cargo has backed itself into a corner, but objects to my further using strings, or trying to encode the information in a more structured but awkward and verbose way. I may have disagreed with @djc on which unpleasant choice too take, but I absolutely do agree that TOML forcing Cargo into this awkward situation is trajic, and no one shouldhave to pick between those unpleasant options in the first place.

Cabal converting from the existing design avoids some of the distortion from TOML's perverse incentives, but I have no doubt the language of Cabal files will continue to evolve, and I don't want "TOML goggles" to mess things up going forward.

Dhalll

Dhall is an actual programming language, and therefore squarely fixes the above issues. And to be clear I would really like to endorse Dhall as it is the right sort of way to make these things conform to a standard. There are two quibbles with Dhall as it currently exists however, that I think should be addressed first:

• imports / IO. As far as I know, Dhall always allows downloading arbitrary stuff, etc. as long as you give it a content address of some sort to make it pure (like fixed output derivations). This is a fine design in general, but I worry about e.g. cabal2nix needing internet access to do it's job, which (ironically giving the nix inspiration!) would be a regression and major pain. If there is a way force Dhall programs to be more self-contained, that would assuage my concern.

• abstract interpretation. The Dhall model somewhat assumes that dhall will evaluate a closed term, spltting out a value for the consuming application to deal with. But this doesn't totally reflect how Cabal works. Today, we have automatic flags, which means we need to vary parameters based on results. With a few flags this can be brute forced, but with more abstract interpretation is much more efficient. Dhall's strong normalization is good to make such static analysis tractable, but we might also want additional restrictions to make it efficient and also easier for humans to understand.

  Maybe that is overkill for manual flags, but @mpickering's and my presentation on what comes after CPP (https://icfp19.sigplan.org/details/hiw-2019-papers/9/Configuration-but-without-CPP, https://www.youtube.com/watch?v=YupkE1vsZ4o) has gotten me thinking about abstract interpretation more broadly. Eventually we want to tackle the goal of "type safe packaging" i.e. ensuring all valid version solving solutions will in fact compile. It's hard, but not tackling it is anathema to our values, and abstract interpretation of various sorts is key to making it work.

So yeah, in conclusion I want Dhall to work, but it's important we we be able to restrict ourselves to a sort of "mini Dhall" so we can do this analysis and we will have to integrate Dhall with Cabal fairly deeply. I'm not sure whether the current Dhall implementation supports such a restricted "mini Dhall", but that can easily be fixed.

[Bodigrim]

Then after a multi-year grace period, the new format would become the standard and projects could drop their autogenerated .cabal files.

Remember that Hackage is an append-only repository. It would be utterly disappointing if a future version of Cabal would be unable to build an old package just because it no longer parses its very own package format. So I don't think it would be wise to abandon a parser of Cabal files even after a very long grace period. And if we are to retain the parser and all its complexity, than what exactly are we to gain? What about other tooling (e. g., Stack)?

With regards to editor support, why aim for a generic JSON autocompletion? These days we should not settle for anything less than a domain-specific language server, and custom format is not a hindrance for it.

I'm sorry if my tone sounds harsh, but I'm afraid we are chasing an ideal to the detriment of compatibility, as it's very customary in Haskell community.