adopt XDG basedir spec

[bos]

(Imported from Trac #688, reported by guest on 2010-05-12)

I want to be able to tell cabal to read the configuration from a different place than $HOME/.cabal/config.

The ultimate goal is to get rid of the non-standard .cabal directory altogether, so far careful --package-db/--prefix etc. juggling served me well; config is the last remaining file there.

> cabal --version
cabal-install version 0.8.2
using version 1.8.0.2 of the Cabal library

[bos]

(Imported comment by @dcoutts on 2010-05-12)

I'm not quite sure why you say it is non-standard. It's the standard unix place to put per-user application files. Where would you consider to be a standard location?

You can use:

cabal --config-file=... [command] [flags]

So it should work to use a shell alias, something like

alias cabal="cabal --config-file=..."

So really the bug here is not that it's hard coded (since it's not) but that these global flags are not easily discoverable. We don't list them all in the global --help so as not to clutter things, but then we need a way to let people know that there are extra global options and provide a way to get the help for them via some extended --help thing.

So let us know about:

• what you think the standard location should be
• if that shell alias (or whatever equivalent you choose to use) works ok

and we should open another ticket on providing extended help for the global flags.

[bos]

(Imported comment by guest on 2010-05-13)

Sorry for neglecting to answer right away, I forgot about this bug for a few days because I didn't bother to jump through the hoops to be able to receive change notification.

If by standard you mean de facto just everyone's been doing it like that, then I agree. And since indeed everyone's been doing that, I end up with hundreds of dotdirs in my home directory. This sucks because their content is essentially unstructured. The problem is that I want to treat files differently according to their purpose; and there are not just application files. There are also configuration files, they are put under version control. There are cache files, they are excluded from backup and indexing.

You know that a software package in Linux is not just dumped into a single directory somewhere, but the files are sorted into directories which each have a certain meaning. FHS governs the semantics for the system layout, FDO's XDG base directory specification governs this for a user's directory. These are real standards, and they enable all sorts of interoperability that would not be possible with unstructured directory contents.

Whether you want to adopt the spec in order to sort files into their appropriate XDG location by default is up to you. Thanks to your hint about --config-file I was able to move away the last file and delete .cabal for good, so certainly a compliant layout is already possible (through careful configuration, as I said before) if a user wants it.

[bos]

(Imported comment by @dcoutts on 2010-05-17)

I thought you might say the XDG spec. Yes I realise it's a "real" standard as opposed to a de facto standard. I must admit I'm slightly sceptical about the attempt to redefine where apps keep their per-user info. So far, very few apps seem to follow this new spec (mainly a few GNOME desktop bits). I will be surprised if they ever manage to get more than the two main desktops to move their dot files.

That said, if the world is going to move that direction then we don't want to get in the way. We should make it easier for users or distros to change where the per-user files go by default.

Thanks for confirming it is at least possible (if not convenient).

[bos]

(Imported comment by guest on 2010-05-17)

we should open another ticket on providing extended help for the global flags

This is now tracked in #690.

[bos]

(Imported comment by @dcoutts on 2010-05-17)

Hmm, I notice the XDG base directory specification does not say anything about program executable files or library files, though the default naming scheme sort-of suggests /.local/bin and /.local/lib.

[bos]

(Imported comment by Syzygies on 2010-05-17)

I found this ticket while researching precedents for a related question.

For various reasons I like to build multiple versions of GHC side by side: Migration between major releases, testing candidate releases. For torture-testing new hardware, building many copies of GHC at once is better than e.g. the usual prime testing. I once literally caused a server power supply to smoke this way; my friend's hardware vendor was in over their head. They took my GHC test in-house to minimize delivery cycles.

A related piece of de facto convention comes up here: It is crazy to sprinkle pieces of a system like GHC into separate global locations like /usr/local/bin, /usr/local/share, unless one will only ever use one version, with no desire to look under the hood or move the installation. In other words, I have no issue with Linux or OS X shipping this way, but major additions should be localized by default. Some are, e.g. TeX. GHC should.

Fortunately, to avoid Siamese twin installations with accidentally shared parts, one can use the ./configure --prefix option while building GHC. Then, it becomes apparent that "cabal install" is the remaining holdout to ideal version localization. There is a perfect storm of tiny factors keeping Cabal from gracefully localizing. I wanted to elicit advice before contributing a fix; this ticket seems to have touched on a key issue.

I did not know about the "--config-file=..." command-line option; I will start using it in scripts. However, we need a better default for global installs.

In /usr/local I now use a symbolic link from ghc to one of the directories ghc-6.10.4, ghc-6.12.1, ghc-6.12.2, ghc-6.12.3-rc1. It then appears that I built GHC using "--prefix=/usr/local/ghc". My ~/.cabal/config file includes lines like "remote-repo-cache: /usr/local/ghc/packages". This works, but I've had amusing debates with unix experts on why my symbolic link is a terrible solution. Basically, when I change this link I pull the rug out from under any ongoing process that wants to see a consistent environment.

Environment variables are one "right" way to handle this. In short, a symbolic link is like changing state, while environment variables are like functional programming.

However, not only does .cabal/config fail to allow external environment variables, it doesn't allow forms like one sees in its own comments. (Who reads documentation? We all read sample code.) For example, even though I see the comment "-- bindir: $prefix/bin" I cannot write "remote-repo-cache: $prefix/packages". I would prefer to be able to write anything that would evaluate correctly in a shell to an absolute path, e.g. which ghc or perl -pe ... and so forth.

Ideally, a global install would by default use a global location for the Cabal config file, but we slam back into the original problem: By default the global install of GHC did not use the --prefix option, so there is no well-defined location parametrized by the version of GHC. On the other hand, this isn't so bad, e.g. put everything in ../share/hackage.haskell.org relative to the currently visible ghc. Saves bandwidth caching packages; if someone hasn't bothered to build ghc using --prefix, they won't mind the Siamese twin effects with Cabal either.

The simplest change that would help my issue would be to allow both absolute and relative paths in .cabal/install, with a relative path relative to the location of which ghc. This would "just work" in simple cases, and allow competing processes to coexist, that chose different versions of ghc via a custom $PATH alone, with no other flags needed. If this change was accepted, I wouldn't mind leaving my config file in ~/.cabal. That location is only mildly annoying; as has been noted, we've already lost the war on keeping that area uncluttered.

However, it just shouldn't matter which authorized user works on a global install. Relying on ~/.cabal/config for a global install should be viewed as a bug, as it prevents several admins from cooperating in consistently maintaining GHC on a shared system.

[bos]

(Imported comment by @dcoutts on 2010-05-28)

Replying to Syzygies:

A related piece of de facto convention comes up here: It is crazy to sprinkle pieces of a system like GHC into separate global locations like /usr/local/bin, /usr/local/share, unless one will only ever use one version, with no desire to look under the hood or move the installation. In other words, I have no issue with Linux or OS X shipping this way, but major additions should be localized by default. Some are, e.g. TeX. GHC should.

You can already do this if you want to. You just set the --prefix to be what you want, e.g. instead of using /usr/local as the prefix, use /opt/ghc/6.12.1 or whatever, then you'll get paths like /opt/ghc/6.12.1/bin, /opt/ghc/6.12.1/lib, /opt/ghc/6.12.1/share etc etc.

Fortunately, to avoid Siamese twin installations with accidentally shared parts, one can use the ./configure --prefix option while building GHC. Then, it becomes apparent that "cabal install" is the remaining holdout to ideal version localization. There is a perfect storm of tiny factors keeping Cabal from gracefully localizing. I wanted to elicit advice before contributing a fix; this ticket seems to have touched on a key issue.

I did not know about the "--config-file=..." command-line option; I will start using it in scripts. However, we need a better default for global installs.

The "default" default, should follow global conventions, meaning using /usr/local. Again you can override the prefix and all other install directories.

It's true that there is not currently a sensible way to have a global cabal config file, in the style of /etc/cabal or something like that. Following the XDG spec in that regard might make some sense.

Environment variables are one "right" way to handle this. In short, a symbolic link is like changing state, while environment variables are like functional programming.

You mean you would like an environment variable to set the location of the Cabal config file? You can set the $CABAL_CONFIG, it acts like setting the "--config-file=..." option.

However, not only does .cabal/config fail to allow external environment variables, it doesn't allow forms like one sees in its own comments. (Who reads documentation? We all read sample code.) For example, even though I see the comment "-- bindir: $prefix/bin" I cannot write "remote-repo-cache: $prefix/packages". I would prefer to be able to write anything that would evaluate correctly in a shell to an absolute path, e.g. which ghc or perl -pe ... and so forth.

The prefix is something that makes sense in the context of an installation. It is not clear that basing the cache location on that makes sense. It means for example that it will not find the default cache if you override the --prefix on the command line. Perhaps what you really want to do is to specify the default prefix and cache dirs in the config file based on some other variable, rather than having the cache dir follow the prefix.

Ideally, a global install would by default use a global location for the Cabal config file,

Do you mean an install run as root, or do you mean an install run as a user? Or do you mean that you'd like to be able to have a shared/global set of connfig defaults that can be overridden in per-user config files?

As a user when you run cabal install --global --su-cmd=sudo you're using your own config, but installing to a global location (the exact prefix etc is set in the config file). It does the build as user and installs as root.

but we slam back into the original problem: By default the global install of GHC did not use the --prefix option, so there is no well-defined location parametrized by the version of GHC.

You can either use versioned ghc binaries, (called ghc-6.10.4, ghc-6.12.1 etc) in a shared prefix like /usr/local or as I described above you can use separate prefixes for different ghc installations. Note that by default, cabal installs libs in dirs including the ghc version, so it all works. As mentioned before you have complete control over the prefix (and other detailed layout of bindir etc) so you can put files where you like, according to some ghc version scheme. You can use the compiler id in the cabal prefix for example.

The simplest change that would help my issue would be to allow both absolute and relative paths in .cabal/install, with a relative path relative to the location of which ghc.

Honestly I don't think that is sensible. Allowing people to use more variables so that you can specify a prefix that happens to correspond to your ghc installation is one thing, making that the meaning for relative paths is rather another. Usually you do not want to install things into the ghc tree, so using relative paths would encourage that dangerous behaviour.

However, it just shouldn't matter which authorized user works on a global install. Relying on ~/.cabal/config for a global install should be viewed as a bug, as it prevents several admins from cooperating in consistently maintaining GHC on a shared system.

I don't understand this bit. Perhaps you can explain what config you would like to use for global installs. Where would it live? Do you assume global installs are only performed by root?

I think it would help if you could explain what you're actually trying to achieve. The mailing list might be the better place for that discussion.

[bos]

(Imported comment by @kosmikus on 2010-05-28)

Moving to 0.16 milestone so that we look at it again sometime soon, but I think this is a candidate for | (meaning that without external concrete patches or suggestions, we probably won't do it).

[eheintzmann]

What is planned now about FreeDesktop XDG basedir specification for cabal?

http://ploum.net/post/207-modify-your-application-to-use-xdg-folders
https://live.gnome.org/GnomeGoals/XDGConfigFolders
http://standards.freedesktop.org/basedir-spec/latest/

[eheintzmann]

According to XDG Base directory specification, cabal should not have its own folder (.cabal) anymore
User data should go into $XDG_DATA_HOME (which default to ~/.local/share),
user preferences should go into $XDG_CONFIG_HOME (which default to ~/.config)
and cached data should go to $XDG_CACHE_HOME (which default to ~/.cache).
More details at :
http://ploum.net/post/207-modify-your-application-to-use-xdg-folders
https://live.gnome.org/GnomeGoals/XDGConfigFolders

Full specification can be found at:
http://standards.freedesktop.org/basedir-spec/latest/

The Freedesktop.org XDG base directory specification have good de facto adoption.
It has been adopted by:

• GNOME ( https://live.gnome.org/GnomeGoals/XDGConfigFolders )
• GTK+ ( https://bugzilla.gnome.org/show_bug.cgi?id=646631 )
• KDE ( http://techbase.kde.org/KDE_System_Administration/XDG_Filesystem_Hierarchy#Freedesktop.org_and_Standard_Directories )
• QT ( http://harmattan-dev.nokia.com/docs/library/html/qt4/qsettings.html#setPath )
• XFCE ( http://docs.xfce.org/xfce/xfce4-session/advanced in Files and Environment Variables )
• LXDE
• Razor-qt
• VLC ( https://trac.videolan.org/vlc/ticket/1267 )
• GStreamer ( https://bugzilla.gnome.org/show_bug.cgi?id=518597 )
• Chrome ( http://code.google.com/p/chromium/issues/detail?id=16976 )
• many more upstream applications
• Ubuntu ( http://brainstorm.ubuntu.com/idea/6557/ & http://packages.ubuntu.com/fr/source/precise/libxdg-basedir )
• Debian ( http://packages.debian.org/squeeze/libxdg-basedir1 )
• Red Hat
• Fedora
• Suse
• many more distributions

I think that cabal should use same locations than the vast majority of Desktop environment and applications.

There are real advantages of following this specification :

• a lot less cluttered $HOME
• Make backups a lot more safer and easier.
  Backuping your $XDG_DATA_HOME along with your files is enough
  (or just excluding $XDG_CACHE_HOME)
• A lot easier to reset a default configuration if you want/need it (and
  without any risk to loose informations). Even for the software itself
  could choose to reset $XDG_CONFIG_HOME if needed.
• Avoid some strange bugs that happens because you had a old version of
  some configuration file
• A lot more of flexibility and portability because no path are hardcoded.

[UnkindPartition]

In general, I tend to agree with these advantages.

However, the XDG spec seems to be designed with typical desktop application in mind, not compilers or package managers. As @dcoutts said above, it doesn't specify where the binaries, libraries, documentation should go. Are there any examples of programming language implementations that use this scheme?

Also, should we try to follow the same spec on Windows as well? (I guess there are no technical reasons not to do so.)

[eheintzmann]

Users binaries, documentation and libraries are users data and thus should go to $XDG_DATA_HOME (which default to ~/.local/share),

"Windows have the Local/Roaming split which could be roughly equated to putting everything in either $XDG_DATA_HOME or $XDG_CACHE_HOME like most Humble Bundle Linux game ports do."

[UnkindPartition]

Users binaries, documentation and libraries are users data and thus should go to $XDG_DATA_HOME (which default to ~/.local/share),

Yes, that's exactly what I meant by the desktop application oriented mindset.

Putting binaries, libraries and docs in one directory is a downgrade from the scheme that we have now.

Or would it be $XDG_DATA_HOME/bin etc.? Then the full path is ~/.local/share/bin, which looks weird and complicated.

[eheintzmann]

I think it is better to avoid users specific binaries, libraries and doc and share these files between all users.

[UnkindPartition]

Whether it is better or not is out of the scope of this issue, I believe.

[eheintzmann]

There is no $XDG_DATA_HOME/bin etc but you are free to use $XDG_DATA_HOME/cabal/bin etc

[UnkindPartition]

So that would be ~/.local/share/cabal/bin, which looks rather disappointing to anyone who ever cares about what these paths are.

I think at this point it would be more realistic for someone who cares to create a simple cabal-install wrapper that would conform to XDG. Then release it and see if it gains popularity. If it does, it would be a convincing argument to switch to that scheme by default.

[nagisa]

Instead of conforming to XDG by default I'd be happy with cabal-install reading an environment variable telling cabal where its config is. Environment variables are much more persistent than some kind of flag+alias for that. Then you can configure all other directories through config anyway.

EDIT Ignore me, reading an issue I found $CABAL_CONFIG. I'd be great if it was documented though.
EDIT2 It'd be great if cabal config evaluated external environment variables as well.

[dag]

Personally I think cabal-install should follow XDG here which implies ${XDG_CONFIG_HOME:-$HOME/.config}/cabal for {config,world} and ${XDG_DATA_HOME:-$HOME/.local/share}/cabal for {bin,lib,logs,share} and ${XDG_CACHE_HOME:-$HOME/.cache}/cabal for {packages,setup-exe-cache}. It can then, separately, symlink binaries to a configurable location defaulting to $HOME/.local/bin (no XDG), which isn't in any official standard I'm aware of but is used by Python (like PREFIX=$HOME/.local) and is on the user $PATH by default in at least Fedora.

I have no idea what those paths should map to on Windows systems but presumably that's a solved problem.

[Earnestly]

You realise that you guys can actually change the XDG_* environment variables? XDG_DATA_HOME doesn't have to be HOME/.local/share. We'll get there eventually, compilers are one of the few remaining bastions supporting the UNIX v2 bug that is "dotfiles"

[rnhmjoj]

@Earnestly Exactly. That is the default but can be changed as you like. So, @nagisa, $CABAL_CONFIGwould be the exact same thing as the XDG variables.

[Earnestly]

Another problem with using a unique environment such as CABAL_CONFIG is we
add to our already bloated environs. For reference here are a few:

export SLRNHOME=$XDG_CONFIG_HOME/slrn
export _RR_TRACE_DIR=$XDG_DATA_HOME/rr
export CARGO_HOME=$XDG_DATA_HOME/cargo
export GNUPGHOME=$XDG_CONFIG_HOME/gnupg
export INPUTRC=$XDG_CONFIG_HOME/inputrc
export RLWRAP_HOME=$XDG_DATA_HOME/rlwrap
export LYNX_CFG_PATH=$XDG_CONFIG_HOME/lynx
export DVDCSS_CACHE =$XDG_CACHE_HOME/dvdcss
export TIGRC_USER=$XDG_CONFIG_HOME/tig/tigrc
export WEECHAT_HOME=$XDG_CONFIG_HOME/weechat
export PERL_CPANM_HOME=$XDG_CACHE_HOME/cpanm
export GIMP2_DIRECTORY=$XDG_CONFIG_HOME/gimp
export ELINKS_CONFDIR=$XDG_CONFIG_HOME/elinks
export CABAL_CONFIG=$XDG_DATA_HOME/cabal/config
export HTTPIE_CONFIG_DIR=$XDG_CONFIG_HOME/httpie
export LESSHISTFILE=$XDG_CACHE_HOME/less/history
export XAUTHORITY=$XDG_RUNTIME_DIR/x11/xauthority
export XCOMPOSEFILE=$XDG_CONFIG_HOME/x11/xcompose
export GUILE_HISTORY=$XDG_CONFIG_HOME/guile/history
export VIMPERATOR_RUNTIME=$XDG_CONFIG_HOME/vimperator
export PENTADACTYL_RUNTIME=$XDG_CONFIG_HOME/pentadactyl
export NOTMUCH_CONFIG=$XDG_CONFIG_HOME/notmuch/notmuchrc
export GTK2_RC_FILES=$XDG_CONFIG_HOME/gtk-2.0/settings.ini
export TERMINFO_DIRS=$XDG_DATA_HOME/terminfo:/usr/share/terminfo
export UNCRUSTIFY_CONFIG=$XDG_CONFIG_HOME/uncrustify/uncrustify.cfg

These are just the ones I use, there are much more.
Clearly this is not sustainable, each export adds to the total memory
consumption of -every- child process on the machine.

Here are some more https://wiki.archlinux.org/index.php/XDG_Base_Directory_support#Partial

[rkraneis]

So no improvements here? Another reason for adopting the XDG scheme is that (some) backup tools know about these and already exclude e.g. .cache from backups. I'm now hunting around to see which of its 3GiB cabal actually needs :-(

[hvr]

@rkraneis I think this mostly needs somebody to take the lead: design a reasonable migration/fallback scheme, and implement it.

[athas]

I submitted a patch to directory for adding support for XDG_STATE_HOME. We'll see how that goes. The draft cabal design as I understand it is then:

• The Cabal configuration file goes in $XDG_CONFIG_HOME/cabal/config.
• cabal install binaries go in $HOME/.local/bin (from the systemd file-hierarchy spec).
• Packages and logs (the current ~/.cabal/packages and ~/.cabal/logs) go in XDG_CACHE_HOME/cabal.
• Everything else (just ~/.cabal/store?) goes in $XDG_STATE_HOME/cabal

People with more Cabal experience can probably judge whether this is a sensible design. There's still two questions to answer:

1. What about Windows? The above will work on Windows because directory supplies fascimiles of the XDG paths, but I'm not sure it's actually what users would want.

2. A migration path for people who have already set CABAL_DIR or CABAL_CONFIG. I think these variables should just take precedence over XDG, if set.

[Mikolaj]

The CABAL_DIR is being discussed at #7455 and I think we should stick to that or discuss there.

[hasufell]

cabal install binaries go in $HOME/.local/bin (from the systemd file-hierarchy spec).

$HOME/.local/bin is accepted in XDG spec: https://gitlab.freedesktop.org/xdg/xdg-specs/-/merge_requests/38

Note that stack also installs into this directory... I'm generally not a fan of this directory anymore. It has caused problems for ghcup as well to the point that I'd like to remove XDG support from it. As an example: if you'd enable XDG for both cabal and ghcup and then run cabal install --overwrite-policy=always --install-method=copy cabal-install, then you'll cause confusion for ghcup by overwriting ~/.local/bin/cabal.

But I'm sure many users already set this as their default installation directory, so yeah. But the (weak) guarantee that only a single tool installs into ~/.whatnot/bin is a useful one.

A migration path for people who have already set CABAL_DIR or CABAL_CONFIG. I think these variables should just take precedence over XDG, if set.

Yes. ghcup on windows sets CABAL_DIR to a short path to avoid MAX_PATH issues. Following the XDG emulations will likely cause problems on windows.

[Mikolaj]

@hasufell: thank you for the remarks. That's very valuable. Let's discuss how to avoid the problems.

There are at least two partially independent proposals here. One is XDG, the other is using the config file instead of environment variables (except CABAL_CONFIG).

[athas]

I think using $HOME/.local/bin is preferable, even though it runs the risk of conflicts. Conflicts will always happen when you have a shared resource, but the Unix/Linux world is for better or for worse standardising on this directory. I don't think it's to the advantage of cabal to be different.

(I would also suggest cabal install adopt common conventions from other tools, namely copying and overwriting by default, but that's unrelated to XDG).

[hasufell]

I think using $HOME/.local/bin is preferable, even though it runs the risk of conflicts. Conflicts will always happen when you have a shared resource, but the Unix/Linux world is for better or for worse standardising on this directory. I don't think it's to the advantage of cabal to be different.

I'm not sure. All of the other XDG directories are usually used in a way where programs create a program specific subdirectory, exactly to avoid conflicts.

~/.local/bin is the only one that breaks this.

Imagine all apps installing their data files into the same dir without subdirectories. Not a great idea.

The only reason this works for /usr/bin is because there's only one program installing into it: your package manager.

[athas]

My argument is not that $HOME/.local/bin is a good design. My argument is that it's the de facto standard. It is used by build tools that are held in high regard by their respective user communities (e.g. cargo, stack), and complaints are few. Is this a fight that cabal cares about?

[hasufell]

My argument is not that $HOME/.local/bin is a good design. My argument is that it's the de facto standard.

Supporting the XDG standard isn't hard. Making it the default is a different thing.

[Mikolaj]

#7819 is an argument for making XDG the default, isn't it?

[Mikolaj]

@phadej explained in #7819 that the relation is not so direct, because we need to support both XDG (>= 9.2) and non-XDG ghcs, so GHC XDG and overall XDG needs to be separated anyway.

[ulysses4ever]

@athas are you still interested in working on this?

[athas]

Yes, but I don't have the time in the near term. It's not difficult to implement, but I got demotivated by having to untangle dependencies in the test suites.

[Mikolaj]

No rush. Please do share the dependencies troubles next time you encounter them.

[athas]

The implementation of this is in #7386 and is pretty much ready to be merged. There's still room to litigate the specific XDG directories though.

I don't have any real issue with the current design (summarised in #7386 and its updated documentation, and also above in this thread), but I should note that the use of XDG_STATE_HOME to keep the Cabal store does seem a bit unusual. XDG_STATE_HOME is not used much by anything (likely because it is so new), and I've seen remarks that it's mostly intended for transient and lightweight state like "current state of the application that can be reused on a restart (view, layout, open files, undo history, …)". In particular, nothing seems to store large files there (which Cabal will certainly do). The classic alternative would be XDG_DATA_HOME (~/.local/share) which is heavily used for all kinds of random data by many programs (e.g. Evolution seems to store email and calendar info there, and all of Steam lives there as well), but it can be argued that this was never supposed to be the point of that directory (it was intended to be more like /usr/share, i.e. mostly read-only data files). This isn't necessarily a problem, and will only be so in the exotic case where a user decides to put XDG_STATE_HOME on a separate file system with very little space, but I feel it should be recorded for posterity.

[athas]

Oh, and another thing: the change in #7386 affects all operating systems, including Windows. I don't think any Windows users have weighed in on whether this is appropriate. directory implements the XDG directories on Windows as best it can, but I have no idea whether what it does is actually useful. It might be a better design to keep using a single unified Cabal directory on Windows - I really don't know.

[athas]

Implemented in #7386.