Store settings and packages in XDG directories instead of ~/.julia

[nalimilan]

Over the last 10 years most Freedesktop environments (GNOME, KDE, XFCE...) and applications around them have moved to store configuration in the XDG config directory (by default ~/.config) instead of creating a hidden directory directly into the home folder. A well-known non-GUI application that does the same is git:
https://github.com/git/git/blob/master/Documentation/RelNotes/1.7.12.txt#L18-23

The Freedesktop specification is here:
http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

and a rationale for this specification can be found here:
https://wiki.gnome.org/GnomeGoals/XDGConfigFolders

I think Julia should follow it too. The idea is very simple, you just need to read an environment variable:

$XDG_CONFIG_HOME defines the base directory relative to which user specific configuration files should be stored. If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used.
http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

Thus, in case $XDG_CONFIG_HOME is not defined, settings would be stored in the default ~/.config/julia.

Packages should likely go to $XDG_DATA_HOME, which defaults to .local/share/ (the equivalent of /usr/share/). One rule of thumb is that configuration is more like text files with limited size and that do not change that much (some people even keep it under a VCS). OTOH $XDG_DATA_HOME is usually conceived as a place to store valuable user data, which needs backup, while packages can easily be downloaded again. After much reading and thinking I still don't know so this probably means it does not really matter.

So the changes would be pretty limited actually. In the documentation, it would be sensible to speak about ~/.config/julia instead of ~/.julia, without mentioning the environment variables trick (users who change the defaults will know).

One of the potential gains of following that XDG spec is that it also defines $XDG_CONFIG_DIRS and $XDG_DATA_DIRS, which could be used by administrators to provide a system-wide package library, and default settings for new users. Julia would read config from there, but only save user config and packages in the user home directory.

[JeffBezanson]

This is a nice and thorough proposal.

[StefanKarpinski]

Yes, this is a very well-presented proposal. However, we're not going to do this. We only just got .julia in the same place on all systems (Windows was the outlier before). It would be crazy to start doing some different idiosyncratic thing on different systems. This standard doesn't even work on all Linuxes, let alone other UNIXes like OS X, FreeBSD and of course Windows. The only thing that is truly universal is putting a bunch of files in a directory in the user's home directory.

[nalimilan]

Thanks for the kind words. ;-)

I understand that changing the path to the config directory is not a nice task. That's why it took ten years and all programs haven't moved yet. But what I'm proposing merely amounts to changing the path from ~/.julia to ~/.config/julia. On 90% of systems it will be that way; only a few UNIX hackers will change the path to XDG_CONFIG_HOME, and they know what they are doing. There's no issue of the standard "working" or not: either the variable is defined, and you use it, or it's not, and you use your default path; in both cases you end up with the same result most of the time: ~/.config. This is my definition of "putting a bunch of files in a directory in the user's home directory". So I don't think this introduces a real inconsistency between OSes.

BTW, it's false that FreeBSD does not support this: see
http://www.freebsd.org/cgi/query-pr.cgi?pr=133825
I also find it unlikely that any Linux distribution fails to support this, as almost all free desktop environments have made the switch for some time. GLib for example provides functions that default to ~/.config if XDG_CONFIG_HOME is not set, and it's used by most GNOME and XFCE apps now.

Julia is still young, so it's not too late to switch. Big and popular cross-platform applications have made or are making the change: Chromium [1], Firefox (although not without debate [2]), VLC[3], LibreOffice, GIMP [4].

Disclaimer: though I'm member of the GNOME Foundation, I have no shares in Freedesktop.org. :-)

1: https://chromiumcodereview.appspot.com/12386077/
2: https://bugzilla.mozilla.org/show_bug.cgi?id=259356#c81
3: https://trac.videolan.org/vlc/ticket/1267
4: https://mail.gnome.org/archives/gimp-developer-list/2012-October/msg00037.html

[StefanKarpinski]

Having read through the spec, I'm not sold on this at all. I've got many issues with it.

1. This spec isn't even finalized and the draft is pretty recent (2012), so it's not like this is a really well-established best practice or anything. This is some stuff that some people made up last year.
2. This is a spec for Linux desktop programs (KDE, Gnome, etc.). Julia does not fit the bill.
3. Even ignoring the non-desktopness of Julia, it's completely unclear that the spec even remotely applies. The spec is about application configs, user data and caches. The installed Julia packages are not any of these things. There is a repo cache directory, and I would maybe consider following the spec for that if the rest applied, but installed packages are not preferences or user data.
4. The package system is explicitly designed so that you will typically work in your .julia directory, creating and developing packages. Moving it to some path like ~/.local/share/julia is a major pain, not to mention being pretty clearly semantically wrong – again, installed packages are neither configuration nor user data.
5. The spec is not designed to be opt-in. If you don't set any environment variables, you're expected to use the long, fiddly locations of things that are scattered all over the place (~/.config/julia, ~/.local/share/julia, /etc/xdg/julia, etc.). If there was some environment variable that allowed people to opt into this – say XDG=true, or something – and this was set by default on standard-conforming systems, then I might consider supporting that, but that's not how the spec works. It assumes everyone has bought into this wholesale, rudely foisting long annoying paths on people who don't give a crap about XDG and just want their Julia stuff in .julia.

Given all these issues and the fact that this causes a fairly major coding headache and fractures the IMO priceless uniformity of our cross-platform behavior, I'm sticking with my original gut reaction, which is to say no.

[StefanKarpinski]

I realize you mentioned that git "conforms" although it isn't a desktop program. Except that it doesn't really conform – git will look in the XDG locations if they exist, but if it needs to create a config file, it still creates ~/.gitconfig.

[perillo]

It is ok for Julia to not follow XDG, however it is a nice thing to allow users to change
the default directories used by Julia from the configuration file, or during julia source compilation.

like go, julia can have a ~/.config/julia/env for storing environment variables. so we can choose which directory to use.

[DilumAluthge]

Quoting Fredrik in #37503 (comment):

You can set JULIA_DEPOT_PATH of you are not happy with ~/.julia.

[StefanKarpinski]

I was kind of harsh in this issue before, but I still think that consistently putting all Julia-related stuff in ~/.julia by default was the right call. The XDG "spec" (if such a tiny, half-baked document can be called a spec) still has all of the issues I called out in 2013. Which is not too surprising since no one has touched it since it was scribbled on the virtual equivalent of the back of a napkin a decade ago.

The "spec" is terribly UNIX-specific and only makes sense on multiuser UNIX systems, which are no longer the common case: these days Linux and macOS dev laptops are the norm and the vast majority of servers are virtualized/containerized single-user systems. This spec really only makes sense on multiuser UNIX systems at research institutions. For the vast majority of users, the complexity of scattering everything around directories that aren't next to each other doesn't gain you anything since those directories are on the same file system on your computer's one and only hard disk. All it does it make it harder to see what's going on and add unnecessary complexity to Julia's Pkg and code-loading logic that works with those directories (which is already too complex).

For those users that do actually work on a multi-user UNIX system where it makes sense to put config, cache and runtime data on different filesystems, it would be simple to create symlinks for the fixed set of subdirectories stored under ~/.julia to the various XDG directories as they feel appropriate. Such a tool could even be open sourced and maintained by people who have need for it! For reference, the directories that are saved under ~/.julia are listed in the glossary entry for "depots" and include:

• artifacts: content-addressed data and binary dependencies
• environments: shared named environments (e.g. v1.0, devtools)
• clones: bare clones of package repositories
• compiled: cached compiled package images (.ji files)
• config: global configuration files (e.g. startup.jl)
• dev: default directory for package development
• logs: log files (e.g. manifest_usage.toml, repl_history.jl)
• packages: installed package versions
• registries: clones of registries (e.g. General)

I'm not entirely sure how to map these onto XDG locations, which I feel is due to lack of clarity of the "spec" since I do understand what these directories are for but the spec doesn't really give clear criteria for what should go where, it just lists a bunch of technical properties of locations by which you're apparently supposed to infer, together with suggestive names, where to put things? I feel like people who want to use the spec can figure this out.

For people who work on a single-user system where all the XDG directories are on the same file system but think that it's distasteful for Julia to clutter up their home directory with it's one ~/.julia directory, it should be sufficient to change JULIA_DEPOT_PATH to point somewhere else.

[jtrakk]

it should be sufficient to change JULIA_DEPOT_PATH to point somewhere else.

Indeed it is nice that we can set a custom path for julia files. Going a bit further would help: currently users can't separate their configuration from the generated like registry clones. For users to be able to apply the spec to their own systems, it would be helpful if the JULIA_DEPOT_PATH could be split up into more than one variable, so I can say JULIA_CONFIG_DIR goes to ~/.config/julia, JULIA_CACHE_DIR goes to ~/.cache/julia, JULIA_DATA_DIR goes to ~/.local/share/julia.

[DilumAluthge]

For those users that do actually work on a multi-user UNIX system where it makes sense to put config, cache and runtime data on different filesystems, it would be simple to create symlinks for the fixed set of subdirectories stored under ~/.julia to the various XDG directories as they feel appropriate. Such a tool could even be open sourced and maintained by people who have need for it!

[DilumAluthge]

It's unlikely that this will be worked on further in Julia itself. A third-party open source tool sounds like the right answer.