What is a Julia standard library?

I have been thinking on this quite a lot recently, and wanted to gather some thoughts. I also feel like this is a question we have sidestepped a bit, and I think it would be nice to collect some ideas about the way forward for Julia's stdlib. In particular I think we should have a good definition of what a stdlib is, and what it means to be a stdlib. What follows are my thoughts on the situation, and I hope you all have some input, and hopefully we can formalize this a bit more.

Should stdlibs be regular packages?

While thinking about this I have come to the following conclusion; stdlibs would benefit a lot by trying to behave more like "regular" packages as much as possible. Why? It is a huge freedom to be a regular package. Packages in Julia work extremely well, and most of the modules in JuliaLang/julia/stdlib would be perfectly fine as packages. Here are some advantages of being a package (over the current JuliaLang/julia/stdlib style):

• Packages have separate repositories, meaning, among other things, that they have dedicated issue trackers such that issues don't get lost in the thousands of issues in JuliaLang/julia. It is also easier to get an overview of the package source code, since it is well contained.
• Packages are free to make new releases whenever they want, without caring about the release schedule of Julia. Most importantly this means you can get bug fix releases almost instantly (well, usually at least within a week!).
• Packages can set up their own docs, which means the focus of the documentation web page will be 100% on the package. As it is now, the stdlib docs are hidden deep within the Julia base manual (and TBH impossible to find).
• Packages have shorter CI times. A typical package does not take more than 10 minutes or so to run the test suite. This leads to faster development, since you don't have to wait for 2 hours to see the result for your PR.

With Pkg3 you will also have complete control over your environment, which IMO solves almost all problems with being a package.

Why do we need stdlibs?

So, then you may ask, why do we need something called a stdlib if packages are so great? Here are some reasons:

1. People like to use "official" packages and, therefore, stdlibs should be packages that are officially endorsed, supported and promoted by the JuliaLang organization. This gives credibility to the package and you should be able to trust that the package contains high quality code that is well tested and well documented.

Actually, that is the only reason I came up with, but it is an important one. What does it then mean to be officially supported by the organization?

• The Julia stdlibs are promoted, and linked to, from the base manual. We also list them on e.g. the homepage as the goto implementations and/or starting points for the different domains.
• Julia stdlibs are given extra consideration when changing things in the base language. I don't think we should necessarily run stdlib tests as part of base Julia CI, but perhaps keep an extra eye on the nightly builds. As with regular packages, I don't think you should expect stdlibs to work on Julia master at all times.
• Julia stdlibs have at least one active official maintainer.

Where should a stdlib live?

I think that JuliaLang/julia/stdlib is just a stopover for the modules that are there now. Eventually we need to move them out of the Julia repo in order to develop them more freely, and to simplify making new releases. We are already pushing the CI limit on the Julia repo, and moving them out would cut CI times by a lot (looking at you LinearAlgebra). In their own repos the stdlibs are better contained, and it will be easier to get an overview of the source. Developing a stdlib will also be much more pleasant when you don't have to build all of julia, and when you don't have to wait for 2 hours to hear back from CI.

Should we have a JuliaStdlib GitHub organization to gather the stdlibs? I don't think so. For example, there have been talks about making HTTP.jl a stdlib (whatever that means, trying to answer that in this post...). Should we then pull out HTTP.jl from the JuliaWeb organization? I think that would be a shame; HTTP.jl fits well in a JuliaWeb organization, and there is no reason to break that up. Likewise it would be a shame to break out StatsBase from JuliaStats.

There are some stdlibs that perhaps don't fit well in any organization (for example Test, Profile, DelimitedFiles etc.), and for those we can have a misc JuliaStdlib organization to host them.

After all, for end users, it does not really matter where the source is hosted and thus I don't think we have to gather them all. Users will read the manual, see that there is a stdlib called Test and then pkg> add Test without a worry in the world about where the source code lives.

How do you install a stdlib?

I don't think a stdlib needs to be installed by default -- it is very simple to pkg> add Package when you need it. Pkg3 makes installation much more pleasant, and pkg> add Package no longer grants you a coffee break, so having to add the stdlibs should not be an annoyance. One thing that is nice right now is that the stdlibs are included in the sysimg, which results in faster load times. At the moment this is needed for some of them, in particular LinearAlgebra and Pkg3 are pretty slow otherwise, but who cares if you can load the Test module instantly or not?

Hopefully we can work on decreasing the load times of packages in the future. We can also have a convenient way to add packages to the sysimg, and regenerate it. I initially thought that this option should only be for stdlibs, since they might not update as regularly as regular packages, but then, why shouldn't you be able to do that for any package?

The way forward

For Julia v0.7/v1.0 I think we should just leave the stdlibs in JuliaLang/julia/stdlib where they are, and state that they might not be included by default in v1.1, but they will still be supported and installable etc in future Julia versions. Then when v1.0 is out the window we simply move them all out to their own repos and reap the benefits.

Addendum

Furthermore, I think stdlibs should be allowed to be pirates.

TL;DR

stdlibs would benefit a lot from being more like regular packages and should live in their own repos.