Description
What is a Julia standard library?
I have been thinking on this quite a lot recently, and wanted to gather some thoughts. I also feel like this is a question we have sidestepped a bit, and I think it would be nice to collect some ideas about the way forward for Julia's stdlib
. In particular I think we should have a good definition of what a stdlib
is, and what it means to be a stdlib
. What follows are my thoughts on the situation, and I hope you all have some input, and hopefully we can formalize this a bit more.
Should stdlib
s be regular packages?
While thinking about this I have come to the following conclusion; stdlib
s would benefit a lot by trying to behave more like "regular" packages as much as possible. Why? It is a huge freedom to be a regular package. Packages in Julia work extremely well, and most of the modules in JuliaLang/julia/stdlib
would be perfectly fine as packages. Here are some advantages of being a package (over the current JuliaLang/julia/stdlib
style):
- Packages have separate repositories, meaning, among other things, that they have dedicated issue trackers such that issues don't get lost in the thousands of issues in
JuliaLang/julia
. It is also easier to get an overview of the package source code, since it is well contained. - Packages are free to make new releases whenever they want, without caring about the release schedule of Julia. Most importantly this means you can get bug fix releases almost instantly (well, usually at least within a week!).
- Packages can set up their own docs, which means the focus of the documentation web page will be 100% on the package. As it is now, the
stdlib
docs are hidden deep within the Julia base manual (and TBH impossible to find). - Packages have shorter CI times. A typical package does not take more than 10 minutes or so to run the test suite. This leads to faster development, since you don't have to wait for 2 hours to see the result for your PR.
With Pkg3
you will also have complete control over your environment, which IMO solves almost all problems with being a package.
Why do we need stdlib
s?
So, then you may ask, why do we need something called a stdlib
if packages are so great? Here are some reasons:
- People like to use "official" packages and, therefore,
stdlib
s should be packages that are officially endorsed, supported and promoted by the JuliaLang organization. This gives credibility to the package and you should be able to trust that the package contains high quality code that is well tested and well documented.
Actually, that is the only reason I came up with, but it is an important one. What does it then mean to be officially supported by the organization?
- The Julia
stdlib
s are promoted, and linked to, from the base manual. We also list them on e.g. the homepage as the goto implementations and/or starting points for the different domains. - Julia
stdlib
s are given extra consideration when changing things in the base language. I don't think we should necessarily runstdlib
tests as part of base Julia CI, but perhaps keep an extra eye on the nightly builds. As with regular packages, I don't think you should expectstdlib
s to work on Julia master at all times. - Julia
stdlib
s have at least one active official maintainer.
Where should a stdlib
live?
I think that JuliaLang/julia/stdlib
is just a stopover for the modules that are there now. Eventually we need to move them out of the Julia repo in order to develop them more freely, and to simplify making new releases. We are already pushing the CI limit on the Julia repo, and moving them out would cut CI times by a lot (looking at you LinearAlgebra
). In their own repos the stdlib
s are better contained, and it will be easier to get an overview of the source. Developing a stdlib
will also be much more pleasant when you don't have to build all of julia, and when you don't have to wait for 2 hours to hear back from CI.
Should we have a JuliaStdlib
GitHub organization to gather the stdlib
s? I don't think so. For example, there have been talks about making HTTP.jl
a stdlib
(whatever that means, trying to answer that in this post...). Should we then pull out HTTP.jl
from the JuliaWeb organization? I think that would be a shame; HTTP.jl
fits well in a JuliaWeb
organization, and there is no reason to break that up. Likewise it would be a shame to break out StatsBase
from JuliaStats
.
There are some stdlib
s that perhaps don't fit well in any organization (for example Test
, Profile
, DelimitedFiles
etc.), and for those we can have a misc JuliaStdlib organization to host them.
After all, for end users, it does not really matter where the source is hosted and thus I don't think we have to gather them all. Users will read the manual, see that there is a stdlib
called Test
and then pkg> add Test
without a worry in the world about where the source code lives.
How do you install a stdlib
?
I don't think a stdlib
needs to be installed by default -- it is very simple to pkg> add Package
when you need it. Pkg3
makes installation much more pleasant, and pkg> add Package
no longer grants you a coffee break, so having to add the stdlib
s should not be an annoyance. One thing that is nice right now is that the stdlib
s are included in the sysimg, which results in faster load times. At the moment this is needed for some of them, in particular LinearAlgebra and Pkg3
are pretty slow otherwise, but who cares if you can load the Test
module instantly or not?
Hopefully we can work on decreasing the load times of packages in the future. We can also have a convenient way to add packages to the sysimg, and regenerate it. I initially thought that this option should only be for stdlib
s, since they might not update as regularly as regular packages, but then, why shouldn't you be able to do that for any package?
The way forward
For Julia v0.7
/v1.0
I think we should just leave the stdlib
s in JuliaLang/julia/stdlib
where they are, and state that they might not be included by default in v1.1
, but they will still be supported and installable etc in future Julia versions. Then when v1.0
is out the window we simply move them all out to their own repos and reap the benefits.
Addendum
Furthermore, I think stdlib
s should be allowed to be pirates.
TL;DR
stdlibs
would benefit a lot from being more like regular packages and should live in their own repos.