Getting BLIS into Ubuntu

[fgvanzee]

Robert asked me recently, "What would it take for BLIS to be incorporated into Ubuntu?"

Some light Googling lead me to understand that there are four main repositories in Ubuntu: main, restricted, universe, multiverse. It sounds like we would be okay being in universe at least at first, especially if we need more time to prove ourselves suitable for main.

I then found this stackexchange thread on the topic. The most up-voted response breaks down some of the options. It appears that getting into Debian is the most comprehensive choice (also indicated here), as it would mean BLIS would be made available to users of both Linux distributions (Debian and Ubuntu).

Thanks to the work of Nico Schlömer, we already have someone involved with the project who knows how to create (and has created) PPA packages. The biggest hurdle, it seems, is to get a Debian Developer (DD) to become our sponsor. This begins with a Request-for-Sponsor (RFS) bug report in Debian's bug-tracking system. (The RFS is like an ad designed to pique the interest of developer to become our sponsor.) But we are not supposed to submit the RFS until we have our source package ready for a potential sponsor to download/build/install. I think, thanks to Nico's work, we are basically there. (Care the comment, @nschloe?)

Note that there is no guarantee that we will find a sponsor. There are a finite number of DDs, each with limited time and specific specializations/interests/preferences.

After we get a sponsor, I think we will need this sponsor to upload any future revisions of the package. This is probably the most concerning to me aspect to me since BLIS has been undergoing relatively frequent updates.

Bottom line, this appears to be a non-trivial amount of work. So we should discuss/think carefully about if and when this would make sense for us.

Please comment, if you like.

[devinamatthews]

Regarding having BLIS in a PPA (either Debian or Ubuntu, or does it matter?): on OpenSUSE there are the regular packages in the "official" repos, but there are also "1-Click" install files which prompt the package manager to both add an external package repo (the PPA in our case) and then install the package. Does something like this exist for Debian/Ubuntu? The upside of this would be not needing to get into the official repo but also making it easy for users to install and get updates. If we don't care about the second part then a bare .deb/.rpm would also suffice.

[fgvanzee]

Does something like this exist for Debian/Ubuntu?

@devinamatthews According to this wiki, it appears that the answer is yes (see section "Adding Personal Package Archives (PPAs)"). However, this may not so much as capture new users as give some of the existing ones another installation option. The way I see it, if the person is willing to pre-configure their Ubuntu GUI package widget to download from, say, Nico's PPA, they would probably already be willing to go straight to GitHub, especially since the latter would allow them access too all configuration options, whereas a binary would restrict them to whatever choices the package maintainer made.

[devinamatthews]

The "1-Click" thing is slightly different in that one can just have a link on a webpage that the user clicks on, and then everything is automated from there. Adding a PPA manually to Ubuntu is relatively much more of a PITA.

[fgvanzee]

@devinamatthews Understood, sorry for that nuance being lost on me. (I've never used OpenSuSE.)

[nschloe]

I myself am a Debian Maintainer (DM) so I have a fair idea of how things are working. A package gets into Debian if (a) it's popular enough for people to want it or (b) if another package depends on it. This approach is typically taken somewhat conservatively since if you're volunteering to maintain a package, it's your task forever. For example, even though I follow BLIS with great interest, I myself wouldn't want to dive into that yet.

If your goal is to get BLIS into Ubuntu, you're already there: There's a PPA, and everyone is just one command away to add it.

If you want to gain some traction, one way would probably be to demonstrate that you do some things better (faster?) than BLAS, given developers of other software arguments for linking against BLIS rather than BLAS. Perhaps get back to SciPy or NumPy, they'll be able to give you some good feedback.

[loveshack]

I think a Debian package should do the same as the openblas one and provide libblas.so.3, to be substitutable at run time (the way the BLIS rpm packaging does). I started doing that along with the rpm spec, but it wasn't clear to me what the actual Debian linear algebra policy is, as I commented somewhere in the issues. Also, I don't like Debian only offering a pthreads version of openblas, not serial or OpenMP, which you probably want for HPC. (I think that was purely for support of numpy for some reason I don't remember, but I don't see why it should exclude the others.)

The specific reason to have BLIS included is the KNL and SKX BLAS support, which OpenBLAS doesn't have.

[fgvanzee]

@loveshack Curious: Why is the soname of the shared library chosen to be 3 (in libblas.so.3)?

[loveshack]

The idea is that it can substitute the netlib library via LD_LIBRARY_PATH or ld.so.conf. If README.Fedora isn't clear (it's inline in the spec file), I'll try to make it so. [It currently has a typo, and I need to make it README.SuSE on SuSE.] Maybe also see https://loveshack.fedorapeople.org/blas-subversion.html if I haven't mentioned it, but it needs some updating, and experimental data mostly don't convince research computing people out of myths about MKL etc., sigh.

[fgvanzee]

Thanks Dave, but I guess my question is why is it a convention that libblas.so.3 use an soname of 3? Is it because BLAS did not change after the introduction of the level-3 operations? I can't help but notice that all of the other netlib libraries seem to use an soname of 3 as well. Maybe it has to do with the current major version number of LAPACK? (It's okay if you don't know the answer.)

[loveshack]

You wrote:

Thanks Dave, but I guess my question is why is it a convention that `libblas.so.3` use an `soname` of 3? Is it because BLAS did not change after the introduction of the level-3 operations?

The soname major version changes when there's an incompatible ABI change. It's nothing specifically to do with BLAS levels. Explained somewhere in the autotools doc, at least.

I can't help but notice that all of the other netlib libraries seem to use an `soname` of 3 as well. Maybe it has to do with the current major version number of LAPACK? (It's okay if you don't know the answer.)

It's related, but not specifically following that. The Debian changelog says this. I don't know what it was initially. ``` lapack3 (3.0.20000531a-1) unstable; urgency=low * Binary incompatible changes in the blas library, against which this library is linked, require an increase in the soname, and a change in package name to allow for support for the old API in both source and binary form to remain in the archives.

…

-- Camm Maguire <camm@enhanced.com> Tue, 23 Dec 2003 15:08:41 +0000 ``` The libblas soversion is the same as liblpack's in both Debian and RHEL(/Fedora?) but the BLAS ABI hasn't changed at all recently, so I think it should actually be 3.0.6, not 3.6.0.

[fgvanzee]

@loveshack Okay, sounds like the "3" is just coincidence then.

(A library increasing its soname occasionally is not surprising to me. But my question was motivated by the fact that I was under the impression that netlib BLAS, which is what I think of when I see libblas, simply did not experience the kinds of changes that would break its ABI, since it was unrolled in a way that did not break old functionality, but rather just added new operations. But I can sleep fine not getting to the bottom of this one.)

[fgvanzee]

@loveshack Do you see any issue with simply providing a symbolic link named libblas.so.3? I'm not totally in love with this idea, mostly due to the aliasing of the soname of 3 onto libblis's (current) soname of 0, but I'm nevertheless open to a symlink.

[loveshack]

You wrote:

@loveshack Okay, sounds like the "3" is just coincidence then. (A library increasing its `soname` occasionally is not surprising to me. But my question was motivated by the fact that I was under the impression that netlib BLAS, which is what I think of when I see `libblas`, simply did not experience the kinds of changes that would break its ABI, since it was unrolled in a way that did not break old functionality, but rather just added new operations. But I can sleep fine not getting to the bottom of this one.)

That's what I think, though I haven't checked the BLAS ABI history. As I said, I don't think it should be libblas.so.3, but at most .1. Anyhow, it can't go backwards now.

[loveshack]

You wrote:

@loveshack Do you see any issue with simply providing a symbolic link named `libblas.so.3`?

For what purpose? It won't work as a dynamic substitute for the original libblas.so.3 because the ELF soname is wrong.

I'm not totally in love with this idea, mostly due to the aliasing of the `soname` of 3 onto libblis's (current) `soname` of 0, but I'm nevertheless open to a symlink.

Never mind. I'll make an effort to get the packaging into Fedora eventually -- and possibly try to shake up up the linear algebra handling in it, but others have failed on that.

[fgvanzee]

For what purpose? It won't work as a dynamic substitute for the original libblas.so.3 because the ELF soname is wrong.

Okay, I suppose I hadn't thought of that.

Hope there aren't any hard feelings, Dave, and that you understand that I'm just trying to balance our own project's conventions with how far we bend over backwards to accommodate existing BLAS conventions (beyond the BLAS API itself). We appreciate your involvement. Let us know how the Fedora thing goes.