Question about Debian packaging

[cdluminate]

I see Nico is working on BLIS packaging. I'm interested in packaging BLIS for Debian based on Nico's work. However I have some questions before doing the actual work:

1. Is BLIS mature enough to be used as a drop-in replacement of libblas.so.3 ?
2. Is there any benchmark of which compares BLIS with other BLAS implementations such as OpenBLAS?

[fgvanzee]

@cdluminate Thank you for your interest in BLIS. We have been eagerly waiting for the right person from the Debian community to come along and "sponsor" our project in the Debian/Ubuntu universe. :)

1. Is BLIS mature enough to be used as a drop-in replacement of libblas.so.3 ?

My first instinct is to answer with an emphatic "yes." However, the framing of your question is ultimately subjective, and also dependent on the actual implementation referred to by libblas.so.3, as my understanding is that libblas.so.3 is merely a generic symlink to the current BLAS library. The actual shared library to which it links would be some specific implementation, whether it is the Fortran-77 reference library from netlib, OpenBLAS, or something else.

The lead developers of the BLIS project exercise great care in their approach to software development. One of our guiding principles is to try to "get it right" the first time so we are much less likely to have to revisit/fix it in the future, and we believe this methodical approach pays dividends in long run. We still make mistakes from time to time, but I think the public record here on github shows that we are quite responsive to the community's feedback, especially with bug reports. We sometimes fix issues within hours of them being reported, and BLIS has multiple tools for checking correctness at our disposal, including a comprehensive BLIS testsuite, a C translation of the netlib BLAS test drivers, and integration with Travis CI that uses the former two mechanisms to test multiple hardware configurations via Intel's software development emulator (SDE).

So, to conclude, the answer to your question is "probably," but it depends on what you expect of libblas.so.3. BLIS is still under active development, even if the core functionality exists in a mostly stable state. Whether BLIS belongs in Debian as an officially-supported package may also depend on how frequently our sponsor will be willing to provide updated packages. The more frequent, the better. (I would hope we would have the opportunity to push our latest code to Debian at least monthly, even if we don't need to exercise each opportunity.)

2. Is there any benchmark of which compares BLIS with other BLAS implementations such as OpenBLAS?

Yes. We have performed many performance experiments that compare BLIS against OpenBLAS. Last week, at our annual BLIS Retreat--a workshop here at UT-Austin centered around BLIS-related topics--Devangi Parikh (@dnparikh) presented performance results for multiple level-3 BLAS operations, floating-point datatypes, and problem sizes, and she did so for Intel Haswell/Broadwell, Intel SkylakeX, and Cavium ThunderX2 (ARMv8). The overall story of the performance results is that BLIS is remarkably competitive and consistent in its performance.

Devangi: could you link our guest to PDFs of your graphs from the Retreat?

Also calling out to @nschloe so he can comment about BLIS in general, if he likes.

[nschloe]

Also calling out to @nschloe so he can comment about BLIS in general, if he likes.

I found BLIS as I was looking for BLAS operations on C-ordered arrays for NumPy. BLIS has that, but even better is the fact that it's developed in the open using a more modern language than Fortran.

The overall story of the performance results is that BLIS is remarkably competitive and consistent in its performance.

Plots about that should definitely go into the main README.

[fgvanzee]

@nschloe Thanks for your comments, Nico. I agree that the time has come for us to include some basic plots in the source distribution.
@dnparikh Perhaps we should skip straight to Nico's idea instead, and then @cdluminate can view them through github or a git clone.

[rvdg]

Comments:

• We just had our "BLIS Retreat" (http://www.cs.utexas.edu/users/flame/BLISRetreat2018/). If you click on "program" and then scroll down to Devangi's talk, you will find http://www.cs.utexas.edu/users/flame/BLISRetreat2018/Slides/Devangi_BLIS_Retreat_2018_poster.pdf.

• Field probably should have pointed out that BLIS is what AMD now distributes as part of its "ACL" replacement of the ACML library. This speaks to the stability of the library.

Robert

[cdluminate]

@fgvanzee This is the background for my first question:

Debian/Ubuntu have an alternatives system, by which the user can switch the BLAS implementation smoothly without recompiling any software, e.g.

$ sudo update-alternatives --config libblas.so.3-x86_64-linux-gnu
There are 4 choices for the alternative libblas.so.3-x86_64-linux-gnu (providing /usr/lib/x86_64-linux-gnu/libblas.so.3).

  Selection    Path                                             Priority   Status
------------------------------------------------------------
  0            /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3   40        auto mode
  1            /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3      35        manual mode
  2            /usr/lib/x86_64-linux-gnu/blas/libblas.so.3       10        manual mode
* 3            /usr/lib/x86_64-linux-gnu/libmkl_rt.so            1         manual mode
  4            /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3   40        manual mode

Press <enter> to keep the current choice[*], or type selection number: ⏎       

All the alternative candidates for the libblas.so.3 shared object provides at least the standard set of BLAS API/ABI. In my question, my term libblas.so.3 means "standard BLAS ABI/API".

If BLIS's CBLAS implementation is mature enough, it could be used as a drop-in replacement of libblas.so.3, and it should be added as another alternative for libblas.so.3.

BLIS is still under active development, even if the core functionality exists in a mostly stable state.

That's good to hear.

Whether BLIS belongs in Debian as an officially-supported package may also depend on how frequently our sponsor will be willing to provide updated packages. The more frequent, the better.

I have enough permission to upload package to Debian. However when Debian is about to release a new version, .e.g. 10.0 Buster, the whole archive will be frozen and nothing can be updated except that some package has severe bug. I can only update packages regularly for debian testing/unstable/experimental. So it makes sense to ask upstream before uploading to make sure the package to be uploaded isn't too buggy.

(I would hope we would have the opportunity to push our latest code to Debian at least monthly, even if we don't need to exercise each opportunity.)

It won't be hard for me to update a Debian package as long as there is neither significant ABI/API change, nor significant change in build system.

Plots about that should definitely go into the main README.

+1

And thanks @rvdg for the plot.

[cdluminate]

One more question about packaging:

As shown in #255, BLIS has the best performance with openmp threading, as long as BLIS_NUM_THREADS is properly configured. This means the default configuration for the package should be

--enable-threading=openmp x86_64

Is this correct?

And does it make sense to provide another --disable-threading BLIS at the same time to avoid threading library clash under some certain conditions?

[fgvanzee]

--enable-threading=openmp x86_64

Seems right, yes. You may want to do a thorough review of all configure options. For example, the BLAS integer size option is important to some people.

And does it make sense to provide another --disable-threading to avoid threading library clash under some certain conditions?

Could you clarify this question?

[cdluminate]

@fgvanzee I spotted significant performance drop when a program used iomp and gomp at the same time.

Besides, Intel MKL provides the sequential threading (single thread) module.
https://software.intel.com/en-us/mkl-linux-developer-guide-calling-intel-mkl-functions-from-multi-threaded-applications

Usage model: disable Intel MKL internal threading for the whole application

When used: Intel MKL internal threading interferes with application's own threading or may slow down the application.

[fgvanzee]

@cdluminate Sounds like you are trying to link your gcc-compiled/linked libblis.so to your application with Intel's icc?

@devinamatthews @jeffhammond You guys have more experience with using icc. Could you comment here? I don't quite understand the issue.

EDIT: BTW everyone, I'm taking the day off today. :) Thanks for your patience, @cdluminate . Hopefully others can step in and help us figure out your issues.

[devinamatthews]

@cdluminate I would guess that when using both iomp5 and gomp at the same time that you are ending up with N^2 threads. What happens when you run with OMP_NUM_THREADS=N and BLIS_NUM_THREADS=1 (assuming there is meaningful multithreading in the calling program)?

But, even if it "works" right now, mixing two different OpenMP runtimes is a recipe for disaster. In the context of a Debian package, I would think that gomp would make a sensible default. I guess that pthreads would be even better but as noted above we don't have a thread pool implementation and so performance suffers.

[jeffhammond]

Intel OpenMP runtime defines the GOMP API so if you link it into a GCC program, you should not end up linking against libgomp.so. Please run ldd on your binary to see what libraries are being linked.

[cdluminate]

@fgvanzee @devinamatthews @jeffhammond Thanks for the pointers. But I'm sorry that observation was found in one of my old cxx code that doesn't use BLIS ... and using different threading libraries at the same time resulted in creation of too many threads. GCC uses gomp by default but clang doesn't ...

Anyway I think the debian package should ship with openmp version first.

[cdluminate]

Just a NOTE: I registered libblis as an alternative candidate to libblas.so.3 and libblas.so, and assigned blis with priority 37. I would be happy to increase the priority of BLIS to higher value if there are strong proof that suggests BLIS is better than OpenBLAS, in terms of cblas_* performance.

The default priority chain for Debian and Ubuntu will look like this:

OpenBLAS > BLIS > Atlas > Netlib
Justification:
OpenBLAS > BLIS: if we don't touch any environment variable at all
BLIS > Atlas: BLIS in single thread (according to my test) is still faster than generic Atlas

~/D/b/blis ❯❯❯ sudo update-alternatives --config libblas.so.3-x86_64-linux-gnu
There are 5 choices for the alternative libblas.so.3-x86_64-linux-gnu (providing /usr/lib/x86_64-linux-gnu/libblas.so.3).

  Selection    Path                                             Priority   Status
------------------------------------------------------------
  0            /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3   40        auto mode
  1            /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3      35        manual mode
  2            /usr/lib/x86_64-linux-gnu/blas/libblas.so.3       10        manual mode
  3            /usr/lib/x86_64-linux-gnu/libblis.so.1            37        manual mode
* 4            /usr/lib/x86_64-linux-gnu/libmkl_rt.so            1         manual mode
  5            /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3   40        manual mode

Press <enter> to keep the current choice[*], or type selection number: ^C⏎                   ~/D/b/blis ❯❯❯ sudo update-alternatives --config libblas.so-x86_64-linux-gnu
There are 4 choices for the alternative libblas.so-x86_64-linux-gnu (providing /usr/lib/x86_64-linux-gnu/libblas.so).

  Selection    Path                                           Priority   Status
------------------------------------------------------------
  0            /usr/lib/x86_64-linux-gnu/openblas/libblas.so   40        auto mode
  1            /usr/lib/x86_64-linux-gnu/blas/libblas.so       10        manual mode
  2            /usr/lib/x86_64-linux-gnu/libblis.so            37        manual mode
* 3            /usr/lib/x86_64-linux-gnu/libmkl_rt.so          1         manual mode
  4            /usr/lib/x86_64-linux-gnu/openblas/libblas.so   40        manual mode

Press <enter> to keep the current choice[*], or type selection number: ^C⏎        

Please ignore the lowest priority 1 assigned to MKL.

[fgvanzee]

I registered libblis as an alternative candidate to libblas.so.3 and libblas.so, and assigned blis with priority 37.

@cdluminate That's great news. We appreciate your willingness to include BLIS in the Debian universe. I'm sure many of our users will be pleased to be able to enable BLIS via the distribution's internal package management tools.

I would be happy to increase the priority of BLIS to higher value if there are strong proof that suggests BLIS is better than OpenBLAS, in terms of cblas_* performance.

It all depends on you hardware, operation, datatype, problem size, and how much threading you need, and maybe other factors as well. That said, Robert has pointed you to evidence that we gathered very recently on ThunderX2 and SkylakeX--two architectures designed for high performance. (Devangi didn't include OpenBLAS on the Haswell graphs because we couldn't get it working in multithreaded mode, despite intense efforts by deeply experienced individuals.) For now, OpenBLAS outperforms BLIS for small problems (less than about 100), but that is one of the only use cases for which OpenBLAS outpaces BLIS. The evidence suggests that, for almost all larger problems, almost all operations of almost all datatypes seem to yield better performance when executed via BLIS than with OpenBLAS.

Also consider that there are other metrics by which to measure "goodness" or "betterness" than raw performance. We believe strongly that BLIS stacks up well against OpenBLAS on virtually all of these other measures of software quality. Most important among these metrics: BLIS provides BLAS (and CBLAS) APIs, but unlike most other BLAS libraries, BLIS provides much more than just BLAS APIs. (The BLAS APIs are quite limiting for many individuals and applications, and BLIS contains APIs that attempt to break free of those limitations and expand the space of parameterization and storage formats.) For a full list of features that BLIS provides, I invite you to read our main github page, whose content is stored in the README.md file.

[jeffhammond]

@cdluminate Clang uses the LLVM OpenMP runtime, which is the Intel OpenMP runtime and thus contains the GOMP symbols so it too should not lead to O(n^2) threads when layered with GCC. Pthreads+OpenMP can definitely hit this, however, and it is the expected behavior.