Netlib BLAS test failures should cause CI failure

[devinamatthews]

See e.g. here. The build passes even though many BLAS tests fail. In this case, I have no idea why they fail but since they do so only for shared builds and not static it is probably a real bug that the CI workflow should indicate.

For context, similar BLAS test failures happen on Windows/AVX512 (#514) and are not reflected in the exit code of make check.

[isuruf]

The way the BLAS tests are setup, it's impossible to have them run on windows with shared builds.

This is because a local xerbla implementation needs to override the one in the blis library which works in Unix and in windows static builds, but with windows DLLs, the xerbla implementation in the DLL takes precedence.

[fgvanzee]

Ah yes, I forgot about that highly unfortunate feature of the BLAS test drivers. Thanks for that reminder, Isuru.

[devinamatthews]

So is the symptom of this seemingly random test failures or is there a more specific outcome?

[isuruf]

So is the symptom of this seemingly random test failures

Yes

[devinamatthews]

Well, that sucks. I'll revert the changes to .appveyor.yml but I still think that BLAS tests should trigger CI failure. I've have them fail in other circumstances where the BLIS tests did not and there was indeed a bug.

[isuruf]

With static libraries, it should work though.

[fgvanzee]

I recommend that we insert a comment somewhere (in the .appveyor.yml file?) that will remind future readers of this issue.

[devinamatthews]

I recommend that we insert a comment somewhere (in the .appveyor.yml file?) that will remind future readers of this issue.

Done

[devinamatthews]

Apparently all test failures are ignored in the Makefile (prepended with "- "). @fgvanzee can you remind me what the rationale for this is? Makes it hard for other people to incorporate "make check" into their CI pipelines.

[fgvanzee]

Apparently all test failures are ignored in the Makefile (prepended with "- "). @fgvanzee can you remind me what the rationale for this is?

I probably did this because I wanted make to continue whatever rule is being executed, even if an error is encountered.

Here's the relevant part of the GNU make documentation, for those who are curious.