openblas build broken on mac when not using binarybuilder

[ViralBShah]

➜  julia git:(master) ✗ make USE_BINARYBUILDER_OPENBLAS=0 VERBOSE=1
make[1]: *** No rule to make target `scratch/objconv/build-compiled', needed by `scratch/openblas-d909f9f3d4fc4ccff36d69f178558df154ba1002/build-compiled'.  Stop.
make: *** [julia-deps] Error 2

[ViralBShah]

I have to do

make USE_BINARYBUILDER_OPENBLAS=0 USE_BINARYBUILDER_OBJCONV=0 VERBOSE=1

It seems that openblas on darwin is unable to use the BB provided objconv (presumably some makefile stuff needs fixing).

[ViralBShah]

Also seems like ZLIB is a problem (USE_BINARYBUILDER_ZLIB=0 doesn't help either):

Undefined symbols for architecture x86_64:
  "_compress2", referenced from:
      llvm::zlib::compress(llvm::StringRef, llvm::SmallVectorImpl<char>&, int) in libLLVMSupport.a(Compression.cpp.o)
  "_compressBound", referenced from:
      llvm::zlib::compress(llvm::StringRef, llvm::SmallVectorImpl<char>&, int) in libLLVMSupport.a(Compression.cpp.o)
  "_crc32", referenced from:
      llvm::zlib::crc32(llvm::StringRef) in libLLVMSupport.a(Compression.cpp.o)
      llvm::crc32(unsigned int, llvm::ArrayRef<unsigned char>) in libLLVMSupport.a(CRC.cpp.o)
      llvm::crc32(llvm::ArrayRef<unsigned char>) in libLLVMSupport.a(CRC.cpp.o)
      llvm::JamCRC::update(llvm::ArrayRef<unsigned char>) in libLLVMSupport.a(CRC.cpp.o)
     (maybe you meant: _ijl_crc32c_sw, _jl_crc32c )
  "_uncompress", referenced from:
      llvm::zlib::uncompress(llvm::StringRef, char*, unsigned long&) in libLLVMSupport.a(Compression.cpp.o)
      llvm::zlib::uncompress(llvm::StringRef, llvm::SmallVectorImpl<char>&, unsigned long) in libLLVMSupport.a(Compression.cpp.o)
     (maybe you meant: _ijl_uncompress_argname_n, _ijl_uncompress_ir , _ijl_uncompress_argnames )
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [/Users/viral/julia/usr/lib/libjulia-internal.1.8.dylib] Error 1
make: *** [julia-src-release] Error 2

[vchuravy]

Ideally we need to have CI for the non-BB builds as well. Otherwise this will keep happening.

[DilumAluthge]

If we run non-BB builds on CI for every PR, that'll eat up a lot of CI time.

Maybe we just have a scheduled CI job that runs once per day and does a non-BB build on each platform? That would use up fewer resources, but we'd still detect the failures within 24 hours of the bad commit being merged to master.

cc: @staticfloat

The risk with this approach (as @JeffBezanson and I were discussing on Slack) is that in theory, a PR can be passing all PR CI (because that PR builds just fine with the PR CI), and then you merge the PR (because CI is green), and then the next day, the scheduled CI job fails because the non-BB build fails.

So I don't know what the right answer is here.

[ViralBShah]

IMO, the right answer here is to fix these as we find them. Once in 24 hour CI is reasonable, if it is possible to set up something like that.

[ViralBShah]

Note that the objconv issue is mac only.

[staticfloat]

I think adding a 24-hour scheduled build with USE_BINARYBUILDER=0 is a good idea, but note that it still wouldn't have found this issue, since this is only triggered if USE_BINARYBUILDER_OPENBLAS=0 USE_BINARYBUILDER_OBJCONV=1 was true. There's just too much surface area in our build system for proper CI of this stuff, we just have to focus on keeping the happy path alive, and fix other issues as we go.

[vchuravy]

Yeah having USE_BINARYBUILDER=0 is sufficient for a smoke tests, but as you said we are facing an infinite build variation.

[DilumAluthge]

I think adding a 24-hour scheduled build with USE_BINARYBUILDER=0 is a good idea

Implemented in #42592, for the purpose of being a very basic smoke test.