Problem when linking to clang and getting the Homebrew libc++ #96915

mppf · 2022-03-15T14:16:14Z

`brew gist-logs <formula>` link OR `brew config` AND `brew doctor` output

% brew gist-logs llvm@13  
Error: No logs.

% brew config
HOMEBREW_VERSION: 3.4.1-67-gb31d8e9
ORIGIN: https://github.com/Homebrew/brew
HEAD: b31d8e929b09562f431f926d1a158753a1df5877
Last commit: 18 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 5731c186dbbe0c0804f7706767029d57a0813a7a
Core tap last commit: 50 minutes ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_CORE_GIT_REMOTE: https://github.com/Homebrew/homebrew-core
HOMEBREW_MAKE_JOBS: 12
Homebrew Ruby: 2.6.8 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.8/bin/ruby
CPU: dodeca-core 64-bit kabylake
Clang: 13.1.6 build 1316
Git: 2.35.1 => /usr/local/bin/git
Curl: 7.77.0 => /usr/bin/curl
macOS: 12.2.1-x86_64
CLT: 13.2.0.0.1.1638488800
Xcode: 13.3

% brew doctor
Your system is ready to brew.

Verification

I ran brew update and am still able to reproduce my issue.
I have resolved all warnings from brew doctor and that did not fix my problem.

What were you trying to do (and why)?

I'm one of the developers for the upstream Chapel project. Chapel was working with llvm@11 but we were running into strange problems with llvm@12 and llvm@13 (see chapel-lang/chapel#19217 ).

It took me a long time but I eventually figured out that the problem was due to mixing libc++ versions. The Chapel compiler, chpl was linking with the libc++ included in the Homebrew LLVM package; but the LLVM and Clang libraries were linking with a libc++ provided by the system. This will occur with any program that is linking with llvm or clang libraries using the usual commands to add a -L path for /usr/local/Cellar/llvm/13.0.1_1/lib, including using the output of /usr/local/opt/llvm@13/bin/llvm-config --ldflags.

I have included a small reproducer example in the step-by-step section below.

What happened (include all command output)?

With the test program provided below, I get this output

% ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

What did you expect to happen?

I expected this output from the test program provided below (without the workaround):

% ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 1

Step-by-step reproduction instructions (by running `brew` commands)

brew install llvm@13

Then, I have saved the following to pretend-in-llvm.cpp and within it are the commands to reproduce this problem.


/*

# first, imagine we are compiling something within LLVM
# this should use the same SDK that Homebrew used when compiling LLVM/clang
# other flags come from `/usr/local/opt/llvm@13/bin/llvm-config --cxxflags`
clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/usr/local/Cellar/llvm/13.0.1_1/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/local/include

# now, link an application with the above
# supposing it is in one of the LLVM/Clang libraries
# other flags come from `/usr/local/opt/llvm@13/bin/llvm-config --ldflags`
clang++ pretend-in-llvm.o -o a.out -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/usr/local/Cellar/llvm/13.0.1_1/lib -L/usr/local/Cellar/llvm/13.0.1_1/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-13 -L/usr/local/lib

./a.out
#  EC.message is No such file or directory
#  EC == no_such_file_or_directory is 0

# We can see the problem with otool -L :
otool -L ./a.out
#  ...
#  /usr/local/opt/llvm/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)

otool -L //usr/local/opt/llvm/lib/libLLVM.dylib
#  ...
#  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)

# (in other words, a.out is using the libc++ installed by llvm@13
#  but LLVM itself is linked with a system libc++)

# Adjusting the link line like this works around the problem:
clang++ pretend-in-llvm.o -o a.out -L/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/usr/local/Cellar/llvm/13.0.1_1/lib -L/usr/local/Cellar/llvm/13.0.1_1/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-13 -L/usr/local/lib

./a.out
#  EC.message is No such file or directory
#  EC == no_such_file_or_directory is 1

*/

#include "llvm/Support/Errc.h"
#include "clang/Basic/FileManager.h"

#include <iostream>

int main() {
  clang::FileManager fileMgr((clang::FileSystemOptions()));

  auto file = fileMgr.getFileRef("./nonexistant.h", /*OpenFile=*/true);

  std::error_code EC = llvm::errorToErrorCode(file.takeError());

  std::cout << "EC.message is " << EC.message() << "\n";

  bool not_no_such = (EC != llvm::errc::no_such_file_or_directory);
  bool is_no_such = !not_no_such;

  std::cout << "EC == no_such_file_or_directory is " << is_no_such << "\n";

  return 0;
}

The text was updated successfully, but these errors were encountered:

carlocab · 2022-03-15T14:54:19Z

I'm not sure what to make of this problem, but I'm pretty sure your problem isn't due to mixed libc++ linkage, for (at least) four reasons:

You have this linking behaviour with llvm@11 as well: libLLVM links with /usr/lib/libc++.1.dylib but linking with libLLVM will typically lead to linkage with a Homebrew-provided libc++ as well.
Many other formulae link with both libLLVM and a Homebrew-provided libc++ with no issues.
It is actually impossible not to link at least indirectly with /usr/lib/libc++.1.dylib. This is because every binary must link with /usr/lib/libSystem.B.dylib, and libSystem actually has a library dependency on /usr/lib/libc++.1.dylib. This means, in particular, that any Homebrew libc++ itself depends on /usr/lib/libc++.1.dylib.
The two-level namespace on macOS often makes loading libraries of different versions in the same process unproblematic, unlike what you get when you try the same on Linux.

My guess is that you've run into a different LLVM bug.

mppf · 2022-03-15T15:55:14Z

You have this linking behaviour with llvm@11 as well: libLLVM links with /usr/lib/libc++.1.dylib but linking with libLLVM will typically lead to linkage with a Homebrew-provided libc++ as well.

Yes, my hypothesis here is that the Mac OS X libc++ happens to be compatible with the one from llvm@11 for the versions used here.

I'm not sure what to make of this problem, but I'm pretty sure your problem isn't due to mixed libc++ linkage, for (at least) four reasons:

Did you try the reproducer? I think the otool -L output is a pretty strong sign that the libc++ versions are different. The fact that the problem goes away with the -L path adjustment (and also makes the libc++ versions the same) seems pretty conclusive to me that, at the very least, mixing libc++ versions is part of the problem. Of course there might be another solution.

Many other formulae link with both libLLVM and a Homebrew-provided libc++ with no issues.

Yes and some things appeared to work in our project. I boiled down to a reproducer above a pattern that was not working. Edit -- it might also matter if static or dynamic linking with clang/llvm libraries is used.

My guess is that you've run into a different LLVM bug.

Note that the clang from llvm@13 is not involved in compiling my example at all.

carlocab · 2022-03-15T16:04:43Z

Yes, my hypothesis here is that the Mac OS X libc++ happens to be compatible with the one from llvm@11 for the versions used here.

If this hypothesis were true, then I would expect to see

this behaviour to differ across different versions of macOS. Does it? Admittedly, Apple aren't terribly transparent about system libc++ versioning, but given that they update the rest of the toolchain on newer macOS versions, it would stand to reason that they update libc++ too.
problems with other formulae that make use of LLVM 13. There are none that I'm aware of so far.

I think the otool -L output is a pretty strong sign that the libc++ versions are different.

Yes, I agree with you that the libc++ versions differ. However,

The fact that the problem goes away with the -L path adjustment (and also makes the libc++ versions the same) seems pretty conclusive to me that, at the very least, mixing libc++ versions is part of the problem.

I'm still not convinced.

Note that the clang from llvm@13 is not involved in compiling my example at all.

Sure, but I don't think this is relevant here. At the very least, this isn't relevant to any claims I've previously made.

mppf · 2022-03-15T16:05:44Z

I'm still not convinced.

Well, feel free to investigate further. I've already spent about a week on this and the workaround I show here is the best I could come up with.

Bo98 · 2022-03-15T16:06:40Z

Note that the clang from llvm@13 is not involved in compiling my example at all.

Does the behaviour change if the clang from llvm@13 is used?

mppf · 2022-03-15T16:10:46Z

Note that the clang from llvm@13 is not involved in compiling my example at all.

Does the behaviour change if the clang from llvm@13 is used?

Not for me, anyway

% /usr/local/Cellar/llvm/13.0.1_1/bin/clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/usr/local/Cellar/llvm/13.0.1_1/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/local/include
% vim pretend-in-llvm.cpp
% /usr/local/Cellar/llvm/13.0.1_1/bin/clang++ pretend-in-llvm.o -o a.out -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/usr/local/Cellar/llvm/13.0.1_1/lib -L/usr/local/Cellar/llvm/13.0.1_1/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-13 -L/usr/local/lib
% ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

carlocab · 2022-03-15T23:28:05Z

Do you mind reporting this upstream (i.e. LLVM)? I think this is an upstream bug.

Linking libLLVM with /usr/lib/libc++.1.dylib is very difficult to avoid. LLVM's build system requires that you build libLLVM (or, at least, its constituent static libraries) before you build libc++. Thus, any standard build of LLVM will have libLLVM linked against /usr/lib/libc++.1.dylib.

If your hypothesis is correct, then it is impossible to link against libLLVM and non-Apple libc++ simultaneously in a standard shared library build, and fixing it would require building LLVM at least twice: once to build a libc++ to link libLLVM against, and another to build the libLLVM you eventually install. (You would still load /usr/lib/libc++.1.dylib indirectly anyway, though, so this might not be a complete fix.)

mppf · 2022-03-16T13:16:46Z

@carlocab - the reason that I have posted it here and not upstream is that I was unable to reproduce the issue when not doing a Homebrew build. (I.e. if I build and install LLVM somewhere myself, manually, outside of homebrew, everything works OK). AFAIK that is due to the -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk arguments that the Homebrew build is using. (AFAIK, this argument comes both from something in the cmake from the Homebrew formula and from superenv).

Additionally I wanted to ask if it was clear to you why building with headers from one version of a library and then linking with a different version of the same library is a problem, in general. I could explain that more / find a reference to link here, if not. We could certainly ask upstream if they expect that to work for libc++ specifically - but I'm not so sure they can even answer the question of whether it is expected to work between Apple's libc++ and the open source one. I think we would have to involve somebody from Apple to answer that. Apple's libc++ could have any number of changes that break compatibility with the open source version.

Linking libLLVM with /usr/lib/libc++.1.dylib is very difficult to avoid. LLVM's build system requires that you build libLLVM (or, at least, its constituent static libraries) before you build libc++. Thus, any standard build of LLVM will have libLLVM linked against /usr/lib/libc++.1.dylib.

If your hypothesis is correct ... fixing it would require building LLVM at least twice: once to build a libc++ to link libLLVM against, and another to build the libLLVM you eventually install. (You would still load /usr/lib/libc++.1.dylib indirectly anyway, though, so this might not be a complete fix.)

Right, that is one way to fix it. Including that, I can think of several ways that Homebrew could fix it:

as you said, it could build LLVM multiple times in order to create a clang that links with the built libc++. This solution makes the least sense to me of these options. However, I do not see why this would be a show-stopper. To me it looks like the formula already builds clang multiple times.

homebrew-core/Formula/llvm.rb

Lines 178 to 179 in 14af399

    
           # We build LLVM a few times first for optimisations. See 
        
           # https://github.com/Homebrew/homebrew-core/issues/77975

package libc++ separately and install it to a separate directory. (Ubuntu does not do this; while libc++-13-dev and llvm-13-dev are separate packages, they both install libraries to /usr/lib/llvm-13/lib. But the issue is not a big deal on Ubuntu since by default the compilers will use the gnu libstdc++). This would address the problem because people would have to opt in to linking with the non-system libc++, somehow.
just don't provide a package that installs a different libc++ from the system one. AFAIK this is the approach FreeBSD uses - and that is relevant because FreeBSD is another system that uses this libc++ as the system C++ library.

Relevant to the last option: Why does the LLVM formula install libc++ at all? Does it need to continue to do that?

~~In the llvm.rb formula I can see that -DLLVM_ENABLE_LIBCXX is only set to ON for Mac OS X builds but not for linux builds. Why is that? It seems to be an odd choice.~~ (Edit: I was confused about this flag; it just adds stdlib=libc++)
There is also a comment about the system libc++ being to old to build LLVM but that no longer seems to be the case, at least not with recent Mac OS X versions. (Edit: I think this is unrelated to building libc++ here, though)

homebrew-core/Formula/llvm.rb

Lines 80 to 81 in 14af399

# Apple's libstdc++ is too old to build LLVM

ENV.libcxx if ENV.compiler == :clang
It looks to me like building LLVM's libc++ started in 856509e as a way to resolve missing system libraries (but we have better ways to handle that now - with things like isysroot) (I might be wrong about the history here).
Also issue llvm: libcxx in LLVM formula sometimes causes problems legacy-homebrew#47149 suggested it will cause problems to install libc++ with LLVM - and that issue is certainly related to this one. That issue suggested not installing libc++ in the OP but then went off in a different direction (to do with libc++abi vs libc++ which is a distinction I don't yet understand). And the first comment included this:

Since libc++ and libc++abi aren't installed by default, there's no immediate harm for most Homebrew users.

But now it is installed by default? When/why did that happen? Was it intentional?

Do you already know the answer? Why does homebrew install libc++ with LLVM by default?

mppf · 2022-03-16T13:51:06Z

I am almost certainly confused -DLLVM_ENABLE_LIBCXX vs -DLLVM_ENABLE_RUNTIMES including libcxx. Also https://github.com/Homebrew/linuxbrew-core/issues/13332 seems related.

carlocab · 2022-03-16T16:03:08Z

@carlocab - the reason that I have posted it here and not upstream is that I was unable to reproduce the issue when not doing a Homebrew build. (I.e. if I build and install LLVM somewhere myself, manually, outside of homebrew, everything works OK). AFAIK that is due to the -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk arguments that the Homebrew build is using. (AFAIK, this argument comes both from something in the cmake from the Homebrew formula and from superenv).

Which version of LLVM did you build? How did you configure it?

Additionally I wanted to ask if it was clear to you why building with headers from one version of a library and then linking with a different version of the same library is a problem, in general. I could explain that more / find a reference to link here, if not.

I'm aware that, in general, this is a recipe for ABI breakage. It just seems unlikely that this is what's happening here.

We know that this doesn't occur with LLVM 11 but does with LLVM 13, and you've suggested that Apple libc++ is ABI compatible with LLVM 11 libc++. This means that LLVM 13 has broken the libc++ ABI, and hasn't bumped the library version accordingly. This strikes me as extremely unlikely, even setting aside the fact that the lead libc++ developer works at Apple. Still, supposing this were the case: this would then really need to be reported upstream.

We also know that this problem does not go away with Homebrew LLVM 13 clang++. This is important because using Homebrew clang++ will make use of Homebrew libc++ headers when compiling your reproduction. (You can check by passing -H to your invocation of clang++.) Thus, the problem can be reproduced when using headers that match the linked library exactly.

We can also try the reverse exercise: use Homebrew LLVM 13 clang++ (and therefore LLVM 13 libc++ headers) but insist on linkage against Apple libc++. This configuration leads to the correct result:

❯ ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 1

If there was indeed ABI breakage in libc++ that is exposed by your example, then it's relatively likely that we should get the wrong result here. But that's not what we're seeing.

Right, that is one way to fix it. Including that, I can think of several ways that Homebrew could fix it:

as you said, it could build LLVM multiple times in order to create a clang that links with the built libc++. This solution makes the least sense to me of these options. However, I do not see why this would be a show-stopper. To me it looks like the formula already builds clang multiple times.

homebrew-core/Formula/llvm.rb

Lines 178 to 179 in 14af399

# We build LLVM a few times first for optimisations. See

# https://github.com/Homebrew/homebrew-core/issues/77975

package libc++ separately and install it to a separate directory. (Ubuntu does not do this; while libc++-13-dev and llvm-13-dev are separate packages, they both install libraries to /usr/lib/llvm-13/lib. But the issue is not a big deal on Ubuntu since by default the compilers will use the gnu libstdc++). This would address the problem because people would have to opt in to linking with the non-system libc++, somehow.

just don't provide a package that installs a different libc++ from the system one. AFAIK this is the approach FreeBSD uses - and that is relevant because FreeBSD is another system that uses this libc++ as the system C++ library.

At this stage, all of these solutions are show-stoppers because they're not the right solutions. They are predicated on a diagnosis that doesn't seem to be correct.

Relevant to the last option: Why does the LLVM formula install libc++ at all?

A few reasons:

Upstream ship it with their binary builds of LLVM.
Users have asked for it.
We aim to provide a feature-complete build of LLVM.

Those are just the one off the top of my head. There's likely more.

Does it need to continue to do that?

Yes, unless we have compelling reasons to do otherwise. I don't think we have any so far.

mppf · 2022-03-16T19:30:33Z

Which version of LLVM did you build? How did you configure it?

I built LLVM 12 with the Homebrew patches applied. I tried to make it as similar as possible to the Homebrew build but was having problems with the lldb build so disabled that. Here is the cmake command I used to configure it:

cmake ../llvm-project/llvm \
-DCMAKE_INSTALL_PREFIX=/usr/local/Cellar/llvm@12/12.0.1_1 \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_FIND_FRAMEWORK=LAST \
-DCMAKE_VERBOSE_MAKEFILE=ON \
-Wno-dev \
-DBUILD_TESTING=OFF \
-DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \
-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;mlir;polly;openmp" \
-DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi;libunwind" \
-DLLVM_POLLY_LINK_INTO_TOOLS=ON \
-DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON \
-DLLVM_LINK_LLVM_DYLIB=ON \
-DLLVM_ENABLE_EH=ON \
-DLLVM_ENABLE_FFI=ON \
-DLLVM_ENABLE_RTTI=ON \
-DLLVM_INCLUDE_DOCS=OFF \
-DLLVM_INCLUDE_TESTS=OFF \
-DLLVM_INSTALL_UTILS=ON \
-DLLVM_ENABLE_Z3_SOLVER=OFF \
-DLLVM_OPTIMIZED_TABLEGEN=ON \
-DLLVM_TARGETS_TO_BUILD=all \
-DLLDB_USE_SYSTEM_DEBUGSERVER=ON \
-DLLDB_ENABLE_PYTHON=ON \
-DLLDB_ENABLE_LUA=OFF \
-DLLDB_ENABLE_LZMA=ON \
-DLLDB_PYTHON_RELATIVE_PATH=libexec/python3.9/site-packages \
-DLIBOMP_INSTALL_ALIASES=OFF \
-DCLANG_PYTHON_BINDINGS_VERSIONS=3.9 \
-DLLVM_CREATE_XCODE_TOOLCHAIN=ON \
-DPACKAGE_VENDOR=Homebrew \
-DBUG_REPORT_URL=https://github.com/Homebrew/homebrew-core/issues \
-DCLANG_VENDOR_UTI=org.homebrew.clang \
-DFFI_INCLUDE_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi \
-DFFI_LIBRARY_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib \
-DLLVM_BUILD_LLVM_C_DYLIB=ON \
-DLLVM_ENABLE_LIBCXX=ON \
-DRUNTIMES_CMAKE_ARGS=-DCMAKE_INSTALL_RPATH=@loader_path/../lib \
-DDEFAULT_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \

We know that this doesn't occur with LLVM 11 but does with LLVM 13, and you've suggested that Apple libc++ is ABI compatible with LLVM 11 libc++.

I don't know that any version of Apple's libc++ is ABI compatible with any version of LLVM's libc++. It could be that there were minor problems that we never ran in to.

This means that LLVM 13 has broken the libc++ ABI, and hasn't bumped the library version accordingly. This strikes me as extremely unlikely, even setting aside the fact that the lead libc++ developer works at Apple. Still, supposing this were the case: this would then really need to be reported upstream.

Do you know how to query the library version? The one reported by otool -L I showed above is always 1 for the Homebrew-built libc++ but the Mac OS X libc++ has something that seems like a real version number (1200.3.0):

  /usr/local/opt/llvm/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)

Is this the same version number you are talking about?

We also know that this problem does not go away with Homebrew LLVM 13 clang++. This is important because using Homebrew clang++ will make use of Homebrew libc++ headers when compiling your reproduction. (You can check by passing -H to your invocation of clang++.) Thus, the problem can be reproduced when using headers that match the linked library exactly.

That is interesting. However I don't think it necessarily rules out header vs library version differences. In particular, the reproducer uses some of clang in a library (and links statically with it). That library is compiled as part of the Homebrew package and so it does not use the Homebrew libc++ today. So, in other words, to do that experiment properly, you would need a version of the Homebrew LLVM 13 clang libraries was compiled with the Homebrew LLVM 13 libc++ at include-time. I bet that will make the reproducer no longer show a problem. Is that an experiment you are able to try?

carlocab · 2022-03-16T23:14:17Z

I don't know that any version of Apple's libc++ is ABI compatible with any version of LLVM's libc++.

This seems to backpedal from your earlier claim that

Yes, my hypothesis here is that the Mac OS X libc++ happens to be compatible with the one from llvm@11 for the versions used here.

It could be that there were minor problems that we never ran in to.

It could, but this is still predicated on one of Apple or upstream LLVM breaking the ABI (both unlikely). In any case, any minor problems, if they exist, appear to be orthogonal to the issue here. They haven't been exposed previously, and are not exposed by your reproducer in LLVM 11.

Do you know how to query the library version? The one reported by otool -L I showed above is always 1 for the Homebrew-built libc++ but the Mac OS X libc++ has something that seems like a real version number (1200.3.0):
  /usr/local/opt/llvm/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)
Is this the same version number you are talking about?

What is important is the compatibility version, which is the same.

That is interesting. However I don't think it necessarily rules out header vs library version differences. In particular, the reproducer uses some of clang in a library (and links statically with it). That library is compiled as part of the Homebrew package and so it does not use the Homebrew libc++ today. So, in other words, to do that experiment properly, you would need a version of the Homebrew LLVM 13 clang libraries was compiled with the Homebrew LLVM 13 libc++ at include-time. I bet that will make the reproducer no longer show a problem. Is that an experiment you are able to try?

You would bet wrong.

❯ ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

This is built from LLVM 13.0.1 (llvm/llvm-project@75e33f7). I invoked CMake using

Details

cmake -S llvm -B testbuild \
    -DCMAKE_INSTALL_PREFIX=/tmp/llvm \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \
    -DLLVM_ENABLE_PROJECTS=clang \
    -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi" \
    -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON \
    -DLLVM_LINK_LLVM_DYLIB=ON \
    -DLLVM_ENABLE_EH=ON \
    -DLLVM_ENABLE_FFI=ON \
    -DLLVM_ENABLE_RTTI=ON \
    -DLLVM_INCLUDE_DOCS=OFF \
    -DLLVM_INCLUDE_TESTS=OFF \
    -DLLVM_ENABLE_Z3_SOLVER=OFF \
    -DLLVM_OPTIMIZED_TABLEGEN=ON \
    -DLLVM_TARGETS_TO_BUILD=Native \
    -DCLANG_PYTHON_BINDINGS_VERSIONS="3.10;3.9;3.8" \
    -DFFI_INCLUDE_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi \
    -DFFI_LIBRARY_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib \
    -DLLVM_BUILD_LLVM_C_DYLIB=ON \
    -DLLVM_ENABLE_LIBCXX=ON \
    -DRUNTIMES_CMAKE_ARGS=-DCMAKE_INSTALL_RPATH="@loader_path/../lib;/opt/homebrew/opt/llvm/lib" \
    -DDEFAULT_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \
    -DCMAKE_C_COMPILER=/opt/homebrew/opt/llvm/bin/clang \
    -DCMAKE_CXX_COMPILER=/opt/homebrew/opt/llvm/bin/clang++ \
    -DCMAKE_PREFIX_PATH=/opt/homebrew \
    -DCMAKE_INSTALL_RPATH="@loader_path/../lib;/opt/homebrew/opt/llvm/lib"

The object file was compiled with

clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/tmp/llvm/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS

and linked with

clang++ pretend-in-llvm.o -o a.out -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/tmp/llvm/lib -L/tmp/llvm/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM -L/tmp/llvm/lib

We can check that this links with the libc++ that we want:

❯ otool -L ./a.out
./a.out:
	@rpath/libLLVM.dylib (compatibility version 1.0.0, current version 13.0.1)
	@rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

❯ otool -l ./a.out | rg -A2 LC_RPATH
          cmd LC_RPATH
      cmdsize 32
         path /tmp/llvm/lib (offset 12)

It also loads our just-built libc++:

❯ DYLD_PRINT_LIBRARIES=1 ./a.out 1> /dev/null 2>&1 | rg 'libc[^A-Za-z]'
dyld[43551]: <355E121E-260B-3931-9825-E8033D109D3F> /private/tmp/llvm/lib/libc++.1.0.dylib
dyld[43551]: <B6B73FA9-4438-3B3D-A3B7-3B8CFA594B16> /opt/homebrew/Cellar/llvm/13.0.1_1/lib/libc++.1.0.dylib
dyld[43551]: <8F8963C1-403D-3282-AE5E-4F6E0E85F5A6> /usr/lib/libc++abi.dylib
dyld[43551]: <0EB3A986-B1BC-3F18-B7CC-51E2E5E00DC3> /usr/lib/libc++.1.dylib
dyld[43551]: <0D566764-7528-3C4B-9526-1C8DA0A9F59B> /opt/homebrew/Cellar/llvm/13.0.1_1/lib/libc++abi.1.0.dylib

This does still load Apple libc++, but I've already explained previously how there is no way to avoid this: every process you run loads Apple libc++. What is important is that it loads our just-built libc++ first.

We can also check that libLLVM has the right linkage:

❯ otool -L /tmp/llvm/lib/libLLVM.dylib
/tmp/llvm/lib/libLLVM.dylib:
	@rpath/libLLVM.dylib (compatibility version 1.0.0, current version 13.0.1)
	/usr/lib/libffi.dylib (compatibility version 1.0.0, current version 27.0.0)
	/usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/opt/homebrew/opt/ncurses/lib/libncursesw.6.dylib (compatibility version 6.0.0, current version 6.0.0)
	/usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version 10.9.0)
	/opt/homebrew/opt/llvm/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)

So, even if we compile libLLVM using Homebrew libc++ (headers included), we still get the wrong output with your reproducer.

To be sure, I did check that I could reproduce the same results as your initial example when using Homebrew LLVM (i.e. not one that I just built locally) and linking with Homebrew libc++ vs Apple libc++.

carlocab · 2022-03-17T04:10:02Z

To try to remove any doubt about mixing libc++ headers and libraries, I used my just-built clang to compile your example:

❯ /tmp/llvm/bin/clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/tmp/llvm/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS

❯ /tmp/llvm/bin/clang++ pretend-in-llvm.o -o a.out -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/tmp/llvm/lib -L/tmp/llvm/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM -L/tmp/llvm/lib

This should guarantee that I am using my just-installed libc++ headers and my just-built libc++.

Here's what I get:

❯ ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

mppf · 2022-03-17T12:20:01Z

OK, at this point you have more information than I do that would be useful to LLVM developers in an upstream bug report. In particular, the bug report to give them is ideally one that does not use Homebrew at all.

Wait, no, the cmake you did uses the Homebrew clang in order to get it to build with that libc++. So it is still not a demonstrator independent of Homebrew.

mppf · 2022-03-17T21:27:28Z

I've been able to create some scripts to reproduce that we can use as a bug report to LLVM. These scripts don't use Homebrew at all. (I haven't tried it on a system without Homebrew installed at all but I didn't have any llvm package installed).

runtest.sh

#!/bin/bash

# assumes ../llvm-project is checked out
#  bug report uses 75e33f71c2dae584b13a7d1186ae0a038ba98838 llvmorg-13.0.1
#
# will make ../llvm-build1 and ../llvm-build2
# will install to /tmp/llvm1 and /tmp/llvm2
./cmake1.sh
./build1.sh
./cmake2.sh
./build2.sh
./repro.sh

cmake1.sh

#!/bin/bash

mkdir -p ../llvm-build1
cd ../llvm-build1

cmake ../llvm-project/llvm \
    -DCMAKE_INSTALL_PREFIX=/tmp/llvm1 \
    -DCMAKE_BUILD_TYPE=RelWithDebInfo \
    -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \
    -DLLVM_ENABLE_PROJECTS=clang \
    -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi" \
    -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON \
    -DLLVM_LINK_LLVM_DYLIB=ON \
    -DLLVM_ENABLE_EH=ON \
    -DLLVM_ENABLE_FFI=ON \
    -DLLVM_ENABLE_RTTI=ON \
    -DLLVM_INCLUDE_DOCS=OFF \
    -DLLVM_INCLUDE_TESTS=OFF \
    -DLLVM_ENABLE_Z3_SOLVER=OFF \
    -DLLVM_OPTIMIZED_TABLEGEN=ON \
    -DLLVM_TARGETS_TO_BUILD=Native \
    -DCLANG_PYTHON_BINDINGS_VERSIONS="3.10;3.9;3.8" \
    -DFFI_INCLUDE_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi \
    -DFFI_LIBRARY_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib \
    -DLLVM_BUILD_LLVM_C_DYLIB=ON \
    -DLLVM_ENABLE_LIBCXX=ON \
    -DDEFAULT_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \

build1.sh

#!/bin/bash

export CMAKE_BUILD_PARALLEL_LEVEL=5
cd ../llvm-build1
cmake --build . $CMAKE_PARBUILD --target install-llvm-headers
cmake --build . $CMAKE_PARBUILD --target install-cmake-exports
cmake --build . $CMAKE_PARBUILD --target LLVMSupport
cmake --build . $CMAKE_PARBUILD --target install-LLVMSupport
cmake --build . $CMAKE_PARBUILD --target llvm-config
#cmake --build . --target install-llvm-config
#cmake --build . --target clang
#cmake --build . --target install-clang
cmake --build . $CMAKE_PARBUILD --target install

cmake2.sh

#!/bin/bash

mkdir -p ../llvm-build2
cd ../llvm-build2

# carlocab's reproducer

cmake ../llvm-project/llvm \
    -DCMAKE_INSTALL_PREFIX=/tmp/llvm2 \
    -DCMAKE_BUILD_TYPE=RelWithDebInfo \
    -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \
    -DLLVM_ENABLE_PROJECTS=clang \
    -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi" \
    -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON \
    -DLLVM_LINK_LLVM_DYLIB=ON \
    -DLLVM_ENABLE_EH=ON \
    -DLLVM_ENABLE_FFI=ON \
    -DLLVM_ENABLE_RTTI=ON \
    -DLLVM_INCLUDE_DOCS=OFF \
    -DLLVM_INCLUDE_TESTS=OFF \
    -DLLVM_ENABLE_Z3_SOLVER=OFF \
    -DLLVM_OPTIMIZED_TABLEGEN=ON \
    -DLLVM_TARGETS_TO_BUILD=Native \
    -DCLANG_PYTHON_BINDINGS_VERSIONS="3.10;3.9;3.8" \
    -DFFI_INCLUDE_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi \
    -DFFI_LIBRARY_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib \
    -DLLVM_BUILD_LLVM_C_DYLIB=ON \
    -DLLVM_ENABLE_LIBCXX=ON \
    -DRUNTIMES_CMAKE_ARGS=-DCMAKE_INSTALL_RPATH="@loader_path/../lib;/tmp/llvm1/lib" \
    -DDEFAULT_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \
    -DCMAKE_C_COMPILER=/tmp/llvm1/bin/clang \
    -DCMAKE_CXX_COMPILER=/tmp/llvm1/bin/clang++ \
    -DCMAKE_PREFIX_PATH=/tmp/llvm1 \
    -DCMAKE_INSTALL_RPATH="@loader_path/../lib;/tmp/llvm1/lib"

exit 0

build2.sh

#!/bin/bash

export CMAKE_BUILD_PARALLEL_LEVEL=5
cd ../llvm-build2
cmake --build . $CMAKE_PARBUILD --target install-llvm-headers
cmake --build . $CMAKE_PARBUILD --target install-cmake-exports
cmake --build . $CMAKE_PARBUILD --target LLVMSupport
cmake --build . $CMAKE_PARBUILD --target install-LLVMSupport
cmake --build . $CMAKE_PARBUILD --target llvm-config
#cmake --build . --target install-llvm-config
#cmake --build . --target clang
#cmake --build . --target install-clang
cmake --build . $CMAKE_PARBUILD --target install

repro.sh

#!/bin/bash

/tmp/llvm2/bin/clang++ -c repro.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/tmp/llvm2/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/local/include

/tmp/llvm2/bin/clang++ repro.o -o a.out -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/tmp/llvm2/lib -L/tmp/llvm2/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM -L/usr/local/lib

./a.out

# This adjustment to the link line allows the test to pass,
# for some reason.
#/tmp/llvm2/bin/clang++ repro.o -o a.out -L/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/tmp/llvm2/lib -L/tmp/llvm2/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM -L/usr/local/lib

repro.cpp

#include "llvm/Support/Errc.h"
#include "clang/Basic/FileManager.h"

#include <iostream>

int main() {
  clang::FileManager fileMgr((clang::FileSystemOptions()));

  auto file = fileMgr.getFileRef("./nonexistant.h", /*OpenFile=*/true);

  std::error_code EC = llvm::errorToErrorCode(file.takeError());

  std::cout << "EC.message is " << EC.message() << "\n";

  bool not_no_such = (EC != llvm::errc::no_such_file_or_directory);
  bool is_no_such = !not_no_such;

  std::cout << "EC == no_such_file_or_directory is " << is_no_such << "\n";

  if (!is_no_such) {
    std::cout << "FAIL\n";
  }

  return 0;
}

mppf · 2022-03-18T13:35:48Z

I have created this llvm project issue: llvm/llvm-project#54438

kencu · 2022-04-02T05:25:54Z

NB:

% otool -L a.out
a.out:
	/usr/local/opt/llvm/lib/libLLVM.dylib (compatibility version 1.0.0, current version 13.0.1)
	/usr/local/opt/llvm/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)

% ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

% install_name_tool -change /usr/lib/libc++.1.dylib  /usr/local/opt/llvm/lib/libc++.1.dylib /usr/local/opt/llvm/lib/libLLVM.dylib

% ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 1

% install_name_tool -change /usr/local/opt/llvm/lib/libc++.1.dylib /usr/lib/libc++.1.dylib /usr/local/opt/llvm/lib/libLLVM.dylib

% ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

Mixing libc++ versions is unsupported upstream.

carlocab · 2022-04-02T05:52:35Z

@kencu As I mentioned in the discussion above, I don't think this is a consequence of mixing libc++ versions.

You will get the erroneous behaviour if you build libc++ first and then link libLLVM against your just-built libc++.

kencu · 2022-04-02T07:03:22Z

If you see what I did with install_name_tool, you can make the error come and go at will just by by forcing mixing or not mixing of libc++ versions…

kencu · 2022-04-02T07:15:18Z

Of course, it’s possible some other factor could be involved too…toolchains are like that!

github-actions · 2022-06-02T00:24:12Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

bradcray · 2022-06-02T00:55:51Z

Thanks for keeping this alive, @Bo98!

carlocab · 2022-08-18T04:54:04Z

I am exploring solutions to this at #106925.

carlocab · 2022-08-21T17:22:01Z

I experimented with removing the libc++ and libc++abi builds from the llvm formula. Results are at https://github.com/Homebrew/homebrew-core/actions/runs/2897386665.

A few notes:

20+ other formulae link with LLVM's libc++ instead of the one in /usr/lib. I'm still suspicious that Chapel is the only one reporting issues with this.
Not shipping libc++[abi] breaks the Catalina build. More specifically, it fails at building compiler-rt. I already have the right flags to pass to the build to make it succeed, but this also means that users must pass these flags in order to use the compiler that ships with the llvm formula. That effectively means shipping a broken clang, unless we add some other fix that makes it work out-of-the box. Some workarounds suggested in this thread.

kencu · 2022-08-21T17:33:16Z

although those projects/runtimes (libcxx/libcxxabi) need to be enabled for the build, there are toggles to allow you to install the libraries or not:

https://github.com/llvm/llvm-project/blob/d9ff670330a80f2d597822bfd574842ad8fc55b3/libcxx/CMakeLists.txt#L157

https://github.com/llvm/llvm-project/blob/d9ff670330a80f2d597822bfd574842ad8fc55b3/libcxxabi/CMakeLists.txt#L80

you probably already knew this, of course.

carlocab · 2022-08-21T18:44:36Z

Yes, I vaguely recalled some LIBCXX_INSTALL_* flags, but I wasn't sure what the exact form was.

However: if we don't want to link with different versions of libc++, then we probably don't want to mix headers from one but use the library of the other either.

kencu · 2022-08-21T19:23:29Z

we at the unnamed always install and use the latest headers that match the compiler version, but (usually) don’t install libcxx/libcxxabi. It has worked generally well for us; you might find the same. MacOS is designed to deploy on older targets, so the newer headers have guards in them for the capabilities of the libcxx you are deploying to.

this install technique was originally set up by one of the Apple compiler engineers who previously was involved way back when — I just continued that along. At one point I installed a newer version of libcxx/libcxxabi as well that can be specifically optionally used during builds on older systems, but there are conflicts sometimes as you have found too…

best of luck! you’re doing great work here!

mppf · 2022-08-28T00:12:24Z

@carlocab -

I'm still suspicious that Chapel is the only one reporting issues with this.

I just wanted to point out some ways that the Chapel package might be unique:

we use libclang and llvm instead of just llvm or just libclang
we use llvm-config to find llvm dependencies (vs cmake stuff)

bradcray · 2022-08-30T00:39:14Z

20+ other formulae link with LLVM's libc++ instead of the one in /usr/lib. I'm still suspicious that Chapel is the only one reporting issues with this.

I just wanted to point out some ways that the Chapel package might be unique...

@carlocab: Following up on Michael's hypothesis, could you share the list of 20+ formulae you're referring to above? I'm curious whether (through inspection or reasoning or inquiries) we could confirm or deny the guess.

carlocab · 2022-08-30T06:53:07Z

Following up on Michael's hypothesis, could you share the list of 20+ formulae you're referring to above? I'm curious whether (through inspection or reasoning or inquiries) we could confirm or deny the guess.

Here's a list:

c2rust
castxml
ccls
chapel
clazy
crystal
dpp
emacs-clang-complete-async
enzyme
faust
halide
include-what-you-use
ispc
ldc
mesa
nvc
odin
pocl
pyside
pyside@2
qt
qt@5
rtags
spirv-llvm-translator
standardese

A number of these also link with libclang-cpp, but I haven't checked them all to see if none link with libclang.

we use libclang and llvm instead of just llvm or just libclang

we use llvm-config to find llvm dependencies (vs cmake stuff)

I notice that brew linkage (or, equivalently, otool) does not report linkage with libclang. Does Chapel dlopen libclang? If so, that may explain the issues you're seeing.

Why would using llvm-config make a difference?

mppf · 2022-08-30T15:53:26Z

Another odd thing about Chapel is that we have some relatively strange workarounds for this issue...

I notice that brew linkage (or, equivalently, otool) does not report linkage with libclang. Does Chapel dlopen libclang? If so, that may explain the issues you're seeing.

Why would using llvm-config make a difference?

We are statically linking with clang and LLVM libraries on Mac OS X because of bugs in llvm-config on Mac OS X.

The static linking is a workaround. From a PR message talking about it ( chapel-lang/chapel#19272 ) :

Note that we statically link both LLVM and clang on Mac OS X (since PR #18727). This PR leaves it this way. Historically, there have been problems with upstream LLVM and Homebrew in building a dynamic library that works with llvm-config (in particular the versioned file, like libLLVM-11.dylib, is missing). See this LLVM bug report. I expect that we will be able to allow dynamic linking on Mac OS X once we are able to establish that the issue is resolved.

It's my understanding that projects using cmake to find clang do not have this issue, but I could be wrong. So, perhaps statically linking with LLVM and clang libraries something strange that we are doing that other projects do not.

carlocab · 2022-08-30T16:39:26Z

Not sure if doing away with static linking will help, but you can definitely do that with llvm@1{2,3,4}:

❯ /usr/local/opt/llvm/bin/llvm-config --link-shared; echo $?
0
❯ /usr/local/opt/llvm/bin/llvm-config --libs
-lLLVM-14
❯ /usr/local/opt/llvm/bin/llvm-config --libfiles
/usr/local/Cellar/llvm/14.0.6_1/lib/libLLVM-14.dylib

In particular, you should no longer see the issue reported at llvm/llvm-project#39599 using a recent enough LLVM packaged with Homebrew.

mppf · 2022-08-30T16:52:12Z

I think a good next step would be to see if the reproducer in this issues original post still has the problem if it is changed to dynamically link.

mppf · 2022-08-31T13:34:20Z

I've reproduced the problem in the OP with static linking and the current LLVM 14 Homebrew package on my system:

$ clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/usr/local/Cellar/llvm/14.0.6_1/include -std=c++14 -stdlib=libc++   -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
$ clang++ pretend-in-llvm.o -o a.out -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/usr/local/Cellar/llvm/14.0.6_1/lib -L/usr/local/Cellar/llvm/14.0.6_1/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-14 -L/usr/local/lib
$ ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

(it is supposed to print out "is 1").

However, I'm still seeing the problem with dynamic linking:

$ clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/usr/local/Cellar/llvm/14.0.6_1/include -std=c++14 -stdlib=libc++   -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
$ clang++ pretend-in-llvm.o -o a.out -lclang-cpp -Wl,-rpath,/usr/local/Cellar/llvm/14.0.6_1/lib -L/usr/local/Cellar/llvm/14.0.6_1/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-14 -L/usr/local/lib
$ ./a.out                               (git)-[remotes/upstream/main] 
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

The workaround (adding -L/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib early in the link command) still seems to work.

So, unfortunately, it does not seem that switching Chapel to dynamic linking will help with this problem. Nonetheless, I'd like to look at doing that anyway, to make it less unique in this regard, and to better match what we do on linux.

carlocab · 2022-09-19T09:06:40Z

Chapel should no longer be linking with LLVM libc++. From the latest version of Chapel distributed in Homebrew (1.28.0_1):

System libraries:
  /usr/lib/libSystem.B.dylib
  /usr/lib/libc++.1.dylib
  /usr/lib/libncurses.5.4.dylib
Homebrew libraries:
  /usr/local/opt/gmp/lib/libgmp.10.dylib (gmp)
  /usr/local/opt/llvm@14/lib/libLLVM.dylib (llvm@14)
  /usr/local/opt/llvm@14/lib/libclang-cpp.dylib (llvm@14)

Compare this with the bottle built at #110783:

System libraries:
  /usr/lib/libSystem.B.dylib
  /usr/lib/libc++.1.dylib
  /usr/lib/libncurses.5.4.dylib
Homebrew libraries:
  /usr/local/opt/gmp/lib/libgmp.10.dylib (gmp)
  /usr/local/opt/llvm/lib/libLLVM.dylib (llvm)
  /usr/local/opt/llvm/lib/libc++.1.dylib (llvm)
  /usr/local/opt/llvm/lib/libclang-cpp.dylib (llvm)

Incidentally, this may be what distinguishes Chapel from the other formulae that use LLVM libc++ referenced above: I don't think any of them have brew linkage reporting both of /usr/lib/libc++.1.dylib and /usr/local/opt/llvm/lib/libc++.1.dylib.

Incidentally, something weird I noticed, in the ARM bottles:

System libraries:
  /usr/lib/libSystem.B.dylib
  /usr/lib/libc++.1.dylib
Homebrew libraries:
  /opt/homebrew/opt/gmp/lib/libgmp.10.dylib (gmp)
  /opt/homebrew/opt/llvm/lib/libLLVM.dylib (llvm)
  /opt/homebrew/opt/llvm/lib/libc++.1.dylib (llvm)
  /opt/homebrew/opt/llvm/lib/libclang-cpp.dylib (llvm)

There is no linkage with ncurses, whereas the Intel bottles have it.

Not sure if this is an issue.

mppf · 2022-09-19T14:21:48Z

@carlocab - that's great, thanks. I see that #106925 went in but I'm not understanding what happened to fix the problem. Could you summarize? Thanks. Also I am understanding that we should be able to remove the workaround now from Chapel, so please speak up if I'm misunderstanding.

Regarding the ncurses linkage, ncurses itself is not a direct dependency for Chapel. And, when I check a local build or the Homebrew chpl on an Intel Mac with otool -L (which has different format from what you show above), I'm not seeing it. I see that the Homebrew LLVM 14 libLLVM.dylib itself depends on ncurses, though. Maybe whatever produced the output you showed above is including the transitive dependency? (If that is the case, perhaps the ncurses dependency was dropped for M1 when LLVM 14 went from llvm to llvm@14 in #106925, for some reason).

mppf · 2022-09-19T14:32:09Z

I can confirm that the reproducer in this issue is no longer misbehaving for me with the updated LLVM 14 or 15 Homebrew packages.

laptop ~ % clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/usr/local/Cellar/llvm@14/14.0.6/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/local/include
laptop ~ % clang++ pretend-in-llvm.o -o a.out -lclang-cpp -Wl,-rpath,/usr/local/Cellar/llvm@14/14.0.6/lib -L/usr/local/Cellar/llvm@14/14.0.6/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-14 -L/usr/local/lib     
laptop ~ % ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 1
laptop ~ % clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/usr/local/Cellar/llvm/15.0.0/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/local/include
laptop ~ % clang++ pretend-in-llvm.o -o a.out -lclang-cpp -Wl,-rpath,/usr/local/Cellar/llvm/15.0.0/lib -L/usr/local/Cellar/llvm/15.0.0/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-15 -L/usr/local/lib
laptop ~ % ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 1

carlocab · 2022-09-19T15:20:21Z

I see that #106925 went in but I'm not understanding what happened to fix the problem. Could you summarize?

That's this one: d3999fc (plus a little extra here, since I am bad at atomic commits)

brew info llvm will also give you another hint.

Also I am understanding that we should be able to remove the workaround now from Chapel, so please speak up if I'm misunderstanding.

I think so; if by workaround you mean what you have in place to try to avoid linkage with both LLVM libc++ and Apple libc++.

Regarding the ncurses linkage, ncurses itself is not a direct dependency for Chapel. And, when I check a local build or the Homebrew chpl on an Intel Mac with otool -L (which has different format from what you show above), I'm not seeing it. I see that the Homebrew LLVM 14 libLLVM.dylib itself depends on ncurses, though. Maybe whatever produced the output you showed above is including the transitive dependency? (If that is the case, perhaps the ncurses dependency was dropped for M1 when LLVM 14 went from llvm to llvm@14 in #106925, for some reason).

Yea, maybe, but all our builds of libLLVM (14, 15, ARM, Intel) link with ncurses, so not really sure what's going on there.

See, for example, the output of brew linkage on Monterey ARM, and Monterey Intel for both LLVM 14 and 15.

mppf · 2022-09-19T15:57:49Z

I see that #106925 went in but I'm not understanding what happened to fix the problem. Could you summarize?

That's this one: d3999fc (plus a little extra here, since I am bad at atomic commits)

brew info llvm will also give you another hint.

OK, so the LLVM libc++ is being installed into a different directory from the LLVM install (e.g. /usr/local/opt/llvm/lib/c++ vs just /usr/local/opt/llvm/lib).

@daviditen

Remove workaround for Homebrew libc++ issue This PR removes a workaround for issue #19217 (which is the same issue as Homebrew/homebrew-core#96915 and llvm/llvm-project#54438 ). The problem is solved in the current Homebrew LLVM 14 and LLVM 15 formulae now that libc++ is installed into a different directory than the LLVM libraries. (This problem would still be present for the LLVM 12 and 13 formulae). Since the workaround is removed, this PR adjusts printchplenv to raise an error if using Mac OS X with system LLVM 11, 12, or 13. Such LLVMs will not be selected as a quickstart configuration. Also, it updates prereqs.rst to indicate that LLVM 14 or newer is required on Mac OS X. Reviewed by @daviditen - thanks! - [x] testing on Mac OS X 10.15 "Catalina" - [x] testing on Mac OS X 11 "Big Sur" - [x] testing on Mac OS X 12 "Monterey" (x86) - [x] testing on Mac OS X 12 "Monterey" (M1)

mppf added the bug Reproducible Homebrew/homebrew-core bug label Mar 15, 2022

mppf mentioned this issue Mar 15, 2022

llvm@12 issues on mac chapel-lang/chapel#19217

Closed

carlocab added the upstream issue An upstream issue report is needed label Mar 15, 2022

This comment was marked as spam.

Sign in to view

This comment was marked as resolved.

Sign in to view

mppf changed the title ~~Linking with clang library in the obvious way leads to mixing libc++ versions~~ Problem when linking to clang and getting the built libc++ Mar 17, 2022

mppf changed the title ~~Problem when linking to clang and getting the built libc++~~ Problem when linking to clang and getting the Homebrew libc++ Mar 17, 2022

This comment was marked as off-topic.

Sign in to view

mppf mentioned this issue Mar 18, 2022

problem using clang/llvm libraries on Mac OS X with a built libc++ llvm/llvm-project#54438

Closed

mppf mentioned this issue Apr 1, 2022

chapel 1.26.0 pre-release #98111

Closed

6 tasks

github-actions bot removed the stale No recent activity label May 11, 2022

github-actions bot added the stale No recent activity label Jun 2, 2022

Bo98 added in progress Stale bot should stay away and removed stale No recent activity labels Jun 2, 2022

kjelljorner mentioned this issue Jun 4, 2022

Build error for lfortran fortran-lang/homebrew-fortran#3

Open

cho-m mentioned this issue Jun 24, 2022

deno 1.23.1 #102394

Closed

carlocab closed this as completed Sep 19, 2022

mppf mentioned this issue Sep 19, 2022

Remove workaround for Homebrew libc++ issue chapel-lang/chapel#20698

Merged

4 tasks

github-actions bot added the outdated PR was locked due to age label Oct 20, 2022

github-actions bot locked as resolved and limited conversation to collaborators Oct 20, 2022

Problem when linking to clang and getting the Homebrew libc++ #96915

Problem when linking to clang and getting the Homebrew libc++ #96915

Comments

mppf commented Mar 15, 2022 • edited Loading

brew gist-logs <formula> link OR brew config AND brew doctor output

Verification

What were you trying to do (and why)?

What happened (include all command output)?

What did you expect to happen?

Step-by-step reproduction instructions (by running brew commands)

carlocab commented Mar 15, 2022

mppf commented Mar 15, 2022 • edited Loading

carlocab commented Mar 15, 2022

mppf commented Mar 15, 2022

Bo98 commented Mar 15, 2022

mppf commented Mar 15, 2022

carlocab commented Mar 15, 2022

mppf commented Mar 16, 2022 • edited Loading

mppf commented Mar 16, 2022

carlocab commented Mar 16, 2022 • edited Loading

mppf commented Mar 16, 2022 • edited Loading

This comment was marked as spam.

This comment was marked as resolved.

carlocab commented Mar 16, 2022 • edited Loading

carlocab commented Mar 17, 2022

mppf commented Mar 17, 2022 • edited Loading

This comment was marked as off-topic.

mppf commented Mar 17, 2022

mppf commented Mar 18, 2022 • edited Loading

kencu commented Apr 2, 2022

carlocab commented Apr 2, 2022

kencu commented Apr 2, 2022

kencu commented Apr 2, 2022

github-actions bot commented Jun 2, 2022

bradcray commented Jun 2, 2022

carlocab commented Aug 18, 2022

carlocab commented Aug 21, 2022 • edited Loading

kencu commented Aug 21, 2022

carlocab commented Aug 21, 2022

kencu commented Aug 21, 2022

mppf commented Aug 28, 2022

bradcray commented Aug 30, 2022

carlocab commented Aug 30, 2022

mppf commented Aug 30, 2022

carlocab commented Aug 30, 2022 • edited Loading

mppf commented Aug 30, 2022

mppf commented Aug 31, 2022

carlocab commented Sep 19, 2022

mppf commented Sep 19, 2022

mppf commented Sep 19, 2022

carlocab commented Sep 19, 2022 • edited Loading

mppf commented Sep 19, 2022

mppf commented Mar 15, 2022 •

edited

Loading

`brew gist-logs <formula>` link OR `brew config` AND `brew doctor` output

Step-by-step reproduction instructions (by running `brew` commands)

mppf commented Mar 15, 2022 •

edited

Loading

mppf commented Mar 16, 2022 •

edited

Loading

carlocab commented Mar 16, 2022 •

edited

Loading

mppf commented Mar 16, 2022 •

edited

Loading

carlocab commented Mar 16, 2022 •

edited

Loading

mppf commented Mar 17, 2022 •

edited

Loading

mppf commented Mar 18, 2022 •

edited

Loading

carlocab commented Aug 21, 2022 •

edited

Loading

carlocab commented Aug 30, 2022 •

edited

Loading

carlocab commented Sep 19, 2022 •

edited

Loading