C++ modules appear to be exceedignly strict with intrinsic headers

[chriselrod]

That is, code using immintrin.h tends to fail to compile when using modules while working fine with headers.
I'll try to produce a minimal example in the next few hours.
For now, I have an example using boost_unordered.
When problems showed up in my own code using intrinsics, I could generally fix it by declaring all arguments as variables, and then passing the lvalues to the intriinsic function.

Hello.cxxm:

#ifndef USE_HEADERS
module;
#endif
#include <boost/unordered/unordered_flat_map.hpp>
#include <iostream>

#ifndef USE_HEADERS
export module Hello;
export {
#endif
  void hello() { std::cout << "Hello World!\n"; }
  template <typename K, typename V> using map = boost::unordered_flat_map<K, V>;
#ifndef USE_HEADERS
}
#endif

user.cpp:

#ifndef USE_HEADERS
import Hello;
#else
#include "hello.cxxm"
#endif

int main() {
  hello();
  int x = 0;
  long y = 0;
  map<int*,long*> m;
  m[&x] = &y;
  [[maybe_unused]] auto f = m.find(&x);
  return 0;
}

Compiling with headers:

$ clang++ -std=c++23 use.cpp -DUSE_HEADERS -o Hello.out
$ ./Hello.out
Hello World!

With modules:

$ clang++ -std=c++23 --precompile hello.cxxm -o M-hello.pcm
$ clang++ -std=c++23 use.cpp -fmodule-file=Hello=M-hello.pcm M-hello.pcm -o Hello_mod.out

results in

/usr/include/boost/unordered/detail/foa/core.hpp:293:7: error: no matching function for call to '_mm_cmpeq_epi8'
  293 |       _mm_cmpeq_epi8(load_metadata(),_mm_setzero_si128()))&0x7FFF;
      |       ^~~~~~~~~~~~~~
/usr/include/boost/unordered/detail/foa/core.hpp:309:14: note: in instantiation of member function 'boost::unordered::detail::foa::group15<boost::unordered::detail::foa::plain_integral>::match_available' requested here
  309 |     return (~match_available())&0x7FFF;

I could file this as a boost_unordered issue or make a PR there, as I've generally found I can work around the problem.
But I'll see about creating a minimal reproducer using #include <immintrin.h> directlry that works with headers but fails with modules.

[llvmbot]

@llvm/issue-subscribers-clang-modules

Author: Chris Elrod (chriselrod)

That is, code using `immintrin.h` tends to fail to compile when using modules while working fine with headers. I'll try to produce a minimal example in the next few hours. For now, I have an example using boost_unordered. When problems showed up in my own code using intrinsics, I could generally fix it by declaring all arguments as variables, and then passing the lvalues to the intriinsic function.

Hello.cxxm:

#ifndef USE_HEADERS
module;
#endif
#include <boost/unordered/unordered_flat_map.hpp>
#include <iostream>

#ifndef USE_HEADERS
export module Hello;
export {
#endif
  void hello() { std::cout << "Hello World!\n"; }
  template <typename K, typename V> using map = boost::unordered_flat_map<K, V>;
#ifndef USE_HEADERS
}
#endif

user.cpp:

#ifndef USE_HEADERS
import Hello;
#else
#include "hello.cxxm"
#endif

int main() {
  hello();
  int x = 0;
  long y = 0;
  map<int*,long*> m;
  m[&x] = &y;
  [[maybe_unused]] auto f = m.find(&x);
  return 0;
}

Compiling with headers:

$ clang++ -std=c++23 use.cpp -DUSE_HEADERS -o Hello.out
$ ./Hello.out
Hello World!

With modules:

$ clang++ -std=c++23 --precompile hello.cxxm -o M-hello.pcm
$ clang++ -std=c++23 use.cpp -fmodule-file=Hello=M-hello.pcm M-hello.pcm -o Hello_mod.out

results in

/usr/include/boost/unordered/detail/foa/core.hpp:293:7: error: no matching function for call to '_mm_cmpeq_epi8'
  293 |       _mm_cmpeq_epi8(load_metadata(),_mm_setzero_si128()))&0x7FFF;
      |       ^~~~~~~~~~~~~~
/usr/include/boost/unordered/detail/foa/core.hpp:309:14: note: in instantiation of member function 'boost::unordered::detail::foa::group15<boost::unordered::detail::foa::plain_integral>::match_available' requested here
  309 |     return (~match_available())&0x7FFF;

I could file this as a boost_unordered issue or make a PR there, as I've generally found I can work around the problem.
But I'll see about creating a minimal reproducer using #include <immintrin.h> directlry that works with headers but fails with modules.

[chriselrod]

Here is the minimal example using immintrin.h:

hello.cxxm:

#ifndef USE_HEADERS
module;
#endif

#include <concepts>
#include <immintrin.h>
#include <iostream>

#ifndef USE_HEADERS
export module Hello;
export {
#endif
  void hello() { std::cout << "Hello World!\n"; }
  template <typename T> auto vload128(const T *p) {
    if constexpr (std::same_as<T, float>) {
      return _mm_loadu_ps(p);
    } else if constexpr (std::same_as<T, double>) {
      return _mm_loadu_pd(p);
    } else {
      return _mm_loadu_si128(reinterpret_cast<const __m128i *>(p));
    }
  }
#ifndef USE_HEADERS
}
#endif

use.cpp:

#ifndef USE_HEADERS
import Hello;
#else
#include "hello.cxxm"
#endif

int main() {
  hello();
  float x[4];
  [[maybe_unused]] auto v = vload128(x);
  return 0;
}

Compiling with headers:

$ clang++ -std=c++23 use.cpp -DUSE_HEADERS -o Hello.
out
$ ./Hello.out
Hello World!

Compiling with modules:

$ clang++ -std=c++23 --precompile hello.cxxm -o M-he
llo.pcm
$ clang++ -std=c++23 use.cpp -fmodule-file=Hello=M-hell
o.pcm M-hello.pcm -o Hello_mod.out
In file included from use.cpp:2:
/home/chriselrod/Documents/progwork/cxx/experiments/modules/hello.cxxm:16:14: error: no matching function for call to '_mm_loadu_ps'
   16 |       return _mm_loadu_ps(p);
      |              ^~~~~~~~~~~~
use.cpp:10:29: note: in instantiation of function template specialization 'vload128<float>' requested here
   10 |   [[maybe_unused]] auto v = vload128(x);
      |                             ^
1 error generated.

[chriselrod]

using something like

template <typename T> auto vload128(const T *p) {
  if constexpr (std::same_as<T, float>) {
    const float *fp = p;
    return _mm_loadu_ps(fp);
  } else if constexpr (std::same_as<T, double>) {
    const double *dp = p;
    return _mm_loadu_pd(dp);
  } else {
    const __m128i *ip = reinterpret_cast<const __m128i *>(p);
    return _mm_loadu_si128(ip);
  }
}

instead allows it to compile, even though fp and dp should have the same type as p.

[chriselrod]

Another workaround is to declare explicit instantiations of the template within the module that defines it.

[ChuanqiXu9]

I can't reproduce this in my local environment with trunk (0182f51).

My standard library is libstdc++ 10.2 (this may not be relevent) in linux.

Can you try again with trunk?

[jiixyj]

I just stumbled across this bug as well, upgrading from a clang trunk from around end of June to this commit: ChuanqiXu9/clangd-for-modules@e583176. I'm also using <boost/unordered/unordered_flat_map.hpp> in my code.

I managed to bisect it to 91d40ef . Maybe something in that commit inadvertently touched some code related to name lookup or module visibility?

It is a bit weird though that the date of that commit is after the report date of this bug...

[jiixyj]

I managed to work around it for now by re-applying 91d40ef on top of ChuanqiXu9/clangd-for-modules@e583176 . I guess reapplying it on main should also work.

[jiixyj]

After reapplying 91d40ef , the reproducer from here now fails again, but luckily my code doesn't seem to trigger that case.

[ChuanqiXu9]

@jiixyj I've relanded #102287. Can you verify if this is still a problem?

[jiixyj]

@ChuanqiXu9 : Sadly, with 3c9e345 I still get the errors when including <boost/unordered/unordered_flat_map.hpp>:

/home/jan/.conan2/p/b/boost77375cc29ad40/p/include/boost/unordered/detail/foa/core.hpp:291:12: error: no matching function for call to '_mm_movemask_epi8'
  291 |     return _mm_movemask_epi8(
      |            ^~~~~~~~~~~~~~~~~

and

/home/jan/.conan2/p/b/boost77375cc29ad40/p/include/boost/unordered/detail/foa/core.hpp:1070:36: error: no matching function for call to 'size_index_for'                                        
 1070 |     auto         groups_size_index=size_index_for<group_type,size_policy>(n);                                                                                                           
      |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                                                                               

...and some more similar ones.

I looked at the differences between the original PR and the relanded version. It turns out if I make Decl::isInAnotherModuleUnit() look like the following, the errors go away:

bool Decl::isInAnotherModuleUnit() const {
  auto *M = getOwningModule();

#if 1
  if (!M || !M->isNamedModule())
    return false;
#else
  if (!M)
    return false;

  // FIXME or NOTE: maybe we need to be clear about the semantics
  // of clang header modules. e.g., if this lives in a clang header
  // module included by the current unit, should we return false
  // here?
  //
  // This is clear for header units as the specification says the
  // header units live in a synthesised translation unit. So we
  // can return false here.
  M = M->getTopLevelModule();
  if (!M->isNamedModule())
    return false;
#endif

  return M != getASTContext().getCurrentNamedModule();
}

The code between #else and #endif is the current code in the main branch.