[clang] Fix assertion failure in injectEmbedTokens

[Fznamznon]

It seems for C++ lexer has some caching ability which doesn't expect injecting "new" tokens in the middle of the cache. Technically #embed tokens are not completely new since they have already been read from the file and there was #embed annotation token in this place, so set reinject flag for them to silence assertion.

[llvmbot]

@llvm/pr-subscribers-clang

Author: Mariya Podchishchaeva (Fznamznon)

Changes

It seems for C++ lexer has some caching ability which doesn't expect injecting "new" tokens in the middle of the cache. Technically #embed tokens are not completely new since they have already been read from the file and there was #embed annotation token in this place, so set reinject flag for them to silence assertion.

---

Full diff: https://github.com/llvm/llvm-project/pull/99624.diff

2 Files Affected:

• (modified) clang/lib/Parse/ParseExpr.cpp (+1-1)
• (modified) clang/test/Preprocessor/embed_weird.cpp (+8)

diff --git a/clang/lib/Parse/ParseExpr.cpp b/clang/lib/Parse/ParseExpr.cpp
index 3d7c58e5b3c3c..a12c375c8d48c 100644
--- a/clang/lib/Parse/ParseExpr.cpp
+++ b/clang/lib/Parse/ParseExpr.cpp
@@ -3597,7 +3597,7 @@ void Parser::injectEmbedTokens() {
     I += 2;
   }
   PP.EnterTokenStream(std::move(Toks), /*DisableMacroExpansion=*/true,
-                      /*IsReinject=*/false);
+                      /*IsReinject=*/true);
   ConsumeAnyToken(/*ConsumeCodeCompletionTok=*/true);
 }
 
diff --git a/clang/test/Preprocessor/embed_weird.cpp b/clang/test/Preprocessor/embed_weird.cpp
index cc73a88e5a657..f533230111b43 100644
--- a/clang/test/Preprocessor/embed_weird.cpp
+++ b/clang/test/Preprocessor/embed_weird.cpp
@@ -115,3 +115,11 @@ void f1() {
   };
 }
 #endif
+
+static_assert(_Generic(
+#embed __FILE__ limit(1)
+  , int : 1, default : 0));
+
+static_assert(alignof(typeof(
+#embed __FILE__ limit(1)
+)) == alignof(int));

[llvm-ci]

LLVM Buildbot has detected a new failure on builder sanitizer-x86_64-linux running on sanitizer-buildbot1 while building clang at step 2 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/66/builds/1938

Here is the relevant piece of the build log for the reference:

Step 2 (annotate) failure: 'python ../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_selector.py' (failure) (timed out)
...
[364/371] Generating FuzzerUtils-x86_64-Test
[365/371] Generating FuzzerTestObjects.gtest-all.cc.x86_64.o
[366/371] Generating Fuzzer-x86_64-Test
[367/371] Generating MSAN_INST_TEST_OBJECTS.msan_test.cpp.x86_64-with-call.o
[368/371] Generating Msan-x86_64-with-call-Test
[369/371] Generating MSAN_INST_TEST_OBJECTS.msan_test.cpp.x86_64.o
[370/371] Generating Msan-x86_64-Test
[370/371] Running compiler_rt regression tests
llvm-lit: /b/sanitizer-x86_64-linux/build/llvm-project/llvm/utils/lit/lit/main.py:72: note: The test suite configuration requested an individual test timeout of 0 seconds but a timeout of 900 seconds was requested on the command line. Forcing timeout to be 900 seconds.
-- Testing: 10048 tests, 88 workers --
command timed out: 1200 seconds without output running [b'python', b'../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_selector.py'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=3315.146076
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
Step 14 (test compiler-rt default) failure: test compiler-rt default (failure)
...
  CMakeLists.txt:6 (include)


CMake Deprecation Warning at /b/sanitizer-x86_64-linux/build/llvm-project/cmake/Modules/CMakePolicy.cmake:11 (cmake_policy):
  The OLD behavior for policy CMP0116 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.
Call Stack (most recent call first):
  CMakeLists.txt:6 (include)
-- Could NOT find LLVM (missing: LLVM_DIR)
-- Could NOT find Clang (missing: Clang_DIR)
-- LLVM host triple: x86_64-unknown-linux-gnu
-- LLVM default target triple: x86_64-unknown-linux-gnu
-- Using libc++abi testing configuration: /b/sanitizer-x86_64-linux/build/llvm-project/libcxxabi/test/configs/llvm-libc++abi-static.cfg.in
-- Using libc++ testing configuration: /b/sanitizer-x86_64-linux/build/llvm-project/libcxx/test/configs/llvm-libc++-static.cfg.in
-- check-runtimes does nothing.
-- Configuring done (1.1s)
-- Generating done (0.1s)
-- Build files have been written to: /b/sanitizer-x86_64-linux/build/build_default/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64
[353/371] Linking CXX static library /b/sanitizer-x86_64-linux/build/build_default/lib/clang/19/lib/i386-unknown-linux-gnu/libclang_rt.fuzzer.a
[354/371] Performing build step for 'libcxx_fuzzer_x86_64'
ninja: no work to do.
[356/371] Linking CXX static library /b/sanitizer-x86_64-linux/build/build_default/lib/clang/19/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer_interceptors.a
[357/371] Linking CXX static library /b/sanitizer-x86_64-linux/build/build_default/lib/clang/19/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer.a
[358/371] Linking CXX static library /b/sanitizer-x86_64-linux/build/build_default/lib/clang/19/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer_no_main.a
[359/371] Generating FuzzedDataProviderTestObjects.FuzzedDataProviderUnittest.cpp.x86_64.o
[360/371] Generating MSAN_INST_GTEST.gtest-all.cc.x86_64-with-call.o
[361/371] Generating FuzzerTestObjects.FuzzerUnittest.cpp.x86_64.o
[362/371] Generating FuzzedDataProviderTestObjects.gtest-all.cc.x86_64.o
[363/371] Generating MSAN_INST_GTEST.gtest-all.cc.x86_64.o
[364/371] Generating FuzzerUtils-x86_64-Test
[365/371] Generating FuzzerTestObjects.gtest-all.cc.x86_64.o
[366/371] Generating Fuzzer-x86_64-Test
[367/371] Generating MSAN_INST_TEST_OBJECTS.msan_test.cpp.x86_64-with-call.o
[368/371] Generating Msan-x86_64-with-call-Test
[369/371] Generating MSAN_INST_TEST_OBJECTS.msan_test.cpp.x86_64.o
[370/371] Generating Msan-x86_64-Test
[370/371] Running compiler_rt regression tests
llvm-lit: /b/sanitizer-x86_64-linux/build/llvm-project/llvm/utils/lit/lit/main.py:72: note: The test suite configuration requested an individual test timeout of 0 seconds but a timeout of 900 seconds was requested on the command line. Forcing timeout to be 900 seconds.
-- Testing: 10048 tests, 88 workers --

command timed out: 1200 seconds without output running [b'python', b'../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_selector.py'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=3315.146076
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..