Assertion error when loading AST of the same file twice under different directories.

[thebesttv]

The assertion ((!RequiresNullTerminator || BufEnd[0] == 0) && "Buffer is not null terminated!") fails (stacktrace #5) when building AST from files provided by compile_commands using Tool.buildASTs(ASTs). Code in main.cpp at the bottom.

Files to reproduce: segFault.zip

The compile_commands.json is provided as an argument:

[
    {
        "command": "cc -c -I. ../poison.c",
        "directory": "/home/thebesttv/vul/llvm-project/segFault/build",
        "file": "/home/thebesttv/vul/llvm-project/segFault/poison.c"
    },
    {
        "command": "cc -c -I. ../poison.c",
        "directory": "/home/thebesttv/vul/llvm-project/segFault/build-static",
        "file": "/home/thebesttv/vul/llvm-project/segFault/poison.c"
    }
]

segFault directory structure:

.
├── build
│   └── config-poison.h
├── build-static
│   └── config-poison.h
├── compile_commands.json
└── poison.c

Here the file poison.c is compiled twice, once under build/, and another under build-static/.
Both directories have config-poison.h, included by poison.c.

Clang version: 6757913

Stacktrace:

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: ./build-debug/bin/demo-main segFault/compile_commands.json
1.      ../poison.h:1:2: current parser token 'include'
 #0 0x00005605aaace184 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /home/thebesttv/vul/llvm-project/llvm/lib/Support/Unix/Signals.inc:723:22
 #1 0x00005605aaace597 PrintStackTraceSignalHandler(void*) /home/thebesttv/vul/llvm-project/llvm/lib/Support/Unix/Signals.inc:798:1
 #2 0x00005605aaacb9cf llvm::sys::RunSignalHandlers() /home/thebesttv/vul/llvm-project/llvm/lib/Support/Signals.cpp:105:20
 #3 0x00005605aaacda6b SignalHandler(int) /home/thebesttv/vul/llvm-project/llvm/lib/Support/Unix/Signals.inc:413:1
 #4 0x00007fe07468b770 (/usr/lib/libc.so.6+0x3c770)
 #5 0x00005605aab11a0e llvm::MemoryBuffer::init(char const*, char const*, bool) /home/thebesttv/vul/llvm-project/llvm/lib/Support/MemoryBuffer.cpp:53:3
 #6 0x00005605aab140b6 (anonymous namespace)::MemoryBufferMMapFile<llvm::MemoryBuffer>::MemoryBufferMMapFile(bool, int, unsigned long, unsigned long, std::error_code&) /home/thebesttv/vul/llvm-project/llvm/lib/Support/MemoryBuffer.cpp:224:3
 #7 0x00005605aab137ae llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>> getOpenFileImpl<llvm::MemoryBuffer>(int, llvm::Twine const&, unsigned long, unsigned long, long, bool, bool, std::optional<llvm::Align>) /home/thebesttv/vul/llvm-project/llvm/lib/Support/MemoryBuffer.cpp:484:25
 #8 0x00005605aab12cc7 llvm::MemoryBuffer::getOpenFile(int, llvm::Twine const&, unsigned long, bool, bool, std::optional<llvm::Align>) /home/thebesttv/vul/llvm-project/llvm/lib/Support/MemoryBuffer.cpp:529:49
 #9 0x00005605aaa60cb9 (anonymous namespace)::RealFile::getBuffer(llvm::Twine const&, long, bool, bool) /home/thebesttv/vul/llvm-project/llvm/lib/Support/VirtualFileSystem.cpp:231:46
#10 0x00005605aaa60219 llvm::vfs::FileSystem::getBufferForFile(llvm::Twine const&, long, bool, bool) /home/thebesttv/vul/llvm-project/llvm/lib/Support/VirtualFileSystem.cpp:126:1
#11 0x00005605ab44d4a0 clang::FileManager::getBufferForFileImpl(llvm::StringRef, long, bool, bool) const /home/thebesttv/vul/llvm-project/clang/lib/Basic/FileManager.cpp:553:43
#12 0x00005605ab44d3a1 clang::FileManager::getBufferForFile(clang::FileEntryRef, bool, bool) /home/thebesttv/vul/llvm-project/clang/lib/Basic/FileManager.cpp:544:53
#13 0x00005605ab48f150 clang::SrcMgr::ContentCache::getBufferOrNone(clang::DiagnosticsEngine&, clang::FileManager&, clang::SourceLocation) const /home/thebesttv/vul/llvm-project/clang/lib/Basic/SourceManager.cpp:124:8
#14 0x00005605ab431a45 clang::SourceManager::getBufferOrNone(clang::FileID, clang::SourceLocation) const /home/thebesttv/vul/llvm-project/clang/include/clang/Basic/SourceManager.h:1050:38
#15 0x00005605ad802245 clang::Preprocessor::EnterSourceFile(clang::FileID, clang::detail::SearchDirIteratorImpl<true>, clang::SourceLocation, bool) /home/thebesttv/vul/llvm-project/clang/lib/Lex/PPLexerChange.cpp:81:8
#16 0x00005605ad7f5933 clang::Preprocessor::HandleHeaderIncludeOrImport(clang::SourceLocation, clang::Token&, clang::Token&, clang::SourceLocation, clang::detail::SearchDirIteratorImpl<true>, clang::FileEntry const*) /home/thebesttv/vul/llvm-project/clang/lib/Lex/PPDirectives.cpp:2568:3
#17 0x00005605ad7f2cd2 clang::Preprocessor::HandleIncludeDirective(clang::SourceLocation, clang::Token&, clang::detail::SearchDirIteratorImpl<true>, clang::FileEntry const*) /home/thebesttv/vul/llvm-project/clang/lib/Lex/PPDirectives.cpp:1989:44
#18 0x00005605ad7eff99 clang::Preprocessor::HandleDirective(clang::Token&) /home/thebesttv/vul/llvm-project/clang/lib/Lex/PPDirectives.cpp:1240:68
#19 0x00005605ad7a90c5 clang::Lexer::LexTokenInternal(clang::Token&, bool) /home/thebesttv/vul/llvm-project/clang/lib/Lex/Lexer.cpp:4481:7
#20 0x00005605ad7a63b3 clang::Lexer::Lex(clang::Token&) /home/thebesttv/vul/llvm-project/clang/lib/Lex/Lexer.cpp:3695:40
#21 0x00005605abcb50bd clang::Preprocessor::CLK_Lexer(clang::Preprocessor&, clang::Token&) /home/thebesttv/vul/llvm-project/clang/include/clang/Lex/Preprocessor.h:2925:3
#22 0x00005605ad83b0c5 clang::Preprocessor::Lex(clang::Token&) /home/thebesttv/vul/llvm-project/clang/lib/Lex/Preprocessor.cpp:872:10
#23 0x00005605abcb76e6 clang::Parser::ConsumeToken() /home/thebesttv/vul/llvm-project/clang/include/clang/Parse/Parser.h:519:12
#24 0x00005605abcab28a clang::Parser::Initialize() /home/thebesttv/vul/llvm-project/clang/lib/Parse/Parser.cpp:582:1
#25 0x00005605abca6e27 clang::ParseAST(clang::Sema&, bool, bool) /home/thebesttv/vul/llvm-project/clang/lib/Parse/ParseAST.cpp:157:28
#26 0x00005605ab6b7d8a clang::ASTFrontendAction::ExecuteAction() /home/thebesttv/vul/llvm-project/clang/lib/Frontend/FrontendAction.cpp:1183:11
#27 0x00005605ab6b7678 clang::FrontendAction::Execute() /home/thebesttv/vul/llvm-project/clang/lib/Frontend/FrontendAction.cpp:1073:38
#28 0x00005605ab58bf91 clang::ASTUnit::Parse(std::shared_ptr<clang::PCHContainerOperations>, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>) /home/thebesttv/vul/llvm-project/clang/lib/Frontend/ASTUnit.cpp:1245:39
#29 0x00005605ab58ec12 clang::ASTUnit::LoadFromCompilerInvocation(std::shared_ptr<clang::PCHContainerOperations>, unsigned int, llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>) /home/thebesttv/vul/llvm-project/clang/lib/Frontend/ASTUnit.cpp:1705:15
#30 0x00005605ab58ef89 clang::ASTUnit::LoadFromCompilerInvocation(std::shared_ptr<clang::CompilerInvocation>, std::shared_ptr<clang::PCHContainerOperations>, llvm::IntrusiveRefCntPtr<clang::DiagnosticsEngine>, clang::FileManager*, bool, clang::CaptureDiagsKind, unsigned int, clang::TranslationUnitKind, bool, bool, bool) /home/thebesttv/vul/llvm-project/clang/lib/Frontend/ASTUnit.cpp:1738:38
#31 0x00005605aba7f16a (anonymous namespace)::ASTBuilderAction::runInvocation(std::shared_ptr<clang::CompilerInvocation>, clang::FileManager*, std::shared_ptr<clang::PCHContainerOperations>, clang::DiagnosticConsumer*) /home/thebesttv/vul/llvm-project/clang/lib/Tooling/Tooling.cpp:658:14
#32 0x00005605aba7d68f clang::tooling::ToolInvocation::runInvocation(char const*, clang::driver::Compilation*, std::shared_ptr<clang::CompilerInvocation>, std::shared_ptr<clang::PCHContainerOperations>) /home/thebesttv/vul/llvm-project/clang/lib/Tooling/Tooling.cpp:440:31
#33 0x00005605aba7d48e clang::tooling::ToolInvocation::run() /home/thebesttv/vul/llvm-project/clang/lib/Tooling/Tooling.cpp:425:23
#34 0x00005605aba7ee1e clang::tooling::ClangTool::run(clang::tooling::ToolAction*) /home/thebesttv/vul/llvm-project/clang/lib/Tooling/Tooling.cpp:623:11
#35 0x00005605aba7f294 clang::tooling::ClangTool::buildASTs(std::vector<std::unique_ptr<clang::ASTUnit, std::default_delete<clang::ASTUnit>>, std::allocator<std::unique_ptr<clang::ASTUnit, std::default_delete<clang::ASTUnit>>>>&) /home/thebesttv/vul/llvm-project/clang/lib/Tooling/Tooling.cpp:671:13
#36 0x00005605aa92e572 main /home/thebesttv/vul/llvm-project/clang/tools/demo/main.cpp:40:12
#37 0x00007fe074674cd0 (/usr/lib/libc.so.6+0x25cd0)
#38 0x00007fe074674d8a __libc_start_main (/usr/lib/libc.so.6+0x25d8a)
#39 0x00005605aa92e0c5 _start (/home/thebesttv/vul/llvm-project/build-debug/bin/demo-main+0x49f0c5)
Bus error (core dumped)

It seems that shortening the content of build/config-poison.h will not cause the assertion to fail; changing the order of the two entries in compile_commands.json will not, either.

After some digging, it seems the problem is caused by SeenFileEntries in FileManager.cpp, which records file by name. It seems the same file manager is used throughout the program, so the second time ./config-poison.h gets processed, build/config-poison.h is used, not build-static/config-poison.h.

Commenting out these if checks seems to bypass the problem.

I encountered this problem when trying to read function declarations in QEMU, and reduced the test cases down to 3 files.
I'm quite new to Clang, and I wonder: is Tool.buildASTs(ASTs) (and FrontendAction) not meant to be used like this, or is this a bug?

Thanks a lot!

Code that loads AST from compile_commands.json: main.cpp

#include "clang/AST/ASTContext.h"
#include "clang/Tooling/CompilationDatabase.h"
#include "clang/Tooling/Tooling.h"
#include "llvm/Support/InitLLVM.h"
#include "llvm/Support/PrettyStackTrace.h"

#include <filesystem>

using namespace clang;
using namespace clang::tooling;
using namespace llvm;
namespace fs = std::filesystem;

std::unique_ptr<CompilationDatabase>
getCompilationDatabase(fs::path buildPath) {
    llvm::errs() << "Getting compilation database from: " << buildPath << "\n";
    std::string errorMsg;
    std::unique_ptr<CompilationDatabase> cb =
        CompilationDatabase::autoDetectFromDirectory(buildPath.string(),
                                                     errorMsg);
    if (!cb) {
        llvm::errs() << "Error while trying to load a compilation database:\n"
                     << errorMsg << "Running without flags.\n";
        exit(1);
    }
    return cb;
}

int main(int argc, const char **argv) {
    llvm::InitLLVM X(argc, argv);

    fs::path compile_commands = fs::canonical(fs::absolute(argv[1])).string();
    std::unique_ptr<CompilationDatabase> cb =
        getCompilationDatabase(compile_commands);

    ClangTool Tool(*cb, cb->getAllFiles());
    std::vector<std::unique_ptr<ASTUnit>> ASTs;
    Tool.buildASTs(ASTs);

    return 0;
}