Write a compile_commands.json from build_ext

[godlygeek]

Summary of changes

Produce a JSON database of the compiler commands executed while building extension modules as build/compile_commands.json. This is usable by various C and C++ language servers, linters, and IDEs, like clangd, clang-tidy, and CLion. These tools need to understand the header search path and macros passed on the compiler command line in order to correctly interpret source files. In the case of Python extension modules, the developer might not even know all of the compiler flags that are being used, since some are inherited from the interpreter via sysconfig.

Closes #1975

Pull Request Checklist

• Changes have tests
• News fragment added in newsfragments/.
  (See documentation for details)

[godlygeek]

Let's call this a proof-of-concept. There are no tests yet. This implementation changes both the setuptools build_ext command and the vendored distutils.unixccompiler module.

If this seems like a reasonable approach to the problem, I guess the changes to unixccompiler would need to be done in https://github.com/pypa/distutils/ instead, and tested there.

Perhaps it should also apply to non-Unix compilers as well, but I'm not sure. compile_commands.json originated from the Clang/LLVM world, and I think most tools that use it expect clang- or gcc-compatible flags.

I'm not very familiar with the architecture of setuptools, so if this approach seems fundamentally wrong or there's a cleaner way to do it (especially one that doesn't require modifying the vendored distutils), please point me in the right direction.

[jaraco]

Let's see if we can figure out an integration that doesn't require multiple touch points, maybe by wrapping spawn. Ideally, the implementation would be one line added somewhere to the existing implementation for integration and the rest of the behavior would be encapsulated in a function or class.

[jaraco]

Thanks for the contrib.

If this seems like a reasonable approach to the problem, I guess the changes to unixccompiler would need to be done in https://github.com/pypa/distutils/ instead, and tested there.

Yes. That's correct.

Let's start by proposing an interface there to collect the commands in distutils. Some things to think about:

• Should it only affect one compiler? If so, what should the behavior be on the unsupported compilers? My preference would be to provide the behavior universally or at least provide a degenerate behavior on compilers that are not relevant or where the implementation is not yet obvious.
• Let's see if we can create a clean hook point to the existing behavior so there's just one touch point.
• We'll want to document and capture in unit tests what the guaranteed interface is meant to be (i.e. what setuptools can rely on).

I'm going to close this for now, but we can re-open or revive in a new PR.