[BUG]: Not detecting the SUFFIX and DEBUG_POSTFIX for cmake correctly

[lpapp-foundry]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.10.4

Problem description

pybind11 uses this to figure out the SUFFIX (extension) for cmake.

pybind11/tools/pybind11NewTools.cmake

Lines 100 to 102 in 0e43fcc

	COMMAND
	"${${_Python}_EXECUTABLE}" "-c"
	"import sys, importlib; s = importlib.import_module('distutils.sysconfig' if sys.version_info < (3, 10) else 'sysconfig'); print(s.get_config_var('EXT_SUFFIX') or s.get_config_var('SO'))"

This is problematic because this can also contain the debug postfix ("_d" on Windows in debug mode) because of this:

https://github.com/python/cpython/blob/a8d69fe92c65d636fc454cfb1825c357eb2e6325/Python/dynload_win.c#L18

What should be done instead is splitting the result into SUFFIX and DEBUG_POSTFIX.

At the moment, we are trying to use pybind11 with a Python that sets the DEBUG_POSTFIX _d when it needs. This is great for all python libraries.

However, cmake combines DEBUG_POSTFIX and SUFFIX, so we end up with double _d, so we _d_d which then crashes since Windows in debug mode is looking for a single _d.

At the moment, as a consumer of pybind11, we have no easy solution to fix this, only to avoid using pybind11_add_module, but then we would need to reinvent quite a bit of boilerplate, which does not look sustainable.

So, the issue is that our Python package sets the DEBUG_POSTFIX and pybind11 sets the SUFFIX. These two get combined in debug mode. They do not overwrite each other.

pybind11_extension sets SUFFIX to _d.something and we set DEBUG_POSTFIX to _d.

_d + _d.something = _d_d.something

Reproducible example code

Call pybind11_add_module() from a consumer cmake extension module in the CMakeLists.txt file on Windows in debug mode.

Is this a regression? Put the last known working version here if it is.

Not a regression

[friendlyanon]

pybind11 should instead use this code to derive the extension:

import sys
import importlib

s = importlib.import_module('distutils.sysconfig' if sys.version_info < (3, 10) else 'sysconfig')

print((s.get_config_var('EXT_SUFFIX') or s.get_config_var('SO')).replace('.', ';.', 1))

This will for example output _d;.cp311-win_amd64.pyd, which is a suitable CMake list with 2 elements: the value for DEBUG_POSTFIX and SUFFIX:

function(pybind11_extension name)
  list(GET PYTHON_MODULE_EXTENSION 0 debug_postfix)
  list(GET PYTHON_MODULE_EXTENSION 1 suffix)
  set_target_properties(${name} PROPERTIES PREFIX "" SUFFIX "${suffix}" DEBUG_POSTFIX "${debug_postfix}")
endfunction()

[macdew]

Are you sure that the fix also works for multi-config generators like Visual Studio? As far as I can tell from the code change, both PYTHON_MODULE_DEBUG_POSTFIX and PYTHON_MODULE_EXTENSION are being generated from the same single python.exe.

But in a multi-config generator, one would need to consider a single build with both python.exe and python_d.exe on Windows. And I don't see this reflected in the logic?

[friendlyanon]

FindPython does not find an interpreter with the python_d basename.