[FR] Allow glob values (or similar) for `ext-modules` in `pyproject.toml`

[sakgoyal]

setuptools version

80.9.0

Python version

3.13 and 3.14

OS

Windows 10 64bit

Additional environment information

No response

Description

Setting the ext-modules in the pyproject.toml causes the parser to crash and installation fails. I also get errors in the editor. but the documentation specifies this is allowed (but experimental)

Expected behavior

It does not crash and it is equivilant to the following in setup.py:

from glob import glob
from setuptools import Extension, setup

setup(
    ext_modules=[
    Extension('pypblib.pblib', sources=
            glob('./Modules/pblib/*.cpp') +
            glob('./External/pblib/*.cpp') +
            glob('./External/pblib/encoder/*.cpp'),
        include_dirs=['./External/pblib', './Modules/pblib'],
        extra_compile_args=get_compiler_args(),
        language='c++'
    )

How to Reproduce

This causes the install to crash

[tool.setuptools]
ext-modules = [
  { name = "pypblib.pblib",
    sources = [
      "Modules/pblib/*.cpp",
      "External/pblib/*.cpp",
      "External/pblib/encoder/*.cpp",
    ],
    include-dirs = ["External/pblib", "Modules/pblib"],
    language = "c++"
  }
]

Output

For python 3.13:

pip install -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///C:/Users/Saksham/Documents/projects/pypblib
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\cli\base_command.py", line 107, in _run_wrapper
    status = _inner_run()
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\cli\base_command.py", line 98, in _inner_run
    return self.run(options, args)
           ~~~~~~~~^^^^^^^^^^^^^^^
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\cli\req_command.py", line 71, in wrapper
    return func(self, options, args)
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\commands\install.py", line 393, in run
    requirement_set = resolver.resolve(
        reqs, check_supported_wheels=not options.target_dir
    )
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 79, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 538, in collect_root_requirements
    reqs = list(
        self._make_requirements_from_install_req(
    ...<2 lines>...
        )
    )
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 494, in _make_requirements_from_install_req
    cand = self._make_base_candidate_from_link(
        ireq.link,
    ...<2 lines>...
        version=None,
    )
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 205, in _make_base_candidate_from_link
    self._editable_candidate_cache[link] = EditableCandidate(
                                           ~~~~~~~~~~~~~~~~~^
        link,
        ^^^^^
    ...<3 lines>...
        version=version,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 334, in __init__
    super().__init__(
    ~~~~~~~~~~~~~~~~^
        link=link,
        ^^^^^^^^^^
    ...<4 lines>...
        version=version,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 162, in __init__
    self.dist = self._prepare()
                ~~~~~~~~~~~~~^^
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 239, in _prepare
    dist = self._prepare_distribution()
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 344, in _prepare_distribution
    return self._factory.preparer.prepare_editable_requirement(self._ireq)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\operations\prepare.py", line 708, in prepare_editable_requirement
    dist = _get_prepared_distribution(
        req,
    ...<3 lines>...
        self.check_build_deps,
    )
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\operations\prepare.py", line 77, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        build_env_installer, build_isolation, check_build_deps
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\distributions\sdist.py", line 42, in prepare_distribution_metadata
    self.req.load_pyproject_toml()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\req\req_install.py", line 515, in load_pyproject_toml
    pyproject_toml_data = load_pyproject_toml(
        self.use_pep517, self.pyproject_toml_path, self.setup_py_path, str(self)
    )
  File "C:\Users\Saksham\AppData\Roaming\Python\Python313\site-packages\pip\_internal\pyproject.py", line 67, in load_pyproject_toml
    pp_toml = tomllib.loads(f.read())
  File "C:\Program Files\Python313\Lib\tomllib\_parser.py", line 102, in loads
    pos = key_value_rule(src, pos, out, header, parse_float)
  File "C:\Program Files\Python313\Lib\tomllib\_parser.py", line 326, in key_value_rule
    pos, key, value = parse_key_value_pair(src, pos, parse_float)
                      ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\tomllib\_parser.py", line 369, in parse_key_value_pair
    pos, value = parse_value(src, pos, parse_float)
                 ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\tomllib\_parser.py", line 616, in parse_value
    return parse_array(src, pos, parse_float)
  File "C:\Program Files\Python313\Lib\tomllib\_parser.py", line 420, in parse_array
    pos, val = parse_value(src, pos, parse_float)
               ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\tomllib\_parser.py", line 620, in parse_value
    return parse_inline_table(src, pos, parse_float)
  File "C:\Program Files\Python313\Lib\tomllib\_parser.py", line 445, in parse_inline_table
    pos, key, value = parse_key_value_pair(src, pos, parse_float)
                      ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\tomllib\_parser.py", line 360, in parse_key_value_pair
    pos, key = parse_key(src, pos)
               ~~~~~~~~~^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\tomllib\_parser.py", line 374, in parse_key
    pos, key_part = parse_key_part(src, pos)
                    ~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\Program Files\Python313\Lib\tomllib\_parser.py", line 404, in parse_key_part
    raise suffixed_err(src, pos, "Invalid initial character for a key part")
tomllib.TOMLDecodeError: Invalid initial character for a key part (at line 35, column 28)

For python 3.14:

py -V:3.14 -m pip install -e .
Obtaining file:///C:/Users/Saksham/Documents/projects/pypblib
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\cli\base_command.py", line 107, in _run_wrapper
    status = _inner_run()
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\cli\base_command.py", line 98, in _inner_run
    return self.run(options, args)
           ~~~~~~~~^^^^^^^^^^^^^^^
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\cli\req_command.py", line 71, in wrapper
    return func(self, options, args)
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\commands\install.py", line 393, in run
    requirement_set = resolver.resolve(
        reqs, check_supported_wheels=not options.target_dir
    )
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\resolution\resolvelib\resolver.py", line 79, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 538, in collect_root_requirements
    reqs = list(
        self._make_requirements_from_install_req(
    ...<2 lines>...
        )
    )
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 494, in _make_requirements_from_install_req
    cand = self._make_base_candidate_from_link(
        ireq.link,
    ...<2 lines>...
        version=None,
    )
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\resolution\resolvelib\factory.py", line 205, in _make_base_candidate_from_link
    self._editable_candidate_cache[link] = EditableCandidate(
                                           ~~~~~~~~~~~~~~~~~^
        link,
        ^^^^^
    ...<3 lines>...
        version=version,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 334, in __init__
    super().__init__(
    ~~~~~~~~~~~~~~~~^
        link=link,
        ^^^^^^^^^^
    ...<4 lines>...
        version=version,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 162, in __init__
    self.dist = self._prepare()
                ~~~~~~~~~~~~~^^
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 239, in _prepare
    dist = self._prepare_distribution()
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\resolution\resolvelib\candidates.py", line 344, in _prepare_distribution
    return self._factory.preparer.prepare_editable_requirement(self._ireq)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\operations\prepare.py", line 708, in prepare_editable_requirement
    dist = _get_prepared_distribution(
        req,
    ...<3 lines>...
        self.check_build_deps,
    )
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\operations\prepare.py", line 77, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        build_env_installer, build_isolation, check_build_deps
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\distributions\sdist.py", line 42, in prepare_distribution_metadata
    self.req.load_pyproject_toml()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\req\req_install.py", line 515, in load_pyproject_toml
    pyproject_toml_data = load_pyproject_toml(
        self.use_pep517, self.pyproject_toml_path, self.setup_py_path, str(self)
    )
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\site-packages\pip\_internal\pyproject.py", line 67, in load_pyproject_toml
    pp_toml = tomllib.loads(f.read())
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\tomllib\_parser.py", line 174, in loads
    pos = key_value_rule(src, pos, out, header, parse_float)
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\tomllib\_parser.py", line 403, in key_value_rule
    pos, key, value = parse_key_value_pair(src, pos, parse_float)
                      ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\tomllib\_parser.py", line 446, in parse_key_value_pair
    pos, value = parse_value(src, pos, parse_float)
                 ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\tomllib\_parser.py", line 695, in parse_value
    return parse_array(src, pos, parse_float)
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\tomllib\_parser.py", line 497, in parse_array
    pos, val = parse_value(src, pos, parse_float)
               ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\tomllib\_parser.py", line 699, in parse_value
    return parse_inline_table(src, pos, parse_float)
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\tomllib\_parser.py", line 522, in parse_inline_table
    pos, key, value = parse_key_value_pair(src, pos, parse_float)
                      ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\tomllib\_parser.py", line 437, in parse_key_value_pair
    pos, key = parse_key(src, pos)
               ~~~~~~~~~^^^^^^^^^^
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\tomllib\_parser.py", line 451, in parse_key
    pos, key_part = parse_key_part(src, pos)
                    ~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\Users\Saksham\AppData\Local\Python\pythoncore-3.14-64\Lib\tomllib\_parser.py", line 481, in parse_key_part
    raise TOMLDecodeError("Invalid initial character for a key part", src, pos)
tomllib.TOMLDecodeError: Invalid initial character for a key part (at line 35, column 28)

[Avasam]

Multiline inline tables are not valid toml.

Can you try placing everything between { and } on a single line ?

Or just not use/reduce usage or inline tables.

---

Note that TOML mutliline inline tables have been approved about 3 years ago: toml-lang/toml#904 . I guess we're still just waiting for TOML 1.1.0 to release (and then Python parsers will have to take the time to adopt it): toml-lang/toml#928

[sakgoyal]

compiling fails even inside a developer powershell.

**********************************************************************
** Visual Studio 2022 Developer PowerShell v17.14.17
** Copyright (c) 2025 Microsoft Corporation
**********************************************************************
PS C:\Users\Saksham\Documents\projects\pypblib> pip install -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///C:/Users/Saksham/Documents/projects/pypblib
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: pypblib
  Building editable for pypblib (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building editable for pypblib (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [17 lines of output]
      C:\Users\Saksham\AppData\Local\Temp\pip-build-env-8uwcktb2\overlay\Lib\site-packages\setuptools\config\pyprojecttoml.py:72: _ExperimentalConfiguration: `[tool.setuptools.ext-modules]` in `pyproject.toml` is still *experimental* and likely to change in future releases.
        config = read_configuration(filepath, True, ignore_option_errors, dist)
      running editable_wheel
      creating C:\Users\Saksham\AppData\Local\Temp\pip-wheel-gpw7y9a8\.tmp-o7a0kvas\pypblib.egg-info
      writing C:\Users\Saksham\AppData\Local\Temp\pip-wheel-gpw7y9a8\.tmp-o7a0kvas\pypblib.egg-info\PKG-INFO
      writing dependency_links to C:\Users\Saksham\AppData\Local\Temp\pip-wheel-gpw7y9a8\.tmp-o7a0kvas\pypblib.egg-info\dependency_links.txt
      writing top-level names to C:\Users\Saksham\AppData\Local\Temp\pip-wheel-gpw7y9a8\.tmp-o7a0kvas\pypblib.egg-info\top_level.txt
      writing manifest file 'C:\Users\Saksham\AppData\Local\Temp\pip-wheel-gpw7y9a8\.tmp-o7a0kvas\pypblib.egg-info\SOURCES.txt'
      reading manifest file 'C:\Users\Saksham\AppData\Local\Temp\pip-wheel-gpw7y9a8\.tmp-o7a0kvas\pypblib.egg-info\SOURCES.txt'
      adding license file 'LICENSE'
      writing manifest file 'C:\Users\Saksham\AppData\Local\Temp\pip-wheel-gpw7y9a8\.tmp-o7a0kvas\pypblib.egg-info\SOURCES.txt'
      creating 'C:\Users\Saksham\AppData\Local\Temp\pip-wheel-gpw7y9a8\.tmp-o7a0kvas\pypblib-0.0.5rc1.dist-info'
      creating C:\Users\Saksham\AppData\Local\Temp\pip-wheel-gpw7y9a8\.tmp-o7a0kvas\pypblib-0.0.5rc1.dist-info\WHEEL
      running build_py
      running build_ext
      building 'pypblib.pblib' extension
      error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building editable for pypblib
Failed to build pypblib
error: failed-wheel-build-for-install

× Failed to build installable wheels for some pyproject.toml based projects
╰─> pypblib
PS C:\Users\Saksham\Documents\projects\pypblib> cl.exe
Microsoft (R) C/C++ Optimizing Compiler Version 19.44.35217 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

usage: cl [ option... ] filename... [ /link linkoption... ]
PS C:\Users\Saksham\Documents\projects\pypblib> clang.exe --version
clang version 19.1.5
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\Llvm\x64\bin

I have clang and cl both available, and well beyond the c++14 requirement specified in the error, but it's not doing this correctly

[sakgoyal]

the syntax does not allow for globbing a directory.

[abravalheri]

the syntax does not allow for globbing a directory.

Yes that is correct. This is not a supported syntax, and therefore not a valid configuration for setuptools right now.

[sakgoyal]

I understand that it is not planned right now, but could it be considered in the future? I think this would be a really nice to have feature, especially when you need to compile more than just 2 or 3 files. (eg: you have 100 files to compile). adding each one individually is quite tedious.

I have not worked with PEP stuff before, but is that something I need to do to suggest an improvement?

[abravalheri]

I understand that it is not planned right now, but could it be considered in the future? I think this would be a really nice to have feature, especially when you need to compile more than just 2 or 3 files. (eg: you have 100 files to compile). adding each one individually is quite tedious.

Sure. I have re-opened the issue and changed it to a Feature Request.
(one thing to consider is that the fastest way of getting a FR implemented is via contribution/collaboration).

I have not worked with PEP stuff before, but is that something I need to do to suggest an improvement?

Right now this is an internal setuptools feature, not subject to standardisation so it does not need a PEP (in the future it may change if other projects/people in PyPA consider this to be an interoperability issue,.)

In terms of what would be the next step, I think that having a clearly communicated design for the ext-modules feature in setuptools is necessary, including how it may interact with other packages like cython and mypyc.