[BUG] `options.packages.find.exclude` not taking effect when `include_package_data = True`

[abravalheri]

setuptools version

62.1.0

Python version

Python 3.10.4 (main, Mar 30 2022, 23:50:39) [GCC 10.2.1 20210110]

OS

Tested in Alpine Linux 3.14, Debian bullseye and Ubuntu 20.04.4 LTS

Additional environment information

Python 3.10.4 (main, Mar 30 2022, 23:50:39) [GCC 10.2.1 20210110]

Package    Version
---------- -------
build      0.7.0
packaging  21.3
pep517     0.12.0
pip        22.0.4
pyparsing  3.0.8
setuptools 62.1.0
tomli      2.0.1
wheel      0.37.1

Description

For some reason, when include_package_data = True, subpackages excluded via options.packages.find.exclude are still added to the wheel.

Expected behavior

When listed via options.packages.find.exclude a subpackage should not be added to the wheel.

How to Reproduce

> docker run --rm -it python:3.10-bullseye /bin/bash

mkdir /tmp/mypkg
cd /tmp/mypkg
mkdir -p mypkg/tests
touch mypkg/__init__.py
touch mypkg/resource.txt
touch mypkg/tests/test_mypkg.py
touch mypkg/tests/test_file.txt

cat <<EOS > pyproject.toml
[build-system]
requires = ['setuptools']
build-backend = 'setuptools.build_meta'
EOS

cat <<EOS > setup.cfg
[metadata]
name = mypkg
version = 42

[options]
include_package_data = True
packages = find:

[options.packages.find]
exclude = *.tests*
EOS

cat <<EOS > MANIFEST.in
global-include *.py *.txt
global-exclude *.py[cod]
prune dist
prune build
EOS

python -m pip install -U pip build setuptools wheel
python -VV
python -m pip list

rm -rf dist build mypkg.egg-info
python -m build --no-isolation
# `--no-isolation` is used to guarantee we have control over the
# versions of the build dependencies
unzip -l dist/*.whl

Output

  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2022-04-11 17:49   mypkg/__init__.py
        0  2022-04-11 17:49   mypkg/resource.txt
        0  2022-04-11 17:49   mypkg/tests/__init__.py
        0  2022-04-11 17:49   mypkg/tests/test_file.txt
        0  2022-04-11 17:49   mypkg/tests/test_mypkg.py
      108  2022-04-11 17:49   mypkg-42.dist-info/METADATA
       92  2022-04-11 17:49   mypkg-42.dist-info/WHEEL
        6  2022-04-11 17:49   mypkg-42.dist-info/top_level.txt
      654  2022-04-11 17:49   mypkg-42.dist-info/RECORD
---------                     -------
      860                     9 files

The expected output should not include:

- mypkg/tests/__init__.py
- mypkg/tests/test_file.txt
- mypkg/tests/test_mypkg.py

[Related: https://github.com//issues/3340]

[jaraco]

The expected output should not include...

The issue I see with this expectation is that under other circumstances, a subfolder in a package is data.

Consider if all you had is:

mypkg/tests/test_file.txt

In that case, the test file would be included as package data. Then if you add a python file to the directory:

mypkg/tests/test_file.txt
mypkg/tests/template.py

Now neither file gets included as data. That behavior would be counterintuitive and conflates two concerns.

Moreover, it seems perfectly legitimate to have a directory of python files as data, especially if they're not intended to be imported through the package, but used in another way (such as a template or scripts or other non-module behavior in a Python syntax).

I don't see a good solution here, except for Setuptools to get out of the business of managing packages, modules, and their data and to instead model discovery of files for the build that could be packages or modules or data, and provide mechanisms to include/exclude those generally.

[abravalheri]

Thank you very much for the detailed explanation @jaraco.

I think that if we change mypkg/tests/test_mypkg.py to mypkg/tests/__init__.py, the example becomes fairly less controversial.

With this change it is more obvious that mypkg.tests is a package, and therefore the expected outcome would be the entire mypkg/tests folder to not be included in the wheel (due to [options.packages.find]). Does this make sense?

In this case, the existing implementation would still behave in an unexpected way and include the directory that is supposed to be excluded.

[abravalheri]

The issue I see with this expectation is that under other circumstances, a subfolder in a package is data.

Would it be better if we consider all the subfolders as nested packages and then require the users to explicitly include this package?

I am OK with this assumption. This would also be coherent with Python's existing behaviour¹.

Moreover, it seems perfectly legitimate to have a directory of python files as data, especially if they're not intended to be imported through the package, but used in another way (such as a template or scripts or other non-module behavior in a Python syntax).

This is a bit tricky... How do we know which Python files are meant to be data files or actual modules?
I think it should be fair to assume that anything that can be reached via the import ... or from ... import ... syntax is a legitimate Python module/package.

I don't see a good solution here, except for Setuptools to get out of the business of managing packages, modules, and their data and to instead model discovery of files for the build that could be packages or modules or data, and provide mechanisms to include/exclude those generally.

Please let me know what are your thoughts about this matter. I would be more than happy to participate in this discussion and see if I can help somehow.

Footnotes

1. I don't know if it is accidental but after PEP 420 a folder nested inside of any sys.path entries will be "importable" even if it does not contain a __init__.py file - as long as all the directories are valid Python identifiers. ↩

[jaraco]

1. I don't know if it is accidental but after PEP 420 a folder ... will be "importable"

It's intentional. During the design, we considered requiring a namespace package to have some sort of indicator that it's a Python package, but ultimately decided that the simpler approach of recognizing any directory as a package was more elegant.

That's why:

$ mkdir foo
$ python -c "import foo; print(foo.__name__)"
foo

That is, an empty directory is an empty Python package. Simple and elegant, though somewhat ambiguous (because a directory of rust code is also a Python package whose sole contents is rust files as data).

require the users to explicitly include this package?

I very much want to avoid users having to uniquely indicate items to be included. I'd like for there to be a discovery mechanism that works for 99% of cases and have options to tweak when the discovery is inaccurate. This is part of the zero-config vision... a user should be able to create a null distribution with no config, then add a folder to it, which creates an empty Python package, then add modules to that package, all having to duplicately tell Setuptools what they just did.

That is, Setuptools should infer the same thing that Python does for imports. So by default, what you create for Python gets packaged.

This is a bit tricky... How do we know which Python files are meant to be data files or actual modules?

I guess we can't reliably. I'd say we'd have to assume everything is a legitimate Python package unless indicated otherwise (such as through an exclusion directive or heuristic).

[abravalheri]

Thank you very much for the information Jason.

I very much want to avoid users having to uniquely indicate items to be included. I'd like for there to be a discovery mechanism that works for 99% of cases and have options to tweak when the discovery is inaccurate. This is part of the zero-config vision... a user should be able to create a null distribution with no config, then add a folder to it, which creates an empty Python package, then add modules to that package, all having to duplicately tell Setuptools what they just did.

If the users adopt include_package_data=True (the default for projects using PEP 621) + setuptools-scm (or relevant VCS plugin), they would be not that far away from this situation.

The difference is that we currently have 2 mechanisms to "trim the excesses":

• exclude_package_data to remove some of the non Python files
• find.packages.exclude to remove entire folders from the build (supposedly)

The second mechanism is broken by the current implementation of include_package_data.

Maybe going forward we would want:

• To add all existing files inside of a package even if the user don't adopt setuptools-scm/relevant VCS plugin.
  When the user does adopt a VCS plugin, instead of adding all files we just add the ones given by the plugin.
  I don't know what would be the implications of this approach with compiled extensions
• A single mechanism for removing any kind of "extra" files and folders, regardless if they are folders, Python scripts or data files.

[abravalheri]

Status update:

In v62.3.0, we introduce a deprecation warning about the conflicting behaviour. After some time we can start tackling this issue.