License files that are inside the source folder get included twice

[SpecLad]

Create a project like this:

pyproject.toml:

[build-system]
requires = ["flit_core >=3.11,<4"]
build-backend = "flit_core.buildapi"

[project]
name = "foobar"
version = "0"
description = "bazquux"
license-files = ["foobar/test.txt"]

foobar/__init__.py: empty
foobar/test.txt: empty

Build it:

> pyproject-build

Examine the results:

> tar -tf .\dist\foobar-0.tar.gz
foobar-0/foobar/test.txt <---
foobar-0/foobar/__init__.py
foobar-0/foobar/test.txt <---
foobar-0/pyproject.toml
foobar-0/PKG-INFO
> tar -tf .\dist\foobar-0-py2.py3-none-any.whl
foobar/__init__.py
foobar/test.txt <---
foobar-0.dist-info/licenses/foobar/test.txt <---
foobar-0.dist-info/WHEEL
foobar-0.dist-info/METADATA
foobar-0.dist-info/RECORD

As you can see, in both the sdist and the wheel, the license file is included twice. The sdist problem I was only able to reproduce on Windows. The wheel problem occurs on both Windows and Linux.

[takluyver]

The wheel problem is, in my view, not a problem. As was just made clearer in the docs (#748), all files within the package directory - foobar/ in your example - are included by default. And they're included in the same relative locations as they are in the source folder (this isn't explicitly stated, but as we don't offer any way to rearrange them, it's the only logical outcome). That's the foobar/test.txt copy. Then files indicated by license-files are included in ...dist-info/licenses like the specification says.

If you don't want the first copy, I think listing it in excludes and building with pyproject-build should work. If it doesn't, that's a bug. If you don't want the second copy, don't put it in license-files.

The sdist issue does seem like a bug, probably something to do with path normalisation on Windows.

[SpecLad]

The wheel problem is, in my view, not a problem.

Okay, that's a fair take.

If you don't want the first copy, I think listing it in excludes and building with pyproject-build should work. If it doesn't, that's a bug.

No, that doesn't work. It actually fails in different ways depending on the OS: on Windows it just doesn't have any effect, while on Linux I get a "Crucial files were excluded from the sdist" exception.

[takluyver]

No, that doesn't work. It actually fails in different ways depending on the OS: on Windows it just doesn't have any effect, while on Linux I get a "Crucial files were excluded from the sdist" exception.

On second thoughts, I'm also going to claim the error you see on Linux makes sense: the license file is needed for building (it has to be copied into the ...dist-info/licenses folder), so you're not allowed to exclude it from the sdist, because that would break building & installing from the sdist. And Flit doesn't provide a way to exclude files from the wheel other than excluding them from the sdist.

The different behaviour on Windows does seem like a bug again, though. As you can probably guess, I don't often use Windows. It will probably be the same underlying cause: comparing Windows native paths with backslashes with POSIX-style paths using forward slashes.

[SpecLad]

On second thoughts, I'm also going to claim the error you see on Linux makes sense

That's fair enough, but then we're back to the original problem - there's no way to put the license only in the *.dist-info/licenses directory.

Do you think flit should provide some way to accomplish that, or is it out of scope?

[takluyver]

I think that's out of scope, at least for the moment.

I see from the linked pip issue that the context for this is vendored packages - I guess that's the only scenario where license files are likely to be inside the package folder itself. I imagine the license files are a fairly small addition to the wheel size, so one might just decide to live with the duplication. Or one could adjust how the vendoring scripts work to move the license files out to a separate folder at the time of vendoring.

If neither of those solutions are acceptable, and particularly if multiple projects are affected, we could look at adding a way to exclude files from wheels separately from sdists. But I believe the USP for Flit compared to other package builders is simplicity and reliability, and that depends on keeping options as few as possible.

[SpecLad]

Alright. Since I've already fixed what could be fixed in #752, there's nothing else to do here. Closing.