sdist with --dist-dir on Windows produces incorrect archive

[ofek]

@jaraco

Using a relative path works as expected (python setup.py sdist --dist-dir dist)

however using an absolute path (python setup.py sdist --dist-dir C:\Users\Ofek\Desktop\mypkg\dist) includes the entire path structure starting at the drive/root

[jaraco]

It seems the issue does not happen on Unix:

$ cat > setup.py
__import__('setuptools').setup(name='mypkg', version='1')
$ python setup.py sdist --dist-dir=$(pwd)/dist --formats=zip,gztar
running sdist
running egg_info
writing mypkg.egg-info/PKG-INFO
writing dependency_links to mypkg.egg-info/dependency_links.txt
writing top-level names to mypkg.egg-info/top_level.txt
reading manifest file 'mypkg.egg-info/SOURCES.txt'
writing manifest file 'mypkg.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md

running check
warning: check: missing required meta-data: url

warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied

creating mypkg-1
creating mypkg-1/mypkg.egg-info
copying files to mypkg-1...
copying setup.py -> mypkg-1
copying mypkg.egg-info/PKG-INFO -> mypkg-1/mypkg.egg-info
copying mypkg.egg-info/SOURCES.txt -> mypkg-1/mypkg.egg-info
copying mypkg.egg-info/dependency_links.txt -> mypkg-1/mypkg.egg-info
copying mypkg.egg-info/top_level.txt -> mypkg-1/mypkg.egg-info
Writing mypkg-1/setup.cfg
creating '/Users/jaraco/yg/work/dist/mypkg-1.zip' and adding 'mypkg-1' to it
adding 'mypkg-1/PKG-INFO'
adding 'mypkg-1/setup.py'
adding 'mypkg-1/setup.cfg'
adding 'mypkg-1/mypkg.egg-info/PKG-INFO'
adding 'mypkg-1/mypkg.egg-info/SOURCES.txt'
adding 'mypkg-1/mypkg.egg-info/top_level.txt'
adding 'mypkg-1/mypkg.egg-info/dependency_links.txt'
Creating tar archive
removing 'mypkg-1' (and everything under it)
$ ls -la dist
total 16
drwxr-xr-x   4 jaraco  staff   128 Nov 13 10:50 .
drwxr-xr-x  10 jaraco  staff   320 Nov 13 10:51 ..
-rw-r--r--   1 jaraco  staff   570 Nov 13 10:51 mypkg-1.tar.gz
-rw-r--r--   1 jaraco  staff  1363 Nov 13 10:51 mypkg-1.zip
$ tar tzf dist/mypkg-1.tar.gz 
mypkg-1/
mypkg-1/PKG-INFO
mypkg-1/mypkg.egg-info/
mypkg-1/mypkg.egg-info/PKG-INFO
mypkg-1/mypkg.egg-info/SOURCES.txt
mypkg-1/mypkg.egg-info/top_level.txt
mypkg-1/mypkg.egg-info/dependency_links.txt
mypkg-1/setup.py
mypkg-1/setup.cfg
$ unzip -l dist/mypkg-1.zip 
Archive:  dist/mypkg-1.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
      209  11-13-2017 10:51   mypkg-1/PKG-INFO
       58  11-13-2017 10:48   mypkg-1/setup.py
       38  11-13-2017 10:51   mypkg-1/setup.cfg
      209  11-13-2017 10:51   mypkg-1/mypkg.egg-info/PKG-INFO
      124  11-13-2017 10:51   mypkg-1/mypkg.egg-info/SOURCES.txt
        1  11-13-2017 10:51   mypkg-1/mypkg.egg-info/top_level.txt
        1  11-13-2017 10:51   mypkg-1/mypkg.egg-info/dependency_links.txt
---------                     -------
      640                     7 files

So the issue seems to be isolated to Windows.

[jaraco]

Since sdist is largely implemented in distutils, I suspect the issue lies in the upstream implementation. Can you confirm the issue happens if you use distutils without setuptools?

[ofek]

@jaraco Sure! What would those imports be?

[ofek]

respectful bump

[jaraco]

I tried replicating the issue on Windows with just distutils and could not. Here's the setup script I created:

__import__('distutils.core').core.setup(name='mypkg', version='1')

However, when I tried to replicate your experience in Setuptools, I can't make it happen:

PS C:\Users\jaraco\m\setuptools-1185> py -m easy_install --version
setuptools 38.5.1 from c:\python36\lib\site-packages (Python 3.6)
PS C:\Users\jaraco\m\setuptools-1185> cat setup.py
__import__('setuptools').setup(name='mypkg', version='1')
PS C:\Users\jaraco\m\setuptools-1185> py setup.py sdist --dist-dir=c:\Users\jaraco\dist
running sdist
running egg_info
writing mypkg.egg-info\PKG-INFO
writing dependency_links to mypkg.egg-info\dependency_links.txt
writing top-level names to mypkg.egg-info\top_level.txt
reading manifest file 'mypkg.egg-info\SOURCES.txt'
writing manifest file 'mypkg.egg-info\SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md

running check
warning: check: missing required meta-data: url

warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied

creating mypkg-1
creating mypkg-1\mypkg.egg-info
copying files to mypkg-1...
copying setup.py -> mypkg-1
copying mypkg.egg-info\PKG-INFO -> mypkg-1\mypkg.egg-info
copying mypkg.egg-info\SOURCES.txt -> mypkg-1\mypkg.egg-info
copying mypkg.egg-info\dependency_links.txt -> mypkg-1\mypkg.egg-info
copying mypkg.egg-info\top_level.txt -> mypkg-1\mypkg.egg-info
Writing mypkg-1\setup.cfg
creating c:\Users\jaraco\dist
Creating tar archive
removing 'mypkg-1' (and everything under it)
PS C:\Users\jaraco\m\setuptools-1185> py -m tarfile -l C:\users\jaraco\dist\mypkg-1.tar.gz
mypkg-1/
mypkg-1/PKG-INFO
mypkg-1/mypkg.egg-info/
mypkg-1/mypkg.egg-info/PKG-INFO
mypkg-1/mypkg.egg-info/SOURCES.txt
mypkg-1/mypkg.egg-info/top_level.txt
mypkg-1/mypkg.egg-info/dependency_links.txt
mypkg-1/setup.py
mypkg-1/setup.cfg

Even on Python 2 with distutils:

PS C:\Users\jaraco\m\setuptools-1185> cat .\setup-distutils.py
__import__('distutils.core').core.setup(name='mypkg', version='1')
PS C:\Users\jaraco\m\setuptools-1185> py -2 setup-distutils.py sdist --dist-dir=c:\users\jaraco\dist --formats=gztar
running sdist
running check
warning: check: missing required meta-data: url

warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied

warning: sdist: manifest template 'MANIFEST.in' does not exist (using default file list)

warning: sdist: standard file not found: should have one of README, README.txt

writing manifest file 'MANIFEST'
creating mypkg-1
copying files to mypkg-1...
copying setup-distutils.py -> mypkg-1
Creating tar archive
removing 'mypkg-1' (and everything under it)
PS C:\Users\jaraco\m\setuptools-1185> py -m tarfile -l C:\users\jaraco\dist\mypkg-1.tar.gz
mypkg-1/
mypkg-1/PKG-INFO
mypkg-1/setuptools-distutils.py

Whatever is happening in your environment isn't happening in mine. I suspect you either have an old version of setuptools or Python that has a bug that's since been fixed.

If you can find a way to replicate the issue with the latest Python and setuptools, please try with just distutils and report back here.

For now I'm going to close this ticket, but I'll certainly re-open if the issue can be replicated.

[Kaiser1989]

Still a problem. Don't use py -m tarfile to test. Just unzip it with 7zip or anything else.
I don't want to provide my datastructure to others

[jaraco]

Kaiser reported more details in the duplicate ticket:

Do NOT test the tar.gz with "python -m tarfile", as this unpacks the tar.gz at once, and therefore showing the correct content. To reproduce, unpack the tar.gz file with 7zip. The "tar" archive contains the correct content. But the "gz" archive provides the full local path.

I also had a question I'd like to see answered:

Do I understand correctly that there is no undesirable user-facing behavior, that the only problem is the leaking of local paths in the gzip file? I do agree it would be nice not to leak that information.

[jaraco]

Here's where the tarfile gets created. It doesn't do anything special with filenames. It just calls TarFile.add.

Can you supply some test that runs on Python that illustrates the undesirable behavior? I'm not even sure it's possible using the Python APIs to affect the name that appears in 7-Zip.

[ArtemSBulgakov]

This is issue of tarfile. It incorrectly writes header for GZIP compression when user gives path with directories as name to TarFile.add. I created an issue at b.p.o. https://bugs.python.org/issue41316 and fixed this by PR python/cpython#21511

[StefanD986]

I also stumbled upon this bug when building using python -m pep517.build . I assume the pep517 package uses dist-dir somewhere internally.

Would be great if this could be merged soon because it limits the usefulness of pep517 on windows platforms.