tests.test_resources:NamespaceTests.test_two_levels_deep fails on OS X because of temp folder strangeness

[ghost]

Originally reported by: msabramo (Bitbucket: msabramo, GitHub: msabramo)

---

❯ tox -e py27 -- -k test_two_levels_deep --tb=short setuptools/tests/test_resources.py
GLOB sdist-make: /Users/marca/dev/hg-repos/setuptools_better_VersionConflict_error/setup.py
py27 inst-nodeps: /Users/marca/dev/hg-repos/setuptools_better_VersionConflict_error/.tox/dist/setuptools-6.0.3dev.zip
py27 runtests: PYTHONHASHSEED='3064395586'
py27 runtests: commands[0] | py.test -k test_two_levels_deep --tb=short setuptools/tests/test_resources.py
============================================================================= test session starts ==============================================================================
platform darwin -- Python 2.7.6 -- py-1.4.25 -- pytest-2.6.3
collected 30 items

setuptools/tests/test_resources.py F

=================================================================================== FAILURES ===================================================================================
_____________________________________________________________________ NamespaceTests.test_two_levels_deep ______________________________________________________________________
setuptools/tests/test_resources.py:612: in test_two_levels_deep
    assert pkg1.pkg2.__path__ == expected
E   AssertionError: assert ['/private/va...gs/pkg1/pkg2'] == ['/var/folders...s2/pkg1/pkg2']
E     At index 0 diff: '/private/var/folders/gw/w0clrs515zx9x_55zgtpv4mm0000gp/T/tests-setuptools-NAWH6D/site-pkgs/pkg1/pkg2' != '/var/folders/gw/w0clrs515zx9x_55zgtpv4mm0000gp/T/tests-setuptools-NAWH6D/site-pkgs/pkg1/pkg2'
E     Left contains more items, first extra item: '/var/folders/gw/w0clrs515zx9x_55zgtpv4mm0000gp/T/tests-setuptools-NAWH6D/site-pkgs/pkg1/pkg2'
=============================================================== 29 tests deselected by '-ktest_two_levels_deep' ================================================================
=================================================================== 1 failed, 29 deselected in 0.18 seconds ====================================================================
ERROR: InvocationError: '/Users/marca/dev/hg-repos/setuptools_better_VersionConflict_error/.tox/py27/bin/py.test -k test_two_levels_deep --tb=short setuptools/tests/test_resources.py'
___________________________________________________________________________________ summary ____________________________________________________________________________________
ERROR:   py27: commands failed

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/271

[ghost]

Original comment by msabramo (Bitbucket: msabramo, GitHub: msabramo):

---

https://bitbucket.org/pypa/setuptools/pull-request/93/fix-test_two_levels_deep-failure-on-os-x/diff fixes this.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

This issue was raised in previously, but only in Pull Request #74. In that commit, a discussion began about the right way to fix this issue.

The issue arises on Mac OS X because the tempdir is in a symlink folder and apparently there's some inconsistency in how these paths resolved.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

hongqn says:

Actually, if I do not change here, the value of pkg1.pkg2.__path__ in test_two_levels_deep() will strangely contains 3 items, with the first and the last item pointing to the same directory but one in realpath form and the other in symlinked form. If it is the expectation should be modified, then it will be like:

expected = [
    os.path.realpath(os.path.join(self._tmpdir, "site-pkgs", "pkg1", "pkg2")),
    os.path.realpath(os.path.join(self._tmpdir, "site-pkgs2", "pkg1", "pkg2")),
]
if os.path.realpath(self._tmpdir) != self._tmpdir:
    expected.append(os.path.join(self._tmpdir, "site-pkgs", "pkg1", "pkg2"))
assert pkg1.pkg2.__path__ == expected

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Then I say:

Aha. So it looks like you may have uncovered a bug (or at least inconsistency) in the code. I don't think the expectation as you've written is one that we want to enforce. Instead, we should (a) update the tests to match the best expectation in the case of a symlinked package root, then (b) update the implementation to behave better in this case.

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Then hongqn again:

The place where realpath comes is in stdlib's pkgutil module: http://hg.python.org/cpython/file/f01413758114/Lib/pkgutil.py#l212

It seems the module loaded by pkgutil will use realpath, but import statement simply uses whatever string in sys.path.

mkdir -p root/pkg
touch root/pkg/__init__.py
ln -s ./root sym
python - <<EOF
import sys, os
sympath = os.path.abspath('sym')
sys.path.append(sympath)
import pkg
print pkg.__path__
import pkgutil
importer = pkgutil.get_importer(sympath)
loader = importer.find_module('pkg')
pkg2 = loader.load_module('pkg')
print pkg2.__path__
EOF

It output:

['/Users/hongqn/tmp/sym/pkg']
['/Users/hongqn/tmp/root/pkg']

Should this be considered as a Python bug?

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Then me again:

I don't know if that's a bug or not. I traced the specification of the package loader to PEP 302, which doesn't say. I suggest sending an e-mail to distutils-sig or maybe python-dev for an opinion. It does seem like an unnecessary inconsistency.

Other avenues to pursue - investigate the loader code and see if it indicates anything about resolving the realpath (is there a comment explaining why it's there?). Run blame on the line where realpath is run to see when it was changed and if the commit message there explained anything. Also, do all Python versions have the same behavior?

[ghost]

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

hongqn also mentions:

The code calling pkgutil in pkg_resources: https://bitbucket.org/pypa/setuptools/src/01ec41e9b983dd7d1fdb81b3d07d7b43217fba3e/pkg_resources.py?at=default#cl-1954