running a single test may have different result than running with all the tests

[ccanepa]

When this can happen ?

• you have a version of nikola installed
• you are testing in another version of nikola that was cloned

why ?

• to run tests without installing we are supposed to add, in each file under tests/, before any import from nikola package, the line

    from .context import nikola

• A test file doing that will get nikola from the cloned directory, while files that don't do that are importing the installed nikola while running as individual test (roughtly)
• Due to import resolution in python, only the first import of nikola in the nosetest run will decide which nikola version is picked , so a test may end using one or another nikola version
• currently, more than half of the files in tests/ don't comply the rule, by example test_integration.py

Possible Solutions:

More of the same

• add the missing line to all files, and another with a comment about including that line in any file in tests/ , so that when people looks in an existing test file as a model to write a new test file they see the line
• mention the rule in the test's readme

Be explict

• delete tests/context.py
• At the beginning of each file in tests/ add

# This code is so you can run the tests without installing the package
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..'))
#

• Add a note in test's readme about including that paragraph at the begining of each file in tests/

I favor the second solution because

• it is not clear to me that 'from context import nikola; import nikola.something' is always the same as the suggested explicit prologue plus import nikola.something, especially in the context of testing, where playing tricks with modules is fair game
• by looking at the test file I don't need to dig further to know what the prologue does

Opinions ?

[ralsina]

+1 on the second one. I never understod what context.py did.

[Kwpolska]

Go for solution 2, i.e. nuke context.py altogether (which openly claims to be a hack).

[ccanepa]

ACK, will do.

[punchagan]

Good to see context.py going! Thanks @ccanepa

[ccanepa]

Now the imports are uniform, still there are two tests that pick the wrong nikola when one nikola is installed, the other version is in a clone and we run the test from the clone.
This is easy to spot by setting a nikola.py module in site-packages with the line assert 0

The failing tests are the two in test_integration.py using subprocess.CalledProcessError to run nikola in a shell.

feeding env to that call, where

            env = os.environ
            path = os.path.normpath(os.path.join(os.path.dirname(__file__), '..'))
            env[str("PYTHONPATH")] = nikola.utils.sys_encode(path)

fixes the remaining two tests. (PYTHONPATH shold be augmented, not replaced, but you get the idea)

ATM I don't see other way to fix those cases

[Kwpolska]

All the subprocess calls of them are calls to the check command, which should be re-implemented using nikola.main.main() or the like. Done on the test-check branch, waiting for Travis’ opinion before pulling.