Exception on poetry build, incorrect gitignore handling logic.

[kawing-chiu]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Arch Linux

• Poetry version: 1.0.2

• Link of a Gist with the contents of your pyproject.toml file: pyproject.toml

Issue

I got an exception when running poetry build -vvv, on the wheel building step:

The issue here is that, poetry seems to be trying to use the /.git directory under my root directory to get ignored files, which does not make sense. I dig into the source and find that the get_vcs() logic here

def get_vcs(directory):  # type: (Path) -> Git
    directory = directory.resolve()
    for p in [directory] + list(directory.parents):
        ...

is incorrect. Here is the content of [directory] + list(directory.parents), the list of directories that get_vcs() searches for .git:

[PosixPath('/tmp/tmp51qstjuk/mypackage-0.3.0.dev3'), PosixPath('/tmp/tmp51qstjuk'), PosixPath('/tmp'), PosixPath('/')]

When running poetry build -vvv, the CompleteBuilder will unpack the sdist file to a tmpdir:

with self.unpacked_tarball(sdist_file) as tmpdir:
    WheelBuilder.make_in(
        Factory().create_poetry(tmpdir),
        ...

Here the value of tmpdir is '/tmp/tmp51qstjuk/mypackage-0.3.0.dev3', there is no .git directory there, so the get_vcs() function is searching for the .git directory in the wrong place. In my specific case, I happen to have a .git directory in my /, so it results in the above exception. Usually, the incorrect search in get_vcs() will return an empty set, so the files ignored by git will not be honored. I suspect this might cause some other issues, like #1660.

[finswimmer]

Hello @kawing-chiu ,

thanks a lot for reporting and finding the reason for this problem. 👍

I hope this is fixed by #1947 . Can you test it?

finswimmer

[kawing-chiu]

Hi @finswimmer , I've tested it. It does fix my issue. Thanks very much.

[kawing-chiu]

@finswimmer Hi, after some more test, I think this issue is not completely fixed. There is still a flaw in get_vcs(): I have a .git directory under / and ~ repectively, to manage the root and user configurations, e.g. systemd units. So if the poetry project that I'm developing don't use git, then this get_vcs() behavior will search up and use my ~/.git 's ignore files, which is not relevant to the poetry project, and will probably cause bugs.

So generally, I think poetry should only work on the directory containing pyproject.toml and downwards by default. It should not go upwards unless the user allows it (maybe through an option).

[finswimmer]

Hey @kawing-chiu,

if you have a valid .git folder somewhere above the project folder all subfolders belong to this repository.

Nevertheless I've changed the way how poetry finds the git root folder. It now does this by running git rev-parse --show-toplevel within the project's folder instead of searching for a folder called .git. This should be more reliable.

fin swimmer

[kawing-chiu]

I'm not sure this method is reliable enough, because the user might have all kinds of settings in that .git repository. For example if I have this directory structure:

directory_A/    <---- has a .git/ directory, but ignores lib/ in .gitignore
  lib/
    some_python_project/   <---- uses hg or some other version control tool instead of git
      pyproject.toml
      pandas/__init__.py

Then running git ls-files --others -i --exclude-standard will return all files under some_python_project.

[finswimmer]

Hello again,

that's an expected behavior. If you want to include files, that are excluded by any git rule, you can do it by using include in the tool.poetry section. In your example like this:

[tool.poetry]
name = "pandas"
version = "0.1.0"
description = ""
authors = ["Fin Swimmer <finswimmer@example.org>"]
include = ["pandas/**/*"]

[kawing-chiu]

I modified my example a bit:

my_project/    <---- uses git as vcs
  .git/
  .gitignore       <---- ignores the whole third_party/ directory
  third_party/
    some_external_python_project/     <---- uses hg as vcs
      .hg/
      .hgignore          <---- poetry should in fact use the ignore rules here
      pyproject.toml     <---- I should not have to touch this, it's thirt_party
      some_external_python_project/__init__.py

The problem is that, now poetry's behavior really depends on where the source directory some_external_python_project/ resides. I don't think the user should be forced to change a third_party project's pyproject.toml to make it work.

In this example, poetry should in fact use the ignore rules provided by hg, just like it normally uses ignore rules from git. However, here hg is just an example. Even if poerty supports hg later, the user might use some other proprietary version control tools, which poetry can never support. Because of this possibility, poetry can't even reliably detect whether a vcs is in use in the current project directory, not to mention in some higher level directories.

So I think a much better and robust solution is to provide an option for the user to override how to find ignore files. For example:

[tool.poetry]
vcs_ignore_files_command = ['hg', 'status', '-i', '-u', '-n']

[ptrcnull]

I believe this isn't fully fixed - the case from the last comment still applies. Poetry shouldn't read random gitignore files above the actual project directory.

Currently we get around this by setting the GIT_DIR variable, but this is just a workaround, not a solution.