Skip to content

Inability to install pytorch3d due to torch not found #1673

Open
@leweex95

Description

@leweex95

🐛 Bugs / Unexpected behaviors

Versions

Python: 3.8.10
OS: Ubuntu 20.04.5 LTS
Poetry: 1.6.1
pip: 23.2.1

Description

I am trying to set up a poetry environment with this library installed. I started off with a completely new, empty environment. Then I attempted to add the poetry3d package:

poetry add -vvv git+https://github.com/facebookresearch/pytorch3d.git

Instead of adding it to the poetry environment, I got the following error:

Cloning https://github.com/facebookresearch/pytorch3d.git at 'HEAD' to /home/myfolder/poetry_env/poetry_env_demo/.venv/src/pytorch3d

  Stack trace:

  1  ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:344 in _run
      342│         try:
      343│             if input_:
    → 344│                 output: str = subprocess.run(
      345│                     cmd,
      346│                     stdout=subprocess.PIPE,

  CalledProcessError

  Command '['/tmp/tmpxdbpwgq0/.venv/bin/python', '-']' returned non-zero exit status 1.

  at /usr/lib/python3.8/subprocess.py:516 in run
       512│             # We don't call process.wait() as .__exit__ does that for us.
       513│             raise
       514│         retcode = process.poll()
       515│         if check and retcode:
    →  516│             raise CalledProcessError(retcode, process.args,
       517│                                      output=stdout, stderr=stderr)
       518│     return CompletedProcess(process.args, retcode, stdout, stderr)
       519│ 
       520│ 

The following error occurred when trying to handle this error:


  Stack trace:

  3  ~/.local/lib/python3.8/site-packages/poetry/inspection/info.py:601 in get_pep517_metadata
      599│                 *PEP517_META_BUILD_DEPS,
      600│             )
    → 601│             venv.run(
      602│                 "python",
      603│                 "-",

  2  ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:314 in run
      312│     def run(self, bin: str, *args: str, **kwargs: Any) -> str:
      313│         cmd = self.get_command_from_bin(bin) + list(args)
    → 314│         return self._run(cmd, **kwargs)
      315│ 
      316│     def run_pip(self, *args: str, **kwargs: Any) -> str:

  1  ~/.local/lib/python3.8/site-packages/poetry/utils/env/virtual_env.py:96 in _run
       94│     def _run(self, cmd: list[str], **kwargs: Any) -> str:
       95│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
    →  96│         return super()._run(cmd, **kwargs)
       97│ 
       98│     def get_temp_environ(

  EnvCommandError

  Command ['/tmp/tmpxdbpwgq0/.venv/bin/python', '-'] errored with the following return code 1
  
  Output:
  Traceback (most recent call last):
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 466, in _handle_backend
      yield
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 356, in get_requires_for_build
      return set(get_requires(config_settings))
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
      return self._call_hook('get_requires_for_build_wheel', {
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 311, in _call_hook
      self._subprocess_runner(
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 302, in _runner
      self._hook_runner(cmd, cwd, extra_environ)
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 71, in quiet_subprocess_runner
      check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)
    File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/usr/lib/python3.8/subprocess.py", line 516, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['/tmp/build-env-8vw3hcw7/bin/python', '/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py', 'get_requires_for_build_wheel', '/tmp/tmp9cn0rpj_']' returned non-zero exit status 1.
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "<stdin>", line 16, in <module>
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 356, in get_requires_for_build
      return set(get_requires(config_settings))
    File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
      self.gen.throw(type, value, traceback)
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 474, in _handle_backend
      raise BuildBackendException(  # noqa: B904 # use raise from
  build.BuildBackendException: Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  
  Input:
  import build
  import build.env
  import pyproject_hooks
  
  source = '/home/myfolder/poetry_env/poetry_env_demo/.venv/src/pytorch3d'
  dest = '/tmp/tmpxdbpwgq0/dist'
  
  with build.env.IsolatedEnvBuilder() as env:
      builder = build.ProjectBuilder(
          srcdir=source,
          scripts_dir=env.scripts_dir,
          python_executable=env.executable,
          runner=pyproject_hooks.quiet_subprocess_runner,
      )
      env.install(builder.build_system_requires)
      env.install(builder.get_requires_for_build('wheel'))
      builder.metadata_path(dest)
  

  at ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:363 in _run
      359│                 output = subprocess.check_output(
      360│                     cmd, stderr=stderr, env=env, text=True, **kwargs
      361│                 )
      362│         except CalledProcessError as e:
    → 363│             raise EnvCommandError(e, input=input_)
      364│ 
      365│         return output
      366│ 
      367│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  Stack trace:

  2  ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:359 in _run
      357│                 output = ""
      358│             else:
    → 359│                 output = subprocess.check_output(
      360│                     cmd, stderr=stderr, env=env, text=True, **kwargs
      361│                 )

  1  /usr/lib/python3.8/subprocess.py:415 in check_output
       413│         kwargs['input'] = empty
       414│ 
    →  415│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
       416│                **kwargs).stdout
       417│ 

  CalledProcessError

  Command '['/tmp/tmpxdbpwgq0/.venv/bin/python', 'setup.py', 'egg_info']' returned non-zero exit status 1.

  at /usr/lib/python3.8/subprocess.py:516 in run
       512│             # We don't call process.wait() as .__exit__ does that for us.
       513│             raise
       514│         retcode = process.poll()
       515│         if check and retcode:
    →  516│             raise CalledProcessError(retcode, process.args,
       517│                                      output=stdout, stderr=stderr)
       518│     return CompletedProcess(process.args, retcode, stdout, stderr)
       519│ 
       520│ 

The following error occurred when trying to handle this error:


  Stack trace:

  3  ~/.local/lib/python3.8/site-packages/poetry/inspection/info.py:622 in get_pep517_metadata
      620│             os.chdir(path)
      621│             try:
    → 622│                 venv.run("python", "setup.py", "egg_info")
      623│                 info = PackageInfo.from_metadata(path)
      624│             except EnvCommandError as fbe:

  2  ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:314 in run
      312│     def run(self, bin: str, *args: str, **kwargs: Any) -> str:
      313│         cmd = self.get_command_from_bin(bin) + list(args)
    → 314│         return self._run(cmd, **kwargs)
      315│ 
      316│     def run_pip(self, *args: str, **kwargs: Any) -> str:

  1  ~/.local/lib/python3.8/site-packages/poetry/utils/env/virtual_env.py:96 in _run
       94│     def _run(self, cmd: list[str], **kwargs: Any) -> str:
       95│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
    →  96│         return super()._run(cmd, **kwargs)
       97│ 
       98│     def get_temp_environ(

  EnvCommandError

  Command ['/tmp/tmpxdbpwgq0/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1
  
  Output:
  Traceback (most recent call last):
    File "setup.py", line 15, in <module>
      import torch
  ModuleNotFoundError: No module named 'torch'
  

  at ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:363 in _run
      359│                 output = subprocess.check_output(
      360│                     cmd, stderr=stderr, env=env, text=True, **kwargs
      361│                 )
      362│         except CalledProcessError as e:
    → 363│             raise EnvCommandError(e, input=input_)
      364│ 
      365│         return output
      366│ 
      367│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  Stack trace:

  18  ~/.local/lib/python3.8/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  17  ~/.local/lib/python3.8/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 

  16  ~/.local/lib/python3.8/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 

  15  ~/.local/lib/python3.8/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code

  14  ~/.local/lib/python3.8/site-packages/cleo/application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  13  ~/.local/lib/python3.8/site-packages/cleo/commands/base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

  12  ~/.local/lib/python3.8/site-packages/cleo/commands/command.py:62 in execute
        60│ 
        61│         try:
     →  62│             return self.handle()
        63│         except KeyboardInterrupt:
        64│             return 1

  11  ~/.local/lib/python3.8/site-packages/poetry/console/commands/add.py:160 in handle
       158│             return 0
       159│ 
     → 160│         requirements = self._determine_requirements(
       161│             packages,
       162│             allow_prereleases=self.option("allow-prereleases"),

  10  ~/.local/lib/python3.8/site-packages/poetry/console/commands/init.py:375 in _determine_requirements
       373│ 
       374│         result = []
     → 375│         for requirement in self._parse_requirements(requires):
       376│             if "git" in requirement or "url" in requirement or "path" in requirement:
       377│                 result.append(requirement)

   9  ~/.local/lib/python3.8/site-packages/poetry/console/commands/init.py:441 in _parse_requirements
       439│             cwd=cwd,
       440│         )
     → 441│         return [parser.parse(requirement) for requirement in requirements]
       442│ 
       443│     def _format_requirements(self, requirements: list[dict[str, str]]) -> Requirements:

   8  ~/.local/lib/python3.8/site-packages/poetry/console/commands/init.py:441 in <listcomp>
       439│             cwd=cwd,
       440│         )
     → 441│         return [parser.parse(requirement) for requirement in requirements]
       442│ 
       443│     def _format_requirements(self, requirements: list[dict[str, str]]) -> Requirements:

   7  ~/.local/lib/python3.8/site-packages/poetry/utils/dependency_specification.py:89 in parse
        87│ 
        88│         specification = (
     →  89│             self._parse_url(requirement)
        90│             or self._parse_path(requirement)
        91│             or self._parse_simple(requirement)

   6  ~/.local/lib/python3.8/site-packages/poetry/utils/dependency_specification.py:149 in _parse_url
       147│ 
       148│         if url_parsed.scheme in GIT_URL_SCHEMES:
     → 149│             return self._parse_git_url(requirement)
       150│ 
       151│         if url_parsed.scheme in ["http", "https"]:

   5  ~/.local/lib/python3.8/site-packages/poetry/utils/dependency_specification.py:133 in _parse_git_url
       131│ 
       132│         source_root = self._env.path.joinpath("src") if self._env else None
     → 133│         package = self._direct_origin.get_package_from_vcs(
       134│             "git",
       135│             url=url.url,

   4  ~/.local/lib/python3.8/site-packages/poetry/packages/direct_origin.py:111 in get_package_from_vcs
       109│             raise ValueError(f"Unsupported VCS dependency {vcs}")
       110│ 
     → 111│         return _get_package_from_git(
       112│             url=url,
       113│             branch=branch,

   3  ~/.local/lib/python3.8/site-packages/poetry/packages/direct_origin.py:46 in _get_package_from_git
        44│         path = path.joinpath(subdirectory)
        45│ 
     →  46│     package = DirectOrigin.get_package_from_directory(path)
        47│     package._source_type = "git"
        48│     package._source_url = url

   2  ~/.local/lib/python3.8/site-packages/poetry/packages/direct_origin.py:75 in get_package_from_directory
        73│     @classmethod
        74│     def get_package_from_directory(cls, directory: Path) -> Package:
     →  75│         return PackageInfo.from_directory(path=directory).to_package(root_dir=directory)
        76│ 
        77│     def get_package_from_url(self, url: str) -> Package:

   1  ~/.local/lib/python3.8/site-packages/poetry/inspection/info.py:499 in from_directory
       497│                         info = cls.from_setup_files(path)
       498│                     else:
     → 499│                         info = get_pep517_metadata(path)
       500│                 except PackageInfoError:
       501│                     if not info:

  PackageInfoError

  Unable to determine package info for path: /home/myfolder/poetry_env/poetry_env_demo/.venv/src/pytorch3d
  
  Command ['/tmp/tmpxdbpwgq0/.venv/bin/python', '-'] errored with the following return code 1
  
  Output:
  Traceback (most recent call last):
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 466, in _handle_backend
      yield
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 356, in get_requires_for_build
      return set(get_requires(config_settings))
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
      return self._call_hook('get_requires_for_build_wheel', {
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 311, in _call_hook
      self._subprocess_runner(
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 302, in _runner
      self._hook_runner(cmd, cwd, extra_environ)
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 71, in quiet_subprocess_runner
      check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)
    File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/usr/lib/python3.8/subprocess.py", line 516, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['/tmp/build-env-8vw3hcw7/bin/python', '/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py', 'get_requires_for_build_wheel', '/tmp/tmp9cn0rpj_']' returned non-zero exit status 1.
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "<stdin>", line 16, in <module>
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 356, in get_requires_for_build
      return set(get_requires(config_settings))
    File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
      self.gen.throw(type, value, traceback)
    File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 474, in _handle_backend
      raise BuildBackendException(  # noqa: B904 # use raise from
  build.BuildBackendException: Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  
  Input:
  import build
  import build.env
  import pyproject_hooks
  
  source = '/home/myfolder/poetry_env/poetry_env_demo/.venv/src/pytorch3d'
  dest = '/tmp/tmpxdbpwgq0/dist'
  
  with build.env.IsolatedEnvBuilder() as env:
      builder = build.ProjectBuilder(
          srcdir=source,
          scripts_dir=env.scripts_dir,
          python_executable=env.executable,
          runner=pyproject_hooks.quiet_subprocess_runner,
      )
      env.install(builder.build_system_requires)
      env.install(builder.get_requires_for_build('wheel'))
      builder.metadata_path(dest)
  
  Fallback egg_info generation failed.
  
  Command ['/tmp/tmpxdbpwgq0/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1
  
  Output:
  Traceback (most recent call last):
    File "setup.py", line 15, in <module>
      import torch
  ModuleNotFoundError: No module named 'torch'

  at ~/.local/lib/python3.8/site-packages/poetry/inspection/info.py:625 in get_pep517_metadata
      621│             try:
      622│                 venv.run("python", "setup.py", "egg_info")
      623│                 info = PackageInfo.from_metadata(path)
      624│             except EnvCommandError as fbe:
    → 625│                 raise PackageInfoError(
      626│                     path, e, "Fallback egg_info generation failed.", fbe
      627│                 )
      628│             finally:
      629│                 os.chdir(cwd)

Alright, so pytorch3d has a dependency on torch but despite this pre-requisite, the installation for some reason doesn't install the dependency. So I went ahead and installed torch:

poetry add torch

Which went successfully. Now I retried the previous installation command, knowing that torch is certainly included in my environment by now:

poetry add -vvv git+https://github.com/facebookresearch/pytorch3d.git

But the error is exactly the same.

Edit 1

I saw this issue from earlier: #1619. But it doesn't offer a clear workaround on how to install pytorch3d with poetry. As I mentioned above, I also installed torch separately before attempting to install pytorch3d, but the installation still fails, as pytorch3d somehow doesn't see the already present torch dependency in the local venv.

Edit 2

I also came across this: #1479 . I assume this change would solve the issue, but it seems to have been never merged for some reason.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions