`micromamba update -f <file>.yml` ignores pip dependencies #2221

peanutfun · 2023-01-13T15:08:42Z

Troubleshooting docs

My problem is not solved in the Troubleshooting docs

Search tried in issue tracker

#2008 mentions that pip dependencies are not exported, so there might be an underlying issue with handling pip dependencies

Latest version of Mamba

My problem is not solved with the latest version

Tried in Conda?

Not reproducible with Conda

Describe your issue

YAML spec files may specify dependencies that are to be installed using pip instead of conda/mamba/micromamba. Unlike conda and mamba, micromamba ignores these dependencies when calling micromamba update.

Two exemplary environment specs:

# env.yml
name: my_env
channels:
  - conda-forge
dependencies:
  - python
  - pip
  - pip:
    - numpy

# update.yml
name: my_env
channels:
  - conda-forge
dependencies:
  - scipy
  - pip:
    - matplotlib

Steps to replicate:

$ micromamba create -n my_env -f env.yml -y
$ micromamba activate my_env
$ python -c "import numpy"  # Works

$ micromamba update -n my_env -f update.yml -y  # Pip dependencies are not installed!
$ python -c "import scipy"  # Works
$ python -c "import matplotlib"  # Fails!
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'matplotlib'

mamba info / micromamba info

                                           __
          __  ______ ___  ____ _____ ___  / /_  ____ _
         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      /_/


            environment : my_env (active)
           env location : /opt/conda/envs/my_env
      user config files : /home/mambauser/.mambarc
 populated config files : 
       libmamba version : 1.1.0
     micromamba version : 1.1.0
           curl version : libcurl/7.86.0 OpenSSL/1.1.1s zlib/1.2.13 libssh2/1.10.0 nghttp2/1.47.0
     libarchive version : libarchive 3.6.1 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.2
       virtual packages : __unix=0=0
                          __linux=5.15.49=0
                          __glibc=2.31=0
                          __archspec=1=aarch64
               channels : 
       base environment : /opt/conda
               platform : linux-aarch64

Logs

__
          __  ______ ___  ____ _____ ___  / /_  ____ _
         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      /_/

info     libmamba Parsing MatchSpec scipy
info     libmamba Parsing MatchSpec pip
info     libmamba Searching index cache file for repo 'https://conda.anaconda.org/conda-forge/linux-aarch64/repodata.json'
info     libmamba Found cache at '/opt/conda/pkgs/cache/d9d93fba.json'
conda-forge/linux-aarch64                                   Using cache
info     libmamba Searching index cache file for repo 'https://conda.anaconda.org/conda-forge/noarch/repodata.json'
info     libmamba Found cache at '/opt/conda/pkgs/cache/09cdf8bf.json'
conda-forge/noarch                                          Using cache
info     libmamba All targets to download are cached
info     libmamba Reading cache files '/opt/conda/pkgs/cache/d9d93fba.*' for repo index 'https://conda.anaconda.org/conda-forge/linux-aarch64'
info     libmamba Reading cache files '/opt/conda/pkgs/cache/09cdf8bf.*' for repo index 'https://conda.anaconda.org/conda-forge/noarch'
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/ld_impl_linux-aarch64-2.39-h16cd69b_1.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/libgomp-12.2.0-h607ecd0_19.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/wheel-0.38.4-pyhd8ed1ab_0.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/libgcc-ng-12.2.0-h607ecd0_19.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/tk-8.6.12-hd8af866_0.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/python-3.11.0-h92ab765_0_cpython.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/ncurses-6.3-headf329_1.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/tzdata-2022g-h191b570_0.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/libzlib-1.2.13-h4e544f5_4.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/xz-5.2.6-h9cdd2b7_0.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/libuuid-2.32.1-hf897c2e_1000.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/libffi-3.4.2-h3557bc0_5.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/libnsl-2.0.0-hf897c2e_0.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/setuptools-65.6.3-pyhd8ed1ab_0.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/libsqlite-3.40.0-hf9034f9_0.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/readline-8.1.2-h38e3740_0.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/openssl-3.0.7-hb4cce97_1.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/ca-certificates-2022.12.7-h4fd8a4c_0.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/bzip2-1.0.8-hf897c2e_4.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/pip-22.3.1-pyhd8ed1ab_0.json"
info     libmamba Loading single package record: "/opt/conda/envs/my_env/conda-meta/_openmp_mutex-4.5-2_gnu.json"
info     libmamba Adding package record to repo __archspec
info     libmamba Adding package record to repo __glibc
info     libmamba Adding package record to repo __linux
info     libmamba Adding package record to repo __unix
info     libmamba Adding package record to repo _openmp_mutex
info     libmamba Adding package record to repo bzip2
info     libmamba Adding package record to repo ca-certificates
info     libmamba Adding package record to repo ld_impl_linux-aarch64
info     libmamba Adding package record to repo libffi
info     libmamba Adding package record to repo libgcc-ng
info     libmamba Adding package record to repo libgomp
info     libmamba Adding package record to repo libnsl
info     libmamba Adding package record to repo libsqlite
info     libmamba Adding package record to repo libuuid
info     libmamba Adding package record to repo libzlib
info     libmamba Adding package record to repo ncurses
info     libmamba Adding package record to repo openssl
info     libmamba Adding package record to repo pip
info     libmamba Adding package record to repo python
info     libmamba Adding package record to repo readline
info     libmamba Adding package record to repo setuptools
info     libmamba Adding package record to repo tk
info     libmamba Adding package record to repo tzdata
info     libmamba Adding package record to repo wheel
info     libmamba Adding package record to repo xz
info     libmamba Parsing MatchSpec scipy
info     libmamba Parsing MatchSpec scipy
info     libmamba Parsing MatchSpec pip
info     libmamba Parsing MatchSpec pip
info     libmamba Parsing MatchSpec scipy
info     libmamba Parsing MatchSpec pip
info     libmamba Parsing MatchSpec python 3.11.*

Pinned packages:
  - python 3.11.*

info     libmamba Problem count: 0
info     libmamba Found python in installed packages 3.11.0
Transaction

  Prefix: /opt/conda/envs/my_env

  All requested packages already installed

info     libmamba Cleaned 0 .mamba_trash files. 0 remaining.

Transaction starting
info     libmamba All targets to download are cached
info     libmamba Waiting for pyc compilation to finish
Transaction finished
info     libmamba Opening history file: "/opt/conda/envs/my_env/conda-meta/history"
info     libmamba Freeing transaction.
info     libmamba Freeing solver.
info     libmamba Freeing pool.

environment.yml

name: my_env
channels:
  - conda-forge
dependencies:
  - scipy
  - pip:
    - matplotlib

~/.condarc

No response

The text was updated successfully, but these errors were encountered:

wolfv · 2023-01-13T15:15:18Z

Interesting. So you'd expect that we call pip install ... --upgrade ?

peanutfun · 2023-01-13T15:23:31Z

Yes, I think this is how conda and mamba behave (though I am not entirely sure about the --upgrade)

avarganov-philips · 2023-01-24T16:32:01Z

The same behavior is observed when using micromamba export command - pip dependencies are missing when compared to results of running mamba export command.

jonashaag · 2023-01-24T16:35:03Z

@peanutfun can you please check what's Conda's behavior here?

avarganov-philips · 2023-01-24T16:41:26Z

Here is the mamba output:

mamba env export -n mambatest
name: mambatest
channels:
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - bzip2=1.0.8=h7f98852_4
  - ca-certificates=2022.12.7=ha878542_0
  - ld_impl_linux-64=2.39=hcc3a1bd_1
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=12.2.0=h65d4601_19
  - libgomp=12.2.0=h65d4601_19
  - libnsl=2.0.0=h7f98852_0
  - libsqlite=3.40.0=h753d276_0
  - libuuid=2.32.1=h7f98852_1000
  - libzlib=1.2.13=h166bdaf_4
  - ncurses=6.3=h27087fc_1
  - openssl=3.0.7=h0b41bf4_1
  - pip=22.3.1=pyhd8ed1ab_0
  - python=3.10.8=h4a9ceb5_0_cpython
  - readline=8.1.2=h0f457ee_0
  - setuptools=66.1.1=pyhd8ed1ab_0
  - tk=8.6.12=h27826a3_0
  - tzdata=2022g=h191b570_0
  - wheel=0.38.4=pyhd8ed1ab_0
  - xz=5.2.6=h166bdaf_0
  - pip:
    - pymupdf==1.21.1

Similar command for the same environment run via micromamba export env would producer shorter output truncated at pip level dependencies.

jonashaag · 2023-01-24T17:00:36Z

@avarganov I think there are other tickets open for this issue. If not, please open a new one

peanutfun · 2023-01-25T07:53:52Z

@avarganov The issue you are referring to is tracked here: #2008

@jonashaag Conda and Mamba both install pip dependencies when executing conda/mamba env update -f update.yml

jonashaag · 2023-01-25T08:29:40Z

Do they run --upgrade?

peanutfun · 2023-01-25T11:00:27Z

@jonashaag Yes, seems to me like they always run -U/--upgrade, see https://github.com/conda/conda/blob/main/conda_env/installers/pip.py#L51. As far as I understand, the same installer is called for both conda env create and conda env update

jonashaag · 2023-01-25T11:28:04Z

Thanks! If you want to come up with a fix I'm happy to help

peanutfun · 2023-01-25T15:15:00Z

As far as I understand, the micromamba create/update commands simply call their mamba counterparts mamba env create/update? I guess the crucial thing is that install_for_other_pkgmgr must be called correctly during installation, but I do not see why this would be the case for mamba and not for micromamba, if both run the same install instructions when creating or updating an environment. Can you point me where to look at exactly?

avarganov-philips · 2023-01-27T15:34:44Z

Should this line be removed https://github.com/mamba-org/mamba#installing-lock-files from README.md until the new release when the bug is fixed? In it's current state the documentation is misleading since conda-lock does account for pip installations

konstantinjdobler · 2023-03-10T16:31:02Z

Is there any update on this? This prevents a nice interplay between conda-lock and micromamba.

jonashaag · 2023-03-10T17:27:08Z

Maybe one of the commenters here has the time to come up with an initial patch?

agilebean · 2023-10-16T09:41:36Z

Any updates on this?
I just reproduces the issues with a tiny .yml file:

name: a1
channels:
  - conda-forge
  - nodefaults
dependencies:
  - pip=23.2.1
  - pip:
      - accelerate==0.23.0

The confusing thing is that the output does state it installs the pip package:

Installing pip packages: accelerate==0.23.0

but at the end, one tiny line hints to the failure:

critical libmamba pip failed to install packages

In this case, as the .yml contained only one pip package, it is clear but it was not with many pip packages.

What is the workaround here?
Do I first need to create the environment, activate it and then can execute something similar like conda env update -f a.yml (which failed btw)?

jjerphan · 2024-10-08T11:49:12Z

This has been fixed in #3419 which has been first distributed in mamba 2.0.

jonashaag added the type::bug Something isn't working label Jan 25, 2023

jjerphan closed this as completed Oct 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`micromamba update -f <file>.yml` ignores pip dependencies #2221

`micromamba update -f <file>.yml` ignores pip dependencies #2221

peanutfun commented Jan 13, 2023 •

edited

Loading

wolfv commented Jan 13, 2023

peanutfun commented Jan 13, 2023

avarganov-philips commented Jan 24, 2023 •

edited

Loading

jonashaag commented Jan 24, 2023

avarganov-philips commented Jan 24, 2023 •

edited

Loading

jonashaag commented Jan 24, 2023

peanutfun commented Jan 25, 2023

jonashaag commented Jan 25, 2023

peanutfun commented Jan 25, 2023

jonashaag commented Jan 25, 2023

peanutfun commented Jan 25, 2023

avarganov-philips commented Jan 27, 2023 •

edited

Loading

konstantinjdobler commented Mar 10, 2023

jonashaag commented Mar 10, 2023

agilebean commented Oct 16, 2023 •

edited

Loading

jjerphan commented Oct 8, 2024

micromamba update -f <file>.yml ignores pip dependencies #2221

micromamba update -f <file>.yml ignores pip dependencies #2221

Comments

peanutfun commented Jan 13, 2023 • edited Loading

Troubleshooting docs

Search tried in issue tracker

Latest version of Mamba

Tried in Conda?

Describe your issue

mamba info / micromamba info

Logs

environment.yml

~/.condarc

wolfv commented Jan 13, 2023

peanutfun commented Jan 13, 2023

avarganov-philips commented Jan 24, 2023 • edited Loading

jonashaag commented Jan 24, 2023

avarganov-philips commented Jan 24, 2023 • edited Loading

jonashaag commented Jan 24, 2023

peanutfun commented Jan 25, 2023

jonashaag commented Jan 25, 2023

peanutfun commented Jan 25, 2023

jonashaag commented Jan 25, 2023

peanutfun commented Jan 25, 2023

avarganov-philips commented Jan 27, 2023 • edited Loading

konstantinjdobler commented Mar 10, 2023

jonashaag commented Mar 10, 2023

agilebean commented Oct 16, 2023 • edited Loading

jjerphan commented Oct 8, 2024

`micromamba update -f <file>.yml` ignores pip dependencies #2221

`micromamba update -f <file>.yml` ignores pip dependencies #2221

peanutfun commented Jan 13, 2023 •

edited

Loading

avarganov-philips commented Jan 24, 2023 •

edited

Loading

avarganov-philips commented Jan 24, 2023 •

edited

Loading

avarganov-philips commented Jan 27, 2023 •

edited

Loading

agilebean commented Oct 16, 2023 •

edited

Loading