Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

An error occurred with Micromamba while creating a workflow containing @pypi #1924

Closed
EveT0n opened this issue Jul 17, 2024 · 2 comments
Closed

Comments

@EveT0n
Copy link

EveT0n commented Jul 17, 2024

When I create a workflow with @pypi by executing the command python3 pypiflow.py --environment=pypi argo-workflows create, the following error occurs:

2024-07-17 19:19:28.652 Bootstrapping virtual environment(s) ...
    Micromamba ran into an error while setting up environment:
    command '/home/hq/.metaflowconfig/micromamba/bin/micromamba create --yes --quiet --dry-run --no-extra-safety-checks --repodata-ttl=86400 --retry-clean-cache --prefix=/tmp/tmptiqsmcnf/prefix --channel=defaults requests==>=2.21.0 boto3==>=1.14.0 python==3.6.8' returned error (1)
    critical libmamba Error parsing version ">=2.21.0". Version contains invalid characters in >=2.21.0.

According to the error message, there is a problem with the use of Micromamba's commands, and there is a problem with the format of requests==>=2.21.0.
I think it might have something to do with the versioning of requests in the metaflow_config.py

def get_pinned_conda_libs(python_version, datastore_type):
    pins = {
        "requests": ">=2.21.0",
    }
    if datastore_type == "s3":
        pins["boto3"] = ">=1.14.0"
    elif datastore_type == "azure":
        pins["azure-identity"] = ">=1.10.0"
        pins["azure-storage-blob"] = ">=12.12.0"
        pins["azure-keyvault-secrets"] = ">=4.7.0"
    elif datastore_type == "gs":
        pins["google-cloud-storage"] = ">=2.5.0"
        pins["google-auth"] = ">=2.11.0"
        pins["google-cloud-secret-manager"] = ">=2.10.0"
    elif datastore_type == "local":
        pass
    else:
        raise MetaflowException(
            msg="conda lib pins for datastore %s are undefined" % (datastore_type,)
        )
    return pins

When I change "requests": ">=2.21.0" to "requests": "2.21.0" in the file, the error disappears.
The python I use locally is 3.6.8. My question is that this seems to be an obvious problem, and I found that the code has not been modified in version 2.12.7, is there a problem with my use?

@savingoyal
Copy link
Collaborator

@cndota123 this can happen if you are relying on the RC version of micromamba. To get around this issue, please nuke - ~/.metaflowconfig/micromamba/, upgrade Metaflow to the latest version and try re-executing the flow.

@EveT0n
Copy link
Author

EveT0n commented Aug 1, 2024

@savingoyal Yes, I found out that the version of micromamba installed via metaflow is 2.0.0, in the new environment I manually installed version 1.5.8 of micromamba, this issue does not appear.
However, I tried modifying the solve method of metaflow/plugins/pypi/micromamba.py to fix the issue as well.

class Micromamba(object):
    def solve(self, id_, packages, python, platform):  
          ... 
           for package, version in packages.items():
                version_specifiers = ['==', '>=', '<=', '>', '<', '!=']
                contains_specifier = any(specifier in version for specifier in version_specifiers)
                if contains_specifier:
                    cmd.append("%s%s" % (package, version))
                else:
                    cmd.append("%s==%s" % (package, version))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants