Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CommandFailed when using conda_install while being offline, even if env is fully setup #313

Closed
smarie opened this issue Apr 21, 2020 · 2 comments · Fixed by #314
Closed

CommandFailed when using conda_install while being offline, even if env is fully setup #313

smarie opened this issue Apr 21, 2020 · 2 comments · Fixed by #314

Comments

@smarie
Copy link
Contributor

smarie commented Apr 21, 2020

As opposed to session.install, that continues to work nicely when no internet connection is present but all required packages are already installed, session.conda_install fails miserably.

session.conda_install('pip')
conda install --yes --prefix C:\_dev\python_ws\my_project\.nox\lint-3-7 pip
Command conda install --yes --prefix C:\_dev\python_ws\my_project\.nox\lint-3-7 pip failed with exit code 1:
Collecting package metadata (current_repodata.json): ...working... failed
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
    Traceback (most recent call last):
      File "C:\Miniconda3\lib\site-packages\urllib3\connection.py", line 157, in _new_conn
        (self._dns_host, self.port), self.timeout, **extra_kw
      File "C:\Miniconda3\lib\site-packages\urllib3\util\connection.py", line 61, in create_connection
        for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
      File "C:\Miniconda3\lib\socket.py", line 748, in getaddrinfo
        for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    socket.gaierror: [Errno 11001] getaddrinfo failed
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "C:\Miniconda3\lib\site-packages\urllib3\connectionpool.py", line 672, in urlopen
        chunked=chunked,
      File "C:\Miniconda3\lib\site-packages\urllib3\connectionpool.py", line 376, in _make_request
        self._validate_conn(conn)
      File "C:\Miniconda3\lib\site-packages\urllib3\connectionpool.py", line 994, in _validate_conn
        conn.connect()
      File "C:\Miniconda3\lib\site-packages\urllib3\connection.py", line 334, in connect
        conn = self._new_conn()
      File "C:\Miniconda3\lib\site-packages\urllib3\connection.py", line 169, in _new_conn
        self, "Failed to establish a new connection: %s" % e
    urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x000001AD90D23F60>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "C:\Miniconda3\lib\site-packages\requests\adapters.py", line 449, in send
        timeout=timeout
      File "C:\Miniconda3\lib\site-packages\urllib3\connectionpool.py", line 760, in urlopen
        **response_kw
      File "C:\Miniconda3\lib\site-packages\urllib3\connectionpool.py", line 760, in urlopen
        **response_kw
      File "C:\Miniconda3\lib\site-packages\urllib3\connectionpool.py", line 760, in urlopen
        **response_kw
      File "C:\Miniconda3\lib\site-packages\urllib3\connectionpool.py", line 720, in urlopen
        method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
      File "C:\Miniconda3\lib\site-packages\urllib3\util\retry.py", line 436, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='repo.anaconda.com', port=443): Max retries exceeded with url: /pkgs/main/win-64/current_repodata.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x000001AD90D23F60>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "C:\Miniconda3\lib\site-packages\conda\core\subdir_data.py", line 485, in fetch_repodata_remote_request
        timeout=timeout)
      File "C:\Miniconda3\lib\site-packages\requests\sessions.py", line 546, in get
        return self.request('GET', url, **kwargs)
      File "C:\Miniconda3\lib\site-packages\requests\sessions.py", line 533, in request
        resp = self.send(prep, **send_kwargs)
      File "C:\Miniconda3\lib\site-packages\requests\sessions.py", line 646, in send
        r = adapter.send(request, **kwargs)
      File "C:\Miniconda3\lib\site-packages\requests\adapters.py", line 516, in send
        raise ConnectionError(e, request=request)
    requests.exceptions.ConnectionError: HTTPSConnectionPool(host='repo.anaconda.com', port=443): Max retries exceeded with url: /pkgs/main/win-64/current_repodata.json (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x000001AD90D23F60>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "C:\Miniconda3\lib\site-packages\conda\exceptions.py", line 1078, in __call__
        return func(*args, **kwargs)
      File "C:\Miniconda3\lib\site-packages\conda\cli\main.py", line 84, in _main
        exit_code = do_call(args, p)
      File "C:\Miniconda3\lib\site-packages\conda\cli\conda_argparse.py", line 82, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "C:\Miniconda3\lib\site-packages\conda\cli\main_install.py", line 20, in execute
        install(args, parser, 'install')
      File "C:\Miniconda3\lib\site-packages\conda\cli\install.py", line 265, in install
        should_retry_solve=(_should_retry_unfrozen or repodata_fn != repodata_fns[-1]),
      File "C:\Miniconda3\lib\site-packages\conda\core\solve.py", line 117, in solve_for_transaction
        should_retry_solve)
      File "C:\Miniconda3\lib\site-packages\conda\core\solve.py", line 158, in solve_for_diff
        force_remove, should_retry_solve)
      File "C:\Miniconda3\lib\site-packages\conda\core\solve.py", line 262, in solve_final_state
        ssc = self._collect_all_metadata(ssc)
      File "C:\Miniconda3\lib\site-packages\conda\common\io.py", line 88, in decorated
        return f(*args, **kwds)
      File "C:\Miniconda3\lib\site-packages\conda\core\solve.py", line 415, in _collect_all_metadata
        index, r = self._prepare(prepared_specs)
      File "C:\Miniconda3\lib\site-packages\conda\core\solve.py", line 1004, in _prepare
        self.subdirs, prepared_specs, self._repodata_fn)
      File "C:\Miniconda3\lib\site-packages\conda\core\index.py", line 229, in get_reduced_index
        repodata_fn=repodata_fn)
      File "C:\Miniconda3\lib\site-packages\conda\core\subdir_data.py", line 105, in query_all
        result = tuple(concat(executor.map(subdir_query, channel_urls)))
      File "C:\Miniconda3\lib\concurrent\futures\_base.py", line 586, in result_iterator
        yield fs.pop().result()
      File "C:\Miniconda3\lib\concurrent\futures\_base.py", line 432, in result
        return self.__get_result()
      File "C:\Miniconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
        raise self._exception
      File "C:\Miniconda3\lib\concurrent\futures\thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
      File "C:\Miniconda3\lib\site-packages\conda\core\subdir_data.py", line 98, in <lambda>
        package_ref_or_match_spec))
      File "C:\Miniconda3\lib\site-packages\conda\core\subdir_data.py", line 110, in query
        self.load()
      File "C:\Miniconda3\lib\site-packages\conda\core\subdir_data.py", line 174, in load
        _internal_state = self._load()
      File "C:\Miniconda3\lib\site-packages\conda\core\subdir_data.py", line 248, in _load
        repodata_fn=self.repodata_fn)
      File "C:\Miniconda3\lib\site-packages\conda\core\subdir_data.py", line 581, in fetch_repodata_remote_request
        """) % maybe_unquote(repr(e))
      File "C:\Miniconda3\lib\site-packages\conda\common\url.py", line 368, in maybe_unquote
        return unquote_plus(remove_auth(url)) if url else url
      File "C:\Miniconda3\lib\site-packages\conda\common\url.py", line 372, in remove_auth
        url_parts = parse_url(url)._asdict()
      File "C:\Miniconda3\lib\site-packages\conda\_vendor\urllib3\util\url.py", line 196, in parse_url
        raise LocationParseError(url)
    conda._vendor.urllib3.exceptions.LocationParseError: Failed to parse: ConnectionError(MaxRetryError("HTTPSConnectionPool(host='repo.anaconda.com', port=443): Max retries exceeded with url: 
`$ C:\Miniconda3\Scripts\conda-script.py install --yes --prefix C:\_dev\python_ws\my_project\.nox\lint-3-7 pip`
  environment variables:
                 CIO_TEST=<not set>
        CONDA_DEFAULT_ENV=tools_py37
                CONDA_EXE=C:\Miniconda3\condabin\..\Scripts\conda.exe
               CONDA_EXES="C:\Miniconda3\condabin\..\Scripts\conda.exe"
             CONDA_PREFIX=C:\Miniconda3\envs\tools_py37
    CONDA_PROMPT_MODIFIER=(tools_py37)
               CONDA_ROOT=C:\Miniconda3
              CONDA_SHLVL=1
                 HOMEPATH=\Users\(...)
                     PATH=C:\Miniconda3;(...)
             PSMODULEPATH=%ProgramFiles%\WindowsPowerShell\Modules;C:\WINDOWS\system32\WindowsPo
                          werShell\v1.0\Modules
  PYTHONDONTWRITEBYTECODE=1
         PYTHONIOENCODING=UTF-8
               PYTHONPATH=(...)
         PYTHONUNBUFFERED=1
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>
    VBOX_MSI_INSTALL_PATH=C:\Program Files\Oracle\VirtualBox\
     active environment : tools_py37
    active env location : C:\Miniconda3\envs\tools_py37
            shell level : 1
       user config file : C:\Users\(...)\.condarc
 populated config files : C:\Users\(...)\.condarc
          conda version : 4.8.0
    conda-build version : not installed
         python version : 3.7.3.final.0
       virtual packages : __cuda=9.2
       base environment : C:\Miniconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Miniconda3\pkgs
                          C:\Users\(...)\.conda\pkgs
                          C:\Users\(...)\AppData\Local\conda\conda\pkgs
       envs directories : C:\Miniconda3\envs
                          C:\Users\(...)\.conda\envs
                          C:\Users\(...)\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.8.0 requests/2.22.0 CPython/3.7.3 Windows/10 Windows/10.0.18362
          administrator : False
             netrc file : None
           offline mode : False
An unexpected error has occurred. Conda has prepared the above report.
Upload did not complete.
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.2\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "C:\_dev\python_ws\_Libs_OpenSource\nox\nox\sessions.py", line 327, in conda_install
    **kwargs
  File "C:\_dev\python_ws\_Libs_OpenSource\nox\nox\sessions.py", line 277, in _run
    return nox.command.run(args, env=env, paths=self.bin_paths, **kwargs)
  File "C:\_dev\python_ws\_Libs_OpenSource\nox\nox\command.py", line 125, in run
    raise CommandFailed("Returned code {}".format(return_code))
nox.command.CommandFailed: Returned code 1

This is of course a conda shortcoming but I think that we could easily solve this in conda_install by detecting when network is unavailable and adding the "--offline" flag when it is the case. I'll propose a PR.
@smarie
Copy link
Contributor Author

smarie commented Apr 21, 2020 •
edited
Loading

One solution to detect that the conda host is there , inspired by this post

from socket import gethostbyname
import requests

CONDA_HOST = 'repo.anaconda.com'

def _is_connected(hostname=CONDA_HOST, dns_only=False):
    try:
        host = gethostbyname(hostname)
        assert host is not None

        # Do not use this: not HTTP proxy compliant
        #   s = socket.create_connection((host, 80), 2)
        #   s.close()

        if not dns_only:
            code = requests.get("https://%s" % REMOTE_HOST).status_code
            assert code is not None

        return True
    except:
        pass
    
    return False

@smarie
Copy link
Contributor Author

smarie commented Apr 21, 2020

Note that a less user-friendly alternative would be to propagate the flag into nox options, so that users wishing to execute a conda based session in an offline context could call nox --offline. This is not my preferred choice but I let you decide of course.

smarie pushed a commit to smarie/nox that referenced this issue Apr 21, 2020
Offline mode is now auto-detected by default by conda_install. This…
04780bc 
… allows users to continue executing nox sessions on already installed environments. This behaviour can be disabled by setting `auto_offline=False`. Fixes wntrblm#313
@smarie smarie mentioned this issue Apr 21, 2020
Merged
theacodes pushed a commit that referenced this issue Jun 21, 2020
@smarie
Auto-offline for conda_install (#314)
3098a7a 
* Offline mode is now auto-detected by default by `conda_install`. This allows users to continue executing nox sessions on already installed environments. This behaviour can be disabled by setting `auto_offline=False`. Fixes #313

* Fixed args order for offline option and added tests

* Black-ened code

* Fixed conda options order again

* Added a log message when doing auto-offline

* Fixed mypy errors

* Fixed mypy errors (2)

* mypy fix (3)

* Fixed last failing test

* Improved coverage

* Last coverage fix ?

* Blackened

* Last Flake8 fix

* removed dependency to `requests`

* removed dependency to `urllib3`

* Simplified offline checks

* fixed import

* fixed test

* Added pragma no cover

Co-authored-by: Sylvain MARIE <sylvain.marie@se.com>
@Tolker-KU Tolker-KU mentioned this issue Dec 29, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Auto-offline for conda_install smarie/nox
1 participant
@smarie