Gym's swan song #9

h-vetinari · 2022-12-14T23:46:08Z

After picking up the history from https://github.com/conda-forge/gym-feedstock, do one last gym build (picking up changes from conda-forge/gym-feedstock#29), before we switch over to the glorious new world of gymnasium.

This reverts commit b36fc53.

conda-forge-linter · 2022-12-14T23:46:13Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

Recipes should usually depend on matplotlib-base as opposed to matplotlib so that runtime environments do not require large packages like qt.

…nda-forge-pinning 2022.12.14.21.36.18

h-vetinari · 2022-12-15T00:12:54Z

+ pip check
gym 0.26.1 requires gym-notices, which is not installed.

OK, I'm going to patch this out for the moment. We can re-add this once we switch to gymnasium (as @thewchan prepared https://github.com/conda-forge/gymnasium-notices-feedstock already 👍)

h-vetinari · 2022-12-15T00:26:27Z

OK, so the segmentation fault I saw in conda-forge/gym-feedstock#29 is still there... 😑

We can try to whittle things down to the various outputs (and then their dependencies), but for now I didn't see a trivial way to just run the test suite for a given extra-package (e.g. gym[box2d]), so I had gone for running it only for gym-all.

Help/input appreciated @pseudo-rnd-thoughts @jkterry1

As mentioned, I can make the artefacts produced in this PR installable locally if that helps with debugging.

h-vetinari · 2022-12-15T03:46:37Z

I'm pretty sure I already ported gymnasium-notices to conda forge?

I literally linked that feedstock in my comment... The point was that gym 0.26.1 still expects to find GYM-notices (for pip check), so unless we specifically want to build gym-notices for this one last release, we'll have to patch things anyway to get pip check to pass.

PS. In case it's not clear from the name & the content of the PR, this is building gym one last time (the release that's supposedly identical with gymnasium), for continuity with the previous history, and then doing the switch to gymnasium.

thewchan · 2022-12-15T03:47:26Z

I'm pretty sure I already ported gymnasium-notices to conda forge?

I literally linked that feedstock in my comment... The point was that gym 0.26.1 still expects to find GYM-notices (for pip check), so unless we specifically want to build gym-notices for this one last release, we'll have to patch things anyway to get pip check to pass.

PS. In case it's not clear from the name & the content of the PR, this is building gym one last time (the release that's supposedly identical with gymnasium), for continuity with the previous history, and then doing the switch to gymnasium.

I know I got confused therefore I deleted the comment sorry

pseudo-rnd-thoughts · 2022-12-15T10:36:03Z

OK, so the segmentation fault I saw in conda-forge/gym-feedstock#29 is still there... 😑

As mentioned, I can make the artefacts produced in this PR installable locally if that helps with debugging.

Yes, could you send me the artefacts, I suspect the issue is with rendering as we have a number of rendering-based tests that probably need xvfc

h-vetinari · 2022-12-15T12:02:24Z

Yes, could you send me the artefacts, I suspect the issue is with rendering as we have a number of rendering-based tests that probably need xvfc

Hey @pseudo-rnd-thoughts, thanks for your help! The artefacts will be downloadable from azure in the CI of this PR (in the previous run you can also see that the segfaults are only on linux; on osx & win there are some failures that look mostly benign - e.g. there's simply no forking on windows, so those tests will need to be skipped).

To install, download the appropriate artefact (for a given platform plus python version), and then do:

conda create -n test_env -c "C:\path\to\where\you\unpacked\the\artefact**" -c conda-forge gym-all python=3.9

** path needs to extend to the point where channeldata.json is found.

Then you can activate the environment and use/test the package as usual.

I suspect the issue is with rendering as we have a number of rendering-based tests that probably need xvfc

Is this a test-only dependency, or one of gym/gymnasium itself?

pseudo-rnd-thoughts

I noticed that you have a windows build however both gym and gymnasium don't officially support it. So we can include it but there might be unknown issues

Also how have you passed all of the tests now?

pseudo-rnd-thoughts · 2022-12-15T12:07:27Z

recipe/patches/0001-disable-requirement-on-gym-notices.patch

+         "cloudpickle >= 1.2.0",
+         "importlib_metadata >= 4.8.0; python_version < '3.10'",
+-        "gym_notices >= 0.0.4",
+        # "gym_notices >= 0.0.4",


Update to gymnasium-notices

Please see this comment.

thewchan · 2022-12-16T17:04:51Z

@h-vetinari do you think I should try to get gym-notices onto conda forge just for backward compatibilities?

h-vetinari · 2022-12-16T20:10:02Z

@h-vetinari do you think I should try to get gym-notices onto conda forge just for backward compatibilities?

I don't think it's worth the effort, but if you feel like doing so, go ahead. :)

h-vetinari · 2022-12-16T20:50:08Z

Btw, as the last run shows, we could publish builds if we don't run the test suite. I'm just not very thrilled about having packages that potentially segfault. But it's the state of the packaging we had before, so...

In any case, if you could have a glance at the artefacts from this PR (@pseudo-rnd-thoughts @jkterry1) that'd be awesome :)

thewchan · 2022-12-20T14:59:04Z

@h-vetinari it is now available: https://github.com/conda-forge/gym-notices-feedstock

h-vetinari · 2022-12-20T22:18:25Z

Thanks @thewchan! :)

@pseudo-rnd-thoughts
Any chance you could look at the segfaults in the test suite on linux with the artefacts¹ from this PR?

how to install those ↩

pseudo-rnd-thoughts · 2022-12-20T23:50:57Z

@h-vetinari Sorry, yes I can do that. Two things, it looks like 3.7 versions doesn't exist and secondly, I feel very stupid but how do I download the artefact and use it for testing?

h-vetinari · 2022-12-21T00:13:58Z

@h-vetinari Sorry, yes I can do that. Two things, it looks like 3.7 versions doesn't exist and secondly, I feel very stupid but how do I download the artefact and use it for testing?

Yes, conda-forge does not build python 3.7 anymore.

My last comment contains a direct link to the download page. Choose your platform and a python version, then, on mouse-over, there's a menu that appears at the right-hand side. Download the artefact somewhere, and unpack it. Then do:

conda create -n test_env -c "path/to/where/you/unpacked/the/artefact*" -c conda-forge gym-all python=3.9**
conda activate test_env
# ready to rumble

* path needs to extend to the folder where channeldata.json is found.
** matching the version you downloaded

jjshoots · 2022-12-21T22:09:45Z

Hey @h-vetinari, I've been delegated by @pseudo-rnd-thoughts to have a stab at this.

I've never used conda before, so I'm pretty unfamiliar with its workflow. Following your steps above, I initially get this:

Collecting package metadata (current_repodata.json): | Unable to retrieve repodata (response: 404) for https://conda.anaconda.org/build_artifacts/noarch/current_repodata.js| Unable to retrieve repodata (response: 404) for https://conda.anaconda.org/build_artifacts/noarch/repodata.jsdone
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): | Unable to retrieve repodata (response: 404) for https://conda.anaconda.org/build_artifacts/noarch/repodata.jsdone
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - conda-forge
  - gym-all

Current channels:

  - https://conda.anaconda.org/build_artifacts/linux-64
  - https://conda.anaconda.org/build_artifacts/noarch
  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

So instead, I did:

conda create -n test_env -c "path/to/where/you/unpacked/the/artefact/channel_data.json" -c conda-forge gym-all python=3.9**

which seemed to work, except it threw this error:

Collecting package metadata (current_repodata.json): failed

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/exceptions.py", line 1082, in __call__
        return func(*args, **kwargs)
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/cli/main.py", line 87, in _main
        exit_code = do_call(args, p)
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/cli/conda_argparse.py", line 84, in do_call
        return getattr(module, func_name)(args, parser)
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/cli/main_create.py", line 41, in execute
        install(args, parser, 'create')
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/cli/install.py", line 260, in install
        unlink_link_transaction = solver.solve_for_transaction(
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 152, in solve_for_transaction
        unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 195, in solve_for_diff
        final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 300, in solve_final_state
        ssc = self._collect_all_metadata(ssc)
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/common/io.py", line 88, in decorated
        return f(*args, **kwds)
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 463, in _collect_all_metadata
        index, r = self._prepare(prepared_specs)
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 1058, in _prepare
        reduced_index = get_reduced_index(self.prefix, self.channels,
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/core/index.py", line 288, in get_reduced_index
        new_records = SubdirData.query_all(spec, channels=channels, subdirs=subdirs,
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 140, in query_all
        result = tuple(concat(executor.map(subdir_query, channel_urls)))
      File "/home/jet/miniconda3/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
        yield fs.pop().result()
      File "/home/jet/miniconda3/lib/python3.9/concurrent/futures/_base.py", line 439, in result
        return self.__get_result()
      File "/home/jet/miniconda3/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
        raise self._exception
      File "/home/jet/miniconda3/lib/python3.9/concurrent/futures/thread.py", line 58, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 132, in <lambda>
        subdir_query = lambda url: tuple(SubdirData(Channel(url), repodata_fn=repodata_fn).query(
      File "/home/jet/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 84, in __call__
        assert not channel.package_filename
    AssertionError

`$ /home/jet/miniconda3/bin/conda create -n test_env -c build_artifacts/channeldata.json conda-forge gym-all python=3.9`

  environment variables:
                 CIO_TEST=<not set>
        CONDA_DEFAULT_ENV=base
                CONDA_EXE=/home/jet/miniconda3/bin/conda
             CONDA_PREFIX=/home/jet/miniconda3
    CONDA_PROMPT_MODIFIER=(base)
         CONDA_PYTHON_EXE=/home/jet/miniconda3/bin/python
               CONDA_ROOT=/home/jet/miniconda3
              CONDA_SHLVL=1
           CURL_CA_BUNDLE=<not set>
            DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
           MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
                     PATH=/home/jet/miniconda3/bin:/home/jet/miniconda3/bin:/home/jet/miniconda3
                          /condabin:/home/jet/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbi
                          n:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>
               WINDOWPATH=2

     active environment : base
    active env location : /home/jet/miniconda3
            shell level : 1
       user config file : /home/jet/.condarc
 populated config files : /home/jet/.condarc
          conda version : 4.12.0
    conda-build version : not installed
         python version : 3.9.12.final.0
       virtual packages : __linux=5.15.0=0
                          __glibc=2.35=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/jet/miniconda3  (writable)
      conda av data dir : /home/jet/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/build_artifacts/channeldata.json/linux-64
                          https://conda.anaconda.org/build_artifacts/channeldata.json/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/jet/miniconda3/pkgs
                          /home/jet/.conda/pkgs
       envs directories : /home/jet/miniconda3/envs
                          /home/jet/.conda/envs
               platform : linux-64
             user-agent : conda/4.12.0 requests/2.27.1 CPython/3.9.12 Linux/5.15.0-56-generic ubuntu/22.04.1 glibc/2.35
                UID:GID : 1000:1000
             netrc file : /home/jet/.netrc
           offline mode : False


An unexpected error has occurred. Conda has prepared the above report.

If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers?

Was this the segfault error that you guys were experiencing? Doesn't seem like it.

Do you have any idea what the above errors are referring to or what I need to do to get to the point of seg fault?

h-vetinari · 2022-12-21T23:06:27Z

Hey @h-vetinari, I've been delegated by @pseudo-rnd-thoughts to have a stab at this.

Hey @jjshoots, thanks for your help!

It seems you must have forgotten a -c in your first incantation (which is short for --channel), because it seems you instructed conda to look for a package named conda-forge. Furthermore, please remove all asterisks (*) from the commands, these were just to annotate/explain. In case it wasn't clear, you need to replace the path/to/where/you/unpacked/the/artefact/channel_data.json with the path where you unpacked the artefact locally.

jjshoots · 2022-12-22T17:36:42Z

Thanks for the help with the install process. I've actually managed to run all tests on gym with no seg faults save for some minor errors regarding np.bool.

h-vetinari · 2022-12-26T04:12:08Z

Hey @jjshoots thanks for testing that's pretty good news - I guess we can go ahead with publishing these builds even without running the tests, but I'd still like to find out why those are segfaulting on our infra, and how to make them pass (or at least skip an appropriate subset, e.g. stuff that cannot pass without an actual screen device).

pseudo-rnd-thoughts · 2023-01-04T19:57:13Z

@h-vetinari Is there anything preventing us from merging this

Also @h-vetinari and @thewchan could you join the gymnasium discord - https://discord.gg/bnJ6kubTg6 then message me PseudoRnd and I can add you as dev to use the conda-dev channel to more discussions

h-vetinari · 2023-01-04T20:07:44Z

Hi @pseudo-rnd-thoughts

Well, mainly I've been waiting for people to continue the discussion here (e.g. why the segfaults in the test suite, and whether we want to just ignore them), but with the holiday period I decided to be patient before repinging.

Please reread the comments above and respond if you can.

Thanks for the invite to the discord; I'll consider it (I'm already drowning in notifications, so I need to be somewhat judicious about whether adding new ones and the signal to noise ratio relative to just packaging gymnasium)

Tobias-Fischer · 2023-01-08T22:35:16Z

Hiya @h-vetinari @pseudo-rnd-thoughts - my two cents: I'd vote for merging here. This PR certainly does not make the situation worse than what it currently is, and @jjshoots was able to run the tests locally without issues, so the builds seem fine. We can then look into issues with our build pipeline when packaging the latest version of gymnasium.

h-vetinari · 2023-01-08T22:44:11Z

I find the lack of responses... confusing, but whatever, let's do it.

…nda-forge-pinning 2023.01.07.12.45.19

conda-forge-webservices · 2023-01-08T22:53:01Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

Recipes should usually depend on matplotlib-base as opposed to matplotlib so that runtime environments do not require large packages like qt.

h-vetinari · 2023-01-08T23:59:24Z

This needs to wait for conda-forge/feedstock-outputs#41 to be merged.

h-vetinari · 2023-01-09T22:36:57Z

@beckermr @ocefpaf
This is failing validation on main somehow:

ERROR getting output validation information from the webservice: JSONDecodeError('Expecting value: line 1 column 1 (char 0)')

I tried a rerender but that didn't change anything. Restart also runs into the same problem.

Any ideas?

h-vetinari · 2023-01-09T22:59:03Z

Should be fixed by conda-forge/admin-requests#562

h-vetinari added 5 commits December 15, 2022 10:29

Revert "temp: add skip to push updated history"

43229a7

This reverts commit b36fc53.

gym 0.22.0

224e295

update dependencies

f7cd0c6

add pip check

a1ddb0d

re-enable gym-atari

680111d

h-vetinari requested review from jkterry1 and thewchan as code owners December 14, 2022 23:46

h-vetinari added 6 commits December 15, 2022 11:03

run test suite

2b7eee8

add setuptool / wheel to build deps

9b32aa6

bump to 0.26.1, switch to github sources

4fa2265

update dependencies

c5bb302

MNT: Re-rendered with conda-build 3.23.3, conda-smithy 3.22.1, and co…

d187b0a

…nda-forge-pinning 2022.12.14.21.36.18

disable gym-atari again

2e2470e

h-vetinari force-pushed the swan_song branch from d211e79 to 2e2470e Compare December 15, 2022 00:04

This was referenced Dec 15, 2022

Fix wrong dependencies conda-forge/gym-feedstock#29

Merged

ARM OSX Migrator #10

Closed

h-vetinari requested a review from pseudo-rnd-thoughts as a code owner December 15, 2022 11:56

h-vetinari marked this pull request as draft December 15, 2022 12:06

pseudo-rnd-thoughts reviewed Dec 15, 2022

View reviewed changes

h-vetinari added 3 commits January 9, 2023 09:48

disable test suite for now

0a096f2

add dependence on gym-notices

7c1734a

MNT: Re-rendered with conda-build 3.23.3, conda-smithy 3.22.1, and co…

0dc8b80

…nda-forge-pinning 2023.01.07.12.45.19

h-vetinari force-pushed the swan_song branch from f74ddd8 to 0dc8b80 Compare January 8, 2023 22:52

This was referenced Jan 8, 2023

allow gym-* to be built as compat outputs of successor gymnasium conda-forge/feedstock-outputs#41

Merged

Switch to gymnasium #13

Merged

h-vetinari marked this pull request as ready for review January 8, 2023 23:22

This was referenced Jan 9, 2023

pillow v9.4.0 conda-forge/pillow-feedstock#132

Merged

libjpeg-turbo and jpeg clobbering each other conda-forge/pillow-feedstock#133

Closed

h-vetinari merged commit 320aa76 into conda-forge:main Jan 9, 2023

h-vetinari deleted the swan_song branch January 9, 2023 18:50

h-vetinari mentioned this pull request Jan 12, 2023

Re-enable gymnasium-atari #16

Merged

h-vetinari mentioned this pull request Oct 22, 2024

Archive this feedstock conda-forge/gym-feedstock#39

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gym's swan song #9

Gym's swan song #9

h-vetinari commented Dec 14, 2022

conda-forge-linter commented Dec 14, 2022

h-vetinari commented Dec 15, 2022

h-vetinari commented Dec 15, 2022

h-vetinari commented Dec 15, 2022

thewchan commented Dec 15, 2022

pseudo-rnd-thoughts commented Dec 15, 2022

h-vetinari commented Dec 15, 2022

pseudo-rnd-thoughts left a comment •

edited

Loading

pseudo-rnd-thoughts Dec 15, 2022

h-vetinari Dec 15, 2022

thewchan commented Dec 16, 2022

h-vetinari commented Dec 16, 2022

h-vetinari commented Dec 16, 2022

thewchan commented Dec 20, 2022

h-vetinari commented Dec 20, 2022

pseudo-rnd-thoughts commented Dec 20, 2022

h-vetinari commented Dec 21, 2022

jjshoots commented Dec 21, 2022

h-vetinari commented Dec 21, 2022

jjshoots commented Dec 22, 2022

h-vetinari commented Dec 26, 2022

pseudo-rnd-thoughts commented Jan 4, 2023

h-vetinari commented Jan 4, 2023

Tobias-Fischer commented Jan 8, 2023

h-vetinari commented Jan 8, 2023

conda-forge-webservices bot commented Jan 8, 2023

h-vetinari commented Jan 8, 2023

h-vetinari commented Jan 9, 2023

h-vetinari commented Jan 9, 2023

Gym's swan song #9

Gym's swan song #9

Conversation

h-vetinari commented Dec 14, 2022

conda-forge-linter commented Dec 14, 2022

h-vetinari commented Dec 15, 2022

h-vetinari commented Dec 15, 2022

h-vetinari commented Dec 15, 2022

thewchan commented Dec 15, 2022

pseudo-rnd-thoughts commented Dec 15, 2022

h-vetinari commented Dec 15, 2022

pseudo-rnd-thoughts left a comment • edited Loading

Choose a reason for hiding this comment

pseudo-rnd-thoughts Dec 15, 2022

Choose a reason for hiding this comment

h-vetinari Dec 15, 2022

Choose a reason for hiding this comment

thewchan commented Dec 16, 2022

h-vetinari commented Dec 16, 2022

h-vetinari commented Dec 16, 2022

thewchan commented Dec 20, 2022

h-vetinari commented Dec 20, 2022

Footnotes

pseudo-rnd-thoughts commented Dec 20, 2022

h-vetinari commented Dec 21, 2022

jjshoots commented Dec 21, 2022

h-vetinari commented Dec 21, 2022

jjshoots commented Dec 22, 2022

h-vetinari commented Dec 26, 2022

pseudo-rnd-thoughts commented Jan 4, 2023

h-vetinari commented Jan 4, 2023

Tobias-Fischer commented Jan 8, 2023

h-vetinari commented Jan 8, 2023

conda-forge-webservices bot commented Jan 8, 2023

h-vetinari commented Jan 8, 2023

h-vetinari commented Jan 9, 2023

h-vetinari commented Jan 9, 2023

pseudo-rnd-thoughts left a comment •

edited

Loading