Skip to content

Async training does not work with macOS and Python 3.8 #572

@muupan

Description

@muupan
% python examples/gym/train_a3c_gym.py 2 --env CartPole-v0
/Users/fujita-rental/.local/share/virtualenvs/chainerrl-TtJB_mwx/lib/python3.8/site-packages/chainer/_environment_check.py:33: UserWarning: Accelerate has been detected as a NumPy backend library.
vecLib, which is a part of Accelerate, is known not to work correctly with Chainer.
We recommend using other BLAS libraries such as OpenBLAS.
For details of the issue, please see
https://docs.chainer.org/en/stable/tips.html#mnist-example-does-not-converge-in-cpu-mode-on-mac-os-x.

Please be aware that Mac OS X is not an officially supported OS.

  warnings.warn('''\
Traceback (most recent call last):
  File "examples/gym/train_a3c_gym.py", line 210, in <module>
    main()
  File "examples/gym/train_a3c_gym.py", line 196, in main
    experiments.train_agent_async(
  File "/Users/fujita-rental/dev/chainerrl/chainerrl/experiments/train_agent_async.py", line 251, in train_agent_async
    async_.run_async(processes, run_func)
  File "/Users/fujita-rental/dev/chainerrl/chainerrl/misc/async_.py", line 138, in run_async
    p.start()
  File "/Users/fujita-rental/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/fujita-rental/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/fujita-rental/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/Users/fujita-rental/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/fujita-rental/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/fujita-rental/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/fujita-rental/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'train_agent_async.<locals>.run_func'

I suspect this is due to the change made by Python 3.8 of the default mode of multiprocessing.

https://docs.python.org/3/library/multiprocessing.html

Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions