Multi-output regression example is broken (potential macOS problem)

[timothygebhard]

Describe the bug

When I try to run the example on multi-output regression from the docs, the .fit() routine crashes with a ValueError.

To Reproduce

Steps to reproduce the behavior:

1. Download or copy code from the docs.
2. Run the code.
3. Watch the .fit() routine crash.

Expected behavior

The code should run through and produce results like in the docs.

Actual behavior, stacktrace or logfile

The .fit() routine crashes with the following error message:

[ERROR] [2022-09-29 16:41:12,619:Client-AutoML(1):synthetic] (' Dummy prediction failed with run state StatusType.CRASHED and additional output: {\'error\': \'Result queue is empty\', \'exit_status\': "<class \'pynisher.limit_function_call.AnythingException\'>", \'subprocess_stdout\': \'\', \'subprocess_stderr\': \'Process pynisher function call:\\nTraceback (most recent call last):\\n  File "/usr/local/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap\\n    self.run()\\n  File "/usr/local/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 108, in run\\n    self._target(*self._args, **self._kwargs)\\n  File "/Users/timothy/.virtualenvs/moml/lib/python3.9/site-packages/pynisher/limit_function_call.py", line 108, in subprocess_func\\n    resource.setrlimit(resource.RLIMIT_AS, (mem_in_b, mem_in_b))\\nValueError: current limit exceeds maximum limit\\n\', \'exitcode\': 1, \'configuration_origin\': \'DUMMY\'}.',)
[ERROR] [2022-09-29 16:41:12,619:Client-AutoML(1):synthetic] (' Dummy prediction failed with run state StatusType.CRASHED and additional output: {\'error\': \'Result queue is empty\', \'exit_status\': "<class \'pynisher.limit_function_call.AnythingException\'>", \'subprocess_stdout\': \'\', \'subprocess_stderr\': \'Process pynisher function call:\\nTraceback (most recent call last):\\n  File "/usr/local/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap\\n    self.run()\\n  File "/usr/local/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 108, in run\\n    self._target(*self._args, **self._kwargs)\\n  File "/Users/timothy/.virtualenvs/moml/lib/python3.9/site-packages/pynisher/limit_function_call.py", line 108, in subprocess_func\\n    resource.setrlimit(resource.RLIMIT_AS, (mem_in_b, mem_in_b))\\nValueError: current limit exceeds maximum limit\\n\', \'exitcode\': 1, \'configuration_origin\': \'DUMMY\'}.',)
Traceback (most recent call last):
  File "/Users/timothy/.virtualenvs/moml/lib/python3.9/site-packages/autosklearn/automl.py", line 765, in fit
    self._do_dummy_prediction()
  File "/Users/timothy/.virtualenvs/moml/lib/python3.9/site-packages/autosklearn/automl.py", line 489, in _do_dummy_prediction
    raise ValueError(msg)
ValueError: (' Dummy prediction failed with run state StatusType.CRASHED and additional output: {\'error\': \'Result queue is empty\', \'exit_status\': "<class \'pynisher.limit_function_call.AnythingException\'>", \'subprocess_stdout\': \'\', \'subprocess_stderr\': \'Process pynisher function call:\\nTraceback (most recent call last):\\n  File "/usr/local/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap\\n    self.run()\\n  File "/usr/local/Cellar/python@3.9/3.9.14/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 108, in run\\n    self._target(*self._args, **self._kwargs)\\n  File "/Users/timothy/.virtualenvs/moml/lib/python3.9/site-packages/pynisher/limit_function_call.py", line 108, in subprocess_func\\n    resource.setrlimit(resource.RLIMIT_AS, (mem_in_b, mem_in_b))\\nValueError: current limit exceeds maximum limit\\n\', \'exitcode\': 1, \'configuration_origin\': \'DUMMY\'}.',)

Environment and installation:

Please give details about your installation:

• OS: macOS Monterey (12.6)
• Environment: virtualenv
• Python version: 3.9.14 (main, Sep 6 2022, 23:29:09) [Clang 13.1.6 (clang-1316.0.21.2.5)]
• Auto-sklearn version: 0.15.0 installed via pip

Additional comments:

When I run the Binder version of the notebook, the problem does not occur. Maybe this means it is a macOS-specific issue? After all, multiprocessing is one of those modules that don't always behave the same on Linux and macOS.

[eddiebergman]

Hi @timothygebhard,

Yup this is a known issue and pynisher had been updated to fix it, but we need to update quite a few things at once to make this work. This will be done at the start of the october once I can return to working on auto-sklearn.

Btw, turns out we just can't limit memory with MacOS: https://github.com/automl/pynisher#details. The future iterations of auto-sklearn will simply just raise a warning when attempting to limit memory on Mac and that's all we can do.

Best,
Eddie

[oantoshchenko]

Hi @eddiebergman is there any update on this issue? Is there any workaround?
My setup:

• Apple M1 Pro
• 16 GB
• OS version 13.1

[daisyyywang]

Hi @eddiebergman! I got the same issues here, is anyway we can solve this like change memory limit or number of cores?

here is my environment:

• Apple M1 Pro
• 16 GB
• MacOS Monterey 12.1

[eddiebergman]

Sorry I missed this! No there's been no update, it's a Mac operating system problem that we can't limit memory, your best bet with auto-sklearn the only solution I know of is to set no memory limit, but this can potentially leads to more problems.

[oantoshchenko]

Sorry I missed this! No there's been no update, it's a Mac operating system problem that we can't limit memory, your best bet with auto-sklearn the only solution I know of is to set no memory limit, but this can potentially leads to more problems.

@eddiebergman thank you for the reply. The issue is not with setting a memory limit, the issue is that the error makes usage of the library on Mac impossible. Even though you would not use Mac in a production setting, but it reduces the number of people who want to experiment with the library, like me. Ultimately I went with another library, as running auto-sklearn in a docker container while debugging via ssh, is not optimal.

[jtlz2]

How does one set no memory limit?

[Sunishchal]

+1