6.0.0rc1 missing dependency on ipyparallel breaks nbconvert

[mmarchetti]

It seems like ipykernel is dependent on ipyparallel, but it is not listed as a package dependency.

Running on MacOS 10.14.6
Python 3.8.5
(please let me know if more information is needed)

virtualenv ./env
source env/bin/activate
pip install --pre nbconvert ipykernel
python -m jupyter nbconvert --to=html --execute Untitled.ipynb
pip freeze

nbconvert fails with:

[NbConvertApp] Converting notebook Untitled.ipynb to html
Traceback (most recent call last):
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/ipykernel/zmqshell.py", line 50, in <module>
    from ipyparallel.engine.datapub import ZMQDataPublisher
ModuleNotFoundError: No module named 'ipyparallel'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/ipykernel/datapub.py", line 18, in <module>
    from ipyparallel.serialize import serialize_object
ModuleNotFoundError: No module named 'ipyparallel'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/ipykernel/serialize.py", line 18, in <module>
    from ipyparallel.serialize.canning import (
ModuleNotFoundError: No module named 'ipyparallel'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mmarchetti/opt/miniconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/mmarchetti/opt/miniconda3/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/ipykernel_launcher.py", line 15, in <module>
    from ipykernel import kernelapp as app
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/ipykernel/kernelapp.py", line 42, in <module>
    from .ipkernel import IPythonKernel
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/ipykernel/ipkernel.py", line 18, in <module>
    from .zmqshell import ZMQInteractiveShell
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/ipykernel/zmqshell.py", line 53, in <module>
    from ipykernel.datapub import ZMQDataPublisher
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/ipykernel/datapub.py", line 21, in <module>
    from ipykernel.serialize import serialize_object
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/ipykernel/serialize.py", line 25, in <module>
    from ipykernel.pickleutil import (
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/ipykernel/pickleutil.py", line 21, in <module>
    from ipyparallel.serialize import codeutil
ModuleNotFoundError: No module named 'ipyparallel'
Traceback (most recent call last):
  File "/Users/mmarchetti/env/bin/jupyter-nbconvert", line 8, in <module>
    sys.exit(main())
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/jupyter_core/application.py", line 254, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/traitlets/config/application.py", line 845, in launch_instance
    app.start()
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 346, in start
    self.convert_notebooks()
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 518, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 483, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbconvert/nbconvertapp.py", line 412, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 181, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 199, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbconvert/exporters/html.py", line 129, in from_notebook_node
    return super().from_notebook_node(nb, resources, **kw)
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbconvert/exporters/templateexporter.py", line 374, in from_notebook_node
    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 143, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbconvert/exporters/exporter.py", line 318, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbconvert/preprocessors/execute.py", line 80, in preprocess
    with self.setup_kernel():
  File "/Users/mmarchetti/opt/miniconda3/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbclient/client.py", line 452, in setup_kernel
    self.start_new_kernel_client()
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbclient/util.py", line 74, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbclient/util.py", line 53, in just_run
    return loop.run_until_complete(coro)
  File "/Users/mmarchetti/opt/miniconda3/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbclient/client.py", line 423, in async_start_new_kernel_client
    await ensure_async(self.kc.wait_for_ready(timeout=self.startup_timeout))
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/nbclient/util.py", line 85, in ensure_async
    result = await obj
  File "/Users/mmarchetti/env/lib/python3.8/site-packages/jupyter_client/client.py", line 180, in _async_wait_for_ready
    raise RuntimeError("Kernel died before replying to kernel_info")
RuntimeError: Kernel died before replying to kernel_info

The output of pip freeze is:

appnope==0.1.2
async-generator==1.10
attrs==21.2.0
backcall==0.2.0
bleach==3.3.0
debugpy==1.3.0
decorator==5.0.9
defusedxml==0.7.1
entrypoints==0.3
ipykernel==6.0.0rc1
ipython==7.24.1
ipython-genutils==0.2.0
jedi==0.18.0
Jinja2==3.0.1
jsonschema==4.0.0a1
jupyter-client==7.0.0a0
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
MarkupSafe==2.0.1
matplotlib-inline==0.1.2
mistune==0.8.4
nbclient==0.5.3
nbconvert==6.1.0
nbformat==5.1.3
nest-asyncio==1.5.1
packaging==20.9
pandocfilters==1.4.3
parso==0.8.2
pexpect==4.8.0
pickleshare==0.7.5
prompt-toolkit==3.0.19
ptyprocess==0.7.0
Pygments==2.9.0
pyparsing==3.0.0b2
pyrsistent==0.17.4rc1
python-dateutil==2.8.1
pyzmq==22.1.0
six==1.16.0
testpath==0.5.0
tornado==6.1
traitlets==5.0.5
wcwidth==0.2.5
webencodings==0.5.1

[asteppke]

Can confirm this on Windows as well. It comes down to pickleutil.py where in 5.5.5 a missing ipyparallel was handled:

try:
    # available since ipyparallel 5.1.1
    from ipyparallel.serialize import codeutil
except ImportError:
    # Deprecated since ipykernel 4.3.1
    from ipykernel import codeutil

and in 6.0.0rc1:

# This registers a hook when it's imported
from ipyparallel.serialize import codeutil

So with this depreciation the ipyparallel dependency is a bit stronger and may require an 'upgrade' from optional to required depending on how the extra_requires is approached here.

[krassowski]

It also break using ipykernel as a kernel in JupyterLab, so this is not contained to use with nbconvert.