Skip to content

voila mode #277

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 101 commits into from
Mar 7, 2023
Merged

voila mode #277

merged 101 commits into from
Mar 7, 2023

Conversation

bcwu
Copy link
Contributor

@bcwu bcwu commented Aug 23, 2022

Description

Add support for Voila

  • write-manifest voila
    • by file
    • by directory
  • deploy manifest voila
  • deploy voila
    • by file
    • by directory
  • voila.json
  • multi-notebook mode

fixes #22006
fixes #22049
fixes #22047

@kgartland-rstudio
Copy link
Contributor

deploying voila with no requirements.txt file results in an error

It seems when deploying voila content we aren't running pip freeze when the requirements.txt file is missing. This results in Connect failing to build the content and throws the following error:

2022/08/29 14:07:32.507057936 Unexpected error while building Python environment: [Errno 2] No such file or directory: 'requirements.txt'
2022/08/29 14:07:32.509883720 Traceback (most recent call last):
2022/08/29 14:07:32.509892493   File "/connect/python/build_environment.py", line 409, in main
2022/08/29 14:07:32.509941456     specs = pip_read_requirements_file(requirements_file)
2022/08/29 14:07:32.509942992   File "/connect/python/report_helpers.py", line 255, in pip_read_requirements_file
2022/08/29 14:07:32.509961389     with open(environment_file, "r") as f:
2022/08/29 14:07:32.509962621 FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'
Build error: exit status 255
An error occurred while building the content (build-failed-error)
Error from Connect server: exit status 255
Error: Task exited with status 1.

@kgartland-rstudio
Copy link
Contributor

When deploying voila from a directory where no voila content exists an internal error is thrown

In this example, I was mistakenly in the parent directory which contained only two folders and no voila content. When I deployed I hit the following error:

> rsconnect deploy voila ./ -n yeti-password
    Warning: Capturing the environment using 'pip freeze'.
             Consider creating a requirements.txt file instead.
Validating server... 	[OK]
Validating app mode... 	[OK]
Making bundle ... 	[ERROR]: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "/Users/kgartland/.pyenv/versions/3.8.2/lib/python3.8/site-packages/rsconnect/main.py", line 86, in wrapper
    result = func(*args, **kwargs)
  File "/Users/kgartland/.pyenv/versions/3.8.2/lib/python3.8/site-packages/rsconnect/main.py", line 811, in deploy_voila
    ce.make_bundle(
  File "/Users/kgartland/.pyenv/versions/3.8.2/lib/python3.8/site-packages/rsconnect/log.py", line 186, in wrapper
    result = method(self, *args, **kw)
  File "/Users/kgartland/.pyenv/versions/3.8.2/lib/python3.8/site-packages/rsconnect/api.py", line 566, in make_bundle
    bundle = func(*args, **kwargs)
  File "/Users/kgartland/.pyenv/versions/3.8.2/lib/python3.8/site-packages/rsconnect/bundle.py", line 849, in make_voila_bundle
    base_dir = dirname(entrypoint)
  File "/Users/kgartland/.pyenv/versions/3.8.2/lib/python3.8/posixpath.py", line 152, in dirname
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Internal error: expected str, bytes or os.PathLike object, not NoneType

@kgartland-rstudio
Copy link
Contributor

inaccurate error text when re-deploying a mis-matched content type

re-deploying voila content where a pre-existing rsconnect-python folder contains a json file with a non-voila app_mode, results in an error mentioning Python Shiny Application:

> rsconnect deploy voila ./ -n yeti-password
    Warning: Capturing the environment using 'pip freeze'.
             Consider creating a requirements.txt file instead.
Validating server... 	[OK]
Validating app mode... 	[ERROR]: Deploying with mode 'Python Shiny Application',
but the existing deployment has mode 'Bokeh Application'.
Use the --new option to create a new deployment of the desired type.
Error: Deploying with mode 'Python Shiny Application',
but the existing deployment has mode 'Bokeh Application'.
Use the --new option to create a new deployment of the desired type.

content of the rsconnect-python/*.json file:

{
    "https://connect.localtest.me/rsc/dev-password": {
        "server_url": "https://connect.localtest.me/rsc/dev-password",
        "filename": "/Users/kgartland/Projects/voila",
        "app_url": "https://connect.localtest.me:443/rsc/dev-password/content/bb4d3939-86ef-4972-8297-4a81a94c4ca9/",
        "app_id": 19,
        "app_guid": "bb4d3939-86ef-4972-8297-4a81a94c4ca9",
        "title": "no-req",
        "app_mode": "python-bokeh"
    }

@bcwu
Copy link
Contributor Author

bcwu commented Aug 29, 2022

deploying voila with no requirements.txt file results in an error

It seems when deploying voila content we aren't running pip freeze when the requirements.txt file is missing. This results in Connect failing to build the content and throws the following error:

2022/08/29 14:07:32.507057936 Unexpected error while building Python environment: [Errno 2] No such file or directory: 'requirements.txt'
2022/08/29 14:07:32.509883720 Traceback (most recent call last):
2022/08/29 14:07:32.509892493   File "/connect/python/build_environment.py", line 409, in main
2022/08/29 14:07:32.509941456     specs = pip_read_requirements_file(requirements_file)
2022/08/29 14:07:32.509942992   File "/connect/python/report_helpers.py", line 255, in pip_read_requirements_file
2022/08/29 14:07:32.509961389     with open(environment_file, "r") as f:
2022/08/29 14:07:32.509962621 FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'
Build error: exit status 255
An error occurred while building the content (build-failed-error)
Error from Connect server: exit status 255
Error: Task exited with status 1.

fixed by 564e82a

When deploying voila from a directory where no voila content exists an internal error is thrown

In this example, I was mistakenly in the parent directory which contained only two folders and no voila content. When I deployed I hit the following error:

> rsconnect deploy voila ./ -n yeti-password
    Warning: Capturing the environment using 'pip freeze'.
             Consider creating a requirements.txt file instead.
Validating server... 	[OK]
Validating app mode... 	[OK]
Making bundle ... 	[ERROR]: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "/Users/kgartland/.pyenv/versions/3.8.2/lib/python3.8/site-packages/rsconnect/main.py", line 86, in wrapper
    result = func(*args, **kwargs)
  File "/Users/kgartland/.pyenv/versions/3.8.2/lib/python3.8/site-packages/rsconnect/main.py", line 811, in deploy_voila
    ce.make_bundle(
  File "/Users/kgartland/.pyenv/versions/3.8.2/lib/python3.8/site-packages/rsconnect/log.py", line 186, in wrapper
    result = method(self, *args, **kw)
  File "/Users/kgartland/.pyenv/versions/3.8.2/lib/python3.8/site-packages/rsconnect/api.py", line 566, in make_bundle
    bundle = func(*args, **kwargs)
  File "/Users/kgartland/.pyenv/versions/3.8.2/lib/python3.8/site-packages/rsconnect/bundle.py", line 849, in make_voila_bundle
    base_dir = dirname(entrypoint)
  File "/Users/kgartland/.pyenv/versions/3.8.2/lib/python3.8/posixpath.py", line 152, in dirname
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Internal error: expected str, bytes or os.PathLike object, not NoneType

fixed by 6fdc811

inaccurate error text when re-deploying a mis-matched content type

re-deploying voila content where a pre-existing rsconnect-python folder contains a json file with a non-voila app_mode, results in an error mentioning Python Shiny Application:

> rsconnect deploy voila ./ -n yeti-password
    Warning: Capturing the environment using 'pip freeze'.
             Consider creating a requirements.txt file instead.
Validating server... 	[OK]
Validating app mode... 	[ERROR]: Deploying with mode 'Python Shiny Application',
but the existing deployment has mode 'Bokeh Application'.
Use the --new option to create a new deployment of the desired type.
Error: Deploying with mode 'Python Shiny Application',
but the existing deployment has mode 'Bokeh Application'.
Use the --new option to create a new deployment of the desired type.

content of the rsconnect-python/*.json file:

{
    "https://connect.localtest.me/rsc/dev-password": {
        "server_url": "https://connect.localtest.me/rsc/dev-password",
        "filename": "/Users/kgartland/Projects/voila",
        "app_url": "https://connect.localtest.me:443/rsc/dev-password/content/bb4d3939-86ef-4972-8297-4a81a94c4ca9/",
        "app_id": 19,
        "app_guid": "bb4d3939-86ef-4972-8297-4a81a94c4ca9",
        "title": "no-req",
        "app_mode": "python-bokeh"
    }

fixed by 0eab04a

@kgartland-rstudio
Copy link
Contributor

kgartland-rstudio commented Sep 6, 2022

The deploy voila help page mentions static mode which isn't available in voila deployments:

rsconnect deploy voila --help
Usage: rsconnect deploy voila [OPTIONS] PATH [EXTRA_FILES]...

Deploy a Voila notebook to RStudio Connect. This may be done by source or as
a static HTML page. If the notebook is deployed as a static HTML page
(--static), it cannot be scheduled or rerun on the Connect server.

I think we can get rid of the second sentence and maybe add something about being able to deploy with customizations by adding a voila.json in the deploy directory.

@kgartland-rstudio
Copy link
Contributor

rsconnect version: 1.10.1.dev19+g26be70b

rsconnect write-manifest requires a filename when it shouldn't. The following command should be able to write a manifest, but instead we're looking for a filename.

rsconnect write-manifest voila ./
Usage: rsconnect write-manifest voila [OPTIONS] FILE [EXTRA_FILES]...
Try 'rsconnect write-manifest voila --help' for help.

Error: Invalid value for 'FILE': File './' is a directory.

@kgartland-rstudio
Copy link
Contributor

kgartland-rstudio commented Sep 8, 2022

Invalid or missing themes cause the notebook to fail to render.

I saw in the example that there's a jupyterlab_miami_nights theme so I added that to my voila.json file but failed to include it in my requirementst.txt file. The theme failed and it caused the notebook to fail to render entirely. Rather than giving the user an error in the Dashboard, a blank page was displayed. The following error was displayed in the application logs.

When using an invalid theme, can we display this error in the dashboard?

ValueError: Could not find lab theme "jupyterlab_miami_nights"

Full Error:

2022/09/08 1:47:05 PM: 404 POST /voila/api/shutdown/f2ab7c9e-1a24-40d8-bbc0-23fdc7662133 (127.0.0.1): Kernel does not exist: f2ab7c9e-1a24-40d8-bbc0-23fdc7662133
2022/09/08 1:47:05 PM: 404 POST /voila/api/shutdown/f2ab7c9e-1a24-40d8-bbc0-23fdc7662133 (127.0.0.1) 2.90ms
2022/09/08 1:47:06 PM: Task exception was never retrieved
2022/09/08 1:47:06 PM: future: .put_html() done, defined at /connect/mnt/app/python/env/lib/python3.10/site-packages/voila/handler.py:202> exception=ValueError('Could not find lab theme "jupyterlab_miami_nights"')>
2022/09/08 1:47:06 PM: Traceback (most recent call last):
2022/09/08 1:47:06 PM: File "/connect/mnt/app/python/env/lib/python3.10/site-packages/voila/handler.py", line 203, in put_html
2022/09/08 1:47:06 PM: async for html_snippet, _ in gen.generate_content_generator(kernel_id, kernel_future):
2022/09/08 1:47:06 PM: File "/connect/mnt/app/python/env/lib/python3.10/site-packages/voila/exporter.py", line 101, in generate_from_notebook_node
2022/09/08 1:47:06 PM: async for output in self.template.generate_async(nb=nb_copy, resources=resources, **extra_context, static_url=self.static_url):
2022/09/08 1:47:06 PM: File "/connect/mnt/app/python/env/lib/python3.10/site-packages/jinja2/environment.py", line 1373, in generate_async
2022/09/08 1:47:06 PM: yield self.environment.handle_exception()
2022/09/08 1:47:06 PM: File "/connect/mnt/app/python/env/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
2022/09/08 1:47:06 PM: raise rewrite_traceback_stack(source=source)
2022/09/08 1:47:06 PM: File "/connect/mnt/app/python/env/share/jupyter/voila/templates/lab/index.html.j2", line 4, in top-level template code
2022/09/08 1:47:06 PM: {% from 'voila_setup.macro.html.j2' import voila_setup_helper_functions, voila_setup_nbextensions with context %}
2022/09/08 1:47:06 PM: File "/connect/mnt/app/python/env/share/jupyter/nbconvert/templates/lab/index.html.j2", line 3, in top-level template code
2022/09/08 1:47:06 PM: {% from 'jupyter_widgets.html.j2' import jupyter_widgets %}
2022/09/08 1:47:06 PM: File "/connect/mnt/app/python/env/share/jupyter/nbconvert/templates/lab/base.html.j2", line 280, in top-level template code
2022/09/08 1:47:06 PM: {% set div_id = uuid4() %}
2022/09/08 1:47:06 PM: File "/connect/mnt/app/python/env/share/jupyter/nbconvert/templates/base/display_priority.j2", line 1, in top-level template code
2022/09/08 1:47:06 PM: {%- extends 'base/null.j2' -%}
2022/09/08 1:47:06 PM: File "/connect/mnt/app/python/env/share/jupyter/nbconvert/templates/base/null.j2", line 24, in top-level template code
2022/09/08 1:47:06 PM: {%- block header -%}
2022/09/08 1:47:06 PM: File "/connect/mnt/app/python/env/share/jupyter/nbconvert/templates/lab/index.html.j2", line 9, in block 'header'
2022/09/08 1:47:06 PM: {%- block html_head -%}
2022/09/08 1:47:06 PM: File "/connect/mnt/app/python/env/share/jupyter/nbconvert/templates/lab/index.html.j2", line 36, in block 'html_head'
2022/09/08 1:47:06 PM: {% block notebook_css %}
2022/09/08 1:47:06 PM: File "/connect/mnt/app/python/env/share/jupyter/nbconvert/templates/lab/index.html.j2", line 43, in block 'notebook_css'
2022/09/08 1:47:06 PM: {{ resources.include_lab_theme(resources.theme) }}
2022/09/08 1:47:06 PM: File "/connect/mnt/app/python/env/lib/python3.10/site-packages/voila/utils.py", line 159, in include_lab_theme
2022/09/08 1:47:06 PM: theme_name, _ = find_lab_theme(name)
2022/09/08 1:47:06 PM: File "/connect/mnt/app/python/env/lib/python3.10/site-packages/nbconvert/exporters/html.py", line 73, in find_lab_theme
2022/09/08 1:47:06 PM: raise ValueError(f'Could not find lab theme "{theme_name}"')
2022/09/08 1:47:06 PM: ValueError: Could not find lab theme "jupyterlab_miami_nights"

@edavidaja
Copy link
Collaborator

Can we make sure that consistent with the other frameworks, calling rsconnect deploy voila returns the help rather than an error?

@edavidaja
Copy link
Collaborator

voila/notebooks on  main [!?]
❯ ls
 basics.ipynb      data.csv               interactive.ipynb   jdash.ipynb           mimerenderers.ipynb      reveal.ipynb
 bqplot.ipynb      gridspecLayout.ipynb   ipympl.ipynb        jupyter_config.json   multiple_widgets.ipynb   stock-report-voila.ipynb
 dashboard.ipynb   hash.ipynb             ipyvolume.ipynb     manifest.json         query-strings.ipynb      xleaflet.ipynb

voila/notebooks on  main [!?]
❯ rsconnect deploy voila -n radixu ./jdash.ipynb
    Warning: the existing manifest.json file will not be used or considered.
    Warning: Capturing the environment using 'pip freeze'.
             Consider creating a requirements.txt file instead.
Validating server... 	[OK]
Validating app mode... 	[OK]
Making bundle ... 	[ERROR]: Path and entrypoint need to match if they are are files.
Error: Path and entrypoint need to match if they are are files.

voila/notebooks on  main [!?]
❯ rsconnect deploy voila -n radixu --entrypoint ./jdash.ipynb ./jdash.ipynb
    Warning: the existing manifest.json file will not be used or considered.
    Warning: Capturing the environment using 'pip freeze'.
             Consider creating a requirements.txt file instead.
Validating server... 	[OK]
Validating app mode... 	[OK]
Making bundle ... 	[ERROR]: Path and entrypoint need to match if they are are files.
Error: Path and entrypoint need to match if they are are files.

@bcwu
Copy link
Contributor Author

bcwu commented Feb 17, 2023

Can we make sure that consistent with the other frameworks, calling rsconnect deploy voila returns the help rather than an error?

Fixed by 8311a1e, which also fixes #277 (comment)

voila/notebooks on  main [!?]
❯ ls
 basics.ipynb      data.csv               interactive.ipynb   jdash.ipynb           mimerenderers.ipynb      reveal.ipynb
 bqplot.ipynb      gridspecLayout.ipynb   ipympl.ipynb        jupyter_config.json   multiple_widgets.ipynb   stock-report-voila.ipynb
 dashboard.ipynb   hash.ipynb             ipyvolume.ipynb     manifest.json         query-strings.ipynb      xleaflet.ipynb

voila/notebooks on  main [!?]
❯ rsconnect deploy voila -n radixu ./jdash.ipynb
    Warning: the existing manifest.json file will not be used or considered.
    Warning: Capturing the environment using 'pip freeze'.
             Consider creating a requirements.txt file instead.
Validating server... 	[OK]
Validating app mode... 	[OK]
Making bundle ... 	[ERROR]: Path and entrypoint need to match if they are are files.
Error: Path and entrypoint need to match if they are are files.

voila/notebooks on  main [!?]
❯ rsconnect deploy voila -n radixu --entrypoint ./jdash.ipynb ./jdash.ipynb
    Warning: the existing manifest.json file will not be used or considered.
    Warning: Capturing the environment using 'pip freeze'.
             Consider creating a requirements.txt file instead.
Validating server... 	[OK]
Validating app mode... 	[OK]
Making bundle ... 	[ERROR]: Path and entrypoint need to match if they are are files.
Error: Path and entrypoint need to match if they are are files.

fixed by 0165f23

@kgartland-rstudio
Copy link
Contributor

Specifying multiple files using --multi-notebook flag fails with a misleading error.

> rsconnect deploy voila ./app.ipynb ./index.ipynb --multi-notebook -n local
 	[ERROR]:
Unable to infer entrypoint.
Multi-notebook deployments need to be specified with the following:
1) A directory as the path
2) Set multi_notebook=True,
    i.e. include --multi-notebook (or -m) in the CLI command.

Error:
Unable to infer entrypoint.
Multi-notebook deployments need to be specified with the following:
1) A directory as the path
2) Set multi_notebook=True,
    i.e. include --multi-notebook (or -m) in the CLI command.

Is mutli-notebook mode supported when using file deployment rather than directory deployment?

@bcwu
Copy link
Contributor Author

bcwu commented Feb 21, 2023

Specifying multiple files using --multi-notebook flag fails with a misleading error.

> rsconnect deploy voila ./app.ipynb ./index.ipynb --multi-notebook -n local
 	[ERROR]:
Unable to infer entrypoint.
Multi-notebook deployments need to be specified with the following:
1) A directory as the path
2) Set multi_notebook=True,
    i.e. include --multi-notebook (or -m) in the CLI command.

Error:
Unable to infer entrypoint.
Multi-notebook deployments need to be specified with the following:
1) A directory as the path
2) Set multi_notebook=True,
    i.e. include --multi-notebook (or -m) in the CLI command.

Is mutli-notebook mode supported when using file deployment rather than directory deployment?

Multi-notebook mode is not supported using file deployment. It must be directory deployment per requirement 1) in the exception message.

bcwu added 2 commits February 24, 2023 11:06
happened when path and entrypoint were provided and there were multiple notebooks in the deploy_dir
@bcwu
Copy link
Contributor Author

bcwu commented Feb 24, 2023

82315e5 fixes #355

bcwu added 2 commits February 24, 2023 11:21
Most of the time this should raise exception.

Adding this line should at least allow the bundle to be created.
@kgartland-rstudio
Copy link
Contributor

Voila deployments from rsconnect-jupyter includes all files in the directory by default. In other notebooks you have to specify which files to include using the Additional Files dialog.

This seems to be publishing as directory rather than by file, is that expected?

Plugin versions:

rsconnect-jupyter server extension version: 1.6.1.dev36+g7d00034
rsconnect-jupyter nbextension version: 1.6.1.dev36+g7d00034
rsconnect-python version:1.14.2.dev164+g6a2973a

Files deployed when deploying as voila:

├── README.md
├── app.ipynb
├── environment.yml
├── manifest.json
├── no-theme
├── none.txt
├── requirements.txt
└── thumbnail.png

Files deployed when deploying the same notebook as a jupyter notebook:

├── app.ipynb
├── manifest.json
└── requirements.txt

@bcwu
Copy link
Contributor Author

bcwu commented Feb 24, 2023

Voila deployments from rsconnect-jupyter includes all files in the directory by default. In other notebooks you have to specify which files to include using the Additional Files dialog.

This seems to be publishing as directory rather than by file, is that expected?

Plugin versions:

rsconnect-jupyter server extension version: 1.6.1.dev36+g7d00034
rsconnect-jupyter nbextension version: 1.6.1.dev36+g7d00034
rsconnect-python version:1.14.2.dev164+g6a2973a

Files deployed when deploying as voila:

├── README.md
├── app.ipynb
├── environment.yml
├── manifest.json
├── no-theme
├── none.txt
├── requirements.txt
└── thumbnail.png

Files deployed when deploying the same notebook as a jupyter notebook:

├── app.ipynb
├── manifest.json
└── requirements.txt

Currently in the CLI we make the following distinction:

  • if you want to deploy a directory:
    • you enter the directory as the path
    • whether you enter an entrypoint or not you'll still get the entire folder
      • it behaves this way because we guess and infer entrypoints, and need to include all filess and sub-folders in multi-notebook mode
  • if you want to deploy just a single file:
    • you have to enter that file as the path

@mmarchetti
Which means the following needs to be changed if we don't want the entire folder:
https://github.com/rstudio/rsconnect-jupyter/blob/7d00034f42721e3225ffbab7faf32d96f0a1e655/rsconnect_jupyter/__init__.py#L205-L206

from

dirname(os_path),
basename(os_path),

to just:

os_path,
None,

@mmarchetti
Copy link
Contributor

I made the change to deploy in single file mode.

@kgartland-rstudio
Copy link
Contributor

I made the change to deploy in single file mode.

Verified rsconnect-jupyter now deploys single file mode.

@bcwu bcwu merged commit c1f163e into master Mar 7, 2023
@bcwu bcwu deleted the bcwu-voila branch March 7, 2023 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants