From fdf488010d46bff25064161e52970a83a493a4f2 Mon Sep 17 00:00:00 2001 From: Jirka Borovec Date: Fri, 7 Oct 2022 03:00:36 +0200 Subject: [PATCH] apply pre-commit (#693) --- .github/ISSUE_TEMPLATE/bug_report.md | 2 +- .github/ISSUE_TEMPLATE/feature_request.md | 2 +- .github/PULL_REQUEST_TEMPLATE.md | 3 +- .github/codecov.yaml | 1 - .github/workflows/ci.yml | 4 +- CODE_OF_CONDUCT.md | 1 - CONTRIBUTING.md | 24 +++--- README.md | 80 +++++++++---------- binder/cli-simple/README.md | 5 +- binder/cli-simple/simple_output.ipynb | 2 +- docs/Makefile | 2 +- docs/changelog.md | 6 +- docs/usage-execute.rst | 10 +-- docs/usage-store.rst | 2 +- papermill/tests/notebooks/binder.ipynb | 2 +- papermill/tests/notebooks/blank-vscode.ipynb | 2 +- .../gcs/gcs_in/gcs-simple_notebook.ipynb | 2 +- papermill/tests/notebooks/systemexit.ipynb | 2 +- .../notebooks/test_notebooknode_io.ipynb | 2 +- papermill/tests/parameters/example.yaml | 2 +- papermill/tests/test_exceptions.py | 2 +- papermill/tests/test_inspect.py | 2 +- papermill/tests/test_translators.py | 2 +- pytest.ini | 1 - requirements/hdfs.txt | 2 +- 25 files changed, 81 insertions(+), 84 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 4b8b49b8..bdd7adb2 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -4,8 +4,8 @@ about: Create a report to help us improve title: '' labels: bug, help wanted assignees: '' - --- + ## 🐛 Bug diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md index d3fa3a98..7cc4d613 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.md +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -4,10 +4,10 @@ about: Suggest an idea for this project title: '' labels: enhancement, help wanted assignees: '' - --- ## 🚀 Feature + ### Motivation diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 2ac2f273..ffcce34a 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -5,4 +5,5 @@ Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> -Fixes # + +Fixes #\ diff --git a/.github/codecov.yaml b/.github/codecov.yaml index 380d2027..a6e1de74 100644 --- a/.github/codecov.yaml +++ b/.github/codecov.yaml @@ -18,4 +18,3 @@ comment: layout: "reach,diff,flags,files,footer" behavior: default require_changes: no - diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7bcfba63..7bd031d5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -30,7 +30,7 @@ jobs: python -VV python -m site python -m pip install --upgrade pip setuptools wheel - python -m pip install --upgrade coverage[toml] virtualenv tox tox-gh-actions + python -m pip install --upgrade coverage[toml] virtualenv tox tox-gh-actions - name: "Run tox targets for ${{ matrix.python-version }}" run: python -m tox @@ -60,4 +60,4 @@ jobs: set -xe python -m pip install virtualenv tox - name: "Run tox targets for ${{ matrix.toxenv }}" - run: python -m tox \ No newline at end of file + run: python -m tox diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index b7152302..e41cc46b 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -1,4 +1,3 @@ # Code of Conduct Please read our entire [Code of Conduct](https://github.com/nteract/nteract/blob/main/CODE_OF_CONDUCT.md) - diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9b0ab8f0..3684225c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -96,8 +96,8 @@ This will generate `.html` files in the `/.tox/docs_out/` directory. Once you ar The general workflow for this will be: 1. Run local tests -2. Pushed changes to your forked repository -3. Open pull request to main repository +1. Pushed changes to your forked repository +1. Open pull request to main repository ### Run Tests Locally @@ -125,17 +125,17 @@ and ensure the remotes point to your GitHub. Don't work on the main branch! 1. Commit changes to local repository: - ```bash - git checkout -b my-feature - git add - git commit - ``` + ```bash + git checkout -b my-feature + git add + git commit + ``` -2. Push changes to your remote repository: +1. Push changes to your remote repository: - ```bash - git push -u origin my-feature - ``` + ```bash + git push -u origin my-feature + ``` ### Create Pull Request @@ -149,6 +149,6 @@ _Note: You might want to set a reference to the main repository to fetch/merge f git remote add upstream https://github.com/nteract/papermill ``` -It's possible you will have conflicts between your repository and main. Here, `main` is meant to be synchronized with the ```upstream``` repository. GitHub has some good [documentation](https://help.github.com/articles/resolving-a-merge-conflict-using-the-command-line/) on merging pull requests from the command line. +It's possible you will have conflicts between your repository and main. Here, `main` is meant to be synchronized with the `upstream` repository. GitHub has some good [documentation](https://help.github.com/articles/resolving-a-merge-conflict-using-the-command-line/) on merging pull requests from the command line. Happy hacking on Papermill! diff --git a/README.md b/README.md index 7506d43d..9b0e4933 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ - -======================================================================================================================================================================= +# + [![CI](https://github.com/nteract/papermill/actions/workflows/ci.yml/badge.svg)](https://github.com/nteract/papermill/actions/workflows/ci.yml) [![CI](https://github.com/nteract/papermill/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/nteract/papermill/actions/workflows/ci.yml) [![image](https://codecov.io/github/nteract/papermill/coverage.svg?branch=main)](https://codecov.io/github/nteract/papermill?branch=main) @@ -18,20 +18,20 @@ Jupyter Notebooks. Papermill lets you: -- **parameterize** notebooks -- **execute** notebooks +- **parameterize** notebooks +- **execute** notebooks This opens up new opportunities for how notebooks can be used. For example: -- Perhaps you have a financial report that you wish to run with - different values on the first or last day of a month or at the - beginning or end of the year, **using parameters** makes this task - easier. -- Do you want to run a notebook and depending on its results, choose a - particular notebook to run next? You can now programmatically - **execute a workflow** without having to copy and paste from - notebook to notebook manually. +- Perhaps you have a financial report that you wish to run with + different values on the first or last day of a month or at the + beginning or end of the year, **using parameters** makes this task + easier. +- Do you want to run a notebook and depending on its results, choose a + particular notebook to run next? You can now programmatically + **execute a workflow** without having to copy and paste from + notebook to notebook manually. Papermill takes an *opinionated* approach to notebook parameterization and execution based on our experiences using notebooks at scale in data @@ -41,14 +41,14 @@ pipelines. From the command line: -``` {.sourceCode .bash} +```{.sourceCode .bash} pip install papermill ``` For all optional io dependencies, you can specify individual bundles -like `s3`, or `azure` -- or use `all`. To use Black to format parameters you can add as an extra requires ['black']. +like `s3`, or `azure` -- or use `all`. To use Black to format parameters you can add as an extra requires \['black'\]. -``` {.sourceCode .bash} +```{.sourceCode .bash} pip install papermill[all] ``` @@ -62,13 +62,13 @@ drop support in the future. ### Parameterizing a Notebook -To parameterize your notebook designate a cell with the tag ``parameters``. +To parameterize your notebook designate a cell with the tag `parameters`. ![enable parameters in Jupyter](docs/img/enable_parameters.gif) -Papermill looks for the ``parameters`` cell and treats this cell as defaults for the parameters passed in at execution time. Papermill will add a new cell tagged with ``injected-parameters`` with input parameters in order to overwrite the values in ``parameters``. If no cell is tagged with ``parameters`` the injected cell will be inserted at the top of the notebook. +Papermill looks for the `parameters` cell and treats this cell as defaults for the parameters passed in at execution time. Papermill will add a new cell tagged with `injected-parameters` with input parameters in order to overwrite the values in `parameters`. If no cell is tagged with `parameters` the injected cell will be inserted at the top of the notebook. -Additionally, if you rerun notebooks through papermill and it will reuse the ``injected-parameters`` cell from the prior run. In this case Papermill will replace the old ``injected-parameters`` cell with the new run's inputs. +Additionally, if you rerun notebooks through papermill and it will reuse the `injected-parameters` cell from the prior run. In this case Papermill will replace the old `injected-parameters` cell with the new run's inputs. ![image](docs/img/parameters.png) @@ -79,7 +79,7 @@ the Python API and (2) through the command line interface. #### Execute via the Python API -``` {.sourceCode .python} +```{.sourceCode .python} import papermill as pm pm.execute_notebook( @@ -94,48 +94,48 @@ pm.execute_notebook( Here's an example of a local notebook being executed and output to an Amazon S3 account: -``` {.sourceCode .bash} +```{.sourceCode .bash} $ papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1 ``` **NOTE:** If you use multiple AWS accounts, and you have [properly configured your AWS credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html), then you can specify which account to use by setting the `AWS_PROFILE` environment variable at the command-line. For example: -``` {.sourceCode .bash} +```{.sourceCode .bash} $ AWS_PROFILE=dev_account papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1 ``` -In the above example, two parameters are set: ``alpha`` and ``l1_ratio`` using ``-p`` (``--parameters`` also works). Parameter values that look like booleans or numbers will be interpreted as such. Here are the different ways users may set parameters: +In the above example, two parameters are set: `alpha` and `l1_ratio` using `-p` (`--parameters` also works). Parameter values that look like booleans or numbers will be interpreted as such. Here are the different ways users may set parameters: -``` {.sourceCode .bash} +```{.sourceCode .bash} $ papermill local/input.ipynb s3://bkt/output.ipynb -r version 1.0 ``` -Using ``-r`` or ``--parameters_raw``, users can set parameters one by one. However, unlike ``-p``, the parameter will remain a string, even if it may be interpreted as a number or boolean. +Using `-r` or `--parameters_raw`, users can set parameters one by one. However, unlike `-p`, the parameter will remain a string, even if it may be interpreted as a number or boolean. -``` {.sourceCode .bash} +```{.sourceCode .bash} $ papermill local/input.ipynb s3://bkt/output.ipynb -f parameters.yaml ``` -Using ``-f`` or ``--parameters_file``, users can provide a YAML file from which parameter values should be read. +Using `-f` or `--parameters_file`, users can provide a YAML file from which parameter values should be read. -``` {.sourceCode .bash} +```{.sourceCode .bash} $ papermill local/input.ipynb s3://bkt/output.ipynb -y " alpha: 0.6 l1_ratio: 0.1" ``` -Using ``-y`` or ``--parameters_yaml``, users can directly provide a YAML string containing parameter values. +Using `-y` or `--parameters_yaml`, users can directly provide a YAML string containing parameter values. -``` {.sourceCode .bash} +```{.sourceCode .bash} $ papermill local/input.ipynb s3://bkt/output.ipynb -b YWxwaGE6IDAuNgpsMV9yYXRpbzogMC4xCg== ``` -Using ``-b`` or ``--parameters_base64``, users can provide a YAML string, base64-encoded, containing parameter values. +Using `-b` or `--parameters_base64`, users can provide a YAML string, base64-encoded, containing parameter values. -When using YAML to pass arguments, through ``-y``, ``-b`` or ``-f``, parameter values can be arrays or dictionaries: +When using YAML to pass arguments, through `-y`, `-b` or `-f`, parameter values can be arrays or dictionaries: -``` {.sourceCode .bash} +```{.sourceCode .bash} $ papermill local/input.ipynb s3://bkt/output.ipynb -y " x: - 0.0 @@ -151,25 +151,23 @@ linear_function: Papermill supports the following name handlers for input and output paths during execution: - * Local file system: `local` +- Local file system: `local` - * HTTP, HTTPS protocol: `http://, https://` +- HTTP, HTTPS protocol: `http://, https://` - * Amazon Web Services: [AWS S3](https://aws.amazon.com/s3/) `s3://` +- Amazon Web Services: [AWS S3](https://aws.amazon.com/s3/) `s3://` - * Azure: [Azure DataLake Store](https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-overview), [Azure Blob Store](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-overview) `adl://, abs://` +- Azure: [Azure DataLake Store](https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-overview), [Azure Blob Store](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-overview) `adl://, abs://` - * Google Cloud: [Google Cloud Storage](https://cloud.google.com/storage/) `gs://` +- Google Cloud: [Google Cloud Storage](https://cloud.google.com/storage/) `gs://` -Development Guide ------------------ +## Development Guide Read [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines on how to setup a local development environment and make code changes back to Papermill. For development guidelines look in the [DEVELOPMENT_GUIDE.md](./DEVELOPMENT_GUIDE.md) file. This should inform you on how to make particular additions to the code base. -Documentation -------------- +## Documentation We host the [Papermill documentation](http://papermill.readthedocs.io) on ReadTheDocs. diff --git a/binder/cli-simple/README.md b/binder/cli-simple/README.md index 5fca4488..23ecfd4b 100644 --- a/binder/cli-simple/README.md +++ b/binder/cli-simple/README.md @@ -1,12 +1,13 @@ # Simple CLI Example This example uses three notebooks: + - simple_input.ipynb - simple_output.ipynb - cli_example.ipynb 1. Open the cli_example.ipynb notebook. -2. Begin executing the cells. +1. Begin executing the cells. -3. One cell demonstrates running papermill from the the command line. +1. One cell demonstrates running papermill from the the command line. diff --git a/binder/cli-simple/simple_output.ipynb b/binder/cli-simple/simple_output.ipynb index ccc28d9e..b36708ed 100644 --- a/binder/cli-simple/simple_output.ipynb +++ b/binder/cli-simple/simple_output.ipynb @@ -138,4 +138,4 @@ }, "nbformat": 4, "nbformat_minor": 1 -} \ No newline at end of file +} diff --git a/docs/Makefile b/docs/Makefile index bc8d6e47..9fea9790 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -17,4 +17,4 @@ help: # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile - @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) \ No newline at end of file + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/changelog.md b/docs/changelog.md index 17984c4f..b634af0c 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -26,7 +26,7 @@ - Update file read to not fail early with boto empty file exception [PR #614](https://github.com/nteract/papermill/pull/614) - Support new version of gcsfs [PR #624](https://github.com/nteract/papermill/pull/624) - Fix an issue where the `PapermillExecutionError` can be pickled but will -not be unpicklable [PR #629](https://github.com/nteract/papermill/pull/624) + not be unpicklable [PR #629](https://github.com/nteract/papermill/pull/624) - Update documentation build and theme - Remove deprecated `pyarrow.hdfs.connect` call from `iorw.py` [PR #615](https://github.com/nteract/papermill/pull/615) - Remove support for python 3.5 @@ -158,8 +158,8 @@ We made it to our [1.0 milestone goals](https://github.com/nteract/papermill/mil - Input and output paths can now reference input parameters. `my_nb_{nb_type}.ipynb out_{nb_type}.ipynb -p nb_type test` will substitute values into the paths passed in with python format application patterns. - `read_notebook`, `read_notebooks`, `record`, and `display` api functions are now removed. -- [upstream] ipywidgets are now supported. See [nbconvert docs](https://nbconvert.readthedocs.io/en/latest/execute_api.html#widget-state) for details. -- [upstream] notebook executions which run out of memory no longer hang indefinitely when the kernel dies. +- \[upstream\] ipywidgets are now supported. See [nbconvert docs](https://nbconvert.readthedocs.io/en/latest/execute_api.html#widget-state) for details. +- \[upstream\] notebook executions which run out of memory no longer hang indefinitely when the kernel dies. ## 0.19.1 diff --git a/docs/usage-execute.rst b/docs/usage-execute.rst index 2ae2509b..2e9fdb4d 100644 --- a/docs/usage-execute.rst +++ b/docs/usage-execute.rst @@ -126,16 +126,16 @@ A similar pattern may be needed for other types of remote storage accounts. Tracking the Execution with Cell Descriptions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -If you want to keep track of the execution in a closer way, you can add cell descriptions to the notebook being executed that will be injected to the TQDM progress bar. +If you want to keep track of the execution in a closer way, you can add cell descriptions to the notebook being executed that will be injected to the TQDM progress bar. Here is an example of the TQDM output during the execution of a 4-cells notebook without cell descriptions: .. code-block:: bash - 10:10 # papermill input_notebook.ipynb output_notebook.ipynb + 10:10 # papermill input_notebook.ipynb output_notebook.ipynb Input Notebook: input_notebook.ipynb Output Notebook: output_notebook.ipynb - Executing: 0%| | 0/4 [00:00= 2.0 \ No newline at end of file +pyarrow >= 2.0