Skip to content

Commit

Permalink
apply pre-commit (#693)
Browse files Browse the repository at this point in the history
  • Loading branch information
Borda authored Oct 7, 2022
1 parent 54f7d7a commit fdf4880
Show file tree
Hide file tree
Showing 25 changed files with 81 additions and 84 deletions.
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ about: Create a report to help us improve
title: ''
labels: bug, help wanted
assignees: ''

---

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ about: Suggest an idea for this project
title: ''
labels: enhancement, help wanted
assignees: ''

---

## 🚀 Feature

<!-- A clear and concise description of the feature proposal -->

### Motivation
Expand Down
3 changes: 2 additions & 1 deletion .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,5 @@ Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context.
List any dependencies that are required for this change.
-->
Fixes #<issue_number>

Fixes #\<issue_number>
1 change: 0 additions & 1 deletion .github/codecov.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,3 @@ comment:
layout: "reach,diff,flags,files,footer"
behavior: default
require_changes: no

4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ jobs:
python -VV
python -m site
python -m pip install --upgrade pip setuptools wheel
python -m pip install --upgrade coverage[toml] virtualenv tox tox-gh-actions
python -m pip install --upgrade coverage[toml] virtualenv tox tox-gh-actions
- name: "Run tox targets for ${{ matrix.python-version }}"
run: python -m tox
Expand Down Expand Up @@ -60,4 +60,4 @@ jobs:
set -xe
python -m pip install virtualenv tox
- name: "Run tox targets for ${{ matrix.toxenv }}"
run: python -m tox
run: python -m tox
1 change: 0 additions & 1 deletion CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# Code of Conduct

Please read our entire [Code of Conduct](https://github.com/nteract/nteract/blob/main/CODE_OF_CONDUCT.md)

24 changes: 12 additions & 12 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,8 +96,8 @@ This will generate `.html` files in the `/.tox/docs_out/` directory. Once you ar
The general workflow for this will be:

1. Run local tests
2. Pushed changes to your forked repository
3. Open pull request to main repository
1. Pushed changes to your forked repository
1. Open pull request to main repository

### Run Tests Locally

Expand Down Expand Up @@ -125,17 +125,17 @@ and ensure the remotes point to your GitHub. Don't work on the main branch!

1. Commit changes to local repository:

```bash
git checkout -b my-feature
git add <updated_files>
git commit
```
```bash
git checkout -b my-feature
git add <updated_files>
git commit
```

2. Push changes to your remote repository:
1. Push changes to your remote repository:

```bash
git push -u origin my-feature
```
```bash
git push -u origin my-feature
```

### Create Pull Request

Expand All @@ -149,6 +149,6 @@ _Note: You might want to set a reference to the main repository to fetch/merge f
git remote add upstream https://github.com/nteract/papermill
```

It's possible you will have conflicts between your repository and main. Here, `main` is meant to be synchronized with the ```upstream``` repository. GitHub has some good [documentation](https://help.github.com/articles/resolving-a-merge-conflict-using-the-command-line/) on merging pull requests from the command line.
It's possible you will have conflicts between your repository and main. Here, `main` is meant to be synchronized with the `upstream` repository. GitHub has some good [documentation](https://help.github.com/articles/resolving-a-merge-conflict-using-the-command-line/) on merging pull requests from the command line.
Happy hacking on Papermill!
80 changes: 39 additions & 41 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<a href="https://github.com/nteract/papermill"><img src="https://media.githubusercontent.com/media/nteract/logos/master/nteract_papermill/exports/images/png/papermill_logo_wide.png" height="48px" /></a>
=======================================================================================================================================================================
# <a href="https://github.com/nteract/papermill"><img src="https://media.githubusercontent.com/media/nteract/logos/master/nteract_papermill/exports/images/png/papermill_logo_wide.png" height="48px" /></a>

<!---(binder links generated at https://mybinder.readthedocs.io/en/latest/howto/badges.html and compressed at https://tinyurl.com) -->

[![CI](https://github.com/nteract/papermill/actions/workflows/ci.yml/badge.svg)](https://github.com/nteract/papermill/actions/workflows/ci.yml)
[![CI](https://github.com/nteract/papermill/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/nteract/papermill/actions/workflows/ci.yml)
[![image](https://codecov.io/github/nteract/papermill/coverage.svg?branch=main)](https://codecov.io/github/nteract/papermill?branch=main)
Expand All @@ -18,20 +18,20 @@ Jupyter Notebooks.

Papermill lets you:

- **parameterize** notebooks
- **execute** notebooks
- **parameterize** notebooks
- **execute** notebooks

This opens up new opportunities for how notebooks can be used. For
example:

- Perhaps you have a financial report that you wish to run with
different values on the first or last day of a month or at the
beginning or end of the year, **using parameters** makes this task
easier.
- Do you want to run a notebook and depending on its results, choose a
particular notebook to run next? You can now programmatically
**execute a workflow** without having to copy and paste from
notebook to notebook manually.
- Perhaps you have a financial report that you wish to run with
different values on the first or last day of a month or at the
beginning or end of the year, **using parameters** makes this task
easier.
- Do you want to run a notebook and depending on its results, choose a
particular notebook to run next? You can now programmatically
**execute a workflow** without having to copy and paste from
notebook to notebook manually.

Papermill takes an *opinionated* approach to notebook parameterization and
execution based on our experiences using notebooks at scale in data
Expand All @@ -41,14 +41,14 @@ pipelines.

From the command line:

``` {.sourceCode .bash}
```{.sourceCode .bash}
pip install papermill
```

For all optional io dependencies, you can specify individual bundles
like `s3`, or `azure` -- or use `all`. To use Black to format parameters you can add as an extra requires ['black'].
like `s3`, or `azure` -- or use `all`. To use Black to format parameters you can add as an extra requires \['black'\].

``` {.sourceCode .bash}
```{.sourceCode .bash}
pip install papermill[all]
```

Expand All @@ -62,13 +62,13 @@ drop support in the future.

### Parameterizing a Notebook

To parameterize your notebook designate a cell with the tag ``parameters``.
To parameterize your notebook designate a cell with the tag `parameters`.

![enable parameters in Jupyter](docs/img/enable_parameters.gif)

Papermill looks for the ``parameters`` cell and treats this cell as defaults for the parameters passed in at execution time. Papermill will add a new cell tagged with ``injected-parameters`` with input parameters in order to overwrite the values in ``parameters``. If no cell is tagged with ``parameters`` the injected cell will be inserted at the top of the notebook.
Papermill looks for the `parameters` cell and treats this cell as defaults for the parameters passed in at execution time. Papermill will add a new cell tagged with `injected-parameters` with input parameters in order to overwrite the values in `parameters`. If no cell is tagged with `parameters` the injected cell will be inserted at the top of the notebook.

Additionally, if you rerun notebooks through papermill and it will reuse the ``injected-parameters`` cell from the prior run. In this case Papermill will replace the old ``injected-parameters`` cell with the new run's inputs.
Additionally, if you rerun notebooks through papermill and it will reuse the `injected-parameters` cell from the prior run. In this case Papermill will replace the old `injected-parameters` cell with the new run's inputs.

![image](docs/img/parameters.png)

Expand All @@ -79,7 +79,7 @@ the Python API and (2) through the command line interface.

#### Execute via the Python API

``` {.sourceCode .python}
```{.sourceCode .python}
import papermill as pm
pm.execute_notebook(
Expand All @@ -94,48 +94,48 @@ pm.execute_notebook(
Here's an example of a local notebook being executed and output to an
Amazon S3 account:

``` {.sourceCode .bash}
```{.sourceCode .bash}
$ papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1
```

**NOTE:**
If you use multiple AWS accounts, and you have [properly configured your AWS credentials](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html), then you can specify which account to use by setting the `AWS_PROFILE` environment variable at the command-line. For example:

``` {.sourceCode .bash}
```{.sourceCode .bash}
$ AWS_PROFILE=dev_account papermill local/input.ipynb s3://bkt/output.ipynb -p alpha 0.6 -p l1_ratio 0.1
```

In the above example, two parameters are set: ``alpha`` and ``l1_ratio`` using ``-p`` (``--parameters`` also works). Parameter values that look like booleans or numbers will be interpreted as such. Here are the different ways users may set parameters:
In the above example, two parameters are set: `alpha` and `l1_ratio` using `-p` (`--parameters` also works). Parameter values that look like booleans or numbers will be interpreted as such. Here are the different ways users may set parameters:

``` {.sourceCode .bash}
```{.sourceCode .bash}
$ papermill local/input.ipynb s3://bkt/output.ipynb -r version 1.0
```

Using ``-r`` or ``--parameters_raw``, users can set parameters one by one. However, unlike ``-p``, the parameter will remain a string, even if it may be interpreted as a number or boolean.
Using `-r` or `--parameters_raw`, users can set parameters one by one. However, unlike `-p`, the parameter will remain a string, even if it may be interpreted as a number or boolean.

``` {.sourceCode .bash}
```{.sourceCode .bash}
$ papermill local/input.ipynb s3://bkt/output.ipynb -f parameters.yaml
```

Using ``-f`` or ``--parameters_file``, users can provide a YAML file from which parameter values should be read.
Using `-f` or `--parameters_file`, users can provide a YAML file from which parameter values should be read.

``` {.sourceCode .bash}
```{.sourceCode .bash}
$ papermill local/input.ipynb s3://bkt/output.ipynb -y "
alpha: 0.6
l1_ratio: 0.1"
```

Using ``-y`` or ``--parameters_yaml``, users can directly provide a YAML string containing parameter values.
Using `-y` or `--parameters_yaml`, users can directly provide a YAML string containing parameter values.

``` {.sourceCode .bash}
```{.sourceCode .bash}
$ papermill local/input.ipynb s3://bkt/output.ipynb -b YWxwaGE6IDAuNgpsMV9yYXRpbzogMC4xCg==
```

Using ``-b`` or ``--parameters_base64``, users can provide a YAML string, base64-encoded, containing parameter values.
Using `-b` or `--parameters_base64`, users can provide a YAML string, base64-encoded, containing parameter values.

When using YAML to pass arguments, through ``-y``, ``-b`` or ``-f``, parameter values can be arrays or dictionaries:
When using YAML to pass arguments, through `-y`, `-b` or `-f`, parameter values can be arrays or dictionaries:

``` {.sourceCode .bash}
```{.sourceCode .bash}
$ papermill local/input.ipynb s3://bkt/output.ipynb -y "
x:
- 0.0
Expand All @@ -151,25 +151,23 @@ linear_function:

Papermill supports the following name handlers for input and output paths during execution:

* Local file system: `local`
- Local file system: `local`

* HTTP, HTTPS protocol: `http://, https://`
- HTTP, HTTPS protocol: `http://, https://`

* Amazon Web Services: [AWS S3](https://aws.amazon.com/s3/) `s3://`
- Amazon Web Services: [AWS S3](https://aws.amazon.com/s3/) `s3://`

* Azure: [Azure DataLake Store](https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-overview), [Azure Blob Store](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-overview) `adl://, abs://`
- Azure: [Azure DataLake Store](https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-overview), [Azure Blob Store](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-overview) `adl://, abs://`

* Google Cloud: [Google Cloud Storage](https://cloud.google.com/storage/) `gs://`
- Google Cloud: [Google Cloud Storage](https://cloud.google.com/storage/) `gs://`

Development Guide
-----------------
## Development Guide

Read [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines on how to setup a local development environment and make code changes back to Papermill.

For development guidelines look in the [DEVELOPMENT_GUIDE.md](./DEVELOPMENT_GUIDE.md) file. This should inform you on how to make particular additions to the code base.

Documentation
-------------
## Documentation

We host the [Papermill documentation](http://papermill.readthedocs.io)
on ReadTheDocs.
5 changes: 3 additions & 2 deletions binder/cli-simple/README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
# Simple CLI Example

This example uses three notebooks:

- simple_input.ipynb
- simple_output.ipynb
- cli_example.ipynb

1. Open the cli_example.ipynb notebook.

2. Begin executing the cells.
1. Begin executing the cells.

3. One cell demonstrates running papermill from the the command line.
1. One cell demonstrates running papermill from the the command line.
2 changes: 1 addition & 1 deletion binder/cli-simple/simple_output.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -138,4 +138,4 @@
},
"nbformat": 4,
"nbformat_minor": 1
}
}
2 changes: 1 addition & 1 deletion docs/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@ help:
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
6 changes: 3 additions & 3 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
- Update file read to not fail early with boto empty file exception [PR #614](https://github.com/nteract/papermill/pull/614)
- Support new version of gcsfs [PR #624](https://github.com/nteract/papermill/pull/624)
- Fix an issue where the `PapermillExecutionError` can be pickled but will
not be unpicklable [PR #629](https://github.com/nteract/papermill/pull/624)
not be unpicklable [PR #629](https://github.com/nteract/papermill/pull/624)
- Update documentation build and theme
- Remove deprecated `pyarrow.hdfs.connect` call from `iorw.py` [PR #615](https://github.com/nteract/papermill/pull/615)
- Remove support for python 3.5
Expand Down Expand Up @@ -158,8 +158,8 @@ We made it to our [1.0 milestone goals](https://github.com/nteract/papermill/mil

- Input and output paths can now reference input parameters. `my_nb_{nb_type}.ipynb out_{nb_type}.ipynb -p nb_type test` will substitute values into the paths passed in with python format application patterns.
- `read_notebook`, `read_notebooks`, `record`, and `display` api functions are now removed.
- [upstream] ipywidgets are now supported. See [nbconvert docs](https://nbconvert.readthedocs.io/en/latest/execute_api.html#widget-state) for details.
- [upstream] notebook executions which run out of memory no longer hang indefinitely when the kernel dies.
- \[upstream\] ipywidgets are now supported. See [nbconvert docs](https://nbconvert.readthedocs.io/en/latest/execute_api.html#widget-state) for details.
- \[upstream\] notebook executions which run out of memory no longer hang indefinitely when the kernel dies.

## 0.19.1

Expand Down
10 changes: 5 additions & 5 deletions docs/usage-execute.rst
Original file line number Diff line number Diff line change
Expand Up @@ -126,16 +126,16 @@ A similar pattern may be needed for other types of remote storage accounts.

Tracking the Execution with Cell Descriptions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you want to keep track of the execution in a closer way, you can add cell descriptions to the notebook being executed that will be injected to the TQDM progress bar.
If you want to keep track of the execution in a closer way, you can add cell descriptions to the notebook being executed that will be injected to the TQDM progress bar.

Here is an example of the TQDM output during the execution of a 4-cells notebook without cell descriptions:

.. code-block:: bash
10:10 # papermill input_notebook.ipynb output_notebook.ipynb
10:10 # papermill input_notebook.ipynb output_notebook.ipynb
Input Notebook: input_notebook.ipynb
Output Notebook: output_notebook.ipynb
Executing: 0%| | 0/4 [00:00<?, ?cell/s]Executing
Executing: 0%| | 0/4 [00:00<?, ?cell/s]Executing
Executing: 25%|█████████ | 1/4 [00:00<?, ?cell/s]Executing
[...]
Expand All @@ -151,9 +151,9 @@ Taking back our previous example, if we add the description 'FirstCell' to cell

.. code-block:: bash
10:12 # papermill input_notebook.ipynb output_notebook.ipynb
10:12 # papermill input_notebook.ipynb output_notebook.ipynb
Input Notebook: input_notebook.ipynb
Output Notebook: output_notebook.ipynb
Executing FirstCell: 0%| | 0/4 [00:00<?, ?cell/s]Executing
Executing FirstCell: 0%| | 0/4 [00:00<?, ?cell/s]Executing
Executing SecondCell: 25%|█████████ | 1/4 [00:00<?, ?cell/s]Executing
[...]
2 changes: 1 addition & 1 deletion docs/usage-store.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ Papermill can store notebooks in a number of locations including
AWS S3, Azure data blobs, and Azure data lakes.

The modular architecture of papermill allows new data stores to be
added over time.
added over time.
2 changes: 1 addition & 1 deletion papermill/tests/notebooks/binder.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -47,4 +47,4 @@
},
"nbformat": 4,
"nbformat_minor": 2
}
}
2 changes: 1 addition & 1 deletion papermill/tests/notebooks/blank-vscode.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,4 @@
"source": []
}
]
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -344,4 +344,4 @@
},
"nbformat": 4,
"nbformat_minor": 0
}
}
2 changes: 1 addition & 1 deletion papermill/tests/notebooks/systemexit.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -49,4 +49,4 @@
},
"nbformat": 4,
"nbformat_minor": 2
}
}
Loading

0 comments on commit fdf4880

Please sign in to comment.