[KED-901] Introduce pre-commit hooks and cheks (#157)

.circleci/config.yml

@@ -22,8 +22,11 @@ jobs:
             # Install requirements
             pip install -r requirements.txt -U
 
+            # venv is required for some pre-commit hooks
+            conda install -y virtualenv
+
             # Install test requirements
-            pip install -r test_requirements.txt -U
+            make install-pre-commit
        - run:
           name: Run pylint and flake8
           command: |

.isort.cfg

@@ -6,6 +6,5 @@ include_trailing_comma=True
 force_grid_wrap=0
 use_parentheses=True
 line_length=88
-skip_glob=**/template/**
 known_first_party=features,kedro,tests
 default_section=THIRDPARTY

.pre-commit-config.yaml

@@ -0,0 +1,75 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+
+default_stages: [commit, push]
+
+repos:
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v2.2.3
+    hooks:
+    -   id: trailing-whitespace
+    -   id: end-of-file-fixer
+    -   id: check-yaml  # Checks yaml files for parseable syntax.
+        exclude: "^kedro/template/"
+    -   id: check-json  # Checks json files for parseable syntax.
+    -   id: check-added-large-files
+    -   id: check-case-conflict  # Check for files that would conflict in case-insensitive filesystems
+    -   id: check-merge-conflict  # Check for files that contain merge conflict strings.
+    -   id: debug-statements  # Check for debugger imports and py37+ `breakpoint()` calls in python source.
+        exclude: "^kedro/template/"
+    -   id: detect-private-key  # Detects the presence of private keys
+    -   id: requirements-txt-fixer  # Sorts entries in requirements.txt
+    -   id: flake8
+        exclude: "^kedro/template/"
+
+-   repo: https://github.com/pre-commit/mirrors-isort
+    rev: v4.3.21
+    hooks:
+    -   id: isort
+        exclude: "^kedro/template/"
+
+-   repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v0.720
+    hooks:
+        - id: mypy
+          args: [--allow-redefinition, --ignore-missing-imports]
+          exclude: |
+            (?x)(
+              ^kedro/template/|
+              ^docs/
+            )
+
+-   repo: local
+    hooks:
+    # It's impossible to specify per-directory configuration, so we just run it many times.
+    # https://github.com/PyCQA/pylint/issues/618
+    -   id: pylint-kedro
+        name: "PyLint on kedro/*"
+        language: system
+        pass_filenames: false
+        stages: [push]
+        entry: pylint -j0 --disable=unnecessary-pass kedro
+    -   id: pylint-features
+        name: "PyLint on features/*"
+        language: system
+        pass_filenames: false
+        stages: [push]
+        entry: pylint -j0 --disable=missing-docstring,no-name-in-module features
+    -   id: pylint-extras
+        name: "PyLint on extras/*"
+        language: system
+        pass_filenames: false
+        stages: [push]
+        entry: pylint -j0 extras
+    -   id: pylint-tests
+        name: "PyLint on tests/*"
+        language: system
+        pass_filenames: false
+        stages: [push]
+        entry: pylint -j0 --disable=missing-docstring,redefined-outer-name,no-self-use,invalid-name tests
+    # We need to make some exceptions for 3.5, that's why it's a custom runner.
+    -   id: black
+        name: "Black"
+        language: system
+        pass_filenames: false
+        entry: python -m tools.black_runner kedro extras features tests

.readthedocs.yml

@@ -26,5 +26,3 @@ python:
       extra_requirements:
         - docs
     - requirements: test_requirements.txt
-
-

CONTRIBUTING.md

@@ -101,10 +101,13 @@ You can add new work to `contrib` if you do not need to create a new Kedro CLI c
 See the [`plugin` development documentation](https://kedro.readthedocs.io/en/latest/04_user_guide/09_developing_plugins.html) for guidance on how to design and develop a Kedro `plugin`.
 
 ## CI / CD and running checks locally
-To run E2E tests you need to install the test requirements which includes `behave`, do this using the following command:
+To run E2E tests you need to install the test requirements which includes `behave`.
+Also we use [pre-commit](https://pre-commit.com) hooks for the repository to run the checks automatically.
+It can all be installed using the following command:
 
 ```bash
-pip install -r test_requirements.txt
+make install-test-requirements
+make install-pre-commit
 ```
 
 ### Running checks locally
@@ -146,3 +149,25 @@ make build-docs
 This command will only work on Unix-like systems and requires `pandoc` to be installed.
 
 > ❗ Running `make build-docs` in a Python 3.5 environment may sometimes yield multiple warning messages like the following: `MemoryDataSet.md: WARNING: document isn't included in any toctree`. You can simply ignore them or switch to Python 3.6+ when building documentation.
+
+## Hints on pre-commit usage
+The checks will automatically run on all the changed files on each commit.
+Even more extensive set of checks (including the heavy set of `pylint` checks)
+will run before the push.
+
+The pre-commit/pre-push checks can be omitted by running with `--no-verify` flag, as per below:
+
+```bash
+git commit --no-verify <...>
+git push --no-verify <...>
+```
+(`-n` alias works for `git commit`, but not for `git push`)
+
+All checks will run during CI build, so skipping checks on push will
+not allow you to merge your code with failing checks.
+
+You can uninstall the pre-commit hooks by running
+```bash
+make uninstall-pre-commit
+```
+`pre-commit` will still be used by `make lint`, but will install the git hooks.

Makefile

@@ -14,13 +14,7 @@ legal:
 	python tools/license_and_headers.py
 
 lint:
-	isort
-	pylint -j 0 --disable=unnecessary-pass kedro
-	pylint -j 0 --disable=missing-docstring,redefined-outer-name,no-self-use,invalid-name tests
-	pylint -j 0 --disable=missing-docstring,no-name-in-module features
-	pylint -j 0 extras
-	flake8 kedro tests features extras --exclude kedro/template*
-	mypy --allow-redefinition --ignore-missing-imports kedro tests features extras
+	pre-commit run -a --hook-stage push
 
 test:
 	pytest tests
@@ -38,3 +32,14 @@ devserver: build-docs
 
 package: clean install
 	python setup.py sdist bdist_wheel
+
+install-test-requirements:
+	pip install -r test_requirements.txt
+
+install-pre-commit: install-test-requirements
+	pre-commit install --install-hooks
+	pre-commit install --hook-type pre-push
+
+uninstall-pre-commit:
+	pre-commit uninstall
+	pre-commit uninstall --hook-type pre-push

README.md

@@ -1,7 +1,7 @@
 ![Kedro Logo Banner](https://raw.githubusercontent.com/quantumblacklabs/kedro/master/img/kedro_banner.jpg)
 
-`develop` | `master`  
-----------|---------  
+`develop` | `master`
+----------|---------
 [![CircleCI](https://circleci.com/gh/quantumblacklabs/kedro/tree/develop.svg?style=shield)](https://circleci.com/gh/quantumblacklabs/kedro/tree/develop) | [![CircleCI](https://circleci.com/gh/quantumblacklabs/kedro/tree/master.svg?style=shield)](https://circleci.com/gh/quantumblacklabs/kedro/tree/master)
 [![Build status](https://ci.appveyor.com/api/projects/status/2u74p5g8fdc45wwh/branch/develop?svg=true)](https://ci.appveyor.com/project/QuantumBlack/kedro/branch/develop) | [![Build status](https://ci.appveyor.com/api/projects/status/2u74p5g8fdc45wwh/branch/master?svg=true)](https://ci.appveyor.com/project/QuantumBlack/kedro/branch/master)
 

docs/README.md

@@ -2,7 +2,7 @@
 
 # Kedro documentation style guide
 
-This is the style guide we have used to create [documentation about Kedro](https://kedro.readthedocs.io/en/latest/). 
+This is the style guide we have used to create [documentation about Kedro](https://kedro.readthedocs.io/en/latest/).
 
 When you are writing documentation for your own project, you may find it useful to follow these rules. We will also ask anyone kind enough to contribute to the Kedro documentation to follow our preferred style to maintain consistency and simplicity. However, we are not over-proscriptive and are happy to take contributions regardless, as long as you are happy if we edit your text to follow these rules.
 
@@ -15,7 +15,7 @@ Please follow these simple rules:
 * Use sentence case in titles. We prefer this, _"Sentence case only has one capital except for names like Kedro"_ and not this, _"Title Case Means Capitalise Every Word"_
 * Mark code blocks with the appropriate language to enable [syntax highlighting](https://support.codebasehq.com/articles/tips-tricks/syntax-highlighting-in-markdown)
 * We use a `bash` lexer for all codeblocks that represent the terminal, and we don't include the prompt
-* Bullet points start with capitals and do not end with full-stops 
+* Bullet points start with capitals and do not end with full-stops
 * Prefer to use symbols for bullets instead of numbers unless you are specifically giving a sequence of instructions
 * Keep your sentences short and easy to read
 * Do not plagiarise other authors. Link to their text and credit them
@@ -24,21 +24,21 @@ If you are in doubt, take a look at how we've written the Kedro documentation. I
 
 ## How do I build your documentation?
 
-If you have installed Kedro, the documentation can be found by running `kedro docs` from the command line or following [this link](https://kedro.readthedocs.io/en/latest/). 
+If you have installed Kedro, the documentation can be found by running `kedro docs` from the command line or following [this link](https://kedro.readthedocs.io/en/latest/).
 
 If you make changes to our documentation, which is stored in the `docs/` folder of your Kedro installation, you can rebuild them within a Unix-like environment (with `pandoc` installed) with:
 
 ```bash
 make build-docs
-``` 
+```
 
-We use the [Sphinx](https://www.sphinx-doc.org) framework to build our documentation. The resulting HTML files can be found in `docs/build/html/`. 
+We use the [Sphinx](https://www.sphinx-doc.org) framework to build our documentation. The resulting HTML files can be found in `docs/build/html/`.
 
-If you are a Windows user, you can still contribute to the documentation, but you cannot rebuild it. This is fine! As long as you have made an effort to verify that your Markdown is rendering correctly, and you have followed our basic guidelines above, we will be happy to take your final draft as a pull request and rebuild it for you. 
+If you are a Windows user, you can still contribute to the documentation, but you cannot rebuild it. This is fine! As long as you have made an effort to verify that your Markdown is rendering correctly, and you have followed our basic guidelines above, we will be happy to take your final draft as a pull request and rebuild it for you.
 
 ## Can I contribute to Kedro documentation?
 
-Yes! If you want to fix or extend our documentation, you'd be welcome to do so. When you are ready to submit, please read the full guide to [contributing to Kedro](../CONTRIBUTING.md).   
+Yes! If you want to fix or extend our documentation, you'd be welcome to do so. When you are ready to submit, please read the full guide to [contributing to Kedro](../CONTRIBUTING.md).
 
 Before you contribute any documentation, please do read the above rules for styling your Markdown. If there's something you think is missing or incorrect, and you'd like to get really meta and contribute to our style guide, please branch this file and submit a PR!
 

docs/conf.py

@@ -256,7 +256,7 @@ def autolink_replacements(what: str) -> List[Tuple[str, str, str]]:
         These ``LambdaDataSet``s load and save
 
     Will convert to:
-        These :class:`kedro.io.LambdaDataSet`\s load and save
+        These :class:`kedro.io.LambdaDataSet` load and save
 
     Args:
         what: The objects to create replacement tuples for. Possible values

docs/source/02_getting_started/02_install.md

@@ -17,6 +17,3 @@ You should see an ASCII art graphic and the Kedro version number. For example:
 ![](images/kedro_graphic.png)
 
 If you do not see the graphic displayed, or have any issues with your installation, see the [FAQs](../06_resources/01_faq.md) for help.
-
-
-

docs/source/03_tutorial/05_package_a_project.md

@@ -19,7 +19,7 @@ You can package your project by running `kedro package` from the command line. T
 
 You can also check out [Kedro-Docker](https://github.com/quantumblacklabs/kedro-docker), an officially supported Kedro plugin for packaging and shipping Kedro projects within [Docker](https://www.docker.com/) containers.
 
-We also support converting your Kedro project into an Airflow project with the [Kedro-Airflow](https://github.com/quantumblacklabs/kedro-airflow) plugin. 
+We also support converting your Kedro project into an Airflow project with the [Kedro-Airflow](https://github.com/quantumblacklabs/kedro-airflow) plugin.
 
 ## What is next?
 

docs/source/04_user_guide/05_nodes_and_pipelines.md

@@ -111,7 +111,7 @@ To tag a node, you can simply specify the `tag` argument, as follows:
 
 ```python
 node(func=add, inputs=["a", "b"], outputs="sum", name="adding_a_and_b", tag="node_tag")
-``` 
+```
 
 Moreover, you can [tag all nodes in a ``Pipeline``](./05_nodes_and_pipelines.md#tagging-pipeline-nodes).
 

docs/source/04_user_guide/07_advanced_io.md

@@ -52,7 +52,7 @@ In order to enable versioning, all of the following conditions must be met:
     4. modify its `_describe`, `_load` and `_save` methods respectively to support versioning (see [`kedro.io.CSVLocalDataSet`](/kedro.io.CSVLocalDataSet) for an example implementation) AND
 2. In the `catalog.yml` config file you must enable versioning by setting `versioned` attribute to `true` for the given dataset.
 
-An example dataset could look similar to the below: 
+An example dataset could look similar to the below:
 
 ```python
 from pathlib import Path

docs/source/04_user_guide/10_ipython.md

@@ -174,4 +174,3 @@ Placing your `Notebook.ipynb` file anywhere in `new-kedro-project/notebooks/`, `
 #### Error handling
 
 In case this script fails to execute any of your Kedro project startup scripts, global variable `load_kedro_errors` will contain a dictionary with the key pointing to the failed script path and the value containing exception object.
-

docs/source/04_user_guide/11_working_with_databricks.md

@@ -9,7 +9,7 @@ Databricks Connect connects your favourite IDE (IntelliJ, Eclipse, [VS Code](01_
 
 You can setup Databricks Connect according to the instructions listed [here](https://docs.databricks.com/user-guide/dev-tools/db-connect.html).
 
-> *Note:* You will need to uninstall PySpark, as Databricks Connect will install it for you. This method only works for 5.x versions of Databricks clusters and disables use of Databricks Notebook. 
+> *Note:* You will need to uninstall PySpark, as Databricks Connect will install it for you. This method only works for 5.x versions of Databricks clusters and disables use of Databricks Notebook.
 
 ## GitHub workflow with Databricks
 

docs/source/05_api_docs/kedro.pipeline.rst

@@ -30,4 +30,3 @@ kedro.pipeline
 
        kedro.pipeline.decorators.log_time
        kedro.pipeline.decorators.mem_profile
-

docs/source/06_resources/01_faq.md

@@ -128,4 +128,3 @@ A local copy of documentation about Kedro can be generated by running `kedro doc
 ## How can I find out more about Kedro?
 
 Kedro is on GitHub, and our preferred community channel for feedback is through [GitHub issues](https://github.com/quantumblacklabs/kedro/issues). We will be updating the codebase regularly, and you can find news about updates and features we introduce by heading over to [RELEASE.md](https://github.com/quantumblacklabs/kedro/blob/develop/RELEASE.md).
-

docs/source/css/qb1-sphinx-rtd.css

@@ -402,4 +402,4 @@ h1, h2, .rst-content .toctree-wrapper p.caption, h3, h4, h5, h6, legend {
 .wy-body-for-nav .rst-versions {
     font-size: 16px;
     line-height: 1;
-}
+}

extras/README.md

@@ -6,4 +6,4 @@
 This script helps to locate Kedro project and run IPython startup scripts in
 it when working with Jupyter Notebooks and IPython sessions.
 This script will automatically identify your Kedro project root directory and execute all Python scripts from `<project_dir>/.ipython/profile_default/startup` directory.
-The details can be found in [the user guide](](https://kedro.readthedocs.io/en/latest/04_user_guide/10_ipython.html)).
+The details can be found in [the user guide](](https://kedro.readthedocs.io/en/latest/04_user_guide/10_ipython.html)).

features/pipeline.feature

@@ -42,5 +42,3 @@ Feature: Pipelines that can be run from a library or a template project
     Given I have executed the kedro command "install"
     When the template pipeline is run
     Then it should successfully produce the results
-
-

features/run.feature

@@ -89,4 +89,3 @@ Feature: Run Project
     And I have deleted the credentials file
     When I execute the kedro command "run"
     Then I should get a successful exit code
-

kedro/contrib/README.md

@@ -1,6 +1,6 @@
 # Kedro contrib
 
-The contrib directory is meant to contain user contributions, these 
+The contrib directory is meant to contain user contributions, these
 contributions might get merged into core Kedro at some point in the future.
 
 When create a new module in `contrib`, place it exactly where it would be if it
@@ -9,7 +9,7 @@ was merged into core Kedro.
 For example, data sets are under the core package `kedro.io`. If you are
 contributing a Data Set you should have the following directory:
 `kedro/contrib/my_project/io/` - i.e., the name of your project before the
-`kedro` package path. 
+`kedro` package path.
 
 This is how a module would look like under `kedro/contrib`:
 ```
@@ -36,4 +36,4 @@ extras_require={
 ```
 
 Please notice that a readme with instructions about how to use your module
-and 100% test coverage are required to accept a PR. 
+and 100% test coverage are required to accept a PR.

kedro/contrib/io/catalog_with_default/README.md

@@ -14,8 +14,8 @@ from kedro.runner import SequentialRunner
 def default_io(name):
     return ParquetLocalDataSet(name)
 
-catalog = DataCatalogWithDefault({}, 
-                                 default=default_io, 
+catalog = DataCatalogWithDefault({},
+                                 default=default_io,
                                  default_prefix='data/')
 
 def my_node(input):
@@ -36,4 +36,3 @@ Less safe during production, very handy during development.
 ### Motivation and Context
 
 Very useful during development, saves a lot of time. Usually most datasets in a pipeline come from a single-source.
-