update and add table menu

iamtodor · iamtodor · commit 145cc3aecf05 · 2022-09-05T22:26:40.000+02:00
diff --git a/article/article.md b/article/article.md
@@ -1,51 +1,82 @@
-Linter helps and advises us about code quality by running sanity checks and displaying warnings and errors about code
-smells. Also, potentially it helps to prevent bugs in a project.
+Hello everyone, I am a Data Platform or DataOps Engineer at [FreshBooks](https://www.freshbooks.com/). In this article I would like to share my experience on configure best practices in CI/CD pipelines. We had a linter configuration that developers could run before submitting PR. We understand that we want to integrate that checks into our regular CI/CD pipeline. This adoption helps to eliminate potential errors, bugs, stylistic errors, and basically have the common code style across the team.  
 
-As we are in FreshBooks using GitHub, so we would like to use it as much as possible.
+We are in FreshBooks using GitHub as a home for our code base, so we would like to use it as much as possible. Recently I finished this configuration so linter and its checks are now part of GitHub actions CI/CD workflow.
 
-Recently I was involved in configuring linters as a part of CI/CD in GitHub actions.
+This article has two major parts: the first one is linter configuration, and the second one is GitHub workflow configuration itself. Feel free to read all the parts, or skip some and jump into specific one you are interested in.
+
+- [Linters configuration](#linters-configuration)
+  * [Disable unwanted checks](#disable-unwanted-checks)
+    + [Documentation](#documentation)
+    + [Import error](#import-error)
+    + [Tweaks for airflow code](#tweaks-for-airflow-code)
+- [GitHub workflow actions CI/CD configurations](#github-workflow-actions-ci-cd-configurations)
+  * [When to run it](#when-to-run-it)
+  * [What files does it run against to](#what-files-does-it-run-against-to)
+  * [Run linter itself](#run-linter-itself)
+- [Conclusion](#conclusion)
 
 ## Linters configuration
 
+**Disclaimer**: author assumes you are familiar with the above-mentioned linters, tools, and checks.
+
+Here are the linters and checks we are going to use:
+
+- [flake8](https://flake8.pycqa.org/en/latest/)
+- [flakeheaven](https://flakeheaven.readthedocs.io/en/latest/)
+- [black](https://github.com/psf/black)
+- [isort](https://github.com/PyCQA/isort)
+
 I would like to share how to configure it for the python project. I prepared a full [github actions python configuration demo repository](https://github.com/iamtodor/github-actions-python-demo).
 
-We use flakeheaven as a flake8 wrapper, which is very easy to configure in one single `pyproject.toml` configuration file.
-The whole `pyproject.toml` configuration file could be found in
+We use flakeheaven as a flake8 wrapper, which is very easy to configure in one single `pyproject.toml`. The whole `pyproject.toml` configuration file could be found in
 a [repo](https://github.com/iamtodor/github-actions-python-configuration-demo/blob/main/pyproject.toml).
 
 ![pyproject.toml](https://github.com/iamtodor/github-actions-python-configuration-demo/blob/main/article/img/flakeheaven-pyproject-config.png?raw=true)
 
-Disclaimer: author assumes you are familiar with the above-mentioned linters, tools, and checks. I would say the config file
-is self-explainable, so I will not stop here for a while. Just a few notes about tiny tweaks.
+I would say the config file is self-explainable, so I will not stop here for a while. Just a few notes about tiny tweaks.
+
+### Disable unwanted checks
 
 A few checks that we don't want to see complain about:
 
-### Documentation
+#### Documentation
 
 We are ok if not every module will be documented. We are ok if not every function or method will be documented.
 
 ![flakeheaven disable docs](https://github.com/iamtodor/github-actions-python-configuration-demo/blob/main/article/img/flakeheaven-disable-docs.png?raw=true)
 
-### Import issues
+#### Import error
+
+Our linter requirements live in a separate file, and we don't aim to mix it with our main production requirements. Hence, linter would complain about import libraries as linter env does not have production libraries, quite obvious.
+
+```
+>>> python -m flakeheaven lint . 
+
+dags/dummy.py
+     3:   1 E0401 Unable to import 'airflow' (import-error) [pylint]
+  from airflow import DAG
+  ^
+     4:   1 E0401 Unable to import 'airflow.operators.dummy_operator' (import-error) [pylint]
+  from airflow.operators.dummy_operator import DummyOperator
+  ^
+```
 
-Our linter requirements live in a separate file, and we don't aim to mix it with our main production requirements.
-Hence, linter could complain about import libraries as linter env does not have production libraries, quite obvious. So
-we need to disable this check. We assume that the developer who writes the code and imports the libs is responsible for
-the tests. So if the test does not pass it means that it's something with import or a code itself. Import checks it's
-not something we would like to put as a linter job.
+So we need to disable this check:
 
 ![flakeheaven disable import checks](https://github.com/iamtodor/github-actions-python-configuration-demo/blob/main/article/img/flakeheaven-disable-import-checks.png?raw=true)
 
-### Tweaks for airflow code
+We assume that the developer who writes the code and imports the libs is responsible for the writing reliable tests. So if the test does not pass it means that it's something with import or a code (logic) itself. Import check is not something we would like to put as a linter job.
+
+#### Tweaks for airflow code
 
 To configure code for Airflow DAGs there are also a few tweaks. Here is the dummy example `dummy.py`.
 
 ![python dummy DAG](https://github.com/iamtodor/github-actions-python-configuration-demo/blob/main/article/img/python-airflow-tasks-order.png?raw=true)
 
-If we run flakeheaven with the default configuration we would see the following error:
+If we run `flakeheaven` with the default configuration we would see the following error:
 
 ```
- python -m flakeheaven lint .                                                       
+>>> python -m flakeheaven lint .                                                       
 
 dags/dummy.py
     17:   9 W503 line break before binary operator [pycodestyle]
@@ -59,49 +90,43 @@ dags/dummy.py
   ^
 ```
 
-However, we want to keep each task specified in a new line, hence we need to disable `W503` from pycodestyle: Disable
-line break before binary operator.
+However, we want to keep each task specified in a new line, hence we need to disable `W503` from pycodestyle.
 
 ![disable W503](https://github.com/iamtodor/github-actions-python-configuration-demo/blob/main/article/img/flakeheaven-diable-line-break-W503.png?raw=true)
 
 Next, with the default configuration we would get the next warning:
 
 ```
-python -m flakeheaven lint .                                                       
+>>> python -m flakeheaven lint .                                                       
 
 dags/dummy.py
     15:   5 W0104 Statement seems to have no effect (pointless-statement) [pylint]
   (
   ^
 ```
 
-The workaround here is to exclude `W0104` from pylint: Statement seems to have no effect (pointless-statement). This is about how we
-specify task order.
+This is about how we specify task order. The workaround here is to exclude `W0104` from pylint.
 
 ![disable W0104](https://github.com/iamtodor/github-actions-python-configuration-demo/blob/main/article/img/flakeheaven-disable-statement-no-effect-W0104.png?raw=true)
 
-## GitHub actions configurations
+## GitHub workflow actions CI/CD configurations
 
 **Disclaimer**: author assumes you are familiar with [GitHub actions](https://github.com/features/actions).
 
 We configure GitHub Workflow to be triggered on every PR against the main (master) branch.
 
-Here are the linters and checks we are going to use:
-
-- [flake8](https://flake8.pycqa.org/en/latest/)
-- [flakeheaven](https://flakeheaven.readthedocs.io/en/latest/)
-- [black](https://github.com/psf/black)
-- [isort](https://github.com/PyCQA/isort)
-
-The whole `py_linter.yml` config could be found in
-a [repo](https://github.com/iamtodor/github-actions-python-demo/blob/main/.github/workflows/py_linter.yml). I will walk you thru it step by step.
+The whole `py_linter.yml` config could be found in a [repo](https://github.com/iamtodor/github-actions-python-demo/blob/main/.github/workflows/py_linter.yml). I will walk you thru it step by step.
 
 ![py_linter.yml](https://github.com/iamtodor/github-actions-python-configuration-demo/blob/main/article/img/gh-config-full.png?raw=true)
 
+### When to run it
+
 We are interested in running linter only when PR has `.py` files. For instance, when we update `README.md` there is no sense to run python linter.
 
 ![configure run workflow on PRs and push](https://github.com/iamtodor/github-actions-python-configuration-demo/blob/main/article/img/gh-config-py-push-pr.png?raw=true)
 
+### What files does it run against to
+
 We are interested in running a linter only against the modified files. Let's say, we take a look at the provided repo, if I update `dags/dummy.py` I don't want to waste a time and resources running linter against `main.py`. For this purpose we use [Paths Filter GitHub Action](https://github.com/dorny/paths-filter), which is very flexible.
 
 ![Paths Filter GitHub Action](https://github.com/iamtodor/github-actions-python-configuration-demo/blob/main/article/img/gh-config-paths-filter.png?raw=true)
@@ -116,7 +141,7 @@ I define the variable where I can find the output (the only `.py` files) from th
 
 ![list files shell](https://github.com/iamtodor/github-actions-python-configuration-demo/blob/main/article/img/gh-config-list-files-shell.png?raw=true)
 
-### Run linter
+### Run linter itself
 
 The next and last step is to run the linter itself.
 
