[AIRFLOW-3611] Breeze

[potiuk]

This is a Draft pull request for initial comments. It's not yet intended to be merged.

The intention is to merge it :)

[potiuk]

Comment to self (@mik-laj ) - convert the doc to .rst format. @mik-laj - maybe you will help with that ?? I am happy to get your help on that one.

[ashb]

Non-complete look, just a few spot comments.

[potiuk]

@mik-laj -> we have .rst Breeze description now.

[potiuk]

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Note to reviewer: Please review only the last commit (use commits section above). This change is based on yet unmerged #4936 change and it contains also commit from that change. It's a github limitation unfortunately.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

[potiuk]

@ashb @Fokko -> I've just added a very nice and easy to setup auto-complete for breee command :). It's becoming a really useful piece of software. Also I am down to just 8 errors in Travis CI.

[potiuk]

@ashb @Fokko -> as a follow up I also implemented auto-complete for "run-test" method inside the container.

This was by far the single biggest problem I had when running the tests. Having to remember module names and the : in the right place was a major pain. Now it's super convenient to start typing the module name and use auto-complete. It goes down to individual tests.

I also renamed and shortened the run_unit_test.sh method to simple 'run-test' that you can run anywhere (it is in the path). I also added --skip-db-init flag - most of the tests do not require to re-run the database initialisation and if you want to re-run single test, it takes awfully lot of time. So you can now pass --skip-db-init flag to skip resetdb/initdb. You can even autocomplete it ;).

I think with those changes, life of developers trying to use breeze/replicate CI tests will be much, much easier.

I used nosetests for now but we can do very similar thing with pytest when we switch to it.

[potiuk]

Hello @ashb @mik-laj @andriisoldatenko @dimberman @kaxil @Fokko - I think breeze environment is now ready to be merged. After all the changes merged so far Breeze now becomes a swiss-army-knife of everything-testing for airflow. It builds on top of the CI images and it gives access to all related to testing, static code check (using pre-commits), image management from a single breeze script. I will be talking about it on Monday in NYC and doing workshops on Tuesday so it would be great to merge it before!

[potiuk]

Comprehensive (and I think very usable) documentation here:
https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/BREEZE.rst

Besides Breeze will guide users in many places in case of problems/missing prerequisites and the like.

[dimberman]

Wooo! Ok I'm gonna roll up my sleeves this morning and dive in on this one.

[dimberman]

First look-through this looks good, but I'm gonna hold off an "Approve" until I've had time to actually play with this environment.

[potiuk]

I simplified pylint - it is now fully integrated in pre-commit including filtering out files from pylint_todo list - that made it quite a bit simpler for running static checks with breeze (but now it is based on not-yet merged two commits with AIRFLOW-5285)

[kaxil]

Awesome job @potiuk . A big chunk of work, really appreciate it.

Small nits. I haven't yet tested it but the docs look really solid. Will add more comments once I do test it next week.

[potiuk]

@kaxil @Fokko @mik-laj @ashb I'd love to merge it today for the workshop today :). No huge problem if it is not merged but it would be great if I can show this from master.

[mik-laj]

Good luck!

[mik-laj]

Good luck!

[potiuk]

Woho! Finally !