[KED-2961] Tidy CircleCI config #1102
Conversation
…-2961' into feature/tidy-circleci-config#ked-2961
antonymilne
force-pushed
the
feature/tidy-circleci-config#ked-2961
branch
from
44a6065
to
ed08221
Compare
…-2961' into feature/tidy-circleci-config#ked-2961 # Conflicts: # .circleci/config.yml
|
I've managed to simplify
|
…-2961' into feature/tidy-circleci-config#ked-2961
…-2961' into feature/tidy-circleci-config#ked-2961
| - unless: | ||
| condition: | ||
| equal: ["3.6", <<parameters.python_version>>] | ||
| steps: | ||
| - run: | ||
| name: Run unit tests without spark | ||
| command: conda activate kedro_builder; make test-no-spark | ||
| - when: | ||
| condition: | ||
| equal: ["3.6", <<parameters.python_version>>] | ||
| steps: | ||
| - run: | ||
| name: Run unit tests without spark or tensorflow | ||
| command: conda activate kedro_builder; pytest tests --no-cov --ignore tests/extras/datasets/spark --ignore tests/extras/datasets/tensorflow --numprocesses 4 --dist loadfile |
There was a problem hiding this comment.
Tensorflow tests actually were working on 3.7 and 3.8 but not 3.6 on windows. I've added a special case for 3.6 to do what is effectively make test-no-spark but also ignoring tensorflow tests. When I do a follow up PR to develop this special conditional bit can be removed
So on Windows we never run spark tests, and additionally on python 3.6 specifically we also don't run tensorflow tests. But previously we were not running neither spark nor tensorflow tests on all python versions on Windows? Is that right?
There was a problem hiding this comment.
Yes, exactly. So now we are running tensorflow tests on 3.7 and 3.8 since they (seem to) work.
There was a problem hiding this comment.
This generally looks good to me but it was very hard to follow the latest set of changes! Moving forward can we please separate functional changes from refactoring changes even for CI config? I expected the change in builds to be either in a separate PR or in one compact commit so I can see the diff better.
|
Yes, agreed - sorry! Unfortunately by its nature this CI stuff requires a lot of different pushes to test stuff out, hence the many, many commits. I tried to make it clearer by linking to just the diff that the latest changes produced: simplify |
Signed-off-by: Rashida Kanchwala <rashida.kanchwala@quantumblack.com>
Signed-off-by: Yetunde Dada <yetudada@users.noreply.github.com>
Signed-off-by: Laurens Vijnck <laurens_vijnck@mckinsey.com>
Description
I started moving the
private-kedro-syncscripts over to this repo but it got very messy very quickly. So I've decided to do a tidy up of what we've got first before we add even more stuff to it. The result is we now have 150 fewer lines in the CircleCI config and it's (I hope) easier to follow and add the new jobs to.Functionally this is almost identical to what we had before. These are the checks we were doing:
The only difference is that I've removed the one marked as removed since I don't think it was there on purpose. Aside from that, all the same checks as before are still there and run identically to before (small exception: I removed install of
pre-pushbut this shouldn't change anything - see inline comments).Development notes
General principles of the refactoring:
commandstojobsmatrixso we don't need a separate job for every version of Python we're testingparameters, which we were already using for Windows, to also choose the docker imageTested by checking through all the CircleCI jobs - all seem to be working correctly.
Checklist
RELEASE.mdfile