CI: Use Ansible Molecule for end-to-end testing #328

mamercad · 2022-12-26T20:38:07Z

Switch to Ansible Molecule for end-to-end testing in favor of Kitchen Ansible and Kitchen Docker (this project is currently without a maintainer and has many known issues). Molecule is the de facto answer to Ansible and testing. This pull request covers convergence (and idempotence) using both Python 3.6 and Python 3.8 against StackStorm stable and unstable. The reason why there are two different jobs (python36 and python38) is mainly to handle the different supported versions of Molecule. Lastly, this pull requests switches to completely running the end-to-end testing as non-root (which is nice since it validates that become: true works correctly); it appeared to me that the current end-to-end tests for Rocky Linux 8 and Focal used the root user (the same pattern for extending the existing Dockerfiles was used -- adding a non-root user named molecule). Here's an example running in my fork.

cognifloyd · 2022-12-28T07:01:05Z

I haven't reviewed yet, but I love getting rid of our dependence on ruby here. It feels like such a huge context switch when I have to touch something with ruby and the ruby-ist APIs of kitchen sheet working exclusively in python and Ansible for awhile.

Yes please. Let's use molecule to test this instead of kitchen.

arm4b

Thanks for working on this!

Test-Kitchen doesn't provide us any headaches, - it just works, but it's nice to switch to actively supported and developed Molecule and rely Ruby -> Python for this part.

It looks like the tests are failing because of how the Molecule runs the tests: iterating through the matrix for each Ansible task and it's falling on ChatOps tests due to rate-limiting.

I'd like to know if there's an option to split each check into a separate GH Job with a matrix and limit parallel runs with a factor of 2 or 3.
From that standpoint usability also matters as having a dedicated and clear check for each OS/platform is more helpful. As a bonus, it also provides an easy understanding of how long the install runs on each OS flavor and which OSes are used for testing without looking into the code.

arm4b · 2022-12-29T12:26:22Z

.github/workflows/e2e.yaml

+          - version: 3.6
+            ansible-core: 2.11.12
+            molecule: 3.6.1
+            molecule-docker: 1.1.0


The amount of versions here looks misleading, let's remove those and test only with one molecule version.
Similar to test-kitchen, as a developer/user who wants to modify or add Ansible tasks I don't care which version of Molecule we run as long as it runs the tests. Let's avoid overloading with this version info.

I mean, one of the main reasons I chose to split the jobs between 3.6 and 3.8 was the Molecule version... would you rather just test both 3.6 and 3.8 against the same (lower) version of Molecule?

Yes, just having one pinned version of Molecule is fine.

requirements.yml

arm4b · 2022-12-29T12:35:20Z

.github/workflows/e2e.yaml

+          HUBOT_SLACK_TOKEN: ${{ secrets.HUBOT_SLACK_TOKEN }}
+          ST2_REPO: ${{ matrix.st2_repo.name }}
+
+  python38:


Interesting split matrix between python3.6 vs python3.8.

Thinking that once we add, change or remove the python version we'll just have more places to change in this repo which is undesired as we want to minimize that.
From another side, highlighting py3.6 vs py3.8 could be useful from the quick-look standpoint.

I think on the Ansible side we care more about the OS in the first place (as before) and it's probably good to leave the python versioning matrix to the st2 repo.

I'm not sure that I agree here, if we support 3.6 and 3.8, we should test them both.

We test python 3.6 and 3.8 in the https://github.com/stackStorm/st2 repository.

In the Ansible-st2 we care about the OS matrix (stable-el7, unstable-u20, etc).

We test python 3.6 and 3.8 in the https://github.com/stackStorm/st2 repository.

You're not testing Ansible with Python 3.6 and 3.8 in that repository, are you?

What I'm getting at is that the purpose of this repository is to ensure that Ansible can deploy StackStorm using Python 3.6 and Python 3.8, which is why, again, I think that we should test that.

In addition to the next big step of turning this repository into an Ansible collection, what we should do is be more formal in regards to the requirements. Having a matrix with local Python and Ansible combinations (as this PR introduces) will facilitate this, i.e., when a new version of Ansible Core is released, or when an older version of Ansible Core is no longer supported, it's simply a matter of bumping a couple of versions and all of a sudden we're testing on the new, and no longer testing on the old.

Yeah, that's why I have it running serially, basically. I'll think on how to mitigate the rate-limiting even more.

Please switch to the previous {repo}-{os} build matrix, there are several benefits behind that and it'll also help us with the rate-limiting naturally.

Sure, do you care about testing both Python 3.6 and 3.8 locally (in CI) or do you want to just test 3.8?

Here's a draft PR which ignores 3.6 locally and matrixes across repo and OS, do you like this better?

At a minimum you would expect our playbooks to be run on our target O/Ses to install StackStorm via ansible locally. So that users don't have to have a separate box to install on. Previously we were using random pythons in the ansible E2E as it was picking up various defaults, such as python3.9, python2.7 etc...

In that regard, then I think we'd need to be able to run with more than just python3.6 or python3.8 locally, as taking focal and bionic then they will have different python available.

Per #328 (comment), Ansible client has a very wide py version support between py2.7-py3.11.

I think we're already in a good spot and can trust the py version compatibility advertised and tested by the Ansible project.

At a minimum you would expect our playbooks to be run on our target O/Ses to install StackStorm via ansible locally

Yeah, I've been thinking about this, and, I like this: not relying on a separate machine to deploy from. There are great conversations in here, for sure. I think that this PR is in a better spot than this one, so, to cut down on the noise, I'm going close this one.

mamercad · 2022-12-29T23:14:06Z

and it's falling on ChatOps tests due to rate-limiting.

Yeah, that's why I have it running serially, basically. I'll think on how to mitigate the rate-limiting even more.

arm4b · 2022-12-30T13:23:47Z

Yeah, that's why I have it running serially, basically. I'll think on how to mitigate the rate-limiting even more.

Please switch to the previous {repo}-{os} build matrix, there are several benefits behind that and it'll also help us with the rate-limiting naturally.

mamercad · 2022-12-31T14:35:49Z

Yeah, that's why I have it running serially, basically. I'll think on how to mitigate the rate-limiting even more.

Please switch to the previous {repo}-{os} build matrix, there are several benefits behind that and it'll also help us with the rate-limiting naturally.

Sure, do you care about testing both Python 3.6 and 3.8 locally (in CI) or do you want to just test 3.8?

arm4b · 2023-01-02T18:14:57Z

I see. We don't care which python version is installed on the client(user) side that runs Ansible.
Ansible role repos usually don't test this.

From https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#node-requirement-summary "Managed node Python" is more important, but it has very wide py version support between py2.7-py3.11. So I wouldn't think about this too as Ansible core already does a good job maintaining compatibility.

From the build matrix, what's important is the Ansible Version compatibility with the roles that we have (#242), OS (el7, U20, etc), and repo (stable/unstable).

mamercad · 2023-01-03T23:18:40Z

Closed in favor of #329.

Add Molecule for testing (remove Kitchen)

1046b33

pull-request-size bot added the size/XL PR that changes 500-999 lines. Consider splitting work into several ones that easier to review. label Dec 26, 2022

mamercad changed the title ~~Use Ansible Molecule for end-to-end testing~~ CI: Use Ansible Molecule for end-to-end testing Dec 26, 2022

arm4b suggested changes Dec 29, 2022

View reviewed changes

arm4b added the enhancement label Dec 29, 2022

Remove community.docker

a4a0049

mamercad mentioned this pull request Dec 31, 2022

CI: Use Ansible Molecule for end-to-end testing (Rework) #329

Merged

mamercad closed this Jan 3, 2023

Uh oh!

CI: Use Ansible Molecule for end-to-end testing #328

CI: Use Ansible Molecule for end-to-end testing #328

Uh oh!

Conversation

mamercad commented Dec 26, 2022

Uh oh!

cognifloyd commented Dec 28, 2022

Uh oh!

arm4b left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

arm4b Dec 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mamercad commented Dec 29, 2022

Uh oh!

arm4b commented Dec 30, 2022

Uh oh!

mamercad commented Dec 31, 2022

Uh oh!

arm4b commented Jan 2, 2023

Uh oh!

mamercad commented Jan 3, 2023

Uh oh!

Uh oh!

arm4b left a comment •

edited

Loading

arm4b Dec 29, 2022 •

edited

Loading