Description
What would you like to be added?
In order to enhance developer experience for this project, I would like the chainsaw e2e tests to be idempotent.
One idea was to patch .spec.preserveInNetbox of the claim, then assert that .spec.preserveInNetbox of the non-Claim is not true before exiting the test (and cleaning up the k8s resources). This is currently not possible for unknown reasons. Observations/Theories:
- Observation: The tests fail non-deterministically and in NetBox there are leftovers.
- Observation: The number of failed tests and the number of leftovers in NetBox does not correlate. e.g. it's possible all tests are successful but there are leftovers.
- There is something weird with booleans, json serialisation and validation in Golang/CRDs: required boolean field will be invalid if the value was
false. go-playground/validator#319 - Theory: The cached client of chainsaw and NetBox Operator are out of sync, chainsaw observes the .spec.preserveInNetbox=false and cleans up the resources while NetBox Operator still has the old state with .spec.preserveInNetbox=true and thus does not clean up things in NetBox.
- Theory: There is an issue in the Claim Controller updating the non-Claim spec in time
Another approach to solve this would be to run a script in the cleanup block of the tests that cleans up resources directly in NetBox e.g. everything that contains the "e2e/" prefix in the description. The advantage of this would be that we don't misuse the chainsaw test itself for cleanup.
A draft PR can be found in #248
Why is this needed?
In order to develop new test cases, it is extremely helpful that a chainsaw test can be executed as many times as one wants without the need to clean up resources in the NetBox backend. This allows for faster development