Move singularity out of main CI tests and into separate docker image

[ashb]

A CI build failed because we couldn't download a release from https://github.com/sylabs/singularity -- which while it was a random network blib, there is no need for the tests for the singularity operators to live in our CI image.

We should instead create "system tests" for singularity, and ensure that the unit tests use mocking.

[ashb]

Wait, I've just looked at the singularity operator tests - it already seems to mock it, so does anyone know why we include this in our CI image? @potiuk ?

[potiuk]

Not sure if it is still needed. Maybe @vsoch can tell more about it whether it is actually needed in the image. I think we had a least few "integration" kind of tests where we run it against a real singluarity installation?

Comment:

In this case it is more of an "integration" tests rather than "system" test (at least in the currently used "integration" test definition that we have). "System" tests (at least how we have defined now) are tests that are reaching out to something that is externally available and we do not have a locally running - via docker compose. Maybe eventually we might want to change those names somehow after we finish all of those "clean-ups" but for now, it's the classification we use :). This is another - unrelated topic on making an order also in the "unit" tests we have - some of them require backend db for example, some of them no, some of them require the whole airflow environment configured, so they are not technically "unit" tests either.

But I am sure we will get there and get the classification right and commonly agreed eventually :). It's just not the right time to do it.

I plan to take a look at this after some of the most important production image changes are handled (this is much higher priority). I created an umbrella task for it #8785 and linked it with all the sub-tasks (minicluster, Hadoop, singularity, kubectl, kind) as all of them can be moved out.