[ci] Remove the duplication in the ci matrix

[vgvassilev]

We can invert the CI matrix to include only the different options and the steps can become the same. For reference see how's done in the vgvassilev/clad project.

[vgvassilev]

@mcbarton, is that something that you can do for 3 minutes with your ever-increasing CI skills ;)

[mcbarton]

@mcbarton, is that something that you can do for 3 minutes with your ever-increasing CI skills ;)

@vgvassilev yes you can assign this to me.