Multiple instances with same name on provider

[maigl]

As we put more load to garm we ran into a situation where garm created multiple (> 5 time) intances with the very same name on our (openstack) provider.

We think this might be happening here:
https://github.com/cloudbase/garm/blob/main/runner/pool/pool.go#L1045

In case the setInstanceStatus fails we simply continue creating the instance and at the same time we don't remember that we already tried to create the same instance. Probably we should use continue to skip creating this instance.
What do you think ?
As the error from setInstanceStaus is not logged we currently don't know why it failed.

_{Michael Kuhnt michael.kuhnt@mercedes-benz.com Mercedes-Benz Tech Innovation GmbH (ProviderInformation)}

[gabriel-samfira]

Hmm. Garm shouldn't try to create an instance more than once, unless the previous create attempt failed. If it did, it tries to clean the instance here: https://github.com/cloudbase/garm/blob/main/runner/pool/pool.go#L621-L629

We record any error here: https://github.com/cloudbase/garm/blob/main/runner/pool/pool.go#L1053-L1055

We re-queue it here: https://github.com/cloudbase/garm/blob/main/runner/pool/pool.go#L864 up to 5 times.

It may be that the cleanup fails? Do you see any failed to cleanup instance messages in the log?

[gabriel-samfira]

But you are right. We should not re-queue the instance unless cleanup succeeds! Will look into this ASAP and create a PR.

[gabriel-samfira]

Could you test out: #77. See if this happens again?

[maigl]

Could you test out: #77. See if this happens again?

This runs for a couple of days now .. so far, everything works like a garm .. charm 😄

[gabriel-samfira]

That is awesome!