Sled Agent service launching should cope with U.2s that might not be ready yet

[smklein]

• RSS / Nexus both tell sled agent to launch services
• If we succeed when starting a service, the sled agent stores "here are the services I should launch on cold boot" in the M.2 configuration dataset
• On reboot, we read this list of services from the M.2 and try to start 'em.

RIGHT NOW if we fail to start those services, we don't try again.

This is a bummer! Especially with the dependencies on U.2s - which might take a sec to load - we should add a retry-with-backoff to service launching, kinda like we already do for the NTP and Switch zones.

[smklein]

See: #2946 (comment)

[davepacheco]

I was trying to figure out why this could fail in a retryable way because that seemed surprising. Thanks for the link to the comment in #2946 that provides more context. I wonder if it would make sense to have Sled Agent track the fact that a zone depends on a ZFS pool and avoid trying to start the zone until that pool becomes available. My worry with this approach is the same as always with retry+backoff: that it retries when it's not helpful to (because the pool still isn't available), doesn't retry when it would be helpful (because the pool has just been imported but we're in a backoff), and potentially papers over some other kind of failure (e.g., an actual bug that will cause a zone never to be able to start up will not be visible until we exhaust the retry policy, after which point we may have clobbered lots of useful state by retrying a bunch of times).

(Sorry if I'm missing more relevant context here.)

[smklein]

Agreed, and I'll rename the name of this bug to be slightly less prescriptive about the solution.

I think that it's viable to set up a system where we only attempt the service launch once the appropriate datasets from U.2s are loaded, but it might require more coordination between the ServiceManager and StorageManager. As you say, however, that mitigates a lot of the downsides that would be encountered with an arbitrary retry.

[andrewjstone]

I wonder if it would make sense to have Sled Agent track the fact that a zone depends on a ZFS pool and avoid trying to start the zone until that pool becomes available

That would be a better solution, indeed. I'm sure it's also somewhat harder to implement, but likely worth it.