Auto-retry upgrades on the edge

[michel-laterman]

Describe the enhancement:

An agent upgrade command needs some form of retry mechanism as discussed in #752
This retry mechanism should work for scheduled actions (may retry within the upgrade window), and other actions (given a default time frame or number of attempts).

The action queue may need to be (re) used for this.

Describe a specific use case for the enhancement or feature:

An upgrade may fail due to temporary network issues. In this case we would expect the upgrade to try again later and succeed.

Currently the upgrade process will remove all non-current version artifacts from the downloads directory when starting the upgrade process; so this may redownload a (verified) artifact

[joshdover]

Depending on the scope of change required for this, we should consider backporting this to the 8.4 release branch.

[ph]

Few notes concerning this features that would need to be considered:

• We should consider some kind of backoff strategy / exponentials to the retry mechanism.
• We need to make sure we don't introduce a more thundering horde effect when we retry the action.

Questions:

• Where would that retry logic goes? Should that be a concerns of queue where all events goes into a local queue and get acked or failed? Should that be something else? @michel-laterman I would love to see something written down in this issues concerning the implementation and the lifecycle of theses events.
• Where would the retry rules should be defined, should they be decided by the emitter of the action?