Agents remains in UPG_SCHEDULED state when upgrade action is cancelled

[juliaElastic]

Bug discovered in scale tests: https://github.com/elastic/ingest-dev/issues/5774

There is a testcase that cancels upgrade action, which leaves agents in UPG_SCHEDULED state.
This causes the next testcase auto upgrade agents fail as auto upgrade doesn't pick up agents with upgrade details in non-terminal state.

This can be reproduced by starting an upgrade action with a rollout duration (e.g. 10m) and then cancelling the action.
The agents are stuck in scheduled state.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[amolnater-qasource]

Hi @juliaElastic
Just an update that we had a similar issue reported here #4293 where agent remains stuck on cancelling the schedule upgrade.

Could you please confirm if these are related?
Thanks!!

[juliaElastic]

@amolnater-qasource Thanks for the link, it seems to be the same issue. cc @cmacknz @ebeahan

[cmacknz]

#4293 (comment) was my original assessment of where this is going wrong. Closing that issue as a duplicate and using this one.

[mbudge]

We've also got a few agents impacted by this.

We'll stop using the scheduled upgrade feature until this is fixed.

Is this highlighted in the release notes?

Thanks!

[cmacknz]

Is this highlighted in the release notes?

Following up on the release notes, confirming the affected versions and working to find a work around. We have the internal support case for this as well now.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

The work around is to force the agent to upgrade via the /fleet/agents API:

curl --request POST \
  --url https://<KIBANA_HOST>/api/fleet/agents/<AGENT_ID>/upgrade \
  --user "<SUPERUSER_NAME>:<SUPERUSER_PASSWORD>" \
  --header 'Content-Type: application/json' \
  --header 'kbn-xsrf: as' \
  --data '{"version": "<VERSION>","force": true}'

[mbudge]

The work around is to force the agent to upgrade via the /fleet/agents API:

curl --request POST \
  --url https://<KIBANA_HOST>/api/fleet/agents/<AGENT_ID>/upgrade \
  --user "<SUPERUSER_NAME>:<SUPERUSER_PASSWORD>" \
  --header 'Content-Type: application/json' \
  --header 'kbn-xsrf: as' \
  --data '{"version": "<VERSION>","force": true}'

On some agents I still get the following errors when I try and do a forced upgrade with the above request.

16:59:43.867
elastic_agent
[elastic_agent][error] failed to get start time for scheduled upgrade action [id = cb41a479-9fe0-4cea-bed3-b4c85e6191f9]
16:59:43.867
elastic_agent
[elastic_agent][warn] Skipping addition to action-queue, issue gathering start time from action id cb41a479-9fe0-4cea-bed3-b4c85e6191f9: action has no start time
17:05:24.985
elastic_agent
[elastic_agent][info] Duplicate upgrade to version 8.18.4 received
17:10:56.966
elastic_agent
[elastic_agent][error] failed to get start time for scheduled upgrade action [id = 2ba115d7-fd01-47bb-abda-de4e358945ef]
17:10:56.966
elastic_agent
[elastic_agent][warn] Skipping addition to action-queue, issue gathering start time from action id 2ba115d7-fd01-47bb-abda-de4e358945ef: action has no start time
17:29:10.471
elastic_agent
[elastic_agent][error] failed to get start time for scheduled upgrade action [id = bb00dda7-bc08-4a72-8bea-d22c44b6bd0a]
17:29:10.471
elastic_agent
[elastic_agent][warn] Skipping addition to action-queue, issue gathering start time from action id bb00dda7-bc08-4a72-8bea-d22c44b6bd0a: action has no start time