Fleet versions
- Discovered: 4.89.2
- Reproduced: Not reproduced in-house. Observed consistently across 3 self-hosted customer instances running Aurora Serverless v2.
Web browser and operating system: N/A — server-side
💥 Actual behavior
MySQL Error 1615 (HY000): Prepared statement needs to be re-prepared is never retried by Fleet. It surfaces in two places.
1. At runtime, losing software install results and stalling the host's activity queue.
POST /api/fleet/orbit/software_install/result
"err": "save host software installation result: activate next activity: delete completed upcoming activity: Error 1615 (HY000): Prepared statement needs to be re-prepared"
SetHostSoftwareInstallResult (server/datastore/mysql/software.go:7594) wraps the result UPDATE and activateNextUpcomingActivity in a single withRetryTxx. The 1615 hits the first statement of the activation step (server/datastore/mysql/activities.go:1060), so the whole transaction rolls back:
- The install result is never recorded. Fleet still shows the install as pending even though it completed on the device.
- The host's upcoming-activity queue does not advance, so subsequent installs and scripts queued for that host never start.
orbit retries the POST up to 5 times (orbit/pkg/installer/installer.go:115), so a one-off occurrence usually self-heals. Beyond that the result is lost and the queue stays stalled.
2. At upgrade, aborting the migration.
Upgrading 4.88.1 → 4.89.2 fails on the first 4.89.x migration:
FAIL 20260626120000_CompressWindowsMDMResponsesColumn.go (backfilling raw_response_gz:
storing compressed response id <ID>: Error 1615 (HY000): Prepared statement needs to be
re-prepared), quitting migration.
🛠️ Expected behavior
MySQL documents 1615 as client-retryable — the client is expected to re-execute. Fleet should retry it in both the shared transaction helper and the migration loop, rather than surfacing it to the caller and aborting.
🧑💻 Steps to reproduce
These steps:
Confirmed on 3 customer instances, not yet reproduced internally. Conditions:
- Run Fleet 4.89.2 against Aurora Serverless v2, where
table_open_cache / table_definition_cache are managed by the platform and resize as ACUs scale.
- Upgrade from 4.88.1 —
20260626120000_CompressWindowsMDMResponsesColumn aborts during the raw_response_gz backfill.
- At runtime, have a host post a software install result — the endpoint intermittently fails as above.
A synthetic repro is likely possible by lowering table_definition_cache well below Fleet's table count and driving install-result traffic.
🕯️ More info (optional)
Root cause. RetryableError (server/platform/mysql/retry.go:130) only matches ER_LOCK_DEADLOCK and ER_LOCK_WAIT_TIMEOUT; every other MySQL error falls through to backoff.Permanent. Adding mysqlerr.ER_NEED_REPREPARE to that switch makes every withRetryTxx path resilient and covers symptom 1.
The migration needs its own handling — migrations don't go through withRetryTxx. The backfill in 20260626120000_CompressWindowsMDMResponsesColumn.go issues one server-side PREPARE/EXECUTE/CLOSE per row (interpolateParams is off, see server/datastore/mysql/mysql.go:406) immediately after ALTERing the same table, and a single failure aborts the run. Batching the updates would also cut prepare churn roughly 100x.
Platform note. Aurora Serverless v2 resizes its table caches as capacity scales, so table metadata versions churn by design, and table_open_cache is not modifiable on that configuration — AWS Support confirmed this with the affected customer after they tried raising it. Customers on Serverless v2 cannot mitigate this themselves, which is why the retry needs to live in Fleet.
Workaround. Re-running the migration makes forward progress: the backfill is resumable via WHERE raw_response_gz IS NULL, and the leading ALTER TABLE forces an implicit commit so completed rows are not rolled back. There is no workaround for the runtime path.
Fleet versions
Web browser and operating system: N/A — server-side
💥 Actual behavior
MySQL
Error 1615 (HY000): Prepared statement needs to be re-preparedis never retried by Fleet. It surfaces in two places.1. At runtime, losing software install results and stalling the host's activity queue.
SetHostSoftwareInstallResult(server/datastore/mysql/software.go:7594) wraps the resultUPDATEandactivateNextUpcomingActivityin a singlewithRetryTxx. The 1615 hits the first statement of the activation step (server/datastore/mysql/activities.go:1060), so the whole transaction rolls back:orbit retries the POST up to 5 times (
orbit/pkg/installer/installer.go:115), so a one-off occurrence usually self-heals. Beyond that the result is lost and the queue stays stalled.2. At upgrade, aborting the migration.
Upgrading 4.88.1 → 4.89.2 fails on the first 4.89.x migration:
🛠️ Expected behavior
MySQL documents 1615 as client-retryable — the client is expected to re-execute. Fleet should retry it in both the shared transaction helper and the migration loop, rather than surfacing it to the caller and aborting.
🧑💻 Steps to reproduce
These steps:
Confirmed on 3 customer instances, not yet reproduced internally. Conditions:
table_open_cache/table_definition_cacheare managed by the platform and resize as ACUs scale.20260626120000_CompressWindowsMDMResponsesColumnaborts during theraw_response_gzbackfill.A synthetic repro is likely possible by lowering
table_definition_cachewell below Fleet's table count and driving install-result traffic.🕯️ More info (optional)
Root cause.
RetryableError(server/platform/mysql/retry.go:130) only matchesER_LOCK_DEADLOCKandER_LOCK_WAIT_TIMEOUT; every other MySQL error falls through tobackoff.Permanent. Addingmysqlerr.ER_NEED_REPREPAREto that switch makes everywithRetryTxxpath resilient and covers symptom 1.The migration needs its own handling — migrations don't go through
withRetryTxx. The backfill in20260626120000_CompressWindowsMDMResponsesColumn.goissues one server-sidePREPARE/EXECUTE/CLOSEper row (interpolateParamsis off, seeserver/datastore/mysql/mysql.go:406) immediately afterALTERing the same table, and a single failure aborts the run. Batching the updates would also cut prepare churn roughly 100x.Platform note. Aurora Serverless v2 resizes its table caches as capacity scales, so table metadata versions churn by design, and
table_open_cacheis not modifiable on that configuration — AWS Support confirmed this with the affected customer after they tried raising it. Customers on Serverless v2 cannot mitigate this themselves, which is why the retry needs to live in Fleet.Workaround. Re-running the migration makes forward progress: the backfill is resumable via
WHERE raw_response_gz IS NULL, and the leadingALTER TABLEforces an implicit commit so completed rows are not rolled back. There is no workaround for the runtime path.