Skip to content

Fix database dead locks in Aggregator #517

Closed
@jpraynaud

Description

@jpraynaud

Issue

We have noticed some database locks occurring on the GCP Aggregator which these error messages:

  • cannot start a transaction within a transaction
  • database is locked

⚠ The problem occurred after activating the pruning feature of the stores #516

Here are the full logs:

mithril-aggregator_1                | {"msg":"STATE MACHINE: an error occurred: ","v":0,"name":"slog-rs","level":50,"time":"2022-09-20T14:44:11.436860471Z","hostname":"1db9e5f989b2","pid":1,"error":"StoreError(AdapterError(QueryError(Error { code: Some(1), message: Some(\"cannot start a transaction within a transaction\") })))"}
mithril-aggregator_1                | {"msg":"… Cycle finished, Sleeping for 60000 ms","v":0,"name":"slog-rs","level":30,"time":"2022-09-20T14:44:11.436936131Z","hostname":"1db9e5f989b2","pid":1}
mithril-aggregator_1                | {"msg":"⇄ HTTP SERVER: certificate_pending","v":0,"name":"slog-rs","level":20,"time":"2022-09-20T14:44:11.436978054Z","hostname":"1db9e5f989b2","pid":1}
mithril-aggregator_1                | {"msg":"certificate_pending::error","v":0,"name":"slog-rs","level":40,"time":"2022-09-20T14:44:11.437023811Z","hostname":"1db9e5f989b2","pid":1,"error":"AdapterError(ParsingDataError(Error { code: Some(5), message: Some(\"database is locked\") }))"}
mithril-aggregator_1                | {"msg":"⇄ HTTP SERVER: certificate_pending","v":0,"name":"slog-rs","level":20,"time":"2022-09-20T14:44:12.626648308Z","hostname":"1db9e5f989b2","pid":1}
mithril-aggregator_1                | {"msg":"certificate_pending::error","v":0,"name":"slog-rs","level":40,"time":"2022-09-20T14:44:12.626705143Z","hostname":"1db9e5f989b2","pid":1,"error":"AdapterError(ParsingDataError(Error { code: Some(5), message: Some(\"database is locked\") }))"}

Workaround

Restart the container

Metadata

Metadata

Labels

D-mediumDifficulty: mediumP-highPriority: highbug ⚠️Something isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions