Cardano transaction database is locked in aggregator

[jpraynaud]

Why

During the release of 2517.0 some networks (release-mainnet, release-preprod, pre-release-preview) had the Cardano transaction database locked and the following failure:

thread 'tokio-runtime-worker' panicked at /home/runner/work/mithril/mithril/internal/mithril-persistence/src/sqlite/cursor.rs:35:51:
database is locked (code 5)
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: <mithril_persistence::sqlite::cursor::EntityCursor<T> as core::iter::traits::iterator::Iterator>::next
   3: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
   4: mithril_persistence::sqlite::connection_extensions::ConnectionExtensions::fetch_collect
   5: mithril_aggregator::database::repository::cardano_transaction_repository::<impl mithril_aggregator::services::cardano_transactions_importer::TransactionStore for mithril_persistence::database::repository::cardano_transaction_repository::CardanoTransactionRepository>::store_transactions::{{closure}}
   6: tokio::runtime::handle::Handle::block_on
   7: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
   8: tokio::runtime::task::core::Core<T,S>::poll
   9: tokio::runtime::task::harness::Harness<T,S>::poll
  10: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Analysis

• Access (read) was possible with the sqlite3 CLI.
• The fix was to stop the aggregator and manually run a vacuum on the database (but vacuum was probably not necessary).
• The problem did not occur when releasing the 2517.1 distribution (the day after releasing 2517.0).

Problem was reproduced on dev-follower-preview network after 6 weeks of not updating the nodes (the problem was fixed after just a restart:):

Starting server...
{"msg":"Started","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.32882563Z","hostname":"11bd18ae8fc6","pid":1,"node_version":"0.7.47+7eb3a8a","run_mode":"dev"}
Press Ctrl+C to stop
{"msg":"SERVE command","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.329725727Z","hostname":"11bd18ae8fc6","pid":1,"config":"ServeCommandConfiguration { environment: Production, cardano_cli_path: \"/app/bin/cardano-cli\", cardano_node_socket_path: \"/ipc/node.socket\", cardano_node_version: \"10.3.1\", network_magic: Some(42), network: \"preview\", chain_observer_type: Pallas, protocol_parameters: ProtocolParameters { k: 5, m: 100, phi_f: 0.65 }, snapshot_uploader_type: Gcp, snapshot_bucket_name: Some(\"cdn.aggregator.dev-follower-preview.api.mithril.network\"), snapshot_use_cdn_domain: false, server_ip: \"0.0.0.0\", server_port: 8080, public_server_url: Some(\"https://aggregator.dev-follower-preview.api.mithril.network/aggregator\"), run_interval: 60000, db_directory: \"/db\", snapshot_directory: \"/mithril-aggregator/mithril/snapshots\", data_stores_directory: \"/mithril-aggregator/mithril/stores\", genesis_verification_key: \"5b3132372c37332c3132342c3136312c362c3133372c3133312c3231332c3230372c3131372c3139382c38352c3137362c3139392c3136322c3234312c36382c3132332c3131392c3134352c31332c3233322c3234332c34392c3232392c322c3234392c3230352c3230352c33392c3233352c34345d\", reset_digests_cache: false, disable_digests_cache: false, store_retention_limit: Some(5), era_reader_adapter_type: CardanoChain, era_reader_adapter_params: Some(\"{\\\"address\\\":\\\"addr_test1qzzngukkj9ydjemqjlgfn42sevy2xnvauay46weushlpuq9thd4ray00csjssf4sxftv04xeequ3xfx72nujg9y4d5ysgkxxlh\\\",\\\"verification_key\\\":\\\"5b35352c3232382c3134342c38372c3133382c3133362c34382c382c31342c3138372c38352c3134382c39372c3233322c3235352c3232392c33382c3234342c3234372c3230342c3139382c31332c33312c3232322c32352c3136342c35322c3130322c39312c3132302c3230382c3134375d\\\"}\"), ancillary_files_signer_config: GcpKms { resource_name: GcpCryptoKeyVersionResourceName { project: \"iog-hydra\", location: \"europe-west1\", key_ring: \"mithril-testing\", key_name: \"mithril-ancillary\", version: \"1\" }, credentials_json_env_var: \"GOOGLE_APPLICATION_CREDENTIALS_GCP_KMS_JSON\" }, signed_entity_types: Some(\"CardanoTransactions,CardanoStakeDistribution,CardanoDatabase\"), snapshot_compression_algorithm: Zstandard, zstandard_parameters: Some(ZstandardCompressionParameters { level: 9, number_of_workers: 4 }), cexplorer_pools_url: Some(\"https://preview-js.cexplorer.io/api-static/pool/list.json\"), signer_importer_run_interval: 720, allow_unparsable_block: false, cardano_transactions_prover_cache_pool_size: 5, cardano_transactions_database_connection_pool_size: 10, cardano_transactions_signing_config: CardanoTransactionsSigningConfig { security_parameter: BlockNumber(0), step: BlockNumber(15) }, cardano_transactions_prover_max_hashes_allowed_by_request: 100, cardano_transactions_block_streamer_max_roll_forwards_per_poll: 10000, enable_metrics_server: true, metrics_server_ip: \"0.0.0.0\", metrics_server_port: 9090, persist_usage_report_interval_in_seconds: 10, leader_aggregator_endpoint: Some(\"https://aggregator.dev-preview.api.mithril.network/aggregator\"), custom_origin_tag_white_list: Some(\"HYDRA,LACE,BLOCKFROST,IBC,DOLOS,AMARU,ACROPOLIS\") }"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.329902053Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/monitoring.sqlite3"}
{"msg":"Applying database migrations","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.340631785Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder"}
{"msg":"Check database version","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.340891825Z","hostname":"11bd18ae8fc6","pid":1,"src":"DatabaseVersionChecker"}
{"msg":"database up to date","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.383111418Z","hostname":"11bd18ae8fc6","pid":1,"src":"DatabaseVersionChecker"}
{"msg":"Starting monitoring event loop to log messages.","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:41.38356537Z","hostname":"11bd18ae8fc6","pid":1,"src":"EventStore"}
{"msg":"Sufficient time has passed since last vacuum","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:41.438049289Z","hostname":"11bd18ae8fc6","pid":1,"min_interval_days":7,"elapsed_days":21,"last_vacuum":"2025-04-28 09:11:19.445171625 UTC"}
{"msg":"Performing vacuum","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:41.438639191Z","hostname":"11bd18ae8fc6","pid":1}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.438774514Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/aggregator.sqlite3"}
{"msg":"Applying database migrations","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.505919252Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder"}
{"msg":"Check database version","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.506034265Z","hostname":"11bd18ae8fc6","pid":1,"src":"DatabaseVersionChecker"}
{"msg":"database up to date","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.507682236Z","hostname":"11bd18ae8fc6","pid":1,"src":"DatabaseVersionChecker"}
{"msg":"Handle discrepancies at startup of epoch settings store, will record epoch settings from the configuration for epoch 936","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.519839382Z","hostname":"11bd18ae8fc6","pid":1,"epoch_settings_configuration":"AggregatorEpochSettings { protocol_parameters: ProtocolParameters { k: 5, m: 100, phi_f: 0.65 }, cardano_transactions_signing_config: CardanoTransactionsSigningConfig { security_parameter: BlockNumber(0), step: BlockNumber(15) } }"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.521557017Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Applying database migrations","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.541378947Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder"}
{"msg":"Check database version","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.541475902Z","hostname":"11bd18ae8fc6","pid":1,"src":"DatabaseVersionChecker"}
{"msg":"database up to date","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.549972307Z","hostname":"11bd18ae8fc6","pid":1,"src":"DatabaseVersionChecker"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.550097174Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.550795814Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.551592242Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.55295833Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.554132434Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.555142617Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.555991878Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.556821514Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.557616874Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.558720351Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Start database vacuum","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:41.559708456Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService"}
{"msg":"Vacuum main database","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:41.560711255Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService"}
{"msg":"Running `vacuum` on the SQLite database","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:41.560858572Z","hostname":"11bd18ae8fc6","pid":1,"src":"SqliteCleaner","src":"AggregatorUpkeepService"}
{"msg":"Vacuum finished","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:43.044106274Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService"}
{"msg":"Vacuum performed","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:43.056574169Z","hostname":"11bd18ae8fc6","pid":1,"last_vacuum":"2025-05-19T16:29:43.044408202+00:00"}
{"msg":"New AggregatorHTTPClient created","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:44.243476022Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorHTTPClient"}
{"msg":"Starts computing the Merkle map pool resource of size 5","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:44.328228008Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService","up_to_block_number":3172724}
{"msg":"Computing the Merkle map pool resource 1/5","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:46.519696533Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"Computing the Merkle map pool resource 3/5","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:46.51973805Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"Computing the Merkle map pool resource 4/5","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:47.808156242Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"Computing the Merkle map pool resource 2/5","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:47.837505627Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"Computing the Merkle map pool resource 5/5","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:49.104610838Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"Draining the Merkle map pool","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:49.888226877Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"Giving back new resources to the Merkle map pool","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:49.888349959Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"Completed computing the Merkle map pool resource of size 5","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:49.888389787Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"New MithrilCertificateVerifier created","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.008763307Z","hostname":"11bd18ae8fc6","pid":1}
{"msg":"New MultiSignerImpl created","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.009168859Z","hostname":"11bd18ae8fc6","pid":1,"src":"MultiSignerImpl"}
{"msg":"Initializing runtime","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:50.009343553Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime"}
{"msg":"Launching State Machine","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:50.010363546Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime"}
{"msg":"Started","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:50.010834184Z","hostname":"11bd18ae8fc6","pid":1,"src":"CardanoTransactionsPreloader"}
{"msg":"Locking signed entity type","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.010855739Z","hostname":"11bd18ae8fc6","pid":1,"src":"CardanoTransactionsPreloader","entity_type":"CardanoTransactions"}
{"msg":"================================================================================","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:50.012389018Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime"}
{"msg":"New cycle: Idle - No TimePoint","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:50.012885027Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime"}
{"msg":"Incrementing 'mithril_aggregator_runtime_cycle_total_since_startup' counter","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.013731489Z","hostname":"11bd18ae8fc6","pid":1,"src":"MetricsService"}
{"msg":">> get_time_point_from_chain","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.013940356Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":"Starting HTTP server for metrics on port 9090","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:50.029024708Z","hostname":"11bd18ae8fc6","pid":1,"src":"MetricsServer"}
{"msg":"Metrics sent, Sleeping for 10 seconds","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:50.032628655Z","hostname":"11bd18ae8fc6","pid":1,"src":"UsageReporter"}
{"msg":"Event received","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.032890926Z","hostname":"11bd18ae8fc6","pid":1,"src":"EventStore","event":"EventMessage { source: \"Metrics\", action: \"mithril_aggregator_runtime_cycle_total_since_startup\", content: Object {\"date\": String(\"2025-05-19T16:29:50.031786144Z\"), \"name\": String(\"mithril_aggregator_runtime_cycle_total_since_startup\"), \"origin\": String(\"\"), \"period\": Object {\"nanos\": Number(0), \"secs\": Number(10)}, \"value\": Number(1)}, headers: {} }"}
{"msg":"Event ID=103504 created","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.032918098Z","hostname":"11bd18ae8fc6","pid":1,"src":"EventStore"}
{"msg":"No last polled point available, falling back to the highest stored chain point","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.148704126Z","hostname":"11bd18ae8fc6","pid":1,"src":"CardanoTransactionsImporter"}
{"msg":"Retrieving Cardano transactions until block numbered '3250290'","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.148739557Z","hostname":"11bd18ae8fc6","pid":1,"src":"CardanoTransactionsImporter","highest_stored_block_number":"Some(BlockNumber(3185425))","starting_slot_number":"Some(SlotNumber(79267146))"}
{"msg":"Connected to a new client","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.15130223Z","hostname":"11bd18ae8fc6","pid":1,"src":"PallasChainReader"}
{"msg":"Has agency, finding intersect point...","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.151328443Z","hostname":"11bd18ae8fc6","pid":1,"src":"PallasChainReader","point":"RawCardanoPoint { slot_number: SlotNumber(79267146), block_hash: \"7e04f60a72c0320a13849f43450d8b3259c18ebcf3a72ad988cb8cb1ca04b2da\" }"}
{"msg":"→ Trying to transition to READY","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:50.233430978Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime","last_time_point":"TimePoint { epoch: Epoch(937), immutable_file_number: 18749, chain_point: ChainPoint { slot_number: SlotNumber(81016186), block_number: BlockNumber(3250290), block_hash: \"4de999f54c68a5ffcec2e62c49cdbb298e2c44252da260f4bf3423f8cc51b70b\" } }"}
{"msg":"Retrieve epoch settings","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.233538955Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorHTTPClient"}
{"msg":">> poll_next","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.271055248Z","hostname":"11bd18ae8fc6","pid":1,"src":"ChainReaderBlockStreamer"}
{"msg":">> close_signer_registration_round","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.309892668Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":">> update_era_checker(Epoch(937))","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:50.310594752Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":"Current Era is pythagoras (Epoch 937).","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.519369019Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":">> update_stake_distribution","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.519516065Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner","time_point":"TimePoint {\n    epoch: Epoch(\n        937,\n    ),\n    immutable_file_number: 18749,\n    chain_point: ChainPoint {\n        slot_number: SlotNumber(\n            81016186,\n        ),\n        block_number: BlockNumber(\n            3250290,\n        ),\n        block_hash: \"4de999f54c68a5ffcec2e62c49cdbb298e2c44252da260f4bf3423f8cc51b70b\",\n    },\n}"}
{"msg":">> inform_new_epoch(Epoch(937))","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.527320207Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":">> inform_epoch(epoch: Epoch(937))","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.527590613Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilCertifierService"}
{"msg":"Informed of a new Epoch: Epoch(937). Cleaned 0 open messages along with their single signatures.","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:51.527834692Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilCertifierService"}
{"msg":">> inform_epoch(epoch: Epoch(937))","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.527936414Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilEpochService"}
{"msg":">> upkeep","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.537572841Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":"Start upkeep of the application","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:51.537594187Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService"}
{"msg":"Pruning stale data","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:51.537603776Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService","current_epoch":"Epoch(937)","pruned_data":"Stake pool"}
{"msg":"Pruning stale data","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:51.537910269Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService","current_epoch":"Epoch(937)","pruned_data":"Epoch settings"}
{"msg":"Pruning stale data","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:51.538397598Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService","current_epoch":"Epoch(937)","pruned_data":"Signer registration"}
{"msg":"Some entities are locked - Skipping database upkeep","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:51.538460944Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService"}
{"msg":"Upkeep finished","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:51.538638131Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService"}
{"msg":">> open_signer_registration_round","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.538728101Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner","time_point":"TimePoint {\n    epoch: Epoch(\n        937,\n    ),\n    immutable_file_number: 18749,\n    chain_point: ChainPoint {\n        slot_number: SlotNumber(\n            81016186,\n        ),\n        block_number: BlockNumber(\n            3250290,\n        ),\n        block_hash: \"4de999f54c68a5ffcec2e62c49cdbb298e2c44252da260f4bf3423f8cc51b70b\",\n    },\n}"}
{"msg":">> update_epoch_settings","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.540167433Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":">> update_epoch_settings","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.540343007Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilEpochService"}
{"msg":"Inserting epoch settings in epoch 939","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.540511601Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilEpochService","epoch_settings":"AggregatorEpochSettings { protocol_parameters: ProtocolParameters { k: 5, m: 100, phi_f: 0.65 }, cardano_transactions_signing_config: CardanoTransactionsSigningConfig { security_parameter: BlockNumber(0), step: BlockNumber(15) } }"}
{"msg":"Retrieve epoch settings","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.540521886Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorHTTPClient"}
{"msg":">> update_next_signers_with_stake","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.656735331Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilEpochService"}
{"msg":">> precompute_epoch_data","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.660726166Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilEpochService"}
{"msg":">> precompute_epoch_data","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.725847373Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":">> precompute_epoch_data","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.725878671Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilEpochService"}
{"msg":">> is_certificate_chain_valid","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:51.781992626Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":"An error occurred, runtime state kept. message = 'certificate chain is invalid'","v":0,"name":"mithril-aggregator","level":50,"time":"2025-05-19T16:29:51.791998432Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime","nested_error":"There is an epoch gap between the last certificate epoch (Epoch(924)) and current epoch (Epoch(937))\n\nStack backtrace:\n   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from\n   1: <mithril_aggregator::services::certifier::certifier_service::MithrilCertifierService as mithril_aggregator::services::certifier::interface::CertifierService>::verify_certificate_chain::{{closure}}\n   2: <mithril_aggregator::services::certifier::buffered_certifier::BufferedCertifierService as mithril_aggregator::services::certifier::interface::CertifierService>::verify_certificate_chain::{{closure}}\n   3: <mithril_aggregator::runtime::runner::AggregatorRunner as mithril_aggregator::runtime::runner::AggregatorRunnerTrait>::is_certificate_chain_valid::{{closure}}\n   4: mithril_aggregator::runtime::state_machine::AggregatorRuntime::cycle::{{closure}}\n   5: mithril_aggregator::commands::serve_command::ServeCommand::execute::{{closure}}::{{closure}}\n   6: tokio::runtime::task::core::Core<T,S>::poll\n   7: tokio::runtime::task::harness::Harness<T,S>::poll\n   8: tokio::runtime::scheduler::multi_thread::worker::Context::run_task\n   9: tokio::runtime::context::scoped::Scoped<T>::set\n  10: tokio::runtime::context::runtime::enter_runtime\n  11: tokio::runtime::scheduler::multi_thread::worker::run\n  12: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll\n  13: tokio::runtime::task::core::Core<T,S>::poll\n  14: tokio::runtime::task::harness::Harness<T,S>::poll\n  15: tokio::runtime::blocking::pool::Inner::run\n  16: std::sys::backtrace::__rust_begin_short_backtrace\n  17: core::ops::function::FnOnce::call_once{{vtable.shim}}\n  18: std::sys::pal::unix::thread::Thread::new::thread_start\n  19: <unknown>\n  20: __clone"}
{"msg":"… Cycle finished","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:51.79205192Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime","run_interval_in_ms":60000,"approximate_next_cycle_time":"16:30:50.012"}
{"msg":"Starting import","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:55.030144735Z","hostname":"11bd18ae8fc6","pid":1,"src":"SignersImporter"}
{"msg":"Retrieving data from source","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:55.030185098Z","hostname":"11bd18ae8fc6","pid":1,"source_url":"https://preview-js.cexplorer.io/api-static/pool/list.json"}
{"msg":"Persisting retrieved data in the database","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:55.178682468Z","hostname":"11bd18ae8fc6","pid":1,"src":"SignersImporter","number_of_signer_to_insert":464}
{"msg":"Cycle finished, Sleeping for 720 min","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:55.182973784Z","hostname":"11bd18ae8fc6","pid":1,"src":"SignersImporter"}

thread 'tokio-runtime-worker' panicked at /home/runner/work/mithril/mithril/internal/mithril-persistence/src/sqlite/cursor.rs:35:51:
database is locked (code 5)
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: <mithril_persistence::sqlite::cursor::EntityCursor<T> as core::iter::traits::iterator::Iterator>::next
   3: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
   4: mithril_persistence::sqlite::connection_extensions::ConnectionExtensions::fetch_collect
   5: mithril_aggregator::database::repository::cardano_transaction_repository::<impl mithril_aggregator::services::cardano_transactions_importer::TransactionStore for mithril_persistence::database::repository::cardano_transaction_repository::CardanoTransactionRepository>::store_transactions::{{closure}}
   6: tokio::runtime::handle::Handle::block_on
   7: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
   8: tokio::runtime::task::core::Core<T,S>::poll
   9: tokio::runtime::task::harness::Harness<T,S>::poll
  10: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
{"msg":"Releasing signed entity type","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.430365466Z","hostname":"11bd18ae8fc6","pid":1,"src":"CardanoTransactionsPreloader","entity_type":"CardanoTransactions"}
{"msg":"Finished","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:29:58.430400067Z","hostname":"11bd18ae8fc6","pid":1,"src":"CardanoTransactionsPreloader"}
{"msg":"OPTIONS /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.888613197Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.888650349Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.888658321Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.888664829Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.907289334Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.907950838Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.909706065Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.909730886Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.921668066Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.921702691Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.922781824Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.936770494Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.937247515Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:29:58.940040327Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"Metrics sent, Sleeping for 10 seconds","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:30:00.029685455Z","hostname":"11bd18ae8fc6","pid":1,"src":"UsageReporter"}
{"msg":"OPTIONS /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:08.89032428Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:08.890504593Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:08.89052543Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:08.890540792Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:08.914003852Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:08.91403894Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:08.915223952Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:08.916098883Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:08.922791596Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:08.922857207Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:08.922872284Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:08.940842927Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:08.94312569Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:08.94316459Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"Metrics sent, Sleeping for 10 seconds","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:30:10.030050098Z","hostname":"11bd18ae8fc6","pid":1,"src":"UsageReporter"}
{"msg":"OPTIONS /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:18.888101795Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:18.888148791Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:18.889420722Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:18.889450585Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:18.908275232Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:18.908529041Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:18.909023735Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:18.909193587Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:18.925100436Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:18.925129272Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:18.925749599Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:18.940592806Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:18.941995869Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:30:18.943989673Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"Metrics sent, Sleeping for 10 seconds","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:30:20.029376732Z","hostname":"11bd18ae8fc6","pid":1,"src":"UsageReporter"}

What

Identify and fix the source of the lock on the Cardano transaction database in the aggregator.

How

• Investigate the source of the lock
• Create a fix for the database lock

[jpraynaud]

Logs of dev-follower-preview after the first restart when everything worked fine:

{"msg":"Started","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:50.292502097Z","hostname":"11bd18ae8fc6","pid":1,"node_version":"0.7.47+7eb3a8a","run_mode":"dev"}
{"msg":"SERVE command","v":0,"name":"mithril-aggregator","level":20Starting server...
Press Ctrl+C to stop
,"time":"2025-05-19T16:35:50.293274532Z","hostname":"11bd18ae8fc6","pid":1,"config":"ServeCommandConfiguration { environment: Production, cardano_cli_path: \"/app/bin/cardano-cli\", cardano_node_socket_path: \"/ipc/node.socket\", cardano_node_version: \"10.3.1\", network_magic: Some(42), network: \"preview\", chain_observer_type: Pallas, protocol_parameters: ProtocolParameters { k: 5, m: 100, phi_f: 0.65 }, snapshot_uploader_type: Gcp, snapshot_bucket_name: Some(\"cdn.aggregator.dev-follower-preview.api.mithril.network\"), snapshot_use_cdn_domain: false, server_ip: \"0.0.0.0\", server_port: 8080, public_server_url: Some(\"https://aggregator.dev-follower-preview.api.mithril.network/aggregator\"), run_interval: 60000, db_directory: \"/db\", snapshot_directory: \"/mithril-aggregator/mithril/snapshots\", data_stores_directory: \"/mithril-aggregator/mithril/stores\", genesis_verification_key: \"5b3132372c37332c3132342c3136312c362c3133372c3133312c3231332c3230372c3131372c3139382c38352c3137362c3139392c3136322c3234312c36382c3132332c3131392c3134352c31332c3233322c3234332c34392c3232392c322c3234392c3230352c3230352c33392c3233352c34345d\", reset_digests_cache: false, disable_digests_cache: false, store_retention_limit: Some(5), era_reader_adapter_type: CardanoChain, era_reader_adapter_params: Some(\"{\\\"address\\\":\\\"addr_test1qzzngukkj9ydjemqjlgfn42sevy2xnvauay46weushlpuq9thd4ray00csjssf4sxftv04xeequ3xfx72nujg9y4d5ysgkxxlh\\\",\\\"verification_key\\\":\\\"5b35352c3232382c3134342c38372c3133382c3133362c34382c382c31342c3138372c38352c3134382c39372c3233322c3235352c3232392c33382c3234342c3234372c3230342c3139382c31332c33312c3232322c32352c3136342c35322c3130322c39312c3132302c3230382c3134375d\\\"}\"), ancillary_files_signer_config: GcpKms { resource_name: GcpCryptoKeyVersionResourceName { project: \"iog-hydra\", location: \"europe-west1\", key_ring: \"mithril-testing\", key_name: \"mithril-ancillary\", version: \"1\" }, credentials_json_env_var: \"GOOGLE_APPLICATION_CREDENTIALS_GCP_KMS_JSON\" }, signed_entity_types: Some(\"CardanoTransactions,CardanoStakeDistribution,CardanoDatabase\"), snapshot_compression_algorithm: Zstandard, zstandard_parameters: Some(ZstandardCompressionParameters { level: 9, number_of_workers: 4 }), cexplorer_pools_url: Some(\"https://preview-js.cexplorer.io/api-static/pool/list.json\"), signer_importer_run_interval: 720, allow_unparsable_block: false, cardano_transactions_prover_cache_pool_size: 5, cardano_transactions_database_connection_pool_size: 10, cardano_transactions_signing_config: CardanoTransactionsSigningConfig { security_parameter: BlockNumber(0), step: BlockNumber(15) }, cardano_transactions_prover_max_hashes_allowed_by_request: 100, cardano_transactions_block_streamer_max_roll_forwards_per_poll: 10000, enable_metrics_server: true, metrics_server_ip: \"0.0.0.0\", metrics_server_port: 9090, persist_usage_report_interval_in_seconds: 10, leader_aggregator_endpoint: Some(\"https://aggregator.dev-preview.api.mithril.network/aggregator\"), custom_origin_tag_white_list: Some(\"HYDRA,LACE,BLOCKFROST,IBC,DOLOS,AMARU,ACROPOLIS\") }"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:50.293710354Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/monitoring.sqlite3"}
{"msg":"Applying database migrations","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:50.295987248Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder"}
{"msg":"Check database version","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:50.296067134Z","hostname":"11bd18ae8fc6","pid":1,"src":"DatabaseVersionChecker"}
{"msg":"database up to date","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:50.337433199Z","hostname":"11bd18ae8fc6","pid":1,"src":"DatabaseVersionChecker"}
{"msg":"Starting monitoring event loop to log messages.","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:50.337817362Z","hostname":"11bd18ae8fc6","pid":1,"src":"EventStore"}
{"msg":"Not enough time elapsed since last vacuum","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:50.363505397Z","hostname":"11bd18ae8fc6","pid":1,"min_interval_days":7,"elapsed_days":0,"last_vacuum":"2025-05-19 16:29:43.044408202 UTC"}
{"msg":"No vacuum needed","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:50.36353646Z","hostname":"11bd18ae8fc6","pid":1,"last_vacuum":"2025-05-19T16:29:43.044408202+00:00"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:50.363545308Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/aggregator.sqlite3"}
{"msg":"Applying database migrations","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:50.363556506Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder"}
{"msg":"Check database version","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:50.363564122Z","hostname":"11bd18ae8fc6","pid":1,"src":"DatabaseVersionChecker"}
{"msg":"database up to date","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:50.363571739Z","hostname":"11bd18ae8fc6","pid":1,"src":"DatabaseVersionChecker"}
{"msg":"Handle discrepancies at startup of epoch settings store, will record epoch settings from the configuration for epoch 936","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:50.371907131Z","hostname":"11bd18ae8fc6","pid":1,"epoch_settings_configuration":"AggregatorEpochSettings { protocol_parameters: ProtocolParameters { k: 5, m: 100, phi_f: 0.65 }, cardano_transactions_signing_config: CardanoTransactionsSigningConfig { security_parameter: BlockNumber(0), step: BlockNumber(15) } }"}
{"msg":"New AggregatorHTTPClient created","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.222448841Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorHTTPClient"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.222572549Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Applying database migrations","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.223670102Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder"}
{"msg":"Check database version","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.223848304Z","hostname":"11bd18ae8fc6","pid":1,"src":"DatabaseVersionChecker"}
{"msg":"database up to date","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.223862608Z","hostname":"11bd18ae8fc6","pid":1,"src":"DatabaseVersionChecker"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.223872225Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.224546237Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.22521463Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.225884036Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.226483285Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.227214538Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.227856566Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.228479673Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.229102493Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Opening SQLite connection","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:51.229741711Z","hostname":"11bd18ae8fc6","pid":1,"src":"ConnectionBuilder","options":"[EnableForeignKeys, EnableWriteAheadLog]","path":"/mithril-aggregator/mithril/stores/cardano-transaction.sqlite3"}
{"msg":"Starts computing the Merkle map pool resource of size 5","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:51.334646697Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService","up_to_block_number":3172724}
{"msg":"Computing the Merkle map pool resource 1/5","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:52.254790702Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"Computing the Merkle map pool resource 3/5","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:52.254910404Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"Computing the Merkle map pool resource 2/5","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:53.262830917Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"Computing the Merkle map pool resource 4/5","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:53.297061751Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"Computing the Merkle map pool resource 5/5","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:54.290874409Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"Draining the Merkle map pool","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:54.948188471Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"Giving back new resources to the Merkle map pool","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:54.948321056Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"Completed computing the Merkle map pool resource of size 5","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:54.948340665Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilProverService"}
{"msg":"New MithrilCertificateVerifier created","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.052065698Z","hostname":"11bd18ae8fc6","pid":1}
{"msg":"New MultiSignerImpl created","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.052103889Z","hostname":"11bd18ae8fc6","pid":1,"src":"MultiSignerImpl"}
{"msg":"Initializing runtime","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:55.052115791Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime"}
{"msg":"Started","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:55.053376059Z","hostname":"11bd18ae8fc6","pid":1,"src":"CardanoTransactionsPreloader"}
{"msg":"Locking signed entity type","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.053841728Z","hostname":"11bd18ae8fc6","pid":1,"src":"CardanoTransactionsPreloader","entity_type":"CardanoTransactions"}
{"msg":"Launching State Machine","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:55.054164771Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime"}
{"msg":"================================================================================","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:55.054300897Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime"}
{"msg":"New cycle: Idle - No TimePoint","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:55.054315195Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime"}
{"msg":"Incrementing 'mithril_aggregator_runtime_cycle_total_since_startup' counter","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.05432431Z","hostname":"11bd18ae8fc6","pid":1,"src":"MetricsService"}
{"msg":">> get_time_point_from_chain","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.054333896Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":"No last polled point available, falling back to the highest stored chain point","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.068691349Z","hostname":"11bd18ae8fc6","pid":1,"src":"CardanoTransactionsImporter"}
{"msg":"Retrieving Cardano transactions until block numbered '3250298'","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.06905505Z","hostname":"11bd18ae8fc6","pid":1,"src":"CardanoTransactionsImporter","highest_stored_block_number":"Some(BlockNumber(3185425))","starting_slot_number":"Some(SlotNumber(79267146))"}
{"msg":"Connected to a new client","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.069949107Z","hostname":"11bd18ae8fc6","pid":1,"src":"PallasChainReader"}
{"msg":"Has agency, finding intersect point...","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.070620621Z","hostname":"11bd18ae8fc6","pid":1,"src":"PallasChainReader","point":"RawCardanoPoint { slot_number: SlotNumber(79267146), block_hash: \"7e04f60a72c0320a13849f43450d8b3259c18ebcf3a72ad988cb8cb1ca04b2da\" }"}
{"msg":"Starting HTTP server for metrics on port 9090","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:55.076171477Z","hostname":"11bd18ae8fc6","pid":1,"src":"MetricsServer"}
{"msg":">> poll_next","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.076540177Z","hostname":"11bd18ae8fc6","pid":1,"src":"ChainReaderBlockStreamer"}
{"msg":"Metrics sent, Sleeping for 10 seconds","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:55.077213125Z","hostname":"11bd18ae8fc6","pid":1,"src":"UsageReporter"}
{"msg":"Event received","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.077575818Z","hostname":"11bd18ae8fc6","pid":1,"src":"EventStore","event":"EventMessage { source: \"Metrics\", action: \"mithril_aggregator_runtime_cycle_total_since_startup\", content: Object {\"date\": String(\"2025-05-19T16:35:55.077040149Z\"), \"name\": String(\"mithril_aggregator_runtime_cycle_total_since_startup\"), \"origin\": String(\"\"), \"period\": Object {\"nanos\": Number(0), \"secs\": Number(10)}, \"value\": Number(1)}, headers: {} }"}
{"msg":"Event ID=103510 created","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.096376724Z","hostname":"11bd18ae8fc6","pid":1,"src":"EventStore"}
{"msg":"→ Trying to transition to READY","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:55.525988809Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime","last_time_point":"TimePoint { epoch: Epoch(937), immutable_file_number: 18749, chain_point: ChainPoint { slot_number: SlotNumber(81016515), block_number: BlockNumber(3250298), block_hash: \"f68757f11eef46ffd4bc85f8b655ae7d92e833c4bccd89c55229740a4c942c62\" } }"}
{"msg":"Retrieve epoch settings","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.526088675Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorHTTPClient"}
{"msg":">> close_signer_registration_round","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.61825689Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":">> update_era_checker(Epoch(937))","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:55.618383296Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":"Current Era is pythagoras (Epoch 937).","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.477962714Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":">> update_stake_distribution","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.478698627Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner","time_point":"TimePoint {\n    epoch: Epoch(\n        937,\n    ),\n    immutable_file_number: 18749,\n    chain_point: ChainPoint {\n        slot_number: SlotNumber(\n            81016515,\n        ),\n        block_number: BlockNumber(\n            3250298,\n        ),\n        block_hash: \"f68757f11eef46ffd4bc85f8b655ae7d92e833c4bccd89c55229740a4c942c62\",\n    },\n}"}
{"msg":">> inform_new_epoch(Epoch(937))","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.485970648Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":">> inform_epoch(epoch: Epoch(937))","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.48599285Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilCertifierService"}
{"msg":"Informed of a new Epoch: Epoch(937). Cleaned 0 open messages along with their single signatures.","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:56.48600271Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilCertifierService"}
{"msg":">> inform_epoch(epoch: Epoch(937))","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.486009765Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilEpochService"}
{"msg":">> upkeep","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.495565382Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":"Start upkeep of the application","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:56.495586142Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService"}
{"msg":"Pruning stale data","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:56.495595421Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService","current_epoch":"Epoch(937)","pruned_data":"Stake pool"}
{"msg":"Pruning stale data","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:56.495954174Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService","current_epoch":"Epoch(937)","pruned_data":"Epoch settings"}
{"msg":"Pruning stale data","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:56.495974929Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService","current_epoch":"Epoch(937)","pruned_data":"Signer registration"}
{"msg":"Some entities are locked - Skipping database upkeep","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:56.496070159Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService"}
{"msg":"Upkeep finished","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:56.496083671Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorUpkeepService"}
{"msg":">> open_signer_registration_round","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.496091289Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner","time_point":"TimePoint {\n    epoch: Epoch(\n        937,\n    ),\n    immutable_file_number: 18749,\n    chain_point: ChainPoint {\n        slot_number: SlotNumber(\n            81016515,\n        ),\n        block_number: BlockNumber(\n            3250298,\n        ),\n        block_hash: \"f68757f11eef46ffd4bc85f8b655ae7d92e833c4bccd89c55229740a4c942c62\",\n    },\n}"}
{"msg":">> update_epoch_settings","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.49731289Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":">> update_epoch_settings","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.497331655Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilEpochService"}
{"msg":"Inserting epoch settings in epoch 939","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.497339624Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilEpochService","epoch_settings":"AggregatorEpochSettings { protocol_parameters: ProtocolParameters { k: 5, m: 100, phi_f: 0.65 }, cardano_transactions_signing_config: CardanoTransactionsSigningConfig { security_parameter: BlockNumber(0), step: BlockNumber(15) } }"}
{"msg":"Retrieve epoch settings","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.498032053Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorHTTPClient"}
{"msg":">> update_next_signers_with_stake","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.572685148Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilEpochService"}
{"msg":">> precompute_epoch_data","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.573865099Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilEpochService"}
{"msg":">> precompute_epoch_data","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.613594206Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":">> precompute_epoch_data","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.61380435Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilEpochService"}
{"msg":">> is_certificate_chain_valid","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.653744517Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":"Verifying certificate","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.65441601Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilCertificateVerifier","certificate_signed_entity_type":"MithrilStakeDistribution(Epoch(937))","certificate_epoch":"Epoch(937)","certificate_previous_hash":"","certificate_hash":"115af9f9c0ccb7b285a064bd1b2ac0dad0a8b16e12c5a9f8ed120a0fdddb5b20"}
{"msg":"Incrementing 'mithril_aggregator_runtime_cycle_success_since_startup' counter","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:56.654939624Z","hostname":"11bd18ae8fc6","pid":1,"src":"MetricsService"}
{"msg":"… Cycle finished","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:35:56.655229169Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime","run_interval_in_ms":60000,"approximate_next_cycle_time":"16:36:55.052"}
{"msg":"OPTIONS /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.889921605Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.890506081Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.891566514Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.891957788Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.912214623Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.912632945Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/epoch-settings 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.913063975Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.913083787Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.914337946Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.922930941Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.923409484Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.924412882Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/epoch-settings 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.944554345Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.946193293Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.946880202Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:35:58.94772997Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"Starting import","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:00.079083254Z","hostname":"11bd18ae8fc6","pid":1,"src":"SignersImporter"}
{"msg":"Retrieving data from source","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:00.079906643Z","hostname":"11bd18ae8fc6","pid":1,"source_url":"https://preview-js.cexplorer.io/api-static/pool/list.json"}
{"msg":"Persisting retrieved data in the database","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:00.238043414Z","hostname":"11bd18ae8fc6","pid":1,"src":"SignersImporter","number_of_signer_to_insert":464}
{"msg":"Cycle finished, Sleeping for 720 min","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:00.274485473Z","hostname":"11bd18ae8fc6","pid":1,"src":"SignersImporter"}
{"msg":">> poll_next","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:02.197265994Z","hostname":"11bd18ae8fc6","pid":1,"src":"ChainReaderBlockStreamer"}
{"msg":"Metrics sent, Sleeping for 10 seconds","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:05.077038168Z","hostname":"11bd18ae8fc6","pid":1,"src":"UsageReporter"}
{"msg":"Event received","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:05.077074076Z","hostname":"11bd18ae8fc6","pid":1,"src":"EventStore","event":"EventMessage { source: \"Metrics\", action: \"mithril_aggregator_runtime_cycle_success_since_startup\", content: Object {\"date\": String(\"2025-05-19T16:36:05.076978675Z\"), \"name\": String(\"mithril_aggregator_runtime_cycle_success_since_startup\"), \"origin\": String(\"\"), \"period\": Object {\"nanos\": Number(0), \"secs\": Number(10)}, \"value\": Number(1)}, headers: {} }"}
{"msg":"Event ID=103511 created","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:05.078053273Z","hostname":"11bd18ae8fc6","pid":1,"src":"EventStore"}
{"msg":"OPTIONS /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:08.888065842Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:08.88811713Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:08.889002823Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:08.889091722Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:08.908392377Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:08.909751314Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:08.909780493Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:08.91126646Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:08.924725933Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:08.924761936Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:08.926423468Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:08.9413456Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:08.942746527Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:08.944033462Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":">> poll_next","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:11.090805992Z","hostname":"11bd18ae8fc6","pid":1,"src":"ChainReaderBlockStreamer"}
{"msg":"Metrics sent, Sleeping for 10 seconds","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:15.07791886Z","hostname":"11bd18ae8fc6","pid":1,"src":"UsageReporter"}
{"msg":"OPTIONS /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:18.887995525Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:18.888031848Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:18.889481682Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:18.890193274Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:18.908320436Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:18.90838786Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:18.908395557Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:18.909490932Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:18.923675262Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:18.925049299Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:18.925164649Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:18.935157709Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:18.939997432Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:18.942061401Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":">> poll_next","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:19.282495984Z","hostname":"11bd18ae8fc6","pid":1,"src":"ChainReaderBlockStreamer"}
{"msg":"Metrics sent, Sleeping for 10 seconds","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:25.077304591Z","hostname":"11bd18ae8fc6","pid":1,"src":"UsageReporter"}
{"msg":">> poll_next","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:26.926877244Z","hostname":"11bd18ae8fc6","pid":1,"src":"ChainReaderBlockStreamer"}
{"msg":"OPTIONS /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:28.888579173Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:28.888632824Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:28.888643565Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:28.888668494Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:28.908100581Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:28.908319516Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:28.911430342Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:28.911468384Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:28.92501165Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:28.925782733Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:28.925807338Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:28.943200721Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:28.945626504Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:28.945676229Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"Metrics sent, Sleeping for 10 seconds","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:35.077096024Z","hostname":"11bd18ae8fc6","pid":1,"src":"UsageReporter"}
{"msg":">> poll_next","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:35.385361556Z","hostname":"11bd18ae8fc6","pid":1,"src":"ChainReaderBlockStreamer"}
{"msg":"OPTIONS /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:39.284793838Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:39.284840413Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:39.286789605Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:39.286817208Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:39.286828608Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:39.28683817Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:39.286938907Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:39.296228129Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:39.302919736Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:39.305062746Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:39.307240378Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:39.308992027Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:39.310228551Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:39.313448727Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":">> poll_next","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:43.579781876Z","hostname":"11bd18ae8fc6","pid":1,"src":"ChainReaderBlockStreamer"}
{"msg":"Metrics sent, Sleeping for 10 seconds","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:45.077061429Z","hostname":"11bd18ae8fc6","pid":1,"src":"UsageReporter"}
{"msg":">> poll_next","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:47.393762339Z","hostname":"11bd18ae8fc6","pid":1,"src":"ChainReaderBlockStreamer"}
{"msg":"Computing Block Range Roots","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:47.429474719Z","hostname":"11bd18ae8fc6","pid":1,"src":"CardanoTransactionsImporter","end_block":3250290,"start_block":3185415}
{"msg":"Releasing signed entity type","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:47.744774781Z","hostname":"11bd18ae8fc6","pid":1,"src":"CardanoTransactionsPreloader","entity_type":"CardanoTransactions"}
{"msg":"Finished","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:47.744803122Z","hostname":"11bd18ae8fc6","pid":1,"src":"CardanoTransactionsPreloader"}
{"msg":"OPTIONS /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:49.283867788Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:49.283905265Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:49.28394466Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:49.283953119Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:49.283959264Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:49.285364414Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"OPTIONS /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:49.285394694Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/snapshots 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:49.312692862Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/status 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:49.312724687Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/mithril-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:49.312731973Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-transactions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:49.312738323Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/certificates 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:49.31274458Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-database 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:49.313152455Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"GET /aggregator/artifact/cardano-stake-distributions 200 OK","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:49.31625338Z","hostname":"11bd18ae8fc6","pid":1,"src":"http_server"}
{"msg":"================================================================================","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:55.053628643Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime"}
{"msg":"New cycle: Ready - TimePoint (epoch: 937, immutable_file_number: 18749, chain_point: ChainPoint (slot_number: 81016515, block_number: 3250298, block_hash: f68757f11eef46ffd4bc85f8b655ae7d92e833c4bccd89c55229740a4c942c62))","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:55.053716694Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime"}
{"msg":"Incrementing 'mithril_aggregator_runtime_cycle_total_since_startup' counter","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.05372868Z","hostname":"11bd18ae8fc6","pid":1,"src":"MetricsService"}
{"msg":">> get_time_point_from_chain","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.053736394Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":"Metrics sent, Sleeping for 10 seconds","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:55.077854336Z","hostname":"11bd18ae8fc6","pid":1,"src":"UsageReporter"}
{"msg":"Event received","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.079182562Z","hostname":"11bd18ae8fc6","pid":1,"src":"EventStore","event":"EventMessage { source: \"Metrics\", action: \"mithril_aggregator_runtime_cycle_total_since_startup\", content: Object {\"date\": String(\"2025-05-19T16:36:55.077796312Z\"), \"name\": String(\"mithril_aggregator_runtime_cycle_total_since_startup\"), \"origin\": String(\"\"), \"period\": Object {\"nanos\": Number(0), \"secs\": Number(10)}, \"value\": Number(1)}, headers: {} }"}
{"msg":"Event ID=103512 created","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.079712575Z","hostname":"11bd18ae8fc6","pid":1,"src":"EventStore"}
{"msg":">> get_current_non_certified_open_message","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.23330876Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner","time_point":"TimePoint {\n    epoch: Epoch(\n        937,\n    ),\n    immutable_file_number: 18749,\n    chain_point: ChainPoint {\n        slot_number: SlotNumber(\n            81016583,\n        ),\n        block_number: BlockNumber(\n            3250301,\n        ),\n        block_hash: \"a41ac7339ecfa97420c98e1826ffd2fa45b1cfc7622992d25c027964a9f6d14c\",\n    },\n}"}
{"msg":">> get_current_open_message_for_signed_entity_type","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.233735819Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner","signed_entity_type":"MithrilStakeDistribution(Epoch(937))"}
{"msg":">> mark_open_message_if_expired","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.233800093Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":">> mark_open_message_if_expired","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.233810154Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilCertifierService"}
{"msg":"Marked expired open messages","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.252804315Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner","expired_open_message":"None"}
{"msg":">> get_open_message(signed_entity_type: MithrilStakeDistribution(Epoch(937)))","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.253255103Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilCertifierService"}
{"msg":">> compute_protocol_message","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.253278973Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":"Compute protocol message for signed entity type: 'MithrilStakeDistribution(Epoch(937))'","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.253287307Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilSignableBuilderService"}
{"msg":">> create_open_message","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.25386299Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRunner"}
{"msg":">> create_open_message(signed_entity_type: MithrilStakeDistribution(Epoch(937)))","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.253963719Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilCertifierService","protocol_message":"ProtocolMessage { message_parts: {NextAggregateVerificationKey: \"7b226d745f636f6d6d69746d656e74223a7b22726f6f74223a5b3138382c3135302c3233382c3130362c3136362c3137372c3235342c3136322c3135352c33342c3139382c34312c3135312c3136372c3130362c3131372c3130302c3135352c3235312c36362c32322c3137392c3137332c3231302c34362c3130312c3139382c3134382c34342c3135382c3135342c35345d2c226e725f6c6561766573223a342c22686173686572223a6e756c6c7d2c22746f74616c5f7374616b65223a33303730383031383139313530397d\", NextProtocolParameters: \"b01de82ca7e57c1bf2a56381ce265f378aeea5f1dde7f824b5ba42125c4adad2\", CurrentEpoch: \"937\"} }"}
{"msg":"create_open_message: created open message for MithrilStakeDistribution(Epoch(937))","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:55.261270198Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilCertifierService"}
{"msg":"create_open_message: created open message ID='018b1303-59c5-4d1c-8070-6c647128b925'","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.261297809Z","hostname":"11bd18ae8fc6","pid":1,"src":"MithrilCertifierService"}
{"msg":"→ Transitioning to SIGNING","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:55.261572538Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime"}
{"msg":"Incrementing 'mithril_aggregator_runtime_cycle_success_since_startup' counter","v":0,"name":"mithril-aggregator","level":20,"time":"2025-05-19T16:36:55.261630416Z","hostname":"11bd18ae8fc6","pid":1,"src":"MetricsService"}
{"msg":"… Cycle finished","v":0,"name":"mithril-aggregator","level":30,"time":"2025-05-19T16:36:55.261645626Z","hostname":"11bd18ae8fc6","pid":1,"src":"AggregatorRuntime","run_interval_in_ms":60000,"approximate_next_cycle_time":"16:37:55.053"}