Skip to content

Commit 6ce2f3e

Browse files
committed
Fix CPU time going backwards when daemonize
When we `daemonize`, we fork the process and cputime metrics get confused about the per-thread resource usage appearing to go backwards because we're comparing the resource usage (`rusage`) from the original process to the forked process. We now kick off the background tasks (`run_as_background_process`) after we have forked the process so the `rusage` we record when we `start` is in the same thread when we `stop`. Bad log examples from before: ``` synapse.logging.context - ERROR - _schedule_next_expiry-0 - utime went backwards! 0.050467 < 0.886526 synapse.logging.context - ERROR - _schedule_db_events-0 - stime went backwards! 0.009941 < 0.155106 synapse.logging.context - ERROR - wake_destinations_needing_catchup-0 - stime went backwards! 0.010175 < 0.130923 synapse.logging.context - ERROR - resume_sync_partial_state_room-0 - utime went backwards! 0.052898 < 0.886526 ``` Testing strategy: 1. Run with `daemonize: true` in your `homeserver.yaml` 1. `poetry run synapse_homeserver --config-path homeserver.yaml` 1. Shutdown the server 1. Look for any bad log entries in your homeserver logs: - `Expected logging context sentinel but found main` - `Expected logging context main was lost` - `utime went backwards!`/`stime went backwards!`
1 parent 93044f4 commit 6ce2f3e

File tree

2 files changed

+7
-7
lines changed

2 files changed

+7
-7
lines changed

synapse/app/_base.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -601,6 +601,12 @@ def run_sighup(*args: Any, **kwargs: Any) -> None:
601601
hs.get_datastores().main.db_pool.start_profiling()
602602
hs.get_pusherpool().start()
603603

604+
# Register background tasks required by this server. This must be done
605+
# somewhat manually due to the background tasks not being registered
606+
# unless handlers are instantiated.
607+
if hs.config.worker.run_background_tasks:
608+
hs.start_background_tasks()
609+
604610
# Log when we start the shut down process.
605611
hs.get_reactor().addSystemEventTrigger(
606612
"before", "shutdown", logger.info, "Shutting down..."

synapse/server.py

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -366,12 +366,6 @@ def setup(self) -> None:
366366
self.datastores = Databases(self.DATASTORE_CLASS, self)
367367
logger.info("Finished setting up.")
368368

369-
# Register background tasks required by this server. This must be done
370-
# somewhat manually due to the background tasks not being registered
371-
# unless handlers are instantiated.
372-
if self.config.worker.run_background_tasks:
373-
self.setup_background_tasks()
374-
375369
def __del__(self) -> None:
376370
"""
377371
Called when an the homeserver is garbage collected.
@@ -410,7 +404,7 @@ def start_listening(self) -> None: # noqa: B027 (no-op by design)
410404
appropriate listeners.
411405
"""
412406

413-
def setup_background_tasks(self) -> None:
407+
def start_background_tasks(self) -> None:
414408
"""
415409
Some handlers have side effects on instantiation (like registering
416410
background updates). This function causes them to be fetched, and

0 commit comments

Comments
 (0)