Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

replication for typing stream restarted from 0 on restart #8524

Open
richvdh opened this issue Oct 12, 2020 · 2 comments
Open

replication for typing stream restarted from 0 on restart #8524

richvdh opened this issue Oct 12, 2020 · 2 comments
Labels
A-Workers Problems related to running Synapse in Worker Mode (or replication) z-p2 (Deprecated Label)

Comments

@richvdh
Copy link
Member

richvdh commented Oct 12, 2020

After restarting a worker, it is emitting log lines as follows:

2020-10-12 18:11:14,942 - synapse.replication.tcp.handler - 520 - INFO - process-replication-data-1 - Fetching replication rows for 'typing' between 0 and 16463350
2020-10-12 18:11:14,943 - synapse.logging.opentracing - 307 - ERROR - process-replication-data-1 - There was no active span when trying to inject the span into a byte dict. Did you forget to start one or did a context slip?
2020-10-12 18:11:15,349 - synapse.http.client - 381 - INFO - process-replication-data-1 - Received response to GET http://10.103.0.2:8126/_synapse/replication/get_repl_stream_updates/typing/DdMdhASQZL?from_token=0&upto_token=16463350: 200
2020-10-12 18:11:15,384 - synapse.replication.tcp.handler - 520 - INFO - process-replication-data-1 - Fetching replication rows for 'typing' between 49300 and 16463350
2020-10-12 18:11:15,400 - synapse.logging.opentracing - 307 - ERROR - process-replication-data-1 - There was no active span when trying to inject the span into a byte dict. Did you forget to start one or did a context slip?
2020-10-12 18:11:15,748 - synapse.http.client - 381 - INFO - process-replication-data-1 - Received response to GET http://10.103.0.2:8126/_synapse/replication/get_repl_stream_updates/typing/McoEWmdCwi?from_token=49300&upto_token=16463350: 200
2020-10-12 18:11:15,784 - synapse.replication.tcp.handler - 520 - INFO - process-replication-data-1 - Fetching replication rows for 'typing' between 107693 and 16463350
2020-10-12 18:11:15,785 - synapse.logging.opentracing - 307 - ERROR - process-replication-data-1 - There was no active span when trying to inject the span into a byte dict. Did you forget to start one or did a context slip?
2020-10-12 18:11:16,629 - synapse.http.client - 381 - INFO - process-replication-data-1 - Received response to GET http://10.103.0.2:8126/_synapse/replication/get_repl_stream_updates/typing/PnoOikhuQZ?from_token=107693&upto_token=16463350: 200
2020-10-12 18:11:16,667 - synapse.replication.tcp.handler - 520 - INFO - process-replication-data-1 - Fetching replication rows for 'typing' between 160170 and 16463350

Apart from the context errors as per #8510, surely it shouldn't start from zero?

@richvdh
Copy link
Member Author

richvdh commented Oct 12, 2020

(it did it again on a second restart)

@erikjohnston
Copy link
Member

This is "known". The basic problem is that workers need the full list of people typing to service /sync requests, and since that list only exists on the typing writer we end up streaming everything on start up.

We should probably figure out a better way of doing it.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A-Workers Problems related to running Synapse in Worker Mode (or replication) z-p2 (Deprecated Label)
Projects
None yet
Development

No branches or pull requests

3 participants