Speed up marking device list changes as converted

[erikjohnston]

The conversion of device_lists_changes_in_room rows into device_lists_outbound_pokes is DB-bound on
mark_redundant_device_lists_pokes. The UPDATE uses the (room_id, stream_id) index, so each call scans the unconverted backlog, getting slower the further behind the conversion is.

Add a partial index matching the query via a background update, and skip the (safe-to-skip) UPDATE until the index has been built. Also add a metric reporting how far behind the conversion is, using the existing inserted_ts column.

Fixes https://github.com/element-hq/backend-internal/issues/286

[github-actions]

Schema Diff

Please check that this looks as expected!

common (sqlite)

Unchanged

common (postgres)

Unchanged

main (sqlite)

--- /tmp/schema_diff_az2vpfi0/before/main/full_schemas/9999/full.sql.sqlite	2001-05-25 12:42:42.000000000 +0000
+++ /tmp/schema_diff_az2vpfi0/after/main/full_schemas/9999/full.sql.sqlite	2001-05-25 12:42:42.000000000 +0000
@@ -1227,20 +1227,21 @@
 CREATE INDEX local_media_repository_sha256 ON local_media_repository (sha256) WHERE sha256 IS NOT NULL;
 CREATE INDEX remote_media_cache_sha256 ON remote_media_cache (sha256) WHERE sha256 IS NOT NULL;
 CREATE INDEX sliding_sync_membership_snapshots_membership_event_id_idx ON sliding_sync_membership_snapshots (membership_event_id) ;
 CREATE INDEX sliding_sync_membership_snapshots_user_id_stream_ordering ON sliding_sync_membership_snapshots (user_id, event_stream_ordering) ;
 CREATE INDEX device_federation_inbox_received_ts_index ON device_federation_inbox (received_ts) ;
 CREATE UNIQUE INDEX event_txn_id_device_id_txn_id2 ON event_txn_id_device_id (user_id, device_id, room_id, txn_id) ;
 CREATE UNIQUE INDEX delayed_events_idx ON delayed_events (delay_id) ;
 CREATE INDEX device_lists_changes_in_room_inserted_ts_idx ON device_lists_changes_in_room (inserted_ts) WHERE inserted_ts IS NOT NULL;
 CREATE INDEX sliding_sync_connections_last_used_ts_idx ON sliding_sync_connections (last_used_ts) WHERE last_used_ts IS NOT NULL;
 CREATE INDEX sliding_sync_connection_lazy_members_conn_pos_idx ON sliding_sync_connection_lazy_members (connection_position) WHERE connection_position IS NOT NULL;
+CREATE INDEX device_lists_changes_in_room_unconverted_idx ON device_lists_changes_in_room (user_id, device_id, room_id, stream_id) WHERE NOT converted_to_destinations;
 /* WARNING: Script requires that SQLITE_DBCONFIG_DEFENSIVE be disabled */
 INSERT INTO appservice_stream_position VALUES('X',0);
 INSERT INTO federation_stream_position VALUES('federation',-1,'master');
 INSERT INTO federation_stream_position VALUES('events',-1,'master');
 INSERT INTO event_push_summary_stream_ordering VALUES('X',0);
 INSERT INTO user_directory_stream_pos VALUES('X',-1);
 INSERT INTO stats_incremental_position VALUES('X',1);
 INSERT INTO event_push_summary_last_receipt_stream_id VALUES('X',0);
 INSERT INTO device_lists_changes_converted_stream_position VALUES('X',1,'',NULL);
 INSERT INTO room_forgetter_stream_pos VALUES('X',0);

main (postgres)

--- /tmp/schema_diff_az2vpfi0/before/main/full_schemas/9999/full.sql.postgres	2001-05-25 12:42:42.000000000 +0000
+++ /tmp/schema_diff_az2vpfi0/after/main/full_schemas/9999/full.sql.postgres	2001-05-25 12:42:42.000000000 +0000
@@ -1654,20 +1654,21 @@
 CREATE INDEX device_auth_providers_devices ON device_auth_providers USING btree (user_id, device_id);
 CREATE INDEX device_auth_providers_sessions ON device_auth_providers USING btree (auth_provider_id, auth_provider_session_id);
 CREATE INDEX device_federation_inbox_received_ts_index ON device_federation_inbox USING btree (received_ts);
 CREATE INDEX device_federation_inbox_sender_id ON device_federation_inbox USING btree (origin, message_id);
 CREATE INDEX device_federation_outbox_destination_id ON device_federation_outbox USING btree (destination, stream_id);
 CREATE INDEX device_federation_outbox_id ON device_federation_outbox USING btree (stream_id);
 CREATE INDEX device_inbox_stream_id_user_id ON device_inbox USING btree (stream_id, user_id);
 CREATE INDEX device_inbox_user_stream_id ON device_inbox USING btree (user_id, device_id, stream_id);
 CREATE INDEX device_lists_changes_in_room_by_room_idx ON device_lists_changes_in_room USING btree (room_id, stream_id);
 CREATE INDEX device_lists_changes_in_room_inserted_ts_idx ON device_lists_changes_in_room USING btree (inserted_ts) WHERE (inserted_ts IS NOT NULL);
+CREATE INDEX device_lists_changes_in_room_unconverted_idx ON device_lists_changes_in_room USING btree (user_id, device_id, room_id, stream_id) WHERE (NOT converted_to_destinations);
 CREATE UNIQUE INDEX device_lists_changes_in_stream_id ON device_lists_changes_in_room USING btree (stream_id, room_id);
 CREATE INDEX device_lists_changes_in_stream_id_unconverted ON device_lists_changes_in_room USING btree (stream_id) WHERE (NOT converted_to_destinations);
 CREATE UNIQUE INDEX device_lists_outbound_last_success_unique_idx ON device_lists_outbound_last_success USING btree (destination, user_id);
 CREATE INDEX device_lists_outbound_pokes_id ON device_lists_outbound_pokes USING btree (destination, stream_id);
 CREATE INDEX device_lists_outbound_pokes_stream ON device_lists_outbound_pokes USING btree (stream_id);
 CREATE INDEX device_lists_outbound_pokes_user ON device_lists_outbound_pokes USING btree (destination, user_id);
 CREATE UNIQUE INDEX device_lists_remote_cache_unique_id ON device_lists_remote_cache USING btree (user_id, device_id);
 CREATE UNIQUE INDEX device_lists_remote_extremeties_unique_idx ON device_lists_remote_extremeties USING btree (user_id);
 CREATE UNIQUE INDEX device_lists_remote_pending_user_device_id ON device_lists_remote_pending USING btree (user_id, device_id);
 CREATE UNIQUE INDEX device_lists_remote_resync_idx ON device_lists_remote_resync USING btree (user_id);

state (sqlite)

Unchanged

state (postgres)

Unchanged

[MadLittleMods]

Perhaps we should just drop mark_redundant_device_lists_pokes as an optimization?

You mentioned that you "don't think it catches many redundant pokes anyway" (internal room)

[erikjohnston]

I think it is useful if you have the same device coming and going, but it's not necessary.

We could probably delete it entirely but I think a) we'd want to investigate it, and b) I think we still would want an index like the above anyway.

[erikjohnston]

@MadLittleMods I've also added this so you can see how many stream IDs we're behind on.

This helps because if there is a large batch of device updates in a short period of time the lag time might increase even though it's catching up.

[MadLittleMods]

We can do the same lookup and update of _completed_background_updates here as well

[erikjohnston]

That's already handled via self._all_done?

[MadLittleMods]

This is have_completed_background_updates(update_names) not has_completed_background_updates()

Updating our cache in have_completed_background_updates(update_names) makes just as much sense as has_completed_background_update(update_name) like we are doing

[erikjohnston]

Thanks @MadLittleMods !