Skip to content

Conversation

@peterbroadhurst
Copy link
Contributor

In long runs, we can see this query becomes very inefficient when a large amount of data has built up in the system:

// Find the messages assocated with that data
var messages []*fftypes.Message
for _, data := range data {
fb := database.MessageQueryFactory.NewFilter(ctx)
filter := fb.And(fb.Eq("confirmed", nil))
messages, _, err = ag.database.GetMessagesForData(ctx, data.ID, filter)
if err != nil {
return err
}
}

[2022-03-31T11:48:27.754Z] DEBUG node_1: SQL-> query: SELECT m.id, m.cid, dbtx=0MOmScfF role=event-manager
[2022-03-31T11:48:28.965Z] DEBUG node_1: SQL<- query dbtx=0MOmScfF role=event-manager

The original intention of this index was it would add a some (unnecessary in my view at this point) additional protection against data ID uniqueness within a message (already enforced strongly in code), while allowing a lookup in either direction to be efficient:

CREATE UNIQUE INDEX messages_data_idx ON messages_data(message_id, data_id);

However, the results clearly indicate to me it's only helping with lookup in the message_id based lookups - and not the data_id based lookups. So this PR proposes we replace it with two separate and simple indexes.

Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
@codecov-commenter
Copy link

Codecov Report

Merging #653 (829a496) into main (b7937f7) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##              main      #653   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          313       313           
  Lines        19175     19175           
=========================================
  Hits         19175     19175           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b7937f7...829a496. Read the comment docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants