chore: Optimize hash queries with lookup table #2933
Conversation
|
This PR may contain changes to database schema of one of the drivers. If you are introducing any changes to the schema, make sure the upgrade from the latest release to this change passes without any errors/issues. Please make sure the label |
|
You can find the image built from this PR at Built from 4f31de1 |
Ivansete-status
force-pushed
the
optimize-hash-queries-with-lookup-table
branch
from
a333adf to
8ad5852
Compare
Ivansete-status
requested review from
SionoiS,
gabrielmer,
NagyZoltanPeter and
richard-ramos
| ); | ||
|
|
||
| -- Put data into lookup table | ||
| INSERT INTO messages_lookup (messageHash, timestamp) SELECT messageHash, timestamp from messages; |
There was a problem hiding this comment.
I do not know how slow this operation is, but since it's doing a batch insert, a way to speed up the insertion could be to create the table without the primary key, and then, after the insertion is complete, do:
ALTER TABLE messages_lookup ADD CONSTRAINT messageIndexLookupTable PRIMARY KEY (messageHash, timestamp)This should be faster because it eliminates the overhead of maintaining the primary key index during each insert operation. And should be 'safe' since the messageHash, timestamp combination should be unique already in the messages table
There was a problem hiding this comment.
(of course, if the insertion takes just a couple of minutes then doing this optimization is an overkill)
There was a problem hiding this comment.
cool thanks! I've applied the suggestion in cb80df4
I think it would take around 15 minutes to complete
Yes, this can be checked through Status . I will suggest to dogfood that in our weekly Status dogfooding session |
Ivansete-status
force-pushed
the
optimize-hash-queries-with-lookup-table
branch
from
efd0e96 to
62d7f5e
Compare
* Upgrade Postgres schema to add messages_lookup table * Perform optimized query for messageHash-only queries
Description
This PR enhances the performance of the most expensive query that we have at the moment, i.e.,
... WHERE messageHash IN (...)The enhancement leverages the use of a lookup table (
messages_lookup)Special thanks to @NagyZoltanPeter and @richard-ramos for sharing that great idea ❤️
Changes
messages_lookuptablemessaged_lookuptable for new messagesgetMessagesprocs when they fetch messages only filtering by themessageHashattributeIssue
closes #2895