-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
fix: Performance and high memory usage of relation hint calculation #4472
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
3149071 to
a1b92a6
Compare
|
I think I've managed to repro, with 125K tables: The server dies when trying to compute the hint: $ curl localhost:3000/xprojects
curl: (52) Empty reply from server
# correct table names do work
$ curl localhost:3000/projects
[{"id":1,"name":"Windows 7","client_id":1},
{"id":2,"name":"Windows 10","client_id":1},
{"id":3,"name":"IOS","client_id":2},
{"id":4,"name":"OSX","client_id":2},
{"id":5,"name":"Orphan","client_id":null}]$ |
Same result with this PR. I'll add a test case. |
Yeah... this PR won't help with the first hint computation (FuzzySet is created anyway). It would only help with subsequent hints (without this PR FuzzySet is created each time). Seems like FuzzySet implementation is memory hungry. Also - it depends on the number of schemas as with this PR there is a separate FuzzySet per schema created (doesn't help if all tables are in a single schema). |
|
@steve-chavez My suggestion would be to either:
Aside: OTOH I understand that if we already have the schema cache then let's use it - trade-offs everywhere :) |
+1 for this also from a dependency perspective: We have to pin an older version of the fuzzy-thing dependency, because a newer version causes even more regression. I'd like to get rid of that dependency, ideally. |
From my experience maintaining a postgREST-as-a-service, before the fuzzy search hint there were lots of support tickets that came from typos but users were quick to to blame postgREST and suspect a malfunction. It was a huge time sink. After the fuzzy search, those complaints just stopped. Nowadays with typescript clients and other typed clients we have in the ecosystem, maybe this is not such a big problem but I'd still argue have the fuzzy search is a net win for 99% use cases.
This looks like the viable short-term solution. Most use cases don't have that many tables too.
IIRC, #3869 was mostly for better error messages, but we have other features that might also reuse it. To not use the schema cache we'd have to implement resource embedding and other features in PostgreSQL itself (possible but would take a loot of time).
Could we instead vendor the dependency? 🤔 (see laserpants/fuzzyset-haskell#9) |
Besides the above, I'm also getting an "empty reply from server" on
Looking at the algorithms used as inspiration on laserpants/fuzzyset-haskell#2 (comment), I wonder if it's optimized for our use case. Maybe we can come up with a better library? (cc @taimoorzaeem) |
Makes sense. [...]
I think it makes sense to merge this PR regardless of these decisions as it is a quite obvious optimization (even though it does not fix the OOM issue). |
I don't think we should merge without a test that proves what's being improved (since it doesn't solve #4463). |
I will be happy to write a library 👍 . I would just need to do some feasibility, maybe a better library already exist on hackage? Needs some investigation. |
a1b92a6 to
20b391f
Compare
Done. See: https://github.com/PostgREST/postgrest/actions/runs/19477317041/attempts/1#summary-55740304508 |
As I mentioned earlier - I've done some research and I couldn't find any interesting alternatives. Fuzzy search algorithms are divided into two subgroups: online and offline. Online algorithms do not require additional memory but require scanning the whole list (so are Offline algorithms require creating an index. Best results are achieved using indexes based on n-grams - this is exactly what I am skeptical we can come up with a solution to 125k tables... |
20b391f to
2057eaa
Compare
@mkleczek We should not modify the mixed loadtest for this, that should stay the same across versions so we can see regressions. Additionally, that doesn't prove there's an enhancement here. Test should be like:
|
I am not sure I understand this. The old So I would say adding the "table not found" scenario to it is an important addition - it makes it more realistic and covering wider set of scenarios.
https://github.com/PostgREST/postgrest/actions/runs/19477317041/attempts/1#summary-55740304508 Am I missing something here?
Hmm... I thought load tests (not @steve-chavez could you advice on the way you would like it to be tested? |
The problem is that "mixed" is already conflating too many scenarios (see #4123), with this addition is even worse as we also conflate "successful traffic" with "error traffic" (note that now we have Another problem is that loadtests are slow, so we shouldn't add too many of them (IMO a dedicated one for errors doesn't seem worth it) if we can instead have an io test.
We do have perf related tests on: postgrest/test/io/test_big_schema.py Lines 10 to 36 in 91abcd4
It looks enough to have an upper bound of expected time for computing the hint to not have regressions.
I'd suggest adding the fixtures on |
ff6d3eb to
c2e1d94
Compare
@steve-chavez |
test/io/test_big_schema.py
Outdated
|
|
||
|
|
||
| def test_second_request_for_non_existent_table_should_be_quick(defaultenv): | ||
| "requesting a non-existent relationship the second time should be quick" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Q: Why is the second time quicker? Does the fuzzy index get populated after hitting an error?
From the code, it looks like the fuzzy index is only populated when the schema cache is built.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Q: Why is the second time quicker? Does the fuzzy index get populated after hitting an error?
From the code, it looks like the fuzzy index is only populated when the schema cache is built.
That's because of Haskell laziness.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, then to avoid confusion I suggest:
| "requesting a non-existent relationship the second time should be quick" | |
| "requesting a non-existent relationship should be quick after the schema cache is loaded (2nd request)" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On second thought, not sure if I understand this.
When we do a request with resource embedding the schema cache is already loaded and works on the first request. Why does this change for the fuzzy index and we need to make 2 requests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe what we need is some comments about this optimization, I suggest adding those on the type definition #4472 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On second thought, not sure if I understand this.
When we do a request with resource embedding the schema cache is already loaded and works on the first request. Why does this change for the fuzzy index and we need to make 2 requests?
This is indeed tricky: schema cache is loaded lazily as well. But there are these two lines in AppState.retryingSchemaCacheLoad:
(t, _) <- timeItT $ observer $ SchemaCacheSummaryObs $ showSummary sCache
observer $ SchemaCacheLoadedObs t
which cause evaluation of SchemaCache fields.
I've decided not to add dbTablesFuzzyIndex to schema cache summary and leave its evaluation till first use.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Introduced the type alias, added comments to SchemaCache field and updated the test description.
c2e1d94 to
ee52b3f
Compare
f8d5cf5 to
6226416
Compare
|
@mkleczek Sorry for the late reply here. Does this fix both #4462 and #4463? If so, maybe we should change the commit prefix to
It does look like the outcome should fix + patch release. Could we add a test or loadtest for the above? |
@steve-chavez |
|
How about a loadtest similar to what I mentioned here: #4523 (comment)?
So, maybe the
There is no reason for it to be @steve-chavez @mkleczek WDYT? |
So while I am open to add some specialized load test for non-existent database objects, I think it is not necessary condition to merge this PR. I would say it is the other way around - it should be merged urgently and - what's more - it should be backported to 14.x. @steve-chavez @taimoorzaeem my 2 cents |
b2e4c42 to
069064f
Compare
Done. See https://github.com/PostgREST/postgrest/actions/runs/20556133512/attempts/1#summary-59040541838 |
From results, this looks like a massive improvement. 🚀 |
I haven't yet dig into the test implemented by @steve-chavez but such huge speedup might be misleading. We have a 500 limit on the number of tables that we index in the FuzzySet. So possibly we just don't calculate any hint in the test hence such a difference. |
I've run What do you think about changing the number of tables in |
🚀 Although the CPU usage is almost x3 on the
The problem is that just using 490 doesn't prove the high memory usage reported on #4462. I think we should merge as is and call #4462 solved. |
See the load test result which shows |
Yeah, that's strange. I examined this and I think that's because on |
Great find. Noted this for later #4123 (comment).
@mkleczek Agree. Let's turn this into a |
069064f to
05375b8
Compare
Done. |
taimoorzaeem
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! We should also add an entry in the CHANGELOG.md.
* Calculation of hint message when requested relation is not present in schema cache requires creation of a FuzzySet (to use fuzzy search to find candidate tables). For schemas with many tables it is costly. This patch introduces dbTablesFuzzyIndex in SchemaCache to memoize the FuzzySet creation. * Additionally, because of FuzzySet large memory requirements, this patch introduces a limit of 500 relations per schema, above which FuzzySet is not created and hint calculation disabled.
05375b8 to
82f6db1
Compare
Done. |
|
Successfully created backport PR for |
| - Fix not returning `Content-Length` on empty HTTP `201` responses by @laurenceisla in #4518 | ||
| - Fix inaccurate Server-Timing header durations by @steve-chavez in #4522 | ||
| - Fix inaccurate "Schema cache queried" logs by @steve-chavez in #4522 | ||
| - Fix performance and high memory usage of relation hint calculation by @mkleczek in #4462 #4463 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Calculation of hint message when requested relation is not present in schema cache requires creation of a FuzzySet (to use fuzzy search to find candidate tables). For schemas with many tables it is costly. This patch introduces dbTablesFuzzyIndex in SchemaCache to memoize the FuzzySet creation.
Fixes #4462
Fixes #4463