-
Notifications
You must be signed in to change notification settings - Fork 233
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
hll memory estimate in MB instead of Bytes #312
base: master
Are you sure you want to change the base?
Conversation
@@ -30,7 +30,7 @@ import ( | |||
) | |||
|
|||
const ( | |||
hllQueryRequiredMemoryInMB = 10 * 1024 | |||
hllQueryRequiredMemoryInBytes = 10 * (1 << 30) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so originally we only reserved 10K for HLL? and now 10GB, isn't a little too bigger?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah we have been reserved 10K.
10GB is to make sure each card only run one hll at a time since we cannot estimate hll effectively. We have agreed on 10G at the first place, but i think at some point the code got mixed up, MB changed to bytes. In production we have enough machines to spread out the large contract queries so that we don't see the problem. but in staging we can actually see the problem
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure, we better do 2 observation:
- how many real gpu memory allocated when running weekly hll contractor in production/staging
- how it impacts production query when hll runs with 10GB reserved. Now as we under-reserve the memory, so other queries may still able to run. After we bump to 10G, other queries in the same machine will be totally blocked even the GPU might have spare space.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #312 +/- ##
========================================
Coverage 71.88% 71.89%
========================================
Files 166 166
Lines 23252 23367 +115
========================================
+ Hits 16714 16799 +85
- Misses 5246 5275 +29
- Partials 1292 1293 +1 ☔ View full report in Codecov by Sentry. |
Jian Shen seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
this drastically under estimate memory usage for hll query and cause out of memory situation during massive hll query hits, such as weekly hll contracts