Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hll memory estimate in MB instead of Bytes #312

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

jshencode
Copy link
Contributor

this drastically under estimate memory usage for hll query and cause out of memory situation during massive hll query hits, such as weekly hll contracts

@@ -30,7 +30,7 @@ import (
)

const (
hllQueryRequiredMemoryInMB = 10 * 1024
hllQueryRequiredMemoryInBytes = 10 * (1 << 30)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so originally we only reserved 10K for HLL? and now 10GB, isn't a little too bigger?

Copy link
Contributor Author

@jshencode jshencode Sep 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah we have been reserved 10K.
10GB is to make sure each card only run one hll at a time since we cannot estimate hll effectively. We have agreed on 10G at the first place, but i think at some point the code got mixed up, MB changed to bytes. In production we have enough machines to spread out the large contract queries so that we don't see the problem. but in staging we can actually see the problem

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, we better do 2 observation:

  1. how many real gpu memory allocated when running weekly hll contractor in production/staging
  2. how it impacts production query when hll runs with 10GB reserved. Now as we under-reserve the memory, so other queries may still able to run. After we bump to 10G, other queries in the same machine will be totally blocked even the GPU might have spare space.

@codecov
Copy link

codecov bot commented Sep 30, 2019

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 71.89%. Comparing base (56108d0) to head (6b14771).
Report is 47 commits behind head on master.

Files with missing lines Patch % Lines
query/aql_processor.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##           master     #312    +/-   ##
========================================
  Coverage   71.88%   71.89%            
========================================
  Files         166      166            
  Lines       23252    23367   +115     
========================================
+ Hits        16714    16799    +85     
- Misses       5246     5275    +29     
- Partials     1292     1293     +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Jian Shen seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants