You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When uploading lots of files using ObjectStorage.Insert(), each file allocates his own buffer. This results in increased CPU usage and high RSS memory.
Graph below shows how many space was allocated during upload of ~30k of 10MB files:
User should be able to pass his own buffer and be responsible for allocations or library shouldn't be so wasteful.
Pull request which adds user facing insert option is here: #632, I was asked to an open issue to discuss it.
cc. @tritone@codyoss
The text was updated successfully, but these errors were encountered:
codyoss
changed the title
High memory allocation footprint during uploading
gensupport: allow passing in buffer for uploads to reduce allocations
Aug 25, 2020
When uploading lots of files using ObjectStorage.Insert(), each file allocates his own buffer. This results in increased CPU usage and high RSS memory.
Graph below shows how many space was allocated during upload of ~30k of 10MB files:
User should be able to pass his own buffer and be responsible for allocations or library shouldn't be so wasteful.
Pull request which adds user facing insert option is here: #632, I was asked to an open issue to discuss it.
cc. @tritone @codyoss
The text was updated successfully, but these errors were encountered: