Skip to content

Conversation

@oleksandr-nc
Copy link
Contributor

@oleksandr-nc oleksandr-nc commented May 28, 2025

I can reproduce locally such error(I was able to reproduce this even a year ago, but I was thinking that it is only on my local machine):
Allowed memory size of 536870912 bytes exhausted (tried to allocate 5000032 bytes) at /var/www/html/lib/private/Log/ExceptionSerializer.php#236

After that always syncing dies, as process crashes and flag onedrive_import_running does not get reset in the database for the next background jobs run.


Why does memory error occur? (a guess)

The Nextcloud HTTP client is a thin wrapper around Guzzle 7.
With Guzzle, the stream => true flag does not stream the response to the passed resource; it only prevents the body from being-decoded, afaik.
The entire response is still buffered in memory before it is handed back as a Psr7\Stream.
When we then iterate over that stream with fread(), every 5 MB chunk we copy is duplicated – the buffered copy inside Guzzle plus your $chunk string – until PHP asks for another 5 MB and finally dies at ~512 MB.

The easiest way is to tell Guzzle to write directly to the file handler, with sink

I have tested this changes and now I do not see memory exhausted errors anymore.

@oleksandr-nc oleksandr-nc force-pushed the fix/memory-overflow branch 3 times, most recently from 8dd48f5 to 801a8f9 Compare May 28, 2025 14:30
Signed-off-by: Oleksander Piskun <oleksandr2088@icloud.com>
@oleksandr-nc oleksandr-nc force-pushed the fix/memory-overflow branch from 801a8f9 to eb979ad Compare May 28, 2025 14:39
Copy link
Member

@julien-nc julien-nc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💥 Nice discovery. We should avoid streaming from now on.

@oleksandr-nc oleksandr-nc marked this pull request as ready for review May 28, 2025 16:26
@oleksandr-nc oleksandr-nc requested a review from marcelklehr as a code owner May 28, 2025 16:26
@oleksandr-nc oleksandr-nc merged commit b17cd17 into main May 28, 2025
14 checks passed
@oleksandr-nc oleksandr-nc deleted the fix/memory-overflow branch May 28, 2025 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants