Skip to content

reduce memory usage when decoding files #2492

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 28, 2015

Conversation

ericLemanissier
Copy link
Contributor

When decoding attached files, passing the whole content of the file to imap_base64 or base64_decode leads to allocating the memory for the whole base 64 decoded file while the binary content of the file is still in memory. This leads easily to Out of memory error on limited resources servers. Using .stream_filter_append to decode the file while writing it in a temporary file uses much less memory. The content of the decoded file is then simply read from the file

When decoding attached files, passing the whole content of the file to imap_base64 or base64_decode leads to allocating the memory for the whole base 64 decoded file while the binary content of the file is still in memory. This leads easily to Out of memory error on limited resources servers. Using .stream_filter_append to decode the file while writing it in a temporary file uses much less memory. The content of the decoded file is then simply read from the file
break;
fclose($f);
unlink($temp);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you probably meant to build in more safeguards in the stream handler. The system should make its best effort to provide base64 decoding if the stream system and temp folder write fail. Perhaps it should be written like:

if (strlen($text) > (1 << 20)) {
    try {
        if (!($temp = tempnam(sys_get_temp_dir(), 'attachments'))
            || !($f = fopen($temp, 'w'))
        ) {
            throw new Exception();
        }
        $s_filter = stream_filter_append($f, 'convert.base64-decode',STREAM_FILTER_WRITE);
        if (!fwrite($f, $text))
            throw new Exception();
        stream_filter_remove($s_filter); 
        fclose($f);
        if (!($f = fopen($temp, 'r')) || !($text = fread($f, filesize($temp)))
            throw new Exception();
        fclose($f);
        unlink($temp);
        break;
    }
    catch (Exception $e) {
        // Noop. Fall through to imap_base64 method below
        @fclose($f);
        @unlink($temp);
    }
}
// imap_base64 implies strict mode. If it refuses to decode the
// data, then fallback to base64_decode in non-strict mode
$text = (($conv=imap_base64($text))) ? $conv : base64_decode($text);
break;

This change handles write and read error in case of decoding to temp file, and falls back to in-memory decoding in case of failure
@ericLemanissier
Copy link
Contributor Author

That's just it, thanks !

protich added a commit that referenced this pull request Sep 28, 2015
reduce memory usage when decoding files

Reviewed-By: Peter Rotich <peter@osticket.com>
@protich protich merged commit 6f866f1 into osTicket:develop Sep 28, 2015
@ericLemanissier ericLemanissier deleted the patch-4 branch September 28, 2015 05:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants