Memory usage: object store

[JeroenVanOort]

I'm looking into a memory problem for a customer of mine and I came across this:

PHP Opencloud seems to be using quite some memory. I've tested it with the code below. It outputs '1048576' which happens to be exactly 1024 * 1024. I've been looking for some kind of destroy function, similar to PHP's GD library, but I can't seem to find one.

I don't know enough about PHP's memory management to troubleshoot this any further. I hope you guys have a clue ;-)

$mem = memory_get_usage(true);

$connection = new \OpenCloud\OpenStack(
    'https://identity.stack.cloudvps.com/v2.0/',
    [
        'username'      => '*SNIP*',
        'password'      => '*SNIP*',
        'tenantName'    => '*SNIP*'
    ]
);
$service = $connection->objectStoreService('swift', 'NL', 'internalURL');
$container = $service->getContainer('test');
$test = $container->objectExists('file.pdf');
//Me trying to reduce memory usage; to no avail.
unset($test);
unset($container);
unset($service);
unset($connection);

echo memory_get_usage(true) - $mem;

[jamiehannaford]

@JeroenVanOort Hi Jeroen, what version of the SDK and PHP are you using? I'll start looking into it.

[JeroenVanOort]

1.12.1 and 5.6.4

Thanks for your fast reply!

[jamiehannaford]

@JeroenVanOort Just starting with the basic questions: 1048576B is just over 1MB, which doesn't seem a lot to me. The default memory_limit in php.ini is 128MB. What figure are you aiming for?

[JeroenVanOort]

I agree if one would use this code in a page request, but we're using this in a Laravel Daemon. It's gathering up memory.

[ycombinator]

Ah. We'll investigate further but, in the interim, you could try calling gc_collect_cycles right after the unsets and see if that helps. Would you mind trying that and pasting here the output again?

[JeroenVanOort]

I'm sorry to say that the output is still 1048576.

[ycombinator]

@JeroenVanOort I just ran your script (only differences being the auth values and gc_collect_cycles) inside a loop of 100 iterations. I see the same number being output each time, implying no growth in memory usage over time. Would it be possible for you to do the same on your end and report the results?

UPDATE: Without the gc_collect_cycles call, the number does increase by exactly 262144 (about 262KB) every few iterations.

[ycombinator]

@JeroenVanOort See my previous comment (including the update I posted in it later) but here's what @jamiehannaford and I think for next steps:

1. This definitely appears to be a memory leak bug. I've labeled this issue as such and will keep it open so we can investigate and fix.
2. Until we have a fix, you could call gc_collect_cycles to explicitly force garbage collection as a stop-gap measure. Does that work for you?

[JeroenVanOort]

gc_collect_cycles does seem to have an effect when executing the code above multiple times. When executing 10 times with it the consumed memory is again 1048576 bytes, but without it memory usage is 1572864.

I've just pushed the use of this function to the production environment to see if it's reducing memory usage. I'll let you know!

[JeroenVanOort]

I doesn't seem to be doing what I want it do to. This is htop's output on the three daemon processes after some images being processed:

[ycombinator]

Hi @JeroenVanOort, I'm not sure how to interpret that htop screenshot. Are you still seeing memory growth over time after adding the unsets and the gc_collect_cycles call to your code?

[JeroenVanOort]

Yes.

What you see in the screenshot are the daemon workers. They process images that are uploaded to our system and I've narrowed down the memory leak to php opencloud as said before.

[JeroenVanOort]

I've put some more time in to this and I think I've got it down to a bug in Guzzle that was fixed a while ago (guzzle/guzzle#813).

Why is php-opencloud stuck at Guzzle 3 while 4 and 5 are out? One thing I can think of is that upgrading demands PHP 5.4, but other than that I'm not into this enough to say anything useful about it.