OPcache recommendation wrong (opcache.enable_cli=1)

[tessus]

I've been meaning to open an issue for this for a while now, but never got around to it.

The recommendation to set opcache.enable_cli=1 is actually bad for performance. I am not sure who came up with this, but I reckon that whoever it was did not understand what it meant.

Enabling OPcache for CLI has the following effect: it will optimize the CLI script (which takes of course more time the first time to do). However, in the case of a CLI script, it is always the first time for every single invocation of the script. The cache only lives until the process terminates, so as soon as the script ends the optimizations are gone.
The only effect it had was that the script took longer to run, because optimization were done during runtime.
You do not have to take my word for it. You can read up on this topic in several articles or contact the PHP developers.
(Full disclosure: There are certain exceptions where perf can be improved for CLI scripts, but please note that those do not come into play for occ or any CLI script that is run by nextcloud.)

Not only the documentation is wrong here, also the check in settings -> admin -> overview. It is rather annoying to get a warning every time, even though the settings are actually better for performance.
But I guess that's a separate ticket, which I will happily open as soon as we have established that my concerns are well-founded.

[skjnldsv]

@nickvergessen @rullzer @MorrisJobke ?

[MorrisJobke]

@tessus Thanks for this. Makes total sense. We should change it.

[MorrisJobke]

Fix is in nextcloud/server#15468

[tessus]

@MorrisJobke Thanks. I just saw that you have already fixed it in the server check. I would have opened another issue in the server repo for this. I originally opened this issue for the documentation: https://docs.nextcloud.com/server/15/admin_manual/installation/server_tuning.html#enable-php-opcache

But I am more than happy that it was fixed in the server. Cheers.

[MichaIng]

@tessus
Many thanks for this, I was thinking the same for a while.

Another question, I hope it's okay I ask it here, is the recommended OPcache size value:

• The whole Nextcloud PHP scripts in OPcache take ~30M (if I am not mistaken?).
• Even when installing a large amount of 3rd party apps, this does not change much.
• So the default max size of 64M (EDIT: It's indeed 128M since PHP7.0) should be totally sufficient. The only real minimum recommendation to me should be 32M, everything above highly depends on possible other webserver usage and should not/cannot be estimated.
• At best the check should simply test if all Nextcloud scripts are indeed in OPcache or not, but not sure how/if that is easily/quickly possible.

[MorrisJobke]

• At best the check should simply test if all Nextcloud scripts are indeed in OPcache or not, but not sure how/if that is easily/quickly possible.

I couldn't find that one when I worked on it.

• So the default max size of 64M should be totally sufficient. The only real minimum recommendation to me should be 32M, everything above highly depends on possible other webserver usage and should not/cannot be estimated.

I actually changed it to 128MB, because I had an instance with some apps enabled that where quite have and reached over the 64 MB.

Does that make sense to you?

[MichaIng]

@MorrisJobke
Hmm, okay in this case it makes sense. I never reached more than 33M AFAIK (monitored with https://github.com/rlerdorf/opcache-status/blob/master/opcache.php) with quite many apps enabled.

I am not sure how this OPcache size is handled. Usually a cache should be not much larger than the amount that is actually used, since larger cache reduces performance for the individual request usually. So if there is any way to have the recommended cache size dependant on the actual amount/size of PHP files of the instance, this would be beneficial. In case it is also a question how much the performance decreases if single scripts are not cached due to max size reached.

[tessus]

In case it is also a question how much the performance decreases if single scripts are not cached due to max size reached.

This is not how a cache works. In that case eviction occurs (by some alogorithm like LRU, LFU, FIFO, ...) and the new csript will be optimized and added to the cache. Thus the perf hit only amounts to removing data from memory. That's it.

[MichaIng]

@tessus
Makes sense. However I know from MariaDB/MySQL caches that larger sizes lead to decreased performance as long as everything stays in cache, since a larger size needs to be handled/searched through for each access. But as said, not sure if this is true for OPcache as well and how this compares to the performance decrease when regularly data needs to be removed due to too low size.

Btw. couldn't some method from opcache.php be used to check currently used OPcache size and file count and recommending the next power of two in admin panel in case max size is full or close to?
EDIT: https://www.php.net/manual/en/function.opcache-get-status.php

---

Perhaps I should open a new issue about this, also based on some new evaluation how much cache is used when installing large amounts of apps.

[tessus]

However I know from MariaDB/MySQL caches that larger sizes lead to decreased performance as long as everything stays in cache, since a larger size needs to be handled/searched through for each access.

Hmm, I worked in DB2 development for many years and data is always manipulated or read from bufferpools (except large objects). This means that data has first to be fetched into the bufferpool to be used. This also means the larger the bufferpool, the better it is for performance.
I haven't looked at the source code for MySQL (I wasn't allowed to look at other code due to code contamination), but I doubt that database engines (for RDBMS) are fundamentally different.
Therefore I do not understand why a larger cache would mean a performance hit.
If this really is the case, something might be wrong with the engine code.

I don't see a bigger OPcache memory allocation to be detrimental to performance. e.g. the max_accelerated_files parameter is actually a prime number (you can check the docs), which means a hash function is used to access data. We all should know what that means for big-O. ;-)

[MichaIng]

@tessus
I tried to find the sources I read some time ago about this. Actually I believe this was indeed limited to the query_cache_size which locks on every update: https://mariadb.com/kb/en/library/query-cache/#limiting-the-size-of-the-query-cache

But lets see how much OPcache is actually used by Nextcloud:

• Test with PHP-FPM 7.3.5, Zend Engine v3.3.5, Nextcloud 16.0.0
• First login after fresh install: 23M
• Enabled literally every available app (on official app store, including the ones that are not yet tested on NC16) that does not deny do be enabled due to e.g. missing backend, unknown SSL issuer or cert revocation (jep there are some that need review 😉). Also I needed to disable 3-4 apps that caused internal server errors, most likely due to missing backend or overlap between apps. I navigated through all apps as well as their settings two times to hopefully call most PHP scripts once: 62.96M, 3764 scripts in cache
• But to be true:

root@VM-Stretch:~# ls /var/www/nextcloud/apps/ | wc -l
204
root@VM-Stretch:~# find /var/www/nextcloud/ -name *.php | wc -l
36551
root@VM-Stretch:~# du -shc $(find /var/www/nextcloud/ -name *[0-9].php) | tail -1
43M     total
root@VM-Stretch:~# du -shc $(find /var/www/nextcloud/ -name *[a-m].php) | tail -1
67M     total
root@VM-Stretch:~# du -shc $(find /var/www/nextcloud/ -name *[n-s].php) | tail -1
93M     total
root@VM-Stretch:~# du -shc $(find /var/www/nextcloud/ -name *[t-z].php) | tail -1
48M     total
root@VM-Stretch:~# du -shc $(find /var/www/nextcloud/ -name *[^0-9a-z].php) | tail -1
8.0M    total

• Not sure if all those scripts could be theoretically cached 😉.

---

Quite interesting btw: