Cache invalidation for scripts in symlinked folders

[silentroach]

Hi!

We are trying to use your opcode cache with our project.
Our code versions are managed in production with symlinks. Sometimes it seems that opcode cacher timestamp invalidation fails after project symlink is changed and we need to clear the cache manually.

/somefolder
    /production -> 20130823_2207
    /20130823_2207
    /20130823_2115
    ...

Can you help us with any advice? Thank you.

Versions:

PHP 5.5.1-1
Zend Engine v2.5.0, Copyright (c) 1998-2013 Zend Technologies
    with Zend OPcache v7.0.2-dev, Copyright (c) 1999-2013, by Zend Technologies

Config:

opcache.blacklist_filename => no value
opcache.consistency_checks => 0
opcache.dups_fix => Off
opcache.enable => On
opcache.enable_cli => Off
opcache.enable_file_override => Off
opcache.error_log => no value
opcache.fast_shutdown => 0
opcache.force_restart_timeout => 180
opcache.inherited_hack => On
opcache.interned_strings_buffer => 4
opcache.load_comments => 1
opcache.log_verbosity_level => 1
opcache.max_accelerated_files => 2000
opcache.max_file_size => 0
opcache.max_wasted_percentage => 5
opcache.memory_consumption => 64
opcache.optimization_level => 0xFFFFFFFF
opcache.preferred_memory_model => no value
opcache.protect_memory => 0
opcache.revalidate_freq => 2
opcache.revalidate_path => Off
opcache.save_comments => 1
opcache.use_cwd => On
opcache.validate_timestamps => On

[dstogov]

Do you pass the real path names (all symlinks resolved) to opcache_invalidate()? Otherwise if you change the symlink, file names might be resolved to different real paths.

[silentroach]

I mean automatic invalidation, not via opcache_invalidate.

[dstogov]

In case you just change symlink, the scripts laying in old directory are
still valid :)

On Mon, Aug 26, 2013 at 11:22 AM, Kalashnikov Igor <notifications@github.com

wrote:

I mean automatic invalidation, not via opcache_invalidate.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/126#issuecomment-23247289
.

[silentroach]

And it will be great if Zend OPCache will handle it.
I think it is a common way to deploy. It is used in Capistrano for example.

[rlerdorf]

The opcode cache uses the realpath of the files, so if a different symlink points to the same file you will get the same set of opcodes. You also have it configured to only check every 2 seconds, so for 2 seconds after pointing your "production" symlink at another target you are going to get the old target files. You can read about how to properly manage a symlinked docroot with opcache here: http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/

[silentroach]

Too complicated, to be the best solution :)

And realpath makes symlinks resolved.

[rlerdorf]

Well, then just turn off opcache.revalidate_freq so it will revalidate on every request. Your deploys won't be atomic, but it should never load the wrong file.

[kayue]

+1 on this, we are using Capifony.org / Capistrano to deploy our project, and it uses symlink...

[rlerdorf]

+1 on what? There is no bug here. What is most likely happening is that the failed requests are the ones that get screwed over when the symlink switch happens while they are executing, or at least within the revalidate_freq window. You can shrink this window by setting opcache.revalidate_freq to 0. It doesn't entirely eliminate the problem, but it comes very close, at least if your site isn't very busy. To completely eliminate the problem, read on:

opcache has no concept of the start of a request. It works on individual opcode arrays. An opcode array is what is generated and cached for each included file and the key for each included file is the fully qualified path for the script that was compiled. How you got there, via a symlink, various relative path specifiers (think ../path/file.php or ../../other/path/file.php) is irrelevant, the fully qualified path (or the realpath) to that file is the same and the access mechanism is not maintained.

So, when you deploy via something like Capistrano which does a symlink swap on the document root, you want all new requests to get the new files, but you don't want to screw over requests that are currently executing as the deploy is happening. What you really need to create a robust deploy environment is to have your web server be in charge of this. The web server is the piece of the stack that understands when a new request is starting. The opcode cache is too deep in the stack to know or care about that.

With nginx this is quite simple. Just add this to your config:

fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
fastcgi_param DOCUMENT_ROOT $realpath_root;

This tells nginx to realpath resolve the docroot symlink meaning that as far as your PHP application knows, the target of the symlink if the real document_root. Now, once a request starts, nginx will resolve the symlink as it stands at that point and for the duration of the request it will use the same docroot directory, even if the symlink switch happening mid-request. This entirely eliminates the symptoms described here and it is the correct approach. This isn't something that can be solved at the opcache level.

Apache doesn't have this same mechanism to resolve a docroot symlink at the start of the request, but I have written an Apache module that does it. See https://github.com/etsy/mod_realdoc
It is slightly more efficient than the nginx approach because it includes a configurable realpath cache so you don't have to do this somewhat expensive realpath on every request.

[kayue]

The $realpath_root is a great solution. Thanks.

FYI the symlink issue doesn't only happen in the 2 seconds opcache.revalidate_freq window; in my case it doesn't update until I restart php-fpm.

Thanks a lot, this should go into FAQ in my opinion. I would submit a pull request if my English is good enough.

[silentroach]

Yes, only service restart or invalidate function call helps.
(closed by mistake, sorry)

$realpath_root isn't a great solution cause it is undocumented.

[rlerdorf]

Actually, invalidating the cache doesn't solve anything. Requests that are already running which started on version A of the code, if you suddenly deploy and invalidate the cache before that request has finished, may very well do another include and at that point it will be including files from version B and you are back to an unknown state for that request. A web server restart, assuming it is a graceful restart that lets existing requests finish, can be made to work, but the timing is a bit tricky. You have to combine the graceful restart with a config change so the new requests will see the new docroot and the requests that are finishing up continue on the previous.

Or you need to get into load balancer tricks where you stop sending new requests to a subset of your servers. Then you wait a while to let existing requests finish, then deploy to that subset and repool them. You may not have enough servers to do this without seriously affecting site performance though, and it also slows down deploys significantly. So I still think having the web server realpath the docroot symlink and setting the effective docroot to the target of that symlink is the slickest and most complete solution to this problem.

[TerryE]

+1 on Rasmus's points here. The issue that you face is that the PHPs Zend VM maps in new sources at runtime as it executes the INCLUDE_OR_EVAL opcodes which reference the source. If you want to avoid scripts barfing in indeterminate ways during version cutover, you must maintain path integrity across all of these includes (and the requested script). If you defer symbolic resolution of paths to the runtime system then you will occasionally hit the asynchronous edge effects, unless you take the steps that Rasmus describes.

Of course, this is also easy to implement at an application level -- but only if you can set coding standards or modify code, e.g. using tricks like using complete install hierarchies and setting:

define('ROOT_DIR', dirname(__FILE__)); 

(or using a realpath(someSymlink) instead of __FILE__) on immediate entry to every request then using ROOT_DIR relative pathing for all includes.

__FILE__ of the requested script is pretty bomb-proof because this is fully resolved by the SAPI and the requested script is loaded from this resolved path, so any paths (which don't embed further symlinks) relative to this will always be consistent with the above. With the someSymlink variant, there is still a small execution window (< a few mSec) where these might get out of step but this is typically orders of magnitude less than the window that a JiT autoloader would experience.

@rasmus, would you agree with this analysis or have I missed something?