Description
Might be hard to believe me, but I'm like 99.999% certain that sometimes php-memcached returns incorrect value from get. Trouble is I have no way to reliably reproduce this, but I have spent hunting the problem on our servers like 4 days, and only sensible conclusion is that sometimes incorrect value is returned. (since php 7.1, php memcached 3.0.3, libmemcached version 1.0.18, previously we were on php 5.6 and did not have such problem).
Some background:
for invalidating cache in large chunks (say all cache entries related to one db table) we have "namespace implementation", certain keys "namespace keys" store numbers, when application needs to get cached value for certain key and a given namespace, we first retrieve value for "namespace key", which is a number, then concatenates "namespace key""namespace value""item key".
Naturally this means that retrievals for "namespace key" are very frequent, for each namespaced get/set/etc request. Still this allows to easily invalidate many keys, by incrementing/changing "namespace key" value.
Trouble is that after upgrading php, and php-memcached, we sometimes somehow have a result of php-memcached -> get("namespace key") returning invalid value (some serialized object, but read on). Needless to say over last 4 days I looked a lot of things. Now biggest problem is that it only happens sometimes, hard to say how often. Still I logged around all our behavior quite a lot, and fail to see application logic failure. One most convincing test that shows me that this is php-memcached and not server issue, is that even when I get errors about invalid value for "namespace key", i get at most one such error per request (but again that "namespace key" is being retrieved multiple times on each request). However if I manually set "namespace key" to invalid value on servers (we have two memcached servers, using consisten hashing) I get like 5-10 notifications about invalid value, as value is really invalid.
One more thing, I have problem with "namespace keys" for some different namespaces, quite some. In any case for some I managed to "work around" this, which simply means where I had cases where I get values for "each row" one by one, I had many such errors, to improve the situation there (not only for cache, but also in case cache goes down and only db is used) I reworked those places to essentially cache whole (sub)table (as those tables are not very large), and store it php static var to not request data from db and/or cache, and I get no more errors from there, even though namespace is still used, but of course just couple of times per http request at most.
Now last thing, for those incorrect values, they seem to be one of fully namespaced values, so say I have calls like
urls_namespace7awesome_page_url it stores value
"www.nicedomain.com/this_is_best_page_ever.html"
then next time (well presumably...) php memcached gets
urls_namespace it also returns "www.nicedomain.com/this_is_best_page_ever.html" while it should return 7 .
So possibly there is some "internal caching" in phpmemcached, that checks if same key was requested recently and if so it does not make request to server? Well this is total speculation on may part, and disregard it if it is nonsense.
I know this is fairly vague, but it is all I have. I tried reproducing this with simple scripts, without whole application, but couldn't do it; I even tried looking at php-memcached C code, but I'm unfamiliar with php extension development, nether libmemcached, so was unable to have any meaningfull understanding of what is happening.