Many oidc_cache_file_clean: unable to open cache entry "/path/to/cache/mod-auth-openidc-s-...tmp" (No Such file or directory)

[rich-page-vtm]

Receiving a high number of the following errors in the Apache error log:

oidc_cache_file_clean: unable to open cache entry "/path/to/efs/apache2/var/oidc_sessions/mod-auth-openidc-s-mdgCJd4-Jdw3r2niEwfsrswQje0nWOEH_zRZ5Dp7-ow.HdfF5q86xuc0SNOt.tmp" (No such file or directory)

Cache files are located on an AWS EFS NFS mount.

mod_auth_openidc: 2.4.16.5
OS: Amazon Linux 2023
Apache: 2.4.62

Thanks in advance for your help.

[rich-page-vtm]

The EFS / NFS filesystem where the cache files are stored was being overloaded with unrelated activity which is was likely causing the increased logging of this error.

While the high NFS read activity was occurring, we were seeing upwards of 400,000 errors of this type in the apache error log.

Now that the heavy read activity has subsided, the error does still appear some 3000-4000 times a day now, but this does not seem to cause any ill effects.

Would still be interested in knowing if anyone has encountered this, what the cause is, and if there is a way to mitigate.

[zandbelt]

the only thing I can offer is to turn it into a warning since as you mentioned it may not actually be an error: perhaps another Apache process cleaned up the file, or if it is corrupted it is merely considered as a non-existing cache entry; would that help?

[rich-page-vtm]

We are running a cluster of 3 or more application servers with each having their own apache with mod_auth_openidc and all configured to share the same OIDCCacheDir on the NFS/EFS share. Each node has a CacheFileCleanInterval of 30 seconds.

Having all the nodes share the cache, is acceptable?

You're suggesting that the cleaners on the nodes are conflicting with each other. Is there a better way to configure this?

[zandbelt]

it can work in most situations but things may be more difficult when (most or) all of the following apply:

• there's a huge number of sessions/entries, all valid (i.e. non expired) and/or a lot expire around the same time
• the network drive access is (relatively) slow
• the Apache servers have all been started at the same time
  we could work on a locking mechanism but it will make things slower for regular cases

[zandbelt]

also: most environments with a massive number of sessions use Memcache or Redis backends