OCSP response cache is not updated in a timely manner

[VWDude]

What happened:

We are using ingress-nginx with the config value "enable-ocsp": true.
In the beginning this works as expected, but the OCSP cache is not updated, when the response expires after 2 days:
Taken from openssl response on 08.Nov.2023 13:53 GMT:

What you expected to happen:
OCSP cache is updated before the expiry and the response is still valid.

NGINX Ingress controller version:

NGINX Ingress controller
  Release:       v1.7.0
  Build:         72ff21ed9e26cb969052c753633049ba8a87ecf9
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

Kubernetes version:

Client Version: v1.27.2
Kustomize Version: v5.0.1
Server Version: v1.26.6

Environment:

• Cloud provider or hardware configuration: Azure, AKS

• OS : Alpine Linux v3.17

• Kernel : 5.15.0-1042-azure

• Install tools: -

• Basic cluster related info: See above

• How was the ingress-nginx-controller installed: -

• Current State of the controller: -

• Current state of ingress object, if applicable: -

• Others: -

How to reproduce this issue:
(Re-)Start Ingress-Nginx pods and wait until the OCSP response is expired.

Anything else we need to know:
Certificate provider: QuoVadis

It seems like the OSCP response is refreshed some time after the expiry (like a day after the expiry). As we just detected this issue I don't have an exact time so far.

[VWDude]

Update:
It seems like the update is fetched a day and some minutes later than expected:

So instead of the expected Nov 8. 5:14:33 GMT,
the answer is fetched on Nov 9. 5:17:00 GMT...

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[VWDude]

Any updates here?

[MrWusa]

Would be interesting to know since we are facing the same problem since we use this feature. We are restarting our ingress very regularly because of this which is kind of annoying...

[dkvaltech]

The issue still persist, we have to manually restart the ingix-ingress controller every couple of hours just to refresh the response. An update on the problem would be much appreaciated!

[longwuyuan]

Do you have any letsencrypt certs ? If yes does the problem occur with the Letsencrypt certs as well ?

[VWDude]

At least in our case we're using QuoVadis Certificates

[VWDude]

Update: It seems like the update is fetched a day and some minutes later than expected:

So instead of the expected Nov 8. 5:14:33 GMT, the answer is fetched on Nov 9. 5:17:00 GMT...

After some search in the code, at least from my point of view, it looks like the update of new ocsp responses is hardcoded to 3 days, which fits my observation from above:

ingress-nginx/rootfs/etc/nginx/lua/certificate.lua

Line 182 in a9c9a9d

local expiry = 3600 * 24 * 3

[longwuyuan]

/triage accepted

@tao12345666333 @rikatz @Gacko @strongjz please comment as it seems that if this is limited to just changing the time period, then it will not be a complicated change.

[VWDude]

/triage accepted

@tao12345666333 @rikatz @Gacko @strongjz please comment as it seems that if this is limited to just changing the time period, then it will not be a complicated change.

It should be the other way (the expiry should be shorter):
For now the ingress is caching the ocsp response for 3 days and after that time it will fetch a new one, regardless if the response itself is already expired or not.
This may be suitable for letsencrypt certificates because they maybe have longer ocsp expiries.
It would be best, if the expiry is calculated dynamically by reading when the ocsp will actually expire and substract a grace time from that (like 5 minutes), so the ocsp response will not have a downtime, while a new one is fetched.

[tao12345666333]

Sorry for the long delay. Let me take a look this week

/assign

[dkvaltech]

Sorry for the long delay. Let me take a look this week

/assign

@tao12345666333 could you verify the issue, do you have some insights for us?