Seeing persistent/repeated 'persistentvolumeclaim "jenkins-home" not found' errors after OSIO env reset

[ldimaggi]

For a user (my user account) provisioned on starter-us-east-2 - after an OSIO env reset, jenkins pod fails to start. Event log includes:

4:48:43 PM | jenkins-1-zvlxn | Pod | Warning | Failed Scheduling | 
persistentvolumeclaim "jenkins-home" not found 152 times in the last 49 minutes
4:03:43 PM | jenkins-1-zvlxn | Pod | Warning | Failed Scheduling | 
persistentvolumeclaim "jenkins-home" is being deleted 5 times in the last 49 minutes
4:03:41 PM | content-repository-1-sf2cr | Pod | Normal | Started | 
Started container

This issue appeared today after the resolution of: #3934

[hrishin]

@ldimaggi we are trying to understand the root cause of this issue. @aslakknutsen would you like shed some lights here?

[piyush-garg]

@ldimaggi Just to correct you, build team is facing this issue from past 5-6 days and Service Delivery was also informed about that. I discussed this issue with @mmclanerh

This issue is not happening after or because of #3934 This is a different issue happening before that fix also.

The fix we just provided in Jenkins version 4.0.97 is related to #3956 and is affecting the #3934 as Jenkins will come up fast now and 503 issue will not block the user as you said build is taking 15-20 minutes in #3934 (comment). We are still working on #3934

[xyntrix]

i would suggest looking at the pvc and clearing off any cruft / old files that arent relevant. i wonder if the amount of files present have ballooned due to stale/failed workspace clean ups?

[aslakknutsen]

Not investigated this specifically, but it looks very similar to another issue from Friday.

Essentially this can happen due to a Reset; Clean & Apply.

Clean return 'deleted ok', but when Apply happens multiple seconds later, the PVC is still not deleted from Openshift, so Apply end up Updating the PVC instead of recreating it. Openshift then later comes around and Deletes the PVC.

A second Reset fixes it after the PVC has been deleted.

[ldimaggi]

Still seeing this:
persistentvolumeclaim "jenkins-home" not found52 times in the last 16 minutes

It looks like Aslak is correct - a 2nd reset is needed.

[ppitonak]

@ldimaggi is this still happening?

[ldimaggi]

I think we can close this one - it's been almost 2 months and I do not think the issue is happening now.