Requeued events are handled even after deletion of entity

[anekdoti]

Describe the bug
If a reconciliation event is requeued with an exponential back-off, it is possible that it is handled even after its entity got deleted in the meantime.

Expected behavior
Reconciliation should not be triggered after an entity got deleted. Or should the existence / deletion timestamp of the entity be checked in the reconciliation handler? If yes, it would be good to document this.

[anekdoti]

From my understanding of https://github.com/buehler/dotnet-operator-sdk/blob/master/src/KubeOps/Operator/Controller/ManagedResourceController%7BTEntity%7D.cs, requeued events for entities that do not exist anymore should be filtered out. So I just added a check for the DeletionTimestamp in the reconciliation handler. However, this does not seem to suffice.

[erin-allison]

The problem here is caused by the fact that while a manually re-queued event will update the resource's data when it processes...

dotnet-operator-sdk/src/KubeOps/Operator/Controller/ManagedResourceController{TEntity}.cs

Lines 80 to 95 in 1e7b53d

	private IObservable<Unit> RequeuedEvents => _requeuedEvents
	.Do(_ => _metrics.RequeuedEvents.Inc())
	.Select(
	data => Observable.Return(data).Delay(data.Delay))
	.Switch()
	.Select(
	data =>
	Observable.FromAsync(
	async () =>
	{
	var queuedEvent = await UpdateResourceData(data.Resource);
	return data.ResourceEvent.HasValue && queuedEvent != null
	? queuedEvent with { ResourceEvent = data.ResourceEvent.Value }
	: queuedEvent;
	}))

... an event which is retried because it threw an exception does not have the same logic.

[github-actions]

🎉 This issue has been resolved in version 6.4.0 🎉

The release is available on:

• GitHub release
• v6.4.0
• v6.4.0

Your semantic-release bot 📦🚀