Lambda Function goes into terminal condition if image doesn't exist at time of update

[jessebye]

Describe the bug
If a Function is created/updated and the imageURI doesn't exist, it goes into terminal condition and won't ever retry reconciliation. We use GitOps workflows to update our k8s manifests, so when a new version is pushed to a git repo, the manifest is synced with the new ECR tag before the ECR build/publish completes. This means Lambda functions will fail to update until we manually restart the lambda-controller every time.

Steps to reproduce

Create a Function using imageURI and use a image tag that doesn't exist yet. Observe the error (have to use debug log level). Push the tag up to ECR - it still doesn't create/update the function. If the controller is manually restarted, then the function is created/updated successfully.

Expected outcome

The lambda-controller should retry periodically (every minute?) until the image is pushed to ECR.

Environment

• Kubernetes version 1.25
• Using EKS (yes/no), if so version? Yes, 1.25
• AWS service targeted (S3, RDS, etc.) Lambda

[a-hilaly]

If the lambda API returns a "validation" error, this is an expected behaviour. However we can always tune the controller to reconcile (do not set a Terminal condition) for some edge cases like this one...
/cc @aws-controllers-k8s/lambda-maintainer