Watch multiple resources: Requeue events for the primary resource

[hasbro17]

Background

A common pattern for operators is to have a sync handler that reconciles events for a primary resource which is usually the Custom Resource(CR) that defines the custom API for the application.
However there are other secondary resources such as:

• Resources that the operator creates e.g deployments, services
• Resources that the operator does not create but depends on e.g user provided secrets for TLS setup.

The operator needs to receive an event for the primary resource to reconcile the state anytime any of the secondary resources are changed.

The current SDK Watch API allows watching multiple resources that would send all events to the same handler but provides no way to define or requeue events for the primary resource.

Goal

The SDK should provide an API that allows the operator to watch a primary resource and receive requeued primary resource events upon changes to any secondary resources.

[hasbro17]

/cc @philips

[philips]

Yes, this seems correct to me. Anything to add @ecordell?

The other thing is how e build the dependency tree for these resources coming off of the CR:

1. Dependent resources could easily be tracked via owner ref
2. Soft dependent resources like user configuration via configmaps though would need a label convention or naming convention though.