[prod-preview] Cluster specific URLs in configurations

[xyntrix]

In order to facilitate seamless planned and unplanned cluster fail-over scenarios, we need to make use of non-cluster specific url references (non-generated hostnames) for public routes and cascade these references into configs (env vars, secrets, configmaps..). We need to have the ability to quickly update DNS during a fail-over without additional host name changes for routes.

An example of how this should work is the foo service url: foo.prod-preview.openshift.io. During a planned/unplanned failover we're able to spin up services at a new cluster, update DNS, and this service continues working. Any apps making use of this URL will also continue working, with no additional configuration changes.

This week we attempted fail-over of the prod-preview cluster (rh-idev to dsaas-stg) and ran into a blocker. We found several cases where routes are hard coded for for the rh-idev cluster and in some cases we have env vars/configurations using those cluster-specific hostnames. Below is a list of what we've been able to identify as using rh-idev host names.

In order to update routes, configmap, or secrets here we'll need housekeeping issues submitted. If we need to create new hostnames and related DNS we're happy to work with folks to get this sorted. Some of these configuration patterns may also exist for production and not just prod-preview. We'll want to get those updated as well. We have need to get off of the current rh-idev cluster sooner rather than later. By March 20 we'll need to have configurations updated to make the next fail-over a transparent success.

Additionally, some of these resources may no longer be in use. If that is the case, let's get that cleaned up by also submitting a housekeeping issue. If there are questions please reach out on the devtools-servicedel chat channel and we can help sort out how to move forward together.

[qodfathr]

Is this a type/bug or type/fundamental? It cannot be both. It looks like type/bug is more appropriate. (I think one could even argue that is is not even a bug, per se, but rather a feature request, but I'm happy with bug.)

[ldimaggi]

Related to - #2229

In this^ issue it looks like the cluster name may be hardcoded.

[xyntrix]

A mis-configuration / coordinated re-configuration perhaps? There's no customer direct value here, but there's sustainability value (so, not really a feature..). We're not supposed to be using generated hostnames by design, so in that loose regard, let's go with a bug here. Happy to re-classify as needed of course.

@ldimaggi I believe you are correct. The scope of this issue is prod-preview, but most certainly this applies to production as well. The complicated/coordinated piece of this is ensuring that any apps making use of a generated hostname may break -- ideally, those type of settings are also coming from a configmap or secret, vs buried in code somewhere.

Thanks!

[sivaavkd]

Added severity. Please modify as needed