when using scheduler and serviceAffinity, unexpected node placements occur

[thoraxe]

Given the following scheduler configuration:

{
  "predicates" : [
    {"name" : "PodFitsResources"},
    {"name" : "PodFitsPorts"},
    {"name" : "NoDiskConflict"},
    {"name" : "Region", "argument" : {"serviceAffinity" : { "label" : "region"}}}
  ],"priorities" : [
    {"name" : "LeastRequestedPriority", "weight" : 1},
    {"name" : "ServiceSpreadingPriority", "weight" : 1},
    {"name" : "Zone", "weight" : 1, "argument" : {"serviceAntiAffinity" : { "label" : "zone" }}}
  ]
}

When creating a pod with a NodeSelector requesting a specific region, pods are placed in the wrong region.

They also fail to start, and I'm not sure if that's related or unrelated.

@abhgupta is aware, this issue is for him.

[sosiouxme]

The scheduler seems to have a bit of split-brain. It will ignore the region predicate when scheduling and put the pod on whatever node; but that pod will immediately be rejected at the node because the predicate is not matched. I end up with a number of failed placements until it actually lands on a satisfactory node.

I deployed the router and updated it to specify "nodeSelector": { "region": "infra"}. Nodes have region tags as below.

# osc get nodes
NAME           LABELS        STATUS
172.16.4.164   Schedulable   region=infra,zone=NA       Ready
172.16.4.167   Schedulable   region=primary,zone=east   Ready
172.16.4.168   Schedulable   region=primary,zone=west   Ready
# osc get pods
POD                       IP         CONTAINER(S)   IMAGE(S)                                                                                    HOST                        LABELS                                                                                  STATUS    CREATED
router-2-6t0yf                       router         docker-buildvm-rhose.usersys.redhat.com:5000/openshift3_beta/ose-haproxy-router:v0.4.3.2    172.16.4.167/               deployment=router-2,deploymentconfig=router,router=router                               Failed    14 seconds
router-2-bd8ro                       router         docker-buildvm-rhose.usersys.redhat.com:5000/openshift3_beta/ose-haproxy-router:v0.4.3.2    172.16.4.168/               deployment=router-2,deploymentconfig=router,router=router                               Failed    15 seconds
router-2-y9tpa            10.1.0.5   router         docker-buildvm-rhose.usersys.redhat.com:5000/openshift3_beta/ose-haproxy-router:v0.4.3.2    172.16.4.164/172.16.4.164   deployment=router-2,deploymentconfig=router,router=router                               Running   10 seconds

master logs:

Apr 15 10:34:47 172.16.4.164 openshift-master[10516]: E0415 10:34:47.243644   10516 endpoints_controller.go:93] Failed to find an IP for pod default/router-2-6t0yf
Apr 15 10:34:47 172.16.4.164 openshift-master[10516]: E0415 10:34:47.243680   10516 endpoints_controller.go:93] Failed to find an IP for pod default/router-2-bd8ro
Apr 15 10:34:47 172.16.4.164 openshift-master[10516]: I0415 10:34:47.424638   10516 factory.go:200] About to try and schedule pod router-2-y9tpa
Apr 15 10:34:47 172.16.4.164 openshift-master[10516]: I0415 10:34:47.424660   10516 scheduler.go:106] Attempting to schedule: &{{ } {router-2-y9tpa router-2- default /api/v1beta3/namespaces/default/pods/router-2
[...]
} {OPENSHIFT_MASTER https://172.16.4.164:8443}] {map[] map[]} [] 0xc20afdf5c0 <nil> <nil> /dev/termination-log false IfNotPresent {[] []}}] Always ClusterFirst map[region:infra]  false} {Pending []    []}}

example node logs:
Apr 15 10:34:46 172.16.4.167 openshift-node[5263]: I0415 10:34:46.831953    5263 config.go:244] Setting pods for source api : {[{{ } {router-2-6t0yf router-2- default /api/v1beta3/namespaces/default/pods/router-2-6t0yf 8ed515d5-e37c-11e4-85ea-fa163ec8dc60 19212 2015-04-15 10:34:43 -0400 EDT <nil> map[deployment:router-2 deploymentconfig:router router:router] map[deployment:router-2 deploymentConfig:router deploymentVersion:2]} {[] [{router docker-buildvm-rhose.usersys.redhat.com:5000/openshift3_beta/ose-haproxy-router:v0.4.3.2 [] []  [{ 80 80 TCP } { 443 443 TCP }] [{OPENSHIFT_CA_DATA -----BEGIN CERTIFICATE-----
[...]
} {OPENSHIFT_MASTER https://172.16.4.164:8443}] {map[] map[]} [] 0xc2084e8690 <nil> <nil> /dev/termination-log false IfNotPresent {[] []}}] Always ClusterFirst map[region:infra] 172.16.4.167 false} {Failed [] Pod cannot be started due to node selector mismatch   []}}] 0 api}

[abhgupta]

I ran my own tests on pods and replication controllers and this seems to be working fine, at least now. Also, I checked with Clayton and it seems that v1beta3 is the only version that supports NodeSelector. You can try the rc below to test out the scheduler.

{
   "kind":"ReplicationController",
   "apiVersion":"v1beta3",
   "metadata":{
      "name":"frontend",
      "labels":{
         "name":"frontend"
      }
   },
   "spec":{
      "replicas":2,
      "selector":{
         "name":"frontend"
      },
      "template":{
         "metadata":{
            "labels":{
               "name":"frontend"
            }
         },
         "spec":{
            "containers":[
               {
                  "name":"hello-openshift",
                  "image":"openshift/hello-openshift"
               }
            ],
            "nodeSelector" : {"zone" : "z2"}
         }
      }
   }
}

Here is the pod

{
   "kind":"Pod",
   "apiVersion":"v1beta3",
   "metadata":{
      "name":"hello-openshift",
      "labels":{
         "name":"frontend"
      }
   },
   "spec":{
      "containers":[
         {
            "name":"hello-openshift",
            "image":"openshift/hello-openshift"
         }
      ],
      "nodeSelector":{"zone" : "z1"}
   }
}

[sosiouxme]

@abhgupta have you tried the example of modifying router dc? At least in my example the json came out wtih "apiVersion": "v1beta1" so maybe that's the problem (I fed it back in with just the labels inserted). Although that doesn't really explain why (per my example above) the scheduler initially ignored the pod nodeSelector and then properly rejected pods according to the nodeSelector after they were placed.

We can try with more recent builds... if I'm not mistaken, we're moving to default v1beta3 API for v0.5 if not before?

[sosiouxme]

@abhgupta just tried the latest build, scheduling a pod as you said works great. The problem I'm having now is that if i just patch the podTemplate in a dc to add a nodeSelector, I still get the above behavior. I even tried changing the "version" in the podTemplate manifest to "v1beta3" from "v1beta2" and it was the same.

If you want to schedule a router to a specific region/zone, how do you do that?

[abhgupta]

@sosiouxme the scheduler ignored it because they weren't (somehow) present in the pod when it checked. However, the error was thrown by the kubelet when it made the exact same check. I am a bit lost here, but it's definitely an api version issue and the client api version (scheduler/master vs the kubelet) is perhaps the issue.

[sosiouxme]

@abhgupta my uninformed hypothesis was that maybe when it's coming from a DC, the scheduler examines it with v1beta2 protocol and doesn't see the nodeSelector; then when the pod is actually scheduled for a node that checks in, it's created with v1beta3 protocol and the nodeSelector appears. Mismatch between pre-create and post-create. But I dunno. Maybe @smarterclayton or @derekwaynecarr can shed some light?

Pretty much everything is going to be deployed via DC, not directly as RCs or pods, so... this kind of needs to work; we were hoping for beta3 :)

[abhgupta]

Apparently, NodeSelector doesn't work well for API versions prior to v1beta3. The router is created from the deployment config and currently we have v1beta1 being specified for the replication controller.

ControllerTemplate v1beta1.ReplicationControllerState

We need to enable v1beta2 for deployment related APIs and use v1beta3 for any K8s resources that they refer. The bulk of this work has already been done by @smarterclayton and I should be able to get to this (for deployment APIs) by tomorrow.

Refer #1694

[brenton]

@abhgupta, Thanks for the update. We'll wait on branch until tomorrow then.

[abhgupta]

@sosiouxme the pod is already created when it comes to the scheduler. The pod binding is also not a separate resource any more (that might have explained this). Either ways, we'll get v1beta2 enabled for deployments and hopefully by tomorrow this issue can be put to rest.

[smarterclayton]

v1beta2 is not going to be able to land for beta3.

----- Original Message -----

@sosiouxme the pod is already created when it comes to the scheduler. The pod
binding is also not a separate resource any more (that might have explained
this). Either ways, we'll get v1beta2 enabled for deployments and hopefully
by tomorrow this issue can be put to rest.

---

Reply to this email directly or view it on GitHub:
#1727 (comment)

[smarterclayton]

If you need this behavior it would be better to create an UPSTREAM patch and add the fields so they can round trip.

----- Original Message -----

v1beta2 is not going to be able to land for beta3.

----- Original Message -----

@sosiouxme the pod is already created when it comes to the scheduler. The
pod
binding is also not a separate resource any more (that might have explained
this). Either ways, we'll get v1beta2 enabled for deployments and hopefully
by tomorrow this issue can be put to rest.

---

Reply to this email directly or view it on GitHub:
#1727 (comment)

[abhgupta]

@smarterclayton @sosiouxme Magically, everything works perfectly with the latest Origin code. Not sure what fixed it, but now the NodeSelector is preserved from dc --> rc --> pod.

Same for HostNetwork attribute that @ramr needed.