testMondogDsbMinikube.sh asserting on VerifyServiceInstancesCountScenario

[mrmccarthy]

I deployed the Mongo DSB via the deployer:

go run deployer.go deploy-mongodsb -local-cluster-ip=$(minikube ip) -namespace=testing
and then tripped over an assert when trying to run the minikube script:

$ ./testMongoDsbMinikube.sh
Root Url - http://192.168.99.100:32708/dsb
Executing Test Case "Create DSB Instance"
Initializing Test Scenario "Verify Service Instances Count"
Executing Test Scenario "Verify Service Instances Count"
Invoking http GET on http://192.168.99.100:32708/dsb/service_instances
Test Scenario "Verify Service Instances Count" PASSED!
Initializing Test Scenario "Create DSB Instance"
Executing Test Scenario "Create DSB Instance"
Invoking http POST on http://192.168.99.100:32708/dsb/service_instances with body:
{
  "instanceId" : "6b6dcaad-96ed-4167-9e00-8247164a5e7f",
  "instanceSettings" : {
    "plan" : "standard"
  }
}
Test Scenario "Create DSB Instance" PASSED!
Initializing Test Scenario "Verify Service Instances Count"
Executing Test Scenario "Verify Service Instances Count"
Invoking http GET on http://192.168.99.100:32708/dsb/service_instances
Test Scenario "Verify Service Instances Count" FAILED!
Test Case "Create DSB Instance" FAILED!
Exception in thread "main" java.lang.AssertionError: expected:<1> but was:<0>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:743)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:555)
	at org.junit.Assert.assertEquals(Assert.java:542)
	at com.emc.ocopea.scenarios.mongodsb.VerifyServiceInstancesCountScenario.executeScenario(VerifyServiceInstancesCountScenario.java:27)
	at com.emc.ocopea.scenarios.ScenarioRunner.executeScenario(ScenarioRunner.java:112)
	at com.emc.ocopea.scenarios.ScenarioRunner.lambda$run$0(ScenarioRunner.java:92)
	at java.util.ArrayList.forEach(ArrayList.java:1249)
	at com.emc.ocopea.scenarios.ScenarioRunner.run(ScenarioRunner.java:90)
	at com.emc.ocopea.mongodsb.MongoDsbTesterMain.createDsbInstanceTest(MongoDsbTesterMain.java:42)
	at com.emc.ocopea.mongodsb.MongoDsbTesterMain.main(MongoDsbTesterMain.java:28)

[mrmccarthy]

Here are the logs from the mongo-k8s-dsb at the time of the assert:

 url https://10.0.0.1:443
namespace:testing
2017/10/23 20:11:02 https://10.0.0.1:443
2017/10/23 20:11:02 here1
2017/10/23 20:11:02 here2
2017/10/23 20:11:02 eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJ0ZXN0aW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tc2RjeDciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImZiODQzNDRiLWI4MmQtMTFlNy1iYzEwLTA4MDAyN2I5ODA4YiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDp0ZXN0aW5nOmRlZmF1bHQifQ.VEk-NjvP_4Pv0amHv8u09irLNRFISacBlxOx5JJOILxGyCMptHEUPZ1sX8Fyhfz2b_-fE0M_9s9crJHxZhaqbzJC6mV5b9WzLH24mQUk7udY0AWTU3a2x5BeL_wgs8w5h0czEtCujucGOVy2OqiFSwLtlcgnyUsp283rm_IYav3ET1T5G8uQdx1Kp_lG627Lq9EIKR6M5mW_oRfta39hnoFNlxJeYLnyLy9KAM2qSqiG8EeYMPsK9ZL7hX6rYQjDYqicUHg1v_tOPESweBrlEABe0RkILfu9ZInjDwOEB7vJnLE6o4T_LjxgVJW4hCvLOc2eaZ2NPb8-vrYk42aR-Q
2017/10/23 20:11:02 GET on  returned 200
2017/10/23 20:11:02 Serving k8s mongodsb at http://[::]:8080
2017/10/23 20:11:58 Creating mongo service with id c0cf4898-864b-4fa7-bc15-ec0b653cacf0
2017/10/23 20:11:58 POST on namespaces/testing/replicationcontrollers returned 201
2017/10/23 20:11:58 POST on namespaces/testing/services returned 201 

[mrmccarthy]

Note: a new pod was created in the testing namespace and was in the ContainerCreating state at the time that the logs were created.

Not sure if the code is expecting the service/pod to be in the running state and is failing if it is still starting up.

[shpandrak]

This probably happen when you don't have the mongo image in the docker cache...
We should make sure the test is consistent even in that case.
Thanks for the investigation @mrmccarthy

[mrmccarthy]

I'm not sure - running this again, I've seen the mongo service get up and running within seconds (as seen when looking at the k8s dashboard).

Here are the docker images:

$ docker images
REPOSITORY                                             TAG                 IMAGE ID            CREATED             SIZE
ocopea/mongo-k8s-dsb                                   latest              6c0296906c6a        5 days ago          861.3 MB
ocopea/go-k8s-psb                                      latest              35b14603d765        5 days ago          706.5 MB
mongo                                                  latest              42e262dc0845        2 weeks ago         360.8 MB
gcr.io/google_containers/k8s-dns-sidecar-amd64         1.14.4              38bac66034a6        4 months ago        41.81 MB
gcr.io/google_containers/k8s-dns-kube-dns-amd64        1.14.4              a8e00546bcf3        4 months ago        49.38 MB
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64   1.14.4              f7f45b9cb733        4 months ago        41.41 MB
gcr.io/google-containers/kube-addon-manager            v6.4-beta.2         0a951668696f        4 months ago        79.23 MB
gcr.io/google_containers/kubernetes-dashboard-amd64    v1.6.1              71dfe833ce74        5 months ago        134.4 MB
gcr.io/google_containers/pause-amd64                   3.0                 99e59f495ffa        17 months ago       746.9 kB
tutum/hello-world                                      latest              31e17b0746e4        22 months ago       17.79 MB

and the environment

$ printenv | grep DOCKER
DOCKER_HOST=tcp://192.168.99.100:2376
DOCKER_API_VERSION=1.23
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/home/mmccarth/.minikube/certs

$ minikube docker-env
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="/home/mmccarth/.minikube/certs"
export DOCKER_API_VERSION="1.23"

I had run the eval $(minikube docker-env) prior to running the test.