che.openshift.io e2e / openshift.io registraion flow is broken - oso namespaces are not provisioned after registration of a new user

[ibuziuk]

Steps to reproduce:

1. Navigate to che.openshift.io and be redirected to https://developers.redhat.com
2. Register a new user
3. get a confirmation email
4. navigate to the confirmation link
5. ERROR layout of account confirmation looks broken:
   
6. press "Activate Account" button
7. receive "Your OpenShift.io environment is being created" email
8. receive "Welcome to OpenShift.io" email
9. ERROR: "Get Started" link points to OSIO Getting started guide [1], previously when registered via che.openshift.io registration flow it pointed to the desired factory url / UD
   
10. Navigate to che.openshift.io
    11.SUCCESS: UD is accessible
    
11. Create and start a workspace
12. ERROR: workspace failed to start
13. Navigate to oso cluster
    15.ERROR: no namespaces have been provisioned
    

Some details:
tested against brand new user / identity_id 19e7ff47-6d9d-4531-a787-9beccae11add
logs from f8-tenant:

{"GET":"/api/tenant","action":"show","ctrl":"TenantController","from":"10.131.10.1","identity_id":"19e7ff47-6d9d-4531-a787-9beccae11add","level":"info","msg":"request started","pkg":"github.com/fabric8-services/fabric8-tenant/vendor/log.LogRequest.func1","req_id":"8Gptm3FRo5-4117608","time":"2019-03-06 21:06:40"}
--
  | time="2019-03-06T21:06:40Z" level=info msg=started GET="https://auth.openshift.io/api/users/19e7ff47-6d9d-4531-a787-9beccae11add " id=xmOuEWgZ req_id=g3I3dKpB
  | time="2019-03-06T21:06:40Z" level=info msg=completed fields.time=14.099304ms id=xmOuEWgZ req_id=g3I3dKpB status=200
  | time="2019-03-06T21:06:40Z" level=info msg=started GET="https://auth.openshift.io/api/token?for=https%3A%2F%2Fapi.starter-us-east-2a.openshift.com%2F&force_pull=false " id=fq/PS+Jy req_id=g3I3dKpB
  | time="2019-03-06T21:06:40Z" level=info msg=completed fields.time=27.448415ms id=fq/PS+Jy req_id=g3I3dKpB status=200
  | {"err":"tenant with id '19e7ff47-6d9d-4531-a787-9beccae11add' not found","file":"/tmp/go/src/github.com/fabric8-services/fabric8-tenant/controller/tenant.go","func":"github.com/fabric8-services/fabric8-tenant/controller.(*TenantController).Show","identity_id":"19e7ff47-6d9d-4531-a787-9beccae11add","level":"error","line":235,"msg":"retrieval of tenant entity from DB failed","pid":1,"pkg":"github.com/fabric8-services/fabric8-tenant/controller","req_headers":{"Accept-Encoding":["gzip"],"Authorization":"*****","User-Agent":["Go-http-client/1.1"],"X-Request-Id":["8Gptm3FRo5-4117608"]},"req_id":"8Gptm3FRo5-4117608","tenantID":"19e7ff47-6d9d-4531-a787-9beccae11add","time":"2019-03-06 21:06:40"}
  | {"err":"tenants with id '19e7ff47-6d9d-4531-a787-9beccae11add' not found","error_message":"tenants with id '19e7ff47-6d9d-4531-a787-9beccae11add' not found","file":"/tmp/go/src/github.com/fabric8-services/fabric8-tenant/jsonapi/jsonapi_utility.go","func":"github.com/fabric8-services/fabric8-tenant/jsonapi.ErrorToJSONAPIError","identity_id":"19e7ff47-6d9d-4531-a787-9beccae11add","level":"error","line":40,"msg":"an error occurred in our api","pid":1,"pkg":"github.com/fabric8-services/fabric8-tenant/jsonapi","req_headers":{"Accept-Encoding":["gzip"],"Authorization":"*****","User-Agent":["Go-http-client/1.1"],"X-Request-Id":["8Gptm3FRo5-4117608"]},"req_id":"8Gptm3FRo5-4117608","time":"2019-03-06 21:06:40"}
  | {"action":"show","bytes":195,"ctrl":"TenantController","duration":44.446965999999996,"duration_unit":"ms","identity_id":"19e7ff47-6d9d-4531-a787-9beccae11add","level":"info","msg":"completed","pkg":"github.com/fabric8-services/fabric8-tenant/vendor/log.LogRequest.func1","req_id":"8Gptm3FRo5-4117608","status":404,"time":"2019-03-06 21:06:40"}
  | {"POST":"/api/tenant","action":"setup","ctrl":"TenantController","from":"10.129.6.1","identity_id":"19e7ff47-6d9d-4531-a787-9beccae11add","level":"info","msg":"request started","pkg":"github.com/fabric8-services/fabric8-tenant/vendor/log.LogRequest.func1","req_id":"WHNeldrX2e-4333533","time":"2019-03-06 21:06:40"}
  | time="2019-03-06T21:06:40Z" level=info msg=started GET="https://auth.openshift.io/api/users/19e7ff47-6d9d-4531-a787-9beccae11add " id=IP1cwNN1 req_id=5+7YhDmL
  | time="2019-03-06T21:06:40Z" level=info msg=completed fields.time=25.91257ms id=IP1cwNN1 req_id=5+7YhDmL status=200
  | time="2019-03-06T21:06:40Z" level=info msg=started GET="https://auth.openshift.io/api/token?for=https%3A%2F%2Fapi.starter-us-east-2a.openshift.com%2F&force_pull=false " id=dYUQcKqD req_id=5+7YhDmL
  | time="2019-03-06T21:06:40Z" level=info msg=completed fields.time=66.090313ms id=dYUQcKqD req_id=5+7YhDmL status=200
  | {"action":"setup","bytes":0,"ctrl":"TenantController","duration":98.932255,"duration_unit":"ms","identity_id":"19e7ff47-6d9d-4531-a787-9beccae11add","level":"info","msg":"completed","pkg":"github.com/fabric8-services/fabric8-tenant/vendor/log.LogRequest.func1","req_id":"WHNeldrX2e-4333533","status":409,"time":"2019-03-06 21:06:40"}
  | {"GET":"/api/tenant","action":"show","ctrl":"TenantController","from":"10.130.8.1","identity_id":"19e7ff47-6d9d-4531-a787-9beccae11add","level":"info","msg":"request started","pkg":"github.com/fabric8-services/fabric8-tenant/vendor/log.LogRequest.func1","req_id":"CvSqAYksho-4186830","time":"2019-03-06 21:06:40"}
  | time="2019-03-06T21:06:40Z" level=info msg=started GET="https://auth.openshift.io/api/users/19e7ff47-6d9d-4531-a787-9beccae11add " id=UPrdbuIr req_id=OuSwhk0Y
  | time="2019-03-06T21:06:40Z" level=info msg=completed fields.time=30.312596ms id=UPrdbuIr req_id=OuSwhk0Y status=200
  | time="2019-03-06T21:06:40Z" level=info msg=started GET="https://auth.openshift.io/api/token?for=https%3A%2F%2Fapi.starter-us-east-2a.openshift.com%2F&force_pull=false " id=r9FtTvGB req_id=OuSwhk0Y
  | time="2019-03-06T21:06:41Z" level=info msg=completed fields.time=20.585252ms id=r9FtTvGB req_id=OuSwhk0Y status=200
  | {"action":"show","bytes":3031,"ctrl":"TenantController","duration":64.68030700000001,"duration_unit":"ms","identity_id":"19e7ff47-6d9d-4531-a787-9beccae11add","level":"info","msg":"completed","pkg":"github.com/fabric8-services/fabric8-tenant/vendor/log.LogRequest.func1","req_id":"CvSqAYksho-4186830","status":200,"time":"2019-03-06 21:06:41"}
  | {"POST":"/api/tenant","action":"setup","ctrl":"TenantController","from":"10.131.10.1","identity_id":"19e7ff47-6d9d-4531-a787-9beccae11add","level":"info","msg":"request started","pkg":"github.com/fabric8-services/fabric8-tenant/vendor/log.LogRequest.func1","req_id":"8Gptm3FRo5-4117609","time":"2019-03-06 21:06:41"}
  | time="2019-03-06T21:06:41Z" level=info msg=started GET="https://auth.openshift.io/api/users/19e7ff47-6d9d-4531-a787-9beccae11add " id=hKz7h93W req_id=3J8E7NKe
  | time="2019-03-06T21:06:41Z" level=info msg=completed fields.time=33.135628ms id=hKz7h93W req_id=3J8E7NKe status=200
  | time="2019-03-06T21:06:41Z" level=info msg=started GET="https://auth.openshift.io/api/token?for=https%3A%2F%2Fapi.starter-us-east-2a.openshift.com%2F&force_pull=false " id=fexEJKzl req_id=3J8E7NKe
  | time="2019-03-06T21:06:41Z" level=info msg=completed fields.time=22.123494ms id=fexEJKzl req_id=3J8E7NKe status=200
  | {"action":"setup","bytes":0,"ctrl":"TenantController","duration":66.022955,"duration_unit":"ms","identity_id":"19e7ff47-6d9d-4531-a787-9beccae11add","level":"info","msg":"completed","pkg":"github.com/fabric8-services/fabric8-tenant/vendor/log.LogRequest.func1","req_id":"8Gptm3FRo5-4117609","status":409,"time":"2019-03-06 21:06:41"}
  | {"GET":"/api/status","action":"show","ctrl":"StatusController","from":"10.130.2.1","level":"info","msg":"request started","pkg":"github.com/fabric8-services/fabric8-tenant/vendor/log.LogRequest.func1","req_id":"Ter1Ngn7Q9-3237","time":"2019-03-06 21:06:43"}
  | {"action":"show","bytes":124,"ctrl":"StatusController","duration":1.276317,"duration_unit":"ms","level":"info","msg":"completed","pkg":"github.com/fabric8-services/fabric8-tenant/vendor/log.LogRequest.func1","req_id":"Ter1Ngn7Q9-3237","status":200,"time":"2019-03-06 21:06:43"}

Another thing is that email templates still have openshift.io branding. IMO it is hight time to change it to che.openshift.io - #3814 (comment)

[1] https://docs.openshift.io/getting-started-guide.html

[alexeykazakov]

@MatousJobanek can you please also take a look at this one?

[ibuziuk]

Some comment from @MatousJobanek:

Ok, I checked the namespaces created in 2a cluster and I see:

svetlanabuziuk               Active
svetlanabuziuk-che        Active
svetlanabuziuk-jenkins  Active
svetlanabuziuk-run        Active
svetlanabuziuk-stage     Active

I gues that these are the expected namespaces
so if @ibuziuk cannot see them in the console, then it means that the rolebindings are not properly created
which is exactly the issue that I posted -  https://gitlab.cee.redhat.com/dtsd/housekeeping/issues/2578
but that would explain state where only run, stage and user's namespace would be created and che + jenkins would be missing, not where all of them are missing

Today I see the following namespaces:

@MatousJobanek @alexeykazakov how is it planed to adress the HK issue, which is probably the root cause of the problem. Currently it is a showstopper for e2e registration flow on che.openshift.io

[1] https://gitlab.cee.redhat.com/dtsd/housekeeping/issues/2578

[ibuziuk]

@rhopp @ScrewTSW @Katka92 according to the dedicated mailing thread the namespace provisioning issue should be fixed by now. Could you please verify?

[Katka92]

@ibuziuk I'll take a look on that today. Thank you for reminding.

[Katka92]

@ibuziuk I've followed your reproduction steps and I created user osio-ci-testcreation. The user was provisioned on 1a cluster and is able to create and start a workspace on a first attempt. I can see all 5 namespaces on OSO cluster. If you need me to verify something else about this issue, please let me know.

[ibuziuk]

great, I believe we can close this issue now in this case