Dashboard deployment does not work with customized namespace

[karlism]

Environment:

• Cloud provider or hardware configuration:
  ESXi VMs

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
  Linux 4.18.0-147.8.1.el8_1.x86_64 x86_64
  NAME="CentOS Linux"
  VERSION="8 (Core)"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="8"
  PLATFORM_ID="platform:el8"
  PRETTY_NAME="CentOS Linux 8 (Core)"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:8"
  HOME_URL="https://www.centos.org/"
  BUG_REPORT_URL="https://bugs.centos.org/"
  CENTOS_MANTISBT_PROJECT="CentOS-8"
  CENTOS_MANTISBT_PROJECT_VERSION="8"
  REDHAT_SUPPORT_PRODUCT="centos"
  REDHAT_SUPPORT_PRODUCT_VERSION="8"

• Version of Ansible (ansible --version):
  ansible 2.9.6
  config file = /home/username/.ansible.cfg
  configured module search path = ['/home/username/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.7.4 (default, Oct 11 2019, 11:15:58) [Clang 8.0.1 (tags/RELEASE_801/final)]

• Version of Python (python --version):
  Python 3.7.4

Kubespray version (commit) (git rev-parse --short HEAD):
01dbc90

Network plugin used:
calico

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):

https://gist.github.com/karlism/653a887fc29f87ea15142e6d8e793e1a

Command used to invoke ansible:
ansible-playbook cluster.yml -i inventory/devk8s/inventory.ini -b

Output of ansible run:

TASK [kubernetes-apps/ansible : Kubernetes Apps | Lay down dashboard template] *****************************
changed: [devkmaster1a] => (item={'file': 'dashboard.yml', 'type': 'deploy', 'name': 'kubernetes-dashboard'})
Thursday 07 May 2020  15:59:28 +0200 (0:00:01.202)       0:18:36.107 ********** 

TASK [kubernetes-apps/ansible : Kubernetes Apps | Start dashboard] *****************************************
failed: [devkmaster1a] (item={'diff': [], 'dest': '/etc/kubernetes/dashboard.yml', 'src': '/home/username/.ansible/tmp/ansible-tmp-1588859967.0010653-176497413600000/source', 'md5sum': '77a49babf65104638a3cd6fa322ee464', 'checksum': '7cf6a19e922a52ba0cec73b4467399424c75492e', 'changed': True, 'uid': 0, 'gid': 0, 'owner': 'root', 'group': 'root', 'mode': '0644', 'state': 'file', 'secontext': 'system_u:object_r:etc_t:s0', 'size': 7674, 'invocation': {'module_args': {'src': '/home/username/.ansible/tmp/ansible-tmp-1588859967.0010653-176497413600000/source', 'dest': '/etc/kubernetes/dashboard.yml', 'mode': None, 'follow': False, '_original_basename': 'dashboard.yml.j2', 'checksum': '7cf6a19e922a52ba0cec73b4467399424c75492e', 'backup': False, 'force': True, 'content': None, 'validate': None, 'directory_mode': None, 'remote_src': None, 'local_follow': None, 'owner': None, 'group': None, 'seuser': None, 'serole': None, 'selevel': None, 'setype': None, 'attributes': None, 'regexp': None, 'delimiter': None, 'unsafe_writes': None}}, 'failed': False, 'item': {'file': 'dashboard.yml', 'type': 'deploy', 'name': 'kubernetes-dashboard'}, 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": true, "checksum": "7cf6a19e922a52ba0cec73b4467399424c75492e", "dest": "/etc/kubernetes/dashboard.yml", "diff": [], "failed": false, "gid": 0, "group": "root", "invocation": {"module_args": {"_original_basename": "dashboard.yml.j2", "attributes": null, "backup": false, "checksum": "7cf6a19e922a52ba0cec73b4467399424c75492e", "content": null, "delimiter": null, "dest": "/etc/kubernetes/dashboard.yml", "directory_mode": null, "follow": false, "force": true, "group": null, "local_follow": null, "mode": null, "owner": null, "regexp": null, "remote_src": null, "selevel": null, "serole": null, "setype": null, "seuser": null, "src": "/home/username/.ansible/tmp/ansible-tmp-1588859967.0010653-176497413600000/source", "unsafe_writes": null, "validate": null}}, "item": {"file": "dashboard.yml", "name": "kubernetes-dashboard", "type": "deploy"}, "md5sum": "77a49babf65104638a3cd6fa322ee464", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 7674, "src": "/home/username/.ansible/tmp/ansible-tmp-1588859967.0010653-176497413600000/source", "state": "file", "uid": 0}, "msg": "error running kubectl (/usr/local/bin/kubectl --namespace=kube-dashboard apply --force --filename=/etc/kubernetes/dashboard.yml) command (rc=1), out='clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created\nclusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created\n', err='Error from server (NotFound): error when creating \"/etc/kubernetes/dashboard.yml\": namespaces \"kube-dashboard\" not found\nError from server (NotFound): error when creating \"/etc/kubernetes/dashboard.yml\": namespaces \"kube-dashboard\" not found\nError from server (NotFound): error when creating \"/etc/kubernetes/dashboard.yml\": namespaces \"kube-dashboard\" not found\nError from server (NotFound): error when creating \"/etc/kubernetes/dashboard.yml\": namespaces \"kube-dashboard\" not found\nError from server (NotFound): error when creating \"/etc/kubernetes/dashboard.yml\": namespaces \"kube-dashboard\" not found\nError from server (NotFound): error when creating \"/etc/kubernetes/dashboard.yml\": namespaces \"kube-dashboard\" not found\nError from server (NotFound): error when creating \"/etc/kubernetes/dashboard.yml\": namespaces \"kube-dashboard\" not found\nError from server (NotFound): error when creating \"/etc/kubernetes/dashboard.yml\": namespaces \"kube-dashboard\" not found\nError from server (NotFound): error when creating \"/etc/kubernetes/dashboard.yml\": namespaces \"kube-dashboard\" not found\nError from server (NotFound): error when creating \"/etc/kubernetes/dashboard.yml\": namespaces \"kube-dashboard\" not found\nError from server (NotFound): error when creating \"/etc/kubernetes/dashboard.yml\": namespaces \"kube-dashboard\" not found\n'"}

error when creating "/etc/kubernetes/dashboard.yml": namespaces "kube-dashboard" not found

Anything else do we need to know:

$ cat inventory/devk8s/group_vars/k8s-cluster/addons.yml | grep dashboard                                                                                   
# Kubernetes dashboard
dashboard_enabled: true
dashboard_namespace: "kube-dashboard"

Removing dashboard_namespace variable from inventory/devk8s/group_vars/k8s-cluster/addons.yml solves the issue but deploys dashboard in kube-system namespace.

[HugoRh]

I had same issue.
The namespace is just not created before applying the manifest, so beside your solution, you could have just created the namespace manually on the cluster:

kubectl create ns kube-dashboard

A step should be added to create the dashboard namespace if it's something else than kube-system

[floryut]

Thank you for reporting this, are you guys ok to make a pr for that ? :)