Certificate ca.crt Is Not Generated

[Adhito]

Hi, i just tried provision the cluster however i ran into this issue.

myuser@vm-ubuntu:~/vagrant-env/kubernetes-the-hard-way$ vagrant up
Bringing machine 'lb' up with 'virtualbox' provider...
Bringing machine 'node0' up with 'virtualbox' provider...
Bringing machine 'node1' up with 'virtualbox' provider...
Bringing machine 'node2' up with 'virtualbox' provider...
==> lb: Cloning VM...
==> lb: Matching MAC address for NAT networking...
==> lb: Checking if box 'ubuntu/jammy64' version '20240821.0.0' is up to date...
==> lb: Setting the name of the VM: k8s-the-hard-way-lb
==> lb: Clearing any previously set network interfaces...
==> lb: Preparing network interfaces based on configuration...
    lb: Adapter 1: nat
    lb: Adapter 2: hostonly
==> lb: Forwarding ports...
    lb: 6443 (guest) => 6443 (host) (adapter 1)
    lb: 22 (guest) => 2222 (host) (adapter 1)
==> lb: Running 'pre-boot' VM customizations...
==> lb: Booting VM...
==> lb: Waiting for machine to boot. This may take a few minutes...
    lb: SSH address: 127.0.0.1:2222
    lb: SSH username: vagrant
    lb: SSH auth method: private key
    lb: Warning: Connection reset. Retrying...
    lb: Warning: Remote connection disconnect. Retrying...
    lb: 
    lb: Vagrant insecure key detected. Vagrant will automatically replace
    lb: this with a newly generated keypair for better security.
    lb: 
    lb: Inserting generated public key within guest...
    lb: Removing insecure key from the guest if it's present...
    lb: Key inserted! Disconnecting and reconnecting using new SSH key...
==> lb: Machine booted and ready!
==> lb: Checking for guest additions in VM...
    lb: The guest additions on this VM do not match the installed version of
    lb: VirtualBox! In most cases this is fine, but in rare cases it can
    lb: prevent things such as shared folders from working properly. If you see
    lb: shared folder errors, please make sure the guest additions within the
    lb: virtual machine match the version of VirtualBox you have installed on
    lb: your host and reload your VM.
    lb: 
    lb: Guest Additions Version: 6.0.0 r127566
    lb: VirtualBox Version: 7.0
==> lb: Setting hostname...
==> lb: Configuring and enabling network interfaces...
==> lb: Mounting shared folders...
    lb: /vagrant => /home/myuser/vagrant-env/kubernetes-the-hard-way
    lb: /downloads => /home/myuser/vagrant-env/kubernetes-the-hard-way/share/dl
==> lb: Running provisioner: shell...
    lb: Running: inline script
    lb: 
    lb: WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
    lb: 
    lb: Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]
    lb: Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease
    lb: Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
    lb: Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB]
    lb: Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1769 kB]
    lb: Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [14.1 MB]
    lb: Get:7 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [290 kB]
    lb: Get:8 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.2 kB]
    lb: Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2308 kB]
    lb: Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [398 kB]
    lb: Get:11 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [890 kB]
    lb: Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [175 kB]
    lb: Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.0 kB]
    lb: Get:14 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB]
    lb: Get:15 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7588 B]
    lb: Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B]
    lb: Get:17 http://archive.ubuntu.com/ubuntu jammy/universe Translation-en [5652 kB]
    lb: Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 c-n-f Metadata [286 kB]
    lb: Get:19 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [217 kB]
    lb: Get:20 http://archive.ubuntu.com/ubuntu jammy/multiverse Translation-en [112 kB]
    lb: Get:21 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 c-n-f Metadata [8372 B]
    lb: Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1988 kB]
    lb: Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB]
    lb: Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB]
    lb: Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1112 kB]
    lb: Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [260 kB]
    lb: Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.0 kB]
    lb: Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB]
    lb: Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB]
    lb: Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B]
    lb: Get:31 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [92.4 kB]
    lb: Get:32 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB]
    lb: Get:33 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B]
    lb: Get:34 http://archive.ubuntu.com/ubuntu jammy-backports/restricted amd64 c-n-f Metadata [116 B]
    lb: Get:35 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [31.4 kB]
    lb: Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB]
    lb: Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B]
    lb: Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/multiverse amd64 c-n-f Metadata [116 B]
    lb: Fetched 30.6 MB in 19s (1574 kB/s)
    lb: Reading package lists...
    lb: Building dependency tree...
    lb: Reading state information...
    lb: 8 packages can be upgraded. Run 'apt list --upgradable' to see them.
    lb: 
    lb: WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
    lb: 
    lb: Reading package lists...
    lb: Building dependency tree...
    lb: Reading state information...
    lb: Calculating upgrade...
    lb: The following packages have been kept back:
    lb:   e2fsprogs libcom-err2 libext2fs2 libss2 logsave
    lb: The following packages will be upgraded:
    lb:   libpython3-stdlib python3 python3-minimal
    lb: 3 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.
    lb: Need to get 53.9 kB of archives.
    lb: After this operation, 0 B of additional disk space will be used.
    lb: Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-minimal amd64 3.10.6-1~22.04.1 [24.3 kB]
    lb: Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3 amd64 3.10.6-1~22.04.1 [22.8 kB]
    lb: Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpython3-stdlib amd64 3.10.6-1~22.04.1 [6812 B]
    lb: dpkg-preconfigure: unable to re-open stdin: No such file or directory
    lb: Fetched 53.9 kB in 6s (9138 B/s)
(Reading database ... 64047 files and directories currently installed.)
    lb: Preparing to unpack .../python3-minimal_3.10.6-1~22.04.1_amd64.deb ...
    lb: Unpacking python3-minimal (3.10.6-1~22.04.1) over (3.10.6-1~22.04) ...
    lb: Setting up python3-minimal (3.10.6-1~22.04.1) ...
(Reading database ... 64047 files and directories currently installed.)
    lb: Preparing to unpack .../python3_3.10.6-1~22.04.1_amd64.deb ...
    lb: running python pre-rtupdate hooks for python3.10...
    lb: Unpacking python3 (3.10.6-1~22.04.1) over (3.10.6-1~22.04) ...
    lb: Preparing to unpack .../libpython3-stdlib_3.10.6-1~22.04.1_amd64.deb ...
    lb: Unpacking libpython3-stdlib:amd64 (3.10.6-1~22.04.1) over (3.10.6-1~22.04) ...
    lb: Setting up libpython3-stdlib:amd64 (3.10.6-1~22.04.1) ...
    lb: Setting up python3 (3.10.6-1~22.04.1) ...
    lb: running python rtupdate hooks for python3.10...
    lb: running python post-rtupdate hooks for python3.10...
    lb: Processing triggers for man-db (2.10.2-1) ...
    lb: 
    lb: Running kernel seems to be up-to-date.
    lb: 
    lb: No services need to be restarted.
    lb: 
    lb: No containers need to be restarted.
    lb: 
    lb: No user sessions are running outdated binaries.
    lb: 
    lb: No VM guests are running outdated hypervisor (qemu) binaries on this host.
==> lb: Running provisioner: ansible...
    lb: Running ansible-playbook...
PYTHONUNBUFFERED=1 ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false ANSIBLE_SSH_ARGS='-o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s' ansible-playbook --connection=ssh --timeout=30 --limit="lb" --inventory-file=/home/myuser/vagrant-env/kubernetes-the-hard-way/.vagrant/provisioners/ansible/inventory -vv bootstrap.yml
ansible-playbook [core 2.16.10]
  config file = /home/myuser/vagrant-env/kubernetes-the-hard-way/ansible.cfg
  configured module search path = ['/home/myuser/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/myuser/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible-playbook
  python version = 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (/usr/bin/python3)
  jinja version = 3.0.3
  libyaml = True
Using /home/myuser/vagrant-env/kubernetes-the-hard-way/ansible.cfg as config file
redirecting (type: modules) ansible.builtin.ufw to community.general.ufw
redirecting (type: modules) ansible.builtin.ufw to community.general.ufw
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/etcd-gateway/tasks/install.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/etcd-gateway/tasks/configure.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/tls-ca/tasks/ca.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/tls-ca/tasks/etcd-ca.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/tls-ca/tasks/etcd-admin.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/tls-ca/tasks/admin.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/tls/tasks/apiserver.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/tls/tasks/kubelet.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/tls/tasks/controller-manager.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/tls/tasks/scheduler.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/tls/tasks/serviceaccount.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/tls/tasks/ca-cert.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/tls/tasks/coredns.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/tls/tasks/admin.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/tls/tasks/kube-proxy.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/kubeconfig/tasks/controlplane.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/kubeconfig/tasks/kubelet.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/kubeconfig/tasks/admin.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/kubeconfig/tasks/kube-proxy.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/etcd/tasks/install.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/etcd/tasks/server-cert.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/etcd/tasks/configure.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/etcd/tasks/kube-apiserver-client-cert.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/etcd/tasks/ca-cert.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/k8s/tasks/install.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/k8s/tasks/configure.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/k8s/tasks/kubelet.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/k8s/tasks/kube-proxy.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/worker/tasks/prepare.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/worker/tasks/containerd.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/worker/tasks/runc.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/worker/tasks/cni-plugins.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/worker/tasks/crictl.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/worker/tasks/kubelet.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/worker/tasks/cni-config.yml
statically imported: /home/myuser/vagrant-env/kubernetes-the-hard-way/worker/tasks/finalize.yml
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: bootstrap.yml ********************************************************
2 plays in bootstrap.yml

PLAY [Configure the Load Balancer] *********************************************

TASK [prerequisites : Enable IP Forwarding permanently] ************************
task path: /home/myuser/vagrant-env/kubernetes-the-hard-way/prerequisites/tasks/main.yml:2
Notification for handler Reload sysctl has been saved.
changed: [lb] => {"changed": true, "checksum": "71111e5788a770311ec475d6c4e5ed7e2d32dfb1", "dest": "/etc/sysctl.d/20-ipforward.conf", "gid": 0, "group": "root", "md5sum": "409a7e4d9ec16d872ccc23215286b8b1", "mode": "0644", "owner": "root", "size": 23, "src": "/home/vagrant/.ansible/tmp/ansible-tmp-1724390311.216068-9225-247639193196719/source", "state": "file", "uid": 0}




...

## (NOTE : Some logs are removed due github comment (maximum is 65536 characters), see the txt file for full logs)

...




TASK [haproxy : Fetch the CA to the HAProxy's cert dir] ************************
task path: /home/myuser/vagrant-env/main.yml:34
fatal: [lb]: FAILED! => {"changed": false, "msg": "Source /vagrant/share/ca.crt not found"}

PLAY RECAP *********************************************************************
lb                         : ok=8    changed=7    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

To recreate issue this is my steps :

1. Clone the developer-friendly/kubernetes-the-hard-way repo
2. Change directory to the repo
3. vagrant up

I also attached the error log into the attachment if anyone curious about the error in a file.
Errorlog-Vagrant-0001.txt

Upon debugging the error logs and reading the author's blog I assume there is a mis match on how to ansible task run order, on the blog the author explained that on Step.1 there is a need for signing certificates

But from the vagrant logs (Errorlog-Vagrant-0001.txt]) it seems that task is skipped and immediately went to the HAProxy provisioning, thus when copying the certificate it is not found, but this my assumption for now as im still trying to debug.

Any help is appreciated.
Regards.

[meysam81]

I confirm that this is an issue.

The order should be modified (verification in progress...).
But the main idea is that the TLS CA should be present by the time HAProxy Ansible role is running. That CA is consequently used for SSL offload to the K8s API server.

As soon as I have a working setup, I'll modify the code. 🙌

[meysam81]

@Adhito
In the meantime, if you do have a working setup on your end, any and all contributions are welcome. 🤗