Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add vsphere-clone builder to decrease build time #394

Merged
merged 2 commits into from
Oct 1, 2020
Merged

Add vsphere-clone builder to decrease build time #394

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1d3263d
Add vsphere-clone builder to decrease build time
EleanorRigby Sep 29, 2020
1f7e1f6
Add docs
EleanorRigby Sep 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions docs/book/src/capi/providers/vsphere.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ The images may be built using one of the following hypervisors:
| macOS | VMware Fusion (vmware-iso)| build-node-ova-local-<OS> |
| ESXi | ESXi | build-node-ova-esx-<OS> |
| vSphere | vSphere >= 6.5 | build-node-ova-vsphere-<OS> |
| vSphere | vSphere >= 6.5 | build-node-ova-vsphere-base-<OS> |
| vSphere Clone | vSphere >= 6.5 | build-node-ova-vsphere-clone-<OS> |
| Linux | VMware Workstation (vmware-vmx) | build-node-ova-local-vmx-<OS> |
| macOS | VMware Fusion (vmware-vmx) | build-node-ova-local-vmx-<OS> |

Expand All @@ -23,6 +25,11 @@ During the dev process it's uncommon for the base OS image to change, but the im

To reduce the image building times during development, one can use the `build-node-ova-local-base-<OS>` target to build the base image from the ISO. By setting `source_path` variable in `vmx.json` to the `*.vmx` file from the output, it can then be re-used with the `build-node-ova-local-vmx-<OS>` build target to speed up the process.


### vsphere-clone builder
`vsphere-base` builder allows you to build one time base OVAs from iso images using the kickstart process. It leaves the user `builder` intact in base OVA to be used by clone builder later. `vSphere-clone` builder builds on top of base OVA by cloning it and ansiblizing it.
This saves time by allowing repeated iteration on base OVA without installing OS from scratch again and again. Also, it uses link cloning and `create_snapshot` feature to clone faster.

### Prerequisites for vSphere builder

Complete the `vsphere.json` configuration file with credentials and informations specific to the remote vSphere hypervisor used to build the `ova` file.
Expand All @@ -37,6 +44,9 @@ This file must have the following format (`cluster` can be replace by `host`):
"cluster": "esxi_cluster_used_for_template_creation",
"network": "network_attached_to_template",
"insecure_connection": "false"
"template": "base_template_used_by_clone_builder",
"create_snbapshot": "creates a snaphot on base OVA after building",
"linked_clone": "Uses link cloning in vsphere-clone builder: true, by default"
}
```

Expand Down
27 changes: 27 additions & 0 deletions images/capi/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,8 @@ NODE_OVA_LOCAL_VMX_BUILD_NAMES := $(addprefix node-ova-local-vmx-,$(PLATFORMS_A
NODE_OVA_LOCAL_BASE_BUILD_NAMES := $(addprefix node-ova-local-base-,$(PLATFORMS_AND_VERSIONS))
NODE_OVA_ESX_BUILD_NAMES := $(addprefix node-ova-esx-,$(PLATFORMS_AND_VERSIONS))
NODE_OVA_VSPHERE_BUILD_NAMES := $(addprefix node-ova-vsphere-,$(PLATFORMS_AND_VERSIONS))
NODE_OVA_VSPHERE_BASE_BUILD_NAMES := $(addprefix node-ova-vsphere-base-,$(PLATFORMS_AND_VERSIONS))
NODE_OVA_VSPHERE_CLONE_BUILD_NAMES := $(addprefix node-ova-vsphere-clone-,$(PLATFORMS_AND_VERSIONS))
HAPROXY_OVA_LOCAL_BUILD_NAMES := $(addprefix haproxy-ova-local-,$(PHOTON_VERSIONS))
HAPROXY_OVA_ESX_BUILD_NAMES := $(addprefix haproxy-ova-esx-,$(PHOTON_VERSIONS))
HAPROXY_OVA_VSPHERE_BUILD_NAMES := $(addprefix haproxy-ova-vsphere-,$(PHOTON_VERSIONS))
Expand All @@ -195,6 +197,8 @@ NODE_OVA_LOCAL_BASE_BUILD_TARGETS := $(addprefix build-,$(NODE_OVA_LOCAL_BASE_BU
NODE_OVA_LOCAL_VALIDATE_TARGETS := $(addprefix validate-,$(NODE_OVA_LOCAL_BUILD_NAMES))
NODE_OVA_ESX_BUILD_TARGETS := $(addprefix build-,$(NODE_OVA_ESX_BUILD_NAMES))
NODE_OVA_VSPHERE_BUILD_TARGETS := $(addprefix build-,$(NODE_OVA_VSPHERE_BUILD_NAMES))
NODE_OVA_VSPHERE_BASE_BUILD_TARGETS := $(addprefix build-,$(NODE_OVA_VSPHERE_BASE_BUILD_NAMES))
NODE_OVA_VSPHERE_CLONE_BUILD_TARGETS := $(addprefix build-,$(NODE_OVA_VSPHERE_CLONE_BUILD_NAMES))
HAPROXY_OVA_LOCAL_BUILD_TARGETS := $(addprefix build-,$(HAPROXY_OVA_LOCAL_BUILD_NAMES))
HAPROXY_OVA_LOCAL_VALIDATE_TARGETS := $(addprefix validate-,$(HAPROXY_OVA_LOCAL_BUILD_NAMES))
HAPROXY_OVA_ESX_BUILD_TARGETS := $(addprefix build-,$(HAPROXY_OVA_ESX_BUILD_NAMES))
Expand Down Expand Up @@ -236,6 +240,14 @@ $(NODE_OVA_ESX_BUILD_TARGETS): deps-ova
$(NODE_OVA_VSPHERE_BUILD_TARGETS): deps-ova
packer build $(PACKER_NODE_FLAGS) -var-file="packer/ova/packer-common.json" -var-file="$(abspath packer/ova/$(subst build-node-ova-vsphere-,,$@).json)" -var-file="packer/ova/vsphere.json" -except=esx -except=local -only=vsphere-iso $(ABSOLUTE_PACKER_VAR_FILES) -only=vsphere packer/ova/packer-node.json

.PHONY: $(NODE_OVA_VSPHERE_BASE_BUILD_TARGETS)
$(NODE_OVA_VSPHERE_BASE_BUILD_TARGETS): deps-ova
packer build $(PACKER_NODE_FLAGS) -var-file="packer/ova/packer-common.json" -var-file="$(abspath packer/ova/$(subst build-node-ova-vsphere-base-,,$@).json)" -var-file="packer/ova/vsphere.json" -except=esx -except=local -except=manifest -except=vsphere -only=vsphere-iso-base $(ABSOLUTE_PACKER_VAR_FILES) packer/ova/packer-node.json

.PHONY: $(NODE_OVA_VSPHERE_CLONE_BUILD_TARGETS)
$(NODE_OVA_VSPHERE_CLONE_BUILD_TARGETS): deps-ova
packer build $(PACKER_NODE_FLAGS) -var-file="packer/ova/packer-common.json" -var-file="$(abspath packer/ova/$(subst build-node-ova-vsphere-clone-,,$@).json)" -var-file="packer/ova/vsphere.json" -except=esx -except=local -only=vsphere-clone $(ABSOLUTE_PACKER_VAR_FILES) packer/ova/packer-node.json

.PHONY: $(HAPROXY_OVA_LOCAL_BUILD_TARGETS)
$(HAPROXY_OVA_LOCAL_BUILD_TARGETS): deps-ova
packer build $(PACKER_HAPROXY_FLAGS) -var-file="packer/ova/packer-common.json" -var-file="$(abspath packer/ova/$(subst build-haproxy-ova-local-,,$@).json)" -except=esx -except=vsphere -only=vmware-iso $(ABSOLUTE_PACKER_VAR_FILES) packer/ova/packer-haproxy.json
Expand Down Expand Up @@ -367,6 +379,21 @@ build-node-ova-vsphere-ubuntu-2004: ## Builds Ubuntu 20.04 Node OVA and template
build-node-ova-vsphere-all: $(NODE_OVA_VSPHERE_BUILD_TARGETS) ## Builds all Node OVAs and templates on vSphere
build-haproxy-ova-vsphere-photon-3: ## Builds Photon 3 HAProxy OVA and template on vSphere

build-node-ova-vsphere-clone-centos-7: ## Builds CentOS 7 Node OVA and template on vSphere
build-node-ova-vsphere-clone-photon-3: ## Builds Photon 3 Node OVA and template on vSphere
build-node-ova-vsphere-clone-rhel-7: ## Builds RHEL 7 Node OVA and template on vSphere
build-node-ova-vsphere-clone-ubuntu-1804: ## Builds Ubuntu 18.04 Node OVA and template on vSphere
build-node-ova-vsphere-clone-ubuntu-2004: ## Builds Ubuntu 20.04 Node OVA and template on vSphere
build-node-ova-vsphere-clone-all: $(NODE_OVA_VSPHERE_CLONE_BUILD_TARGETS) ## Builds all Node OVAs and templates on vSphere

build-node-ova-vsphere-base-centos-7: ## Builds CentOS 7 Node OVA and template on vSphere
build-node-ova-vsphere-base-photon-3: ## Builds Photon 3 Node OVA and template on vSphere
build-node-ova-vsphere-base-rhel-7: ## Builds RHEL 7 Node OVA and template on vSphere
build-node-ova-vsphere-base-ubuntu-1804: ## Builds Ubuntu 18.04 Node OVA and template on vSphere
build-node-ova-vsphere-base-ubuntu-2004: ## Builds Ubuntu 20.04 Node OVA and template on vSphere
build-node-ova-vsphere-base-all: $(NODE_OVA_VSPHERE_BASE_BUILD_TARGETS) ## Builds all Node OVAs and templates on vSphere


build-node-ova-local-vmx-photon-3: ## Builds Photon 3 Node OVA from VMX file w local hypervisor
build-node-ova-local-vmx-centos-7: ## Builds Centos 7 Node OVA from VMX file w local hypervisor
build-node-ova-local-vmx-rhel-7: ## Builds RHEL 7 Node OVA from VMX file w local hypervisor
Expand Down
98 changes: 93 additions & 5 deletions images/capi/packer/ova/packer-node.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
"ansible_extra_vars": "guestinfo_datasource_slug={{user `guestinfo_datasource_slug`}} guestinfo_datasource_ref={{user `guestinfo_datasource_ref`}} guestinfo_datasource_script={{user `guestinfo_datasource_script`}}",
"build_timestamp": "{{timestamp}}",
"build_version": "{{user `build_name`}}-kube-{{user `kubernetes_semver`}}",
"base_build_version": "base-{{user `build_name`}}-kube-{{user `kubernetes_semver`}}",
EleanorRigby marked this conversation as resolved.
Show resolved Hide resolved
"cluster": "",
"containerd_version": null,
"containerd_sha256": null,
Expand Down Expand Up @@ -32,6 +33,7 @@
"kubernetes_rpm_gpg_check": null,
"kubernetes_container_registry": null,
"output_dir": "./output/{{user `build_version`}}",
"base_output_dir": "./output/{{user `base_build_version`}}",
"username": "",
"vcenter_server": ""
},
Expand Down Expand Up @@ -67,9 +69,9 @@
},
{
"name": "vmware-iso-base",
"vm_name": "{{user `build_version`}}",
"vmdk_name": "{{user `build_version`}}",
"output_directory": "{{user `output_dir`}}",
"vm_name": "{{user `base_build_version`}}",
"vmdk_name": "{{user `base_build_version`}}",
"output_directory": "{{user `base_output_dir`}}",
"type": "vmware-iso",
"version": "{{user `vmx_version`}}",
"cpus": 1,
Expand Down Expand Up @@ -152,7 +154,60 @@
"vmx_data": {
"ethernet0.networkName": "{{user `network`}}"
}
},{
"type": "vsphere-iso",
"name": "vsphere-iso-base",
"vcenter_server": "{{user `vcenter_server`}}",
"username": "{{user `username`}}",
"password": "{{user `password`}}",
"insecure_connection": "{{user `insecure_connection`}}",
"vm_name": "{{user `base_build_version`}}",
"datastore": "{{user `datastore`}}",
"datacenter": "{{user `datacenter`}}",
"folder": "{{user `folder`}}",
"host": "{{user `host`}}",
"convert_to_template": "{{user `convert_to_template`}}",
"cluster": "{{user `cluster`}}",
"create_snapshot": "{{user `create_snapshot`}}",
"network_adapters": [
{
"network": "{{user `network`}}",
"network_card": "{{user `network_card`}}"
}
],
"vm_version": "{{user `vmx_version`}}",
"CPUs": 1,
"cpu_cores": 1,
"RAM": 2048,
"disk_controller_type": "{{user `disk_controller_type`}}",
"guest_os_type": "{{user `vsphere_guest_os_type`}}",
"boot_wait": "{{user `boot_wait`}}",
"http_directory": "./packer/ova/linux/{{user `distro_name`}}/http/",
"iso_urls": "{{user `iso_url`}}",
"iso_checksum": "{{user `iso_checksum_type`}}:{{user `iso_checksum`}}",
"communicator": "ssh",
"ssh_username": "{{user `ssh_username`}}",
"ssh_password": "{{user `ssh_password`}}",
"ssh_timeout": "4h",
"ssh_clear_authorized_keys": "false",
"storage": [
{
"disk_size": 20480,
"disk_thin_provisioned": "{{user `disk_thin_provisioned`}}"
}
],
"boot_command": [
"{{user `boot_command_prefix`}}",
"http://{{ .HTTPIP }}:{{ .HTTPPort }}",
"{{user `boot_command_suffix`}}"
],
"shutdown_command": "echo '{{user `ssh_password`}}' | sudo -S -E sh -c '{{user `shutdown_command`}}'",
"export": {
"force": true,
"output_directory": "{{user `base_output_dir`}}"
}
},

{
"type": "vsphere-iso",
"name": "vsphere",
Expand Down Expand Up @@ -203,11 +258,44 @@
"force": true,
"output_directory": "{{user `output_dir`}}"
}
},
{
"type": "vsphere-clone",
"name": "vsphere-clone",
"vcenter_server": "{{user `vcenter_server`}}",
"create_snapshot": "{{user `create_snapshot`}}",
"username": "{{user `username`}}",
"password": "{{user `password`}}",
"insecure_connection": "{{user `insecure_connection`}}",
"vm_name": "{{user `build_version`}}",
"datastore": "{{user `datastore`}}",
"datacenter": "{{user `datacenter`}}",
"folder": "{{user `folder`}}",
"host": "{{user `host`}}",
"convert_to_template": "{{user `convert_to_template`}}",
"cluster": "{{user `cluster`}}",
"network": "{{user `network`}}",
"template": "{{user `template`}}",
"CPUs": 4,
"cpu_cores": 12,
"RAM": 16384,
"linked_clone": "{{user `linked_clone`}}",
"communicator": "ssh",
"ssh_username": "{{user `ssh_username`}}",
"ssh_password": "{{user `ssh_password`}}",
"ssh_timeout": "4h",
"shutdown_command": "echo '{{user `ssh_password`}}' | sudo -S -E sh -c 'usermod -L {{user `ssh_username`}} && {{user `shutdown_command`}}'",
"export": {
"force": true,
"output_directory": "{{user `output_dir`}}"
}
}

],
"provisioners": [
{
"type": "ansible",
"except": ["vmware-iso-base", "vsphere-iso-base"],
"playbook_file": "./ansible/firstboot.yml",
"user": "{{user `ssh_username`}}",
"ansible_env_vars": [
Expand All @@ -234,7 +322,7 @@
},
{
"type": "ansible",
"except": ["vmware-iso-base"],
"except": ["vmware-iso-base", "vsphere-iso-base"],
"playbook_file": "./ansible/node.yml",
"user": "{{user `ssh_username`}}",
"ansible_env_vars": [
Expand All @@ -250,7 +338,7 @@
},
{
"type": "goss",
"except": ["vmware-iso-base"],
"except": ["vmware-iso-base", "vsphere-iso-base"],
"arch": "{{user `goss_arch`}}",
"format": "{{user `goss_format`}}",
"format_options": "{{user `goss_format_options`}}",
Expand Down
3 changes: 3 additions & 0 deletions images/capi/packer/ova/vsphere.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,8 @@
"cluster": "",
"network": "",
"convert_to_template": "false",
"linked_clone": "true",
"create_snapshot": "true",
"template": "",
"insecure_connection": "false"
}