fix(upgrade): make upgrades from master to master possible

[starbops]

Problem:

Sometimes we need to verify the bug fixes or feature deliveries in the upgrade area, it's essential to upgrade from the master-head to master-head with the exact same ISO image downloaded from https://releases.rancher.com/harvester/master/harvester-master-amd64.iso. But the server-version does not match with the image tag for rancher/harvester-upgrade image for such cases. The upgrade will then fail at the image-preloading phase as it cannot find the right image on the node nor on the Internet (if not air-gapped).

This is due to a different VERSION argument passed into the harvester-upgrade docker build command in .drone.yaml:

...
- name: docker-publish-upgrade-branch
  image: plugins/docker
  settings:
    build_args:
    - ARCH=amd64
    - VERSION=${DRONE_BRANCH}-${DRONE_COMMIT_SHA:0:8}-head
    context: package/upgrade
    custom_dns: 1.1.1.1
    dockerfile: package/upgrade/Dockerfile
    password:
      from_secret: docker_password
    repo: "rancher/harvester-upgrade"
    tag: ${DRONE_BRANCH}-head-amd64
    username:
      from_secret: docker_username
  when:
    ref:
      include:
      - "refs/heads/master"
      - "refs/heads/v*"
    event:
    - push
...

This step will be triggered every time when there's a new push to the master or v* branch. The image tag ${DRONE_BRANCH}-${DRONE_COMMIT_SHA:0:8}-head only exists in an intermediate state and is never published. The one that gets published is the ${DRONE_BRANCH}-head-amd64. However, the tag ${DRONE_BRANCH}-${DRONE_COMMIT_SHA:0:8}-head tag is stored in the server-version CR and is then referenced as the previousVersion in the following upgrades.

Solution:

Auto builds triggered by push events to branches like "master" or "v1.1" etc. will have rancher/harvester-upgrade:<branch>-<commit>-head image added into the bundle, therefore making the upgrade from master-head to master-head or v1.1-head to v1.1-head possible.

Related Issue:

harvester/harvester#3610

Test plan:

For reviewers:

1. Checkout to this branch
2. Build the ISO with the command make
3. Check the ISO image artifact with isoinfo command, it should contain the additional harvester-upgrade image with the master-<commit>-head tag

$ isoinfo -J -i dist/artifacts/harvester-master-amd64.iso -x /bundle/harvester/images-lists/harvester-extra-master.txt
docker.io/rancher/harvester-upgrade:544cb2b
docker.io/rancher/harvester-upgrade:master-544cb2b-head

For QAs:

1. Download the ISO image from https://releases.rancher.com/harvester/master/harvester-master-amd64.iso, please be sure that the image already has the fix included
2. Provision a Harvester cluster with the ISO image (single-node is fine)
3. Upgrade the cluster with the same ISO image
4. The upgrade should succeed

[bk201]

LGTM, thanks.

[Vicente-Cheng]

Tested with the upgrade path. LGTM thanks!

[starbops]

@Mergifyio backport v1.1

[mergify]

backport v1.1

✅ Backports have been created

• #449 fix(upgrade): make upgrades from master to master possible (backport #447) has been created for branch v1.1

[bk201]

@starbops I'm still getting ErrImagePull looks like there is one character difference.

rancher/harvester-upgrade:master-152642c-head       <--- what we pack after this PR
rancher/harvester-upgrade:master-152642c2-head      <--- server version

[bk201]

@Vicente-Cheng share that this might be because commish prefix conflict. We might need to use git rev-parse --short=8 HEAD in the version file as you suggested before.

[starbops]

@bk201 @Vicente-Cheng If you build the ISO image within the harvester-installer repo via make, you'll get a 7-character commit hash. If you build the ISO image within the harvester repo via make && make build-iso, you'll get 8 characters instead. I think that's because there are much more objects in the harvester repo compared to the harvester-installer repo, so it has to use at least 8 characters to prevent the collision.

For drone CI-built images, it should be fine since the CI builds the ISO image within the harvester repo. Did you build the ISO image which got rancher/harvester-upgrade:master-152642c-head within the harvester-installer repo by yourself?

git rev-parse --short=8 HEAD is not a perfect solution though, it's good enough for us now. I'll fix this in a new PR.

[bk201]

I got rancher/harvester-upgrade:master-152642c-head by downloading the drone-build ISO.

[bk201]

I guess the ISO is the one built in the harvester/harvester-installer drone job and it clones a single branch.
Thus there is no collision and produces an 8-character-length hash.