testing: new release on 2020-06-11 (32.20200601.2.2)

[dustymabe]

First, verify that you meet all the prerequisites

Name this issue testing: new release on YYYY-MM-DD with today's date. Once the pipeline spits out the new version ID, you can append it to the title e.g. (31.20191117.2.0).

Pre-release

Promote testing-devel changes to testing

From the checkout for fedora-coreos-config (replace upstream below with
whichever remote name tracks coreos/):

• git fetch upstream
• git checkout testing
• git reset --hard upstream/testing
• /path/to/fedora-coreos-releng-automation/scripts/promote-config.sh testing-devel
• Sanity check promotion with git show
• Open PR against the testing branch on https://github.com/coreos/fedora-coreos-config
• Post a link to the PR as a comment to this issue
• Ideally have at least one other person check it and approve
• Once CI has passed, merge it

Build

• Start a pipeline build (select testing, leave all other defaults)
• Post a link to the job as a comment to this issue
• Wait for the job to finish

Sanity-check the build

Using the the build browser for the testing stream:

• Verify that the parent commit and version match the previous testing release (in the future, we'll want to integrate this check in the release job)
• Check kola AWS run to make sure it didn't fail
• Check kola GCP run to make sure it didn't fail

⚠️ Release ⚠️

IMPORTANT: this is the point of no return here. Once the OSTree commit is
imported into the unified repo, any machine that manually runs rpm-ostree upgrade will have the new update.

Run the release job

• Run the release job, filling in for parameters testing and the new version ID
• Post a link to the job as a comment to this issue
• Wait for job to finish
• Verify that the OSTree commit and its signature are present and valid by booting a VM at the previous release (e.g. cosa run -d /path/to/previous.qcow2) and verifying that rpm-ostree upgrade works and rpm-ostree status shows a valid signature.

At this point, Cincinnati will see the new release on its next refresh and create a corresponding node in the graph without edges pointing to it yet.

Refresh metadata (stream and updates)

From a checkout of this repo:

• Update stream metadata, by running:

fedora-coreos-stream-generator -releases=https://fcos-builds.s3.amazonaws.com/prod/streams/testing/releases.json  -output-file=streams/testing.json -pretty-print

• Update the updates metadata, editing updates/testing.json:
  • Find the last-known-good release (whose rollout has a start_percentage of 1.0) and set its version to the most recent completed rollout
  • Delete releases with completed rollouts
  • Add a new rollout:
    • Set version field to the new version
    • Set start_epoch field to a future timestamp for the rollout start (e.g. date -d '2019/09/10 14:30UTC' +%s)
    • Set start_percentage field to 0.0
    • Set duration_minutes field to a reasonable rollout window (e.g. 2880 for 48h)
  • Update the last-modified field to current time (e.g. date -u +%Y-%m-%dT%H:%M:%SZ)

A reviewer can validate the start_epoch time by running date -u -d @<EPOCH>. An example of encoding and decoding in one step: date -d '2019/09/10 14:30UTC' +%s | xargs -I{} date -u -d @{}.

• Commit the changes and open a PR against the repo.
• Post a link to the PR as a comment to this issue
• Wait for the PR to be approved.
• Once approved, merge it and verify that the sync-stream-metadata job syncs the contents to S3
• Verify the new version shows up on the download page
• Verify the incoming edges are showing up in the update graph:

curl -H 'Accept: application/json' 'https://updates.coreos.fedoraproject.org/v1/graph?basearch=x86_64&stream=testing&rollout_wariness=0'

NOTE: In the future, most of these steps will be automated.

Open an issue for the next release

• Open an issue in this repo with the approximate date in the title of the next release in this stream.
  • Add the jira label to the ticket

[dustymabe]

This is a build done to pull in fixes that address coreos/fedora-coreos-tracker#525 by removing the modular repos as inputs to the rpm-ostree compose.

[dustymabe]

coreos/fedora-coreos-config#467

[dustymabe]

https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/13141/

[dustymabe]

https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline/13141/

The ostree import failed because the old coreos-ostree-importer was still running in the phx2 datacenter and accepting some of the requests (via fedmsg). It failed to import, which was good because it allowed us to identify this problem and fix it.

I've manually kicked coreos-ostree-importer and the import was successful. I also manually started AWS/GCP kola runs.

[dustymabe]

The AWS run is still failing because of a known bug.

[dustymabe]

PR to stop rollouts for testing/next: #121

[dustymabe]

https://jenkins-fedora-coreos.apps.ci.centos.org/job/fedora-coreos/job/fedora-coreos-fedora-coreos-pipeline-release/551

[dustymabe]

[core@localhost-live ~]$ sudo rpm-ostree status 
State: idle
Deployments:
● ostree://fedora:fedora/x86_64/coreos/testing
                   Version: 32.20200601.2.2 (2020-06-11T18:23:48Z)
                    Commit: 2f9f73534e012e2da57e1afae4e5684fa797e871bc9400f4af506cea291400da
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

  ostree://fedora:fedora/x86_64/coreos/testing
                   Version: 32.20200601.2.1 (2020-06-03T02:40:08Z)
                    Commit: f482e2f1870124abf1f0094b7fcc66f86db8394a4096e1e66984b80178e84b34
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

[dustymabe]

#122

[dustymabe]

Ok the rollout is scheduled for 7am EDT on 06/15/2020

[dustymabe]

Going to close this out since all seems to be going well.