Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get information that system rolled back #118

Open
pmtk opened this issue Sep 24, 2023 · 3 comments
Open

Get information that system rolled back #118

pmtk opened this issue Sep 24, 2023 · 3 comments
Labels
enhancement New feature or request

Comments

@pmtk
Copy link

pmtk commented Sep 24, 2023

Hello

In MicroShift we're looking for a sure way to know that system rolled back so we can perform certain actions.

We have some ideas so far:

  1. In red script, if boot_counter == 0 then create some kind of file on disk to persist that information and read it on next boot. This assumes that:
    • Existence of boot_counter means we're in the middle of "new deployment hasn't been determined yet to be okay, so greenboot might reboot the system" - especially when red script runs (as it will be followed by reboot)
    • boot_counter == 0 means it's a last attempt, when system reboots, grub will see that value and select second boot entry (rollback)
  2. Inspect journalctl --boot 0 -u greenboot-rpm-ostree-grub2-check-fallback for existence of FALLBACK BOOT DETECTED! Default rpm-ostree deployment has been rolled back message
    • Slightly worried about timing, we would probably want After=greenboot-rpm-ostree-grub2-check-fallback, but we'd have to check if non-ostree has any impacts, or if RemainAfterExit=yes would affect that as well
  3. Extend greenboot so greenboot-rpm-ostree-grub2-check-fallback to create a file like /run/rolled-back, this file would be removed by greenboot-grub2-set-counter or be cleaned automatically by reboot (if new deployment wasn't staged, but machine was simply rebooted)

Do you have any other ideas how could we make it a robust mechanism?
Would you happen to know about any other source of this information like grub or (rpm-)ostree?

@say-paul
Copy link
Member

say-paul commented Sep 26, 2023

Does the for a sure way to know that system rolled back needs to be known on the boot(T) when greenboot will attempt rollback or on reboot(T+1) after the rollback happened. It will determine which approach needs to be taken.
Though I agree there should be a cleaner way to determine it.

@pmtk
Copy link
Author

pmtk commented Sep 26, 2023

The latter: on reboot(T+1) after the rollback happened

@say-paul
Copy link
Member

Then the possible workaround for now can be to check if the the boot_counter is unset/boot_success=1 combined with journalctl --boot 1 -u greenboot-rpm-ostree-grub2-check-fallback has the rollback message.

@say-paul say-paul added the enhancement New feature or request label Sep 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants