Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reset is not removing content under /usr/local #2442

Closed
vipsharm opened this issue Apr 6, 2024 · 8 comments
Closed

Reset is not removing content under /usr/local #2442

vipsharm opened this issue Apr 6, 2024 · 8 comments
Assignees
@Itxaka
Labels
area/agent bug Something isn't working prio: high uki verified

Comments

@vipsharm
Copy link
Collaborator

vipsharm commented Apr 6, 2024

While testing FDE cluster reset , noticed the binaries copied under /usr/local were still there after reset. The cloud config file embedded in the ISO did not have the step to copy stuff from /usr/local to /oem after-install stage.

So it was strange how this data was not cleaned up and still intact after reset. Some other config files generated under /system/oem got cleaned up though.

Shared the installer ISO and provider image I used to repro through email.

Kairos version:
3.0.4
@vipsharm vipsharm added bug Something isn't working triage Add this label to issues that should be triaged and prioretized in the next planning call unconfirmed labels Apr 6, 2024
@Itxaka
Copy link
Member

Itxaka commented Apr 6, 2024

cloud-config:

#cloud-config
install:
  device: "/dev/vda"
  auto: true
  partitions:
    oem:
      size: 4000
      fs: ext4
    state:
      size: 5000
      fs: ext4

users:
- name: "kairos"
  passwd: "kairos"

stylus:
  site:
    debug: true
    insecureSkipVerify: false
    paletteEndpoint: REDACTED
    edgeHostToken: REDACTED
stages:
  initramfs:
    - users:
        kairos:
          groups:
            - sudo
          passwd: kairos

@Itxaka
Copy link
Member

Itxaka commented Apr 6, 2024

  • created a file under /usr/local
root@edge-1fefe67570ac44398111bd143dac7350:/usr/local# ls -ltra
total 146984
drwxr-xr-x 12 root root        240 Apr 16  2020 ..
drwx------  2 root root      16384 Apr  6 09:36 lost+found
-rw-------  1 root root  150465024 Apr  6 09:36 stylus-image.tar
drwxrwx---  2 root admin      4096 Apr  6 09:36 cloud-config
drwxr-xr-x 32 root root       4096 Apr  6 09:38 .state
drwxr-xr-x  2 root root       4096 Apr  6 09:38 bin
drwxr-xr-x  2 root root       4096 Apr  6 09:38 etc
d---------  3 root root       4096 Apr  6 09:38 lib
drw-------  2 root root       4096 Apr  6 09:38 .kairos
-rw-r--r--  1 root root          0 Apr  6 09:39 itxaka-was-here
drwxr-xr-x  9 root root       4096 Apr  6 09:39 .
  • selected statereset bootentry
  • reboot
  • entry is selected
  • reset is done, reboots automatically to active
  • list contents of /usr/local
root@edge-1fefe67570ac44398111bd143dac7350:/usr/local# ls -ltra
total 146984
drwxr-xr-x 12 root root        240 Apr 16  2020 ..
drwx------  2 root root      16384 Apr  6 09:36 lost+found
-rw-------  1 root root  150465024 Apr  6 09:36 stylus-image.tar
drwxrwx---  2 root admin      4096 Apr  6 09:36 cloud-config
drwxr-xr-x 32 root root       4096 Apr  6 09:38 .state
drwxr-xr-x  2 root root       4096 Apr  6 09:38 etc
d---------  3 root root       4096 Apr  6 09:38 lib
drw-------  2 root root       4096 Apr  6 09:38 .kairos
-rw-r--r--  1 root root          0 Apr  6 09:39 itxaka-was-here
drwxr-xr-x  9 root root       4096 Apr  6 09:39 .
drwxr-xr-x  2 root root       4096 Apr  6 09:41 bin

File its still there. This looks like the reset didnt work as expected somehow?

@Itxaka
Copy link
Member

Itxaka commented Apr 6, 2024

booting on recovery adn running the reset manually seems to work but gives us some errors in the stages related to the mounts

2024-04-06T09:44:36Z DBG Loaded reset-uki spec: &v1.ResetUkiSpec{
  FormatPersistent: true,
  FormatOEM: false,
  Reboot: false,
  PowerOff: false,
  Partitions: v1.ElementalPartitions{
    BIOS: nil,
    EFI: &v1.Partition{
      Name: "vda1",
      FilesystemLabel: "COS_GRUB",
      Size: 15360,
      FS: "vfat",
      Flags: nil,
      MountPoint: "/efi",
      Path: "/dev/vda1",
      Disk: "/dev/vda",
    },
    OEM: &v1.Partition{
      Name: "",
      FilesystemLabel: "COS_OEM",
      Size: 4177526784,
      FS: "ext4",
      Flags: nil,
      MountPoint: "/oem",
      Path: "/dev/disk/by-label/COS_OEM",
      Disk: "/dev/vda",
    },
    Recovery: nil,
    State: nil,
    Persistent: &v1.Partition{
      Name: "",
      FilesystemLabel: "COS_PERSISTENT",
      Size: 28124905472,
      FS: "ext4",
      Flags: nil,
      MountPoint: "",
      Path: "/dev/disk/by-label/COS_PERSISTENT",
      Disk: "/dev/vda",
    },
  },
}
2024-04-06T09:44:36Z DBG Cloud-init paths set to [/system/oem /oem/ /usr/local/cloud-config/]
2024-04-06T09:44:36Z INF Running stage: kairos-uki-reset.pre.before

2024-04-06T09:44:36Z INF Done executing stage 'kairos-uki-reset.pre.before'

2024-04-06T09:44:36Z INF Running stage: kairos-uki-reset.pre

2024-04-06T09:44:36Z INF Done executing stage 'kairos-uki-reset.pre'

2024-04-06T09:44:36Z INF Running stage: kairos-uki-reset.pre.after

2024-04-06T09:44:36Z INF Done executing stage 'kairos-uki-reset.pre.after'

2024-04-06T09:44:36Z INF Running stage: kairos-uki-reset.pre.before

2024-04-06T09:44:36Z INF Done executing stage 'kairos-uki-reset.pre.before'

2024-04-06T09:44:36Z INF Running stage: kairos-uki-reset.pre

2024-04-06T09:44:36Z INF Done executing stage 'kairos-uki-reset.pre'

2024-04-06T09:44:36Z INF Running stage: kairos-uki-reset.pre.after

2024-04-06T09:44:36Z INF Done executing stage 'kairos-uki-reset.pre.after'

2024-04-06T09:44:36Z INF Unmounting disk partitions
2024-04-06T09:44:36Z DBG Unmounting partition COS_OEM
2024-04-06T09:44:36Z DBG Unmounting partition COS_GRUB
2024-04-06T09:44:36Z INF Formatting 'COS_PERSISTENT' partition
2024-04-06T09:44:36Z DBG Running cmd: 'mkfs.ext4 -L COS_PERSISTENT /dev/disk/by-label/COS_PERSISTENT'
2024-04-06T09:44:36Z DBG Mounting partition COS_GRUB
2024-04-06T09:44:37Z DBG Conf file /efi/loader/entries/active_install-mode_stylus.registration.conf has values map[string]string{
  "efi": "/EFI/kairos/recovery_install-mode_stylus.registration.efi",
  "title": "Palette eXtended Kubernetes Edge recovery",
}
2024-04-06T09:44:37Z DBG Conf file /efi/loader/entries/active_install-mode_stylus.registration.conf new values map[string]string{
  "efi": "/EFI/kairos/active_install-mode_stylus.registration.efi",
  "title": "Palette eXtended Kubernetes Edge recovery",
}
2024-04-06T09:44:37Z INF Setting default boot entry to cos
2024-04-06T09:44:37Z DBG Checking file /efi/loader/entries
2024-04-06T09:44:37Z DBG Checking file /efi/loader/entries/active_install-mode_stylus.registration.conf
2024-04-06T09:44:37Z DBG Checking file /efi/loader/entries/passive_install-mode_stylus.registration.conf
2024-04-06T09:44:37Z DBG Checking file /efi/loader/entries/recovery_install-mode_stylus.registration.conf
2024-04-06T09:44:37Z DBG Checking file /efi/loader/entries/statereset_install-mode_stylus.registration.conf
2024-04-06T09:44:37Z INF Default boot entry set to cos
2024-04-06T09:44:37Z DBG Mounting partition COS_OEM
2024-04-06T09:44:37Z INF Running after-reset hook
2024-04-06T09:44:37Z DBG Cloud-init paths set to [/system/oem /oem/ /usr/local/cloud-config/]
2024-04-06T09:44:37Z INF Running stage: after-reset.before

2024-04-06T09:44:37Z INF Done executing stage 'after-reset.before'

2024-04-06T09:44:37Z INF Running stage: after-reset

2024-04-06T09:44:37Z INF Processing stage step 'Copy files from oem to persistent'. ( commands: 1, files: 0, ... )
2024-04-06T09:44:37Z WRN (conditional) Skip 'Skipping stage (if statement error: failed to run grep -vq "rd.immucore.uki" /proc/cmdline: exit status 1)' stage name: Mount state
2024-04-06T09:44:38Z ERR Copying files from oem to persistent
semid 3: semop failed for cookie 0xd4d11de: incorrect semaphore state
Failed to set a proper state for notification semaphore identified by cookie value 223154654 (0xd4d11de) to initialize waiting for incoming notifications.
semid 4: semop failed for cookie 0xd4d974f: incorrect semaphore state
Failed to set a proper state for notification semaphore identified by cookie value 223188815 (0xd4d974f) to initialize waiting for incoming notifications.
/bin/bash: line 13: luet: command not found
: failed to run /bin/bash <<'EOF'
#!/bin/bash
set -e
echo "Copying files from oem to persistent"
# /oem was mounted in my tests. Let's umount it to be sure.
umount /oem || true
# Close all encrypted partitions
for p in $(ls /dev/mapper/vda*); do cryptsetup close $p; done
/usr/lib/systemd/systemd-cryptsetup attach persistent $(findfs PARTLABEL=persistent) - tpm2-device=auto
/usr/lib/systemd/systemd-cryptsetup attach oem $(findfs PARTLABEL=oem) - tpm2-device=auto
mount /dev/mapper/persistent /usr/local
mount /dev/mapper/oem /oem
mkdir -p /usr/local/.state/opt.bind
luet util unpack file://oem/.opt/stylus-image.tar /
umount /dev/mapper/persistent
umount /dev/mapper/oem
cryptsetup close /dev/mapper/persistent
cryptsetup close /dev/mapper/oem
: exit status 127
2024-04-06T09:44:38Z ERR 1 error occurred:
	* failed to run /bin/bash <<'EOF'
#!/bin/bash
set -e
echo "Copying files from oem to persistent"
# /oem was mounted in my tests. Let's umount it to be sure.
umount /oem || true
# Close all encrypted partitions
for p in $(ls /dev/mapper/vda*); do cryptsetup close $p; done
/usr/lib/systemd/systemd-cryptsetup attach persistent $(findfs PARTLABEL=persistent) - tpm2-device=auto
/usr/lib/systemd/systemd-cryptsetup attach oem $(findfs PARTLABEL=oem) - tpm2-device=auto
mount /dev/mapper/persistent /usr/local
mount /dev/mapper/oem /oem
mkdir -p /usr/local/.state/opt.bind
luet util unpack file://oem/.opt/stylus-image.tar /
umount /dev/mapper/persistent
umount /dev/mapper/oem
cryptsetup close /dev/mapper/persistent
cryptsetup close /dev/mapper/oem
: exit status 127


2024-04-06T09:44:38Z WRN (conditional) Skip 'Skipping stage (if statement error: failed to run ! grep -q "grub_boot_assessment" /tmp/mnt/STATE/grubcustom && grep -vq "rd.immucore.uki" /proc/cmdline
: exit status 1)' stage name: Hook boot assessment grub configuration
2024-04-06T09:44:38Z WRN (conditional) Skip 'Skipping stage (if statement error: failed to run [ -e "/etc/kairos/branding/grubmenu.cfg" ] && grep -vq "rd.immucore.uki" /proc/cmdline: exit status 1)' stage name: Grub branding
2024-04-06T09:44:38Z WRN (conditional) Skip 'Skipping stage (if statement error: failed to run grep -vq "rd.immucore.uki" /proc/cmdline: exit status 1)' stage name: Add boot assessment grub configuration
2024-04-06T09:44:38Z WRN (conditional) Skip 'Skipping stage (if statement error: failed to run grep -vq "rd.immucore.uki" /proc/cmdline: exit status 1)' stage name: umount state
2024-04-06T09:44:38Z INF Processing stage step 'Copy files from oem to persistent'. ( commands: 1, files: 0, ... )
2024-04-06T09:44:38Z ERR Copying files from oem to persistent
ls: cannot access '/dev/mapper/vda*': No such file or directory
Volume persistent already active.
Volume oem already active.
mount: /usr/local: /dev/mapper/persistent already mounted on /usr/local.
       dmesg(1) may have more information after failed mount system call.
: failed to run /bin/bash <<'EOF'
#!/bin/bash
set -e
echo "Copying files from oem to persistent"
# /oem was mounted in my tests. Let's umount it to be sure.
umount /oem || true
# Close all encrypted partitions
for p in $(ls /dev/mapper/vda*); do cryptsetup close $p; done
/usr/lib/systemd/systemd-cryptsetup attach persistent $(findfs PARTLABEL=persistent) - tpm2-device=auto
/usr/lib/systemd/systemd-cryptsetup attach oem $(findfs PARTLABEL=oem) - tpm2-device=auto
mount /dev/mapper/persistent /usr/local
mount /dev/mapper/oem /oem
mkdir -p /usr/local/.state/opt.bind
luet util unpack file://oem/.opt/stylus-image.tar /
umount /dev/mapper/persistent
umount /dev/mapper/oem
cryptsetup close /dev/mapper/persistent
cryptsetup close /dev/mapper/oem
: exit status 32
2024-04-06T09:44:38Z ERR 1 error occurred:
	* failed to run /bin/bash <<'EOF'
#!/bin/bash
set -e
echo "Copying files from oem to persistent"
# /oem was mounted in my tests. Let's umount it to be sure.
umount /oem || true
# Close all encrypted partitions
for p in $(ls /dev/mapper/vda*); do cryptsetup close $p; done
/usr/lib/systemd/systemd-cryptsetup attach persistent $(findfs PARTLABEL=persistent) - tpm2-device=auto
/usr/lib/systemd/systemd-cryptsetup attach oem $(findfs PARTLABEL=oem) - tpm2-device=auto
mount /dev/mapper/persistent /usr/local
mount /dev/mapper/oem /oem
mkdir -p /usr/local/.state/opt.bind
luet util unpack file://oem/.opt/stylus-image.tar /
umount /dev/mapper/persistent
umount /dev/mapper/oem
cryptsetup close /dev/mapper/persistent
cryptsetup close /dev/mapper/oem
: exit status 32


2024-04-06T09:44:38Z INF Done executing stage 'after-reset'

2024-04-06T09:44:38Z INF Running stage: after-reset.after

2024-04-06T09:44:38Z INF Done executing stage 'after-reset.after'

2024-04-06T09:44:38Z INF Running stage: after-reset.before

2024-04-06T09:44:38Z INF Done executing stage 'after-reset.before'

2024-04-06T09:44:38Z INF Running stage: after-reset

2024-04-06T09:44:38Z INF Done executing stage 'after-reset'

2024-04-06T09:44:38Z INF Running stage: after-reset.after

2024-04-06T09:44:38Z INF Done executing stage 'after-reset.after'

2024-04-06T09:44:38Z INF Some errors found but were ignored. Enable --strict mode to fail on those or --debug to see them in the log
2024-04-06T09:44:38Z WRN 4 errors occurred:
	* failed to run /bin/bash <<'EOF'
#!/bin/bash
set -e
echo "Copying files from oem to persistent"
# /oem was mounted in my tests. Let's umount it to be sure.
umount /oem || true
# Close all encrypted partitions
for p in $(ls /dev/mapper/vda*); do cryptsetup close $p; done
/usr/lib/systemd/systemd-cryptsetup attach persistent $(findfs PARTLABEL=persistent) - tpm2-device=auto
/usr/lib/systemd/systemd-cryptsetup attach oem $(findfs PARTLABEL=oem) - tpm2-device=auto
mount /dev/mapper/persistent /usr/local
mount /dev/mapper/oem /oem
mkdir -p /usr/local/.state/opt.bind
luet util unpack file://oem/.opt/stylus-image.tar /
umount /dev/mapper/persistent
umount /dev/mapper/oem
cryptsetup close /dev/mapper/persistent
cryptsetup close /dev/mapper/oem
: exit status 127
	* failed to run /bin/bash <<'EOF'
#!/bin/bash
set -e
echo "Copying files from oem to persistent"
# /oem was mounted in my tests. Let's umount it to be sure.
umount /oem || true
# Close all encrypted partitions
for p in $(ls /dev/mapper/vda*); do cryptsetup close $p; done
/usr/lib/systemd/systemd-cryptsetup attach persistent $(findfs PARTLABEL=persistent) - tpm2-device=auto
/usr/lib/systemd/systemd-cryptsetup attach oem $(findfs PARTLABEL=oem) - tpm2-device=auto
mount /dev/mapper/persistent /usr/local
mount /dev/mapper/oem /oem
mkdir -p /usr/local/.state/opt.bind
luet util unpack file://oem/.opt/stylus-image.tar /
umount /dev/mapper/persistent
umount /dev/mapper/oem
cryptsetup close /dev/mapper/persistent
cryptsetup close /dev/mapper/oem
: exit status 32
	* yaml: unmarshal errors:
  line 1: cannot unmarshal !!str `/usr/lo...` into schema.YipConfig
	* yaml: unmarshal errors:
  line 1: cannot unmarshal !!str `/usr/lo...` into schema.YipConfig


2024-04-06T09:44:38Z DBG Cloud-init paths set to [/system/oem /oem/ /usr/local/cloud-config/]
2024-04-06T09:44:38Z INF Running stage: kairos-uki-reset.after.before

2024-04-06T09:44:38Z INF Done executing stage 'kairos-uki-reset.after.before'

2024-04-06T09:44:38Z INF Running stage: kairos-uki-reset.after

2024-04-06T09:44:38Z INF Done executing stage 'kairos-uki-reset.after'

2024-04-06T09:44:38Z INF Running stage: kairos-uki-reset.after.after

2024-04-06T09:44:38Z INF Done executing stage 'kairos-uki-reset.after.after'

2024-04-06T09:44:38Z INF Running stage: kairos-uki-reset.after.before

2024-04-06T09:44:38Z INF Done executing stage 'kairos-uki-reset.after.before'

2024-04-06T09:44:38Z INF Running stage: kairos-uki-reset.after

2024-04-06T09:44:38Z INF Done executing stage 'kairos-uki-reset.after'

2024-04-06T09:44:38Z INF Running stage: kairos-uki-reset.after.after

2024-04-06T09:44:38Z INF Done executing stage 'kairos-uki-reset.after.after'

2024-04-06T09:44:38Z DBG Unmounting partition COS_GRUB
Provider stylus at /usr/local/bin/agent-provider-stylus had an error: error while executing plugin: fork/exec /usr/local/bin/agent-provider-stylus: no such file or directory

@Itxaka
Copy link
Member

Itxaka commented Apr 6, 2024

After the manual reset the file is no longer there, so something is happening in the automated one

@jimmykarily jimmykarily added prio: high and removed triage Add this label to issues that should be triaged and prioretized in the next planning call labels Apr 8, 2024
@Itxaka
Copy link
Member

Itxaka commented Apr 8, 2024

not happening in non-uki

@Itxaka Itxaka self-assigned this Apr 8, 2024
@Itxaka
Copy link
Member

Itxaka commented Apr 8, 2024

umm, master build from kairos alos doesnt trigger this, but I think that its missing a bumped package

boot mode is unknown when selecting autoreset:

kairos@localhost:~$ ls /run/cos/
cos-layout.env  uki_boot_mode  unknown

KAIROS_VERSION="v3.0.4-2-g3fba4f4"
KAIROS_VERSION_ID="v3.0.4-2-g3fba4f4"
KAIROS_FLAVOR="ubuntu"
KAIROS_TARGETARCH="amd64"
KAIROS_RELEASE="v3.0.4-2-g3fba4f4"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_NAME="kairos-standard-ubuntu-23.10"
KAIROS_ARTIFACT="kairos-ubuntu-23.10-standard-amd64-generic-v3.0.4-2-g3fba4f4"
KAIROS_FLAVOR_RELEASE="23.10"
KAIROS_MODEL="generic"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_IMAGE_REPO="quay.io/kairos/ubuntu:23.10-standard-amd64-generic-v3.0.4-2-g3fba4f4"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_VARIANT="standard"
KAIROS_ID_LIKE="kairos-standard-ubuntu-23.10"
KAIROS_PRETTY_NAME="kairos-standard-ubuntu-23.10 v3.0.4-2-g3fba4f4"
KAIROS_IMAGE_LABEL="23.10-standard-amd64-gen

@Itxaka
Copy link
Member

Itxaka commented Apr 8, 2024

immucore v0.1.23
agent v2.8.8

@Itxaka Itxaka mentioned this issue Apr 8, 2024
Merged
@Itxaka
Copy link
Member

Itxaka commented Apr 8, 2024

can confirm that kairos-io/kairos-sdk#83 fixes it by triggering the autoreset properly.

Is now on master @vipsharm and on framework v2.7.25

@Itxaka Itxaka closed this as completed Apr 8, 2024
@mudler mudler mentioned this issue Apr 10, 2024
Closed
25 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Itxaka Itxaka

Labels
area/agent bug Something isn't working prio: high uki verified
Projects
🧙Issue tracking board
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Itxaka @jimmykarily @vipsharm