installer: unable to boot from second drive when installing ZFS mirror

[crealte]

I have read the contributing guidelines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
I have searched the existing issues and I am convinced that mine is new.

Describe the bug:
Unable to boot from second drive when installing with ZFS mirror on installer. Upon completion of install and rebooting gracefully, the UEFI menu only presents drive 1 as a bootable option. Upon removal of drive 1, drive 2 which was installed as part of the mirror does not boot into opnsense.

Hardware Tested:

• Generic Mini PC (1xNVME 1xSATA)
• Supermicro Industrial PC (2xSATA)

Attempted on multiple different environments. Both of these devices work with ZFS mirror boot configurations as tested with baremetal mirror ZFS Proxmox install.

To Reproduce:
Install ZFS mirror onto two drives.

Expected behavior
ZFS mirror install should be able to boot from either drive.

Workaround:
Performing UFS install and then re-installing with ZFS mirror seems to fix the install and both drives are boot options.

Environment
OPNsense-23.7
OPNsense-23.1
OPNsense-22.7

Did not try earlier releases

Tried DVD, VGA, and NANO images, same behavior exhibited.

[fichtner]

Does this also happen when you use the other install method provided by FreeBSD (not the main hybrid option). I would suspect it does so there is an issue in the ZFS install glue of FreeBSD. If not it should be easy to spot.

[fichtner]

Also important: does it work if you turn UEFI off (hybrid mode install only)

[fichtner]

Similar issue here https://forum.opnsense.org/index.php?topic=37058.0

[crealte]

I installed the ZFS mirror in hybrid mode with FreeBSD 13.2 and experienced the same issue where only one device has a boot option.

Same issue when I installed the ZFS mirror in the default hybrid mode with opnsense, The default hybrid mode is GPT(BIOS+UEFI). I disabled UEFI boot in the bios and didn't see any boot options at all.

I don't see any option for an another hybrid mode other than GPT+ACTIVE(BIOS). That one also didn't yield a different result.

To this point I have been using Ventoy so just to rule that out as an issue I made a dedicated bootable USB with the latest opnsense and the same issue was present. I'm happy to try any other steps to troubleshoot further, perhaps I should take the issue to FreeBSD? I couldn't find any active cases of this issue.

Also wondering if I'm missing something or if this really is a bug, curious if anyone else can reproduce? I've tried three different PCs now. If it is an actual issue I'm afraid there's a lot of people out there who do a mirror install but never actually test it afterwards.

[fichtner]

FreeBSD doesn't have a "hybrid" mode.

I guess this all confirms there is a standard installer issue with the first disk getting in the way which is a bit surprising and I can try to reproduce, but it will have to wait till January when we traditionally start testing images for the new major release.

The picture may be different on FreeBSD 14.0, but I haven't checked that out yet either.

[crealte]

I did a clean install of 24.1 and the same issue is observed.

For anyone who is wondering, the workaround I mentioned in the original post still works.