verity-boot: append hash device to rootfs (#2142)

* verity-boot: append hash device to rootfs

Turned out that dev nodes for SCSI devices may not be
determenistic, where the hash device and rootfs may end
up appearing under /dev/sda and /dev/sdb respectively.

Instead of mounting a separate hash device, append the
verity Merkle tree to rootfs ext4 filesystem, similarly
to how it's done for layer VHDs and mount single VHD.
Remove redundant hash device code.

The default `GuestStateFile` filename was changed to `kernel.vmgs`.

Update the IVGM kernel init to reflect the changes.

The kernel command looks something like this:

 8250_core.nr_uarts=0 panic=-1 debug loglevel=7 root=/dev/dm-0 \
     dm-mod.create="dmverity,,,ro,0 173768 verity \
         1 /dev/sda /dev/sda 4096 4096 21721 21721 sha256 \
         42896a788a58da77b6acb8ddf53aa744bd269c19146cfdf48eb8fc5529a52e62 \
         a1c38923e44adffdd21f84e9185248c884fa28e767795d1025e5804e1c3df905" \
     init=/startup.sh

To break this down a little further:

  dm-mod.create="<name>,<uuid>,<minor>,<flags>,[table {verity_params}]"
  table="<start_sector> <num_sectors> <target_type> verity_params"
  verity_params="<version> <data_device> <hash_device> <data_block_size> \
                 <hash_block_size> <num_data_blocks> <hash_start_block> \
                 <algorithm> <root_digest> <salt> [<opt_params>]"

With the example above we get:

  name: "dmverity"
  uuid: ""
  minor: ""
  flags: "ro"
  table: 0 0 173768 verity <verity_params>
  verity_params:
    version: 1
    data_device: /dev/sda
    hash_device: /dev/sda
    data_block_size: 4096
    hash_block_size: 4096
    num_data_blocks: 21721
    hash_start_block: 21721
    algorithm: "sha256"
    root_digest: "42896a788a58da77b6acb8ddf53aa744bd269c19146cfdf48eb8fc5529a52e62"
    salt: "a1c38923e44adffdd21f84e9185248c884fa28e767795d1025e5804e1c3df905"

The support for booting non-SNP UVMs with dm-verity has also been added
as part of this PR. A new annotation can be used to pass the `dm-mod.create`
parameters to kernel. The assumption that the rootfs VHD will also have Merkle
tree appended after ext4 filesystem still holds. The new annotation is
"io.microsoft.virtualmachine.lcow.dmverity-create-args" and must be used
in conjunction with an existing "io.microsoft.virtualmachine.lcow.dmverity-mode"
annotation.

Add an internal "io.microsoft.virtualmachine.console.pipe" annotation, which
can be used to set the serial for the UVM for debugging purposes.

Note that dm-verity boot has a dependency on `CONFIG_DM_INIT` kernel config.

---------

Signed-off-by: Maksim An <maksiman@microsoft.com>

Makefile.bootfiles

@@ -20,6 +20,9 @@ IGVM_TOOL:=
 KERNEL_PATH:=
 TAR2EXT4_TOOL:=bin/cmd/tar2ext4
 
+ROOTFS_DEVICE:=/dev/sda
+HASH_DEVICE:=/dev/sdb
+
 .PHONY: all always rootfs test snp simple
 
 .DEFAULT_GOAL := all
@@ -32,7 +35,7 @@ clean:
 
 rootfs: out/rootfs.vhd
 
-snp: out/kernelinitrd.vmgs out/rootfs.hash.vhd out/rootfs.vhd out/v2056.vmgs
+snp: out/kernel.vmgs out/rootfs-verity.vhd out/v2056.vmgs out/v2056combined.vmgs
 
 simple: out/simple.vmgs snp
 
@@ -52,23 +55,103 @@ out/simple.bin: out/initrd.img $(PATH_PREFIX)/$(KERNEL_PATH) boot/startup_simple
 		-rdinit out/initrd.img \
 		-vtl 0
 
-ROOTFS_DEVICE:=/dev/sda
-VERITY_DEVICE:=/dev/sdb
-# Debug build for use with uvmtester. UVM with dm-verity protected vhd disk mounted directly via the kernel command line. Ignores corruption in dm-verity protected disk. (Use dmesg to see if dm-verity is ignoring data corruption.)
+# The boot performance is optimized by supplying rootfs as a SCSI attachment. In this case the kernel boots with
+# dm-verity to ensure the integrity. Similar to layer VHDs the verity Merkle tree is appended to ext4 filesystem.
+# It transpires that the /dev/sd* order is not deterministic wrt the scsi device order. Thus build a single userland
+# fs + merkle tree device and boot that.
+#
+# From https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/dm-init.html
+#
+# dm-mod.create=<name>,<uuid>,<minor>,<flags>,<table>[,<table>+][;<name>,<uuid>,<minor>,<flags>,<table>[,<table>+]+]
+#
+# where:
+# <name>          ::= The device name.
+# <uuid>          ::= xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | ""
+# <minor>         ::= The device minor number | ""
+# <flags>         ::= "ro" | "rw"
+# <table>         ::= <start_sector> <num_sectors> <target_type> <target_args>
+# <target_type>   ::= "verity" | "linear" | ... (see list below)
+#
+# From https://docs.kernel.org/admin-guide/device-mapper/verity.html
+# <version> <dev> <hash_dev>
+# <data_block_size> <hash_block_size>
+# <num_data_blocks> <hash_start_block>
+# <algorithm> <digest> <salt>
+# [<#opt_params> <opt_params>]
+#
+# typical igvm tool line once all the macros are expanded
+# python3 /home/user/igvmfile.py -o out/v2056.bin -kernel /hose/user/bzImage -append "8250_core.nr_uarts=0 panic=-1 debug loglevel=9 ignore_loglevel dev.scsi.logging_level=9411 root=/dev/dm-0 dm-mod.create=\"dmverity,,,ro,0 196744 verity 1 /dev/sda /dev/sdb 4096 4096 24593 0 sha256 6d625a306aafdf73125a84388b7bfdd2c3a154bd8d698955f4adffc736bdfd66 b9065c23231f0d8901cc3a68e1d3b8d624213e76d6f9f6d3ccbcb829f9c710ba 1 ignore_corruption\" init=/startup_v2056.sh"  -vtl 0
+#
+# so a kernel command line of:
+# 8250_core.nr_uarts=0 panic=-1 debug loglevel=9 ignore_loglevel dev.scsi.logging_level=9411 root=/dev/dm-0 dm-mod.create=\"dmverity,,,ro,0 196744 verity 1 /dev/sda /dev/sdb 4096 4096 24593 0 sha256 6d625a306aafdf73125a84388b7bfdd2c3a154bd8d698955f4adffc736bdfd66 b9065c23231f0d8901cc3a68e1d3b8d624213e76d6f9f6d3ccbcb829f9c710ba 1 ignore_corruption\" init=/startup_v2056.sh
+#
+# and a dm-mod.create of:
+# dmverity,,,ro,0 196744 verity 1 /dev/sda /dev/sdb 4096 4096 24593 0 sha256 6d625a306aafdf73125a84388b7bfdd2c3a154bd8d698955f4adffc736bdfd66 b9065c23231f0d8901cc3a68e1d3b8d624213e76d6f9f6d3ccbcb829f9c710ba 1 ignore_corruption
+#
+# which breaks down to:
+#
+# name = "dmverity"
+# uuid = ""
+# minor = ""
+# flags = "ro"
+# table = 0 196744 verity "args"
+#     start_sector = 0
+#     num_sectors = 196744
+#     target_type = verity
+#     target_args = 1 /dev/sda /dev/sdb 4096 4096 24593 0 sha256 6d625a306aafdf73125a84388b7bfdd2c3a154bd8d698955f4adffc736bdfd66 b9065c23231f0d8901cc3a68e1d3b8d624213e76d6f9f6d3ccbcb829f9c710ba 1 ignore_corruption
+# args:
+#     version               1
+#     dev                   /dev/sda
+#     hash_dev              /dev/sdb
+#     data_block_size       4096
+#     hash_block_size       4096
+#     num_data_blocks       24593
+#     hash_start_block      0
+#     algorithm             sha256
+#     digest                6d625a306aafdf73125a84388b7bfdd2c3a154bd8d698955f4adffc736bdfd66
+#     salt                  b9065c23231f0d8901cc3a68e1d3b8d624213e76d6f9f6d3ccbcb829f9c710ba
+#     opt_params
+#         count = 1
+#         ignore_corruption
+#
+# combined typical (not bigger count of sectors for the whole device)
+# dmverity,,,ro,0 199672 verity 1 /dev/sda /dev/sda 4096 4096 24959 24959 sha256 4aa6e79866ee946ddbd9cddd6554bc6449272942fcc65934326817785a3bd374 adc4956274489c936395bab046a2d476f21ef436e571ba53da2fdf3aee59bf0a
+#
+# A few notes:
+#  - num_sectors is the size of the final (aka target) verity device, i.e. the size of our rootfs excluding the Merkle
+#    tree.
+#  - We don't add verity superblock, so the <hash_start_block> will be exactly at the end of ext4 filesystem and equal
+#    to its size. In the case when verity superblock is present an extra block should be added to the offset value,
+#    i.e. 24959 becomes 24960.
+
+
+# Debug build for use with uvmtester. UVM with dm-verity protected vhd disk mounted directly via the kernel command line.
+# Ignores corruption in dm-verity protected disk. (Use dmesg to see if dm-verity is ignoring data corruption.)
 out/v2056.bin: out/rootfs.vhd out/rootfs.hash.vhd $(PATH_PREFIX)/$(KERNEL_PATH) out/rootfs.hash.datasectors out/rootfs.hash.datablocksize out/rootfs.hash.hashblocksize out/rootfs.hash.datablocks out/rootfs.hash.rootdigest out/rootfs.hash.salt boot/startup_v2056.sh
 	rm -f $@
 	python3 $(PATH_PREFIX)/$(IGVM_TOOL) \
-		-o $@ -kernel $(PATH_PREFIX)/$(KERNEL_PATH) \
-		-append "8250_core.nr_uarts=0 panic=-1 debug loglevel=7 root=/dev/dm-0 dm-mod.create=\"dmverity,,,ro,0 $(shell cat out/rootfs.hash.datasectors) verity 1 $(ROOTFS_DEVICE) $(VERITY_DEVICE) $(shell cat out/rootfs.hash.datablocksize) $(shell cat out/rootfs.hash.hashblocksize) $(shell cat out/rootfs.hash.datablocks) 0 sha256 $(shell cat out/rootfs.hash.rootdigest) $(shell cat out/rootfs.hash.salt) 1 ignore_corruption\" init=/startup_v2056.sh" \
+		-o $@ \
+		-kernel $(PATH_PREFIX)/$(KERNEL_PATH) \
+		-append "8250_core.nr_uarts=0 panic=-1 debug loglevel=9 root=/dev/dm-0 dm-mod.create=\"dmverity,,,ro,0 $(shell cat out/rootfs.hash.datasectors) verity 1 $(ROOTFS_DEVICE) $(HASH_DEVICE) $(shell cat out/rootfs.hash.datablocksize) $(shell cat out/rootfs.hash.hashblocksize) $(shell cat out/rootfs.hash.datablocks) $(shell cat out/rootfs.hash.datablocks) sha256 $(shell cat out/rootfs.hash.rootdigest) $(shell cat out/rootfs.hash.salt) 1 ignore_corruption\" init=/startup_v2056.sh" \
+		-vtl 0
+
+out/v2056combined.bin: out/rootfs-verity.vhd $(PATH_PREFIX)/$(KERNEL_PATH) out/rootfs.hash.datablocksize out/rootfs.hash.hashblocksize out/rootfs.hash.datablocks out/rootfs.hash.rootdigest out/rootfs.hash.salt boot/startup_v2056.sh
+	rm -f $@
+	echo root=/dev/dm-0 dm-mod.create=\"dmverity,,,ro,0 $(shell cat out/rootfs.hash.datasectors) verity 1 $(ROOTFS_DEVICE) $(ROOTFS_DEVICE) $(shell cat out/rootfs.hash.datablocksize) $(shell cat out/rootfs.hash.hashblocksize) $(shell cat out/rootfs.hash.datablocks) $(shell cat out/rootfs.hash.datablocks) sha256 $(shell cat out/rootfs.hash.rootdigest) $(shell cat out/rootfs.hash.salt) 1 ignore_corruption\"
+	python3 $(PATH_PREFIX)/$(IGVM_TOOL) \
+		-o $@ \
+		-kernel $(PATH_PREFIX)/$(KERNEL_PATH) \
+		-append "8250_core.nr_uarts=0 panic=-1 debug loglevel=9 ignore_loglevel dev.scsi.logging_level=9411 root=/dev/dm-0 dm-mod.create=\"dmverity,,,ro,0 $(shell cat out/rootfs.hash.datasectors) verity 1 $(ROOTFS_DEVICE) $(ROOTFS_DEVICE) $(shell cat out/rootfs.hash.datablocksize) $(shell cat out/rootfs.hash.hashblocksize) $(shell cat out/rootfs.hash.datablocks) $(shell cat out/rootfs.hash.datablocks) sha256 $(shell cat out/rootfs.hash.rootdigest) $(shell cat out/rootfs.hash.salt) 1 ignore_corruption\" init=/startup_v2056.sh" \
 		-vtl 0
 
 # Full UVM with dm-verity protected vhd disk mounted directly via the kernel command line.
-out/kernelinitrd.bin: out/rootfs.vhd out/rootfs.hash.vhd out/rootfs.hash.datasectors out/rootfs.hash.datablocksize out/rootfs.hash.hashblocksize out/rootfs.hash.datablocks out/rootfs.hash.rootdigest out/rootfs.hash.salt $(PATH_PREFIX)/$(KERNEL_PATH) boot/startup.sh
+out/kernel.bin: out/rootfs-verity.vhd $(PATH_PREFIX)/$(KERNEL_PATH) out/rootfs.hash.datasectors out/rootfs.hash.datablocksize out/rootfs.hash.hashblocksize out/rootfs.hash.datablocks out/rootfs.hash.rootdigest out/rootfs.hash.salt boot/startup.sh
 	rm -f $@
+	echo root=/dev/dm-0 dm-mod.create=\"dmverity,,,ro,0 $(shell cat out/rootfs.hash.datasectors) verity 1 $(ROOTFS_DEVICE) $(ROOTFS_DEVICE) $(shell cat out/rootfs.hash.datablocksize) $(shell cat out/rootfs.hash.hashblocksize) $(shell cat out/rootfs.hash.datablocks) $(shell cat out/rootfs.hash.datablocks) sha256 $(shell cat out/rootfs.hash.rootdigest) $(shell cat out/rootfs.hash.salt)\"
 	python3 $(PATH_PREFIX)/$(IGVM_TOOL) \
 		-o $@ \
 		-kernel $(PATH_PREFIX)/$(KERNEL_PATH) \
-		-append "8250_core.nr_uarts=0 panic=-1 debug loglevel=7 root=/dev/dm-0 dm-mod.create=\"dmverity,,,ro,0 $(shell cat out/rootfs.hash.datasectors) verity 1 $(ROOTFS_DEVICE) $(VERITY_DEVICE) $(shell cat out/rootfs.hash.datablocksize) $(shell cat out/rootfs.hash.hashblocksize) $(shell cat out/rootfs.hash.datablocks) 0 sha256 $(shell cat out/rootfs.hash.rootdigest) $(shell cat out/rootfs.hash.salt)\" init=/startup.sh" \
+		-append "8250_core.nr_uarts=0 panic=-1 debug loglevel=7 root=/dev/dm-0 dm-mod.create=\"dmverity,,,ro,0 $(shell cat out/rootfs.hash.datasectors) verity 1 $(ROOTFS_DEVICE) $(ROOTFS_DEVICE) $(shell cat out/rootfs.hash.datablocksize) $(shell cat out/rootfs.hash.hashblocksize) $(shell cat out/rootfs.hash.datablocks) $(shell cat out/rootfs.hash.datablocks) sha256 $(shell cat out/rootfs.hash.rootdigest) $(shell cat out/rootfs.hash.salt)\" init=/startup.sh" \
 		-vtl 0
 
 # Rule to make a vhd from a file. This is used to create the rootfs.hash.vhd from rootfs.hash.
@@ -97,6 +180,10 @@ out/rootfs.ext4: out/rootfs.tar.gz $(TAR2EXT4_TOOL)
 	gzip -f -d ./out/rootfs.tar.gz
 	$(TAR2EXT4_TOOL) -i ./out/rootfs.tar -o $@
 
+out/rootfs-verity.ext4: out/rootfs.ext4 out/rootfs.hash
+	cp out/rootfs.ext4 $@
+	cat out/rootfs.hash >> $@
+
 out/rootfs.tar.gz: out/initrd.img
 	rm -rf rootfs-conv
 	mkdir rootfs-conv

internal/annotations/annotations.go

@@ -44,4 +44,8 @@ const (
 
 	// ExtraVSockPorts adds additional ports to the list of ports that the UVM is allowed to use.
 	ExtraVSockPorts = "io.microsoft.virtualmachine.lcow.extra-vsock-ports"
+
+	// UVMConsolePipe is the name of the named pipe that the UVM console is connected to. This works only for non-SNP
+	// scenario, for SNP use a debugger.
+	UVMConsolePipe = "io.microsoft.virtualmachine.console.pipe"
 )

internal/oci/uvm.go

@@ -209,13 +209,14 @@ func handleSecurityPolicy(ctx context.Context, a map[string]string, lopts *uvm.O
 		lopts.SecurityPolicyEnabled = true
 
 		// There are two possible ways to boot SNP mode. Either kernelinitrd.vmgs which consists of kernel plus initrd.cpio.gz
-		// Or a kernel vmgs file (without an initrd) plus a separate vhd file which is dmverity protected via a hash vhd file.
+		// Or a kernel.vmgs file (without an initrd) plus a separate vhd file which is dmverity protected via a hash tree
+		// appended to rootfs ext4 filesystem.
 		// We only currently support using the dmverity scheme. Note that the dmverity file name may be explicitly specified via
 		// an annotation this is deliberately not the same annotation as the non-SNP rootfs vhd file.
+		// The default behavior is to use kernel.vmgs and a rootfs-verity.vhd file with Merkle tree appended to ext4 filesystem.
 		lopts.PreferredRootFSType = uvm.PreferredRootFSTypeNA
 		lopts.RootFSFile = ""
 		lopts.DmVerityRootFsVhd = uvm.DefaultDmVerityRootfsVhd
-		lopts.DmVerityHashVhd = uvm.DefaultDmVerityHashVhd
 		lopts.DmVerityMode = true
 	}
 
@@ -278,6 +279,7 @@ func SpecToUVMCreateOpts(ctx context.Context, s *specs.Spec, id, owner string) (
 		lopts.UVMReferenceInfoFile = ParseAnnotationsString(s.Annotations, annotations.UVMReferenceInfoFile, lopts.UVMReferenceInfoFile)
 		lopts.KernelBootOptions = ParseAnnotationsString(s.Annotations, annotations.KernelBootOptions, lopts.KernelBootOptions)
 		lopts.DisableTimeSyncService = ParseAnnotationsBool(ctx, s.Annotations, annotations.DisableLCOWTimeSyncService, lopts.DisableTimeSyncService)
+		lopts.ConsolePipe = ParseAnnotationsString(s.Annotations, iannotations.UVMConsolePipe, lopts.ConsolePipe)
 		handleAnnotationPreferredRootFSType(ctx, s.Annotations, lopts)
 		handleAnnotationKernelDirectBoot(ctx, s.Annotations, lopts)
 		handleAnnotationFullyPhysicallyBacked(ctx, s.Annotations, lopts)
@@ -286,11 +288,11 @@ func SpecToUVMCreateOpts(ctx context.Context, s *specs.Spec, id, owner string) (
 		// Eg VMPem device count, overridden kernel option cannot be respected.
 		handleSecurityPolicy(ctx, s.Annotations, lopts)
 
-		// override the default GuestState and DmVerityRootFs/HashVhd filenames if specified
+		// override the default GuestState and DmVerityRootFs filenames if specified
 		lopts.GuestStateFile = ParseAnnotationsString(s.Annotations, annotations.GuestStateFile, lopts.GuestStateFile)
 		lopts.DmVerityRootFsVhd = ParseAnnotationsString(s.Annotations, annotations.DmVerityRootFsVhd, lopts.DmVerityRootFsVhd)
-		lopts.DmVerityHashVhd = ParseAnnotationsString(s.Annotations, annotations.DmVerityHashVhd, lopts.DmVerityHashVhd)
 		lopts.DmVerityMode = ParseAnnotationsBool(ctx, s.Annotations, annotations.DmVerityMode, lopts.DmVerityMode)
+		lopts.DmVerityCreateArgs = ParseAnnotationsString(s.Annotations, annotations.DmVerityCreateArgs, lopts.DmVerityCreateArgs)
 		// Set HclEnabled if specified. Else default to a null pointer, which is omitted from the resulting JSON.
 		lopts.HclEnabled = ParseAnnotationsNullableBool(ctx, s.Annotations, annotations.HclEnabled)
 		return lopts, nil

internal/uvm/create_lcow.go

@@ -69,20 +69,20 @@ const (
 	InitrdFile = "initrd.img"
 	// VhdFile is the default file name for a rootfs.vhd used to boot LCOW.
 	VhdFile = "rootfs.vhd"
-	// DmVerityVhdFile is the default file name for a dmverity_rootfs.vhd which
-	// is mounted by the GuestStateFile during boot and used as the root file
-	// system when booting in the SNP case.
+	// DefaultDmVerityRootfsVhd is the default file name for a dmverity_rootfs.vhd,
+	// which is mounted by the GuestStateFile during boot and used as the root file
+	// system when booting in the SNP case. Similar to layer VHDs, the Merkle tree
+	// is appended after ext4 filesystem ends.
 	DefaultDmVerityRootfsVhd = "rootfs.vhd"
-	DefaultDmVerityHashVhd   = "rootfs.hash.vhd"
 	// KernelFile is the default file name for a kernel used to boot LCOW.
 	KernelFile = "kernel"
 	// UncompressedKernelFile is the default file name for an uncompressed
 	// kernel used to boot LCOW with KernelDirect.
 	UncompressedKernelFile = "vmlinux"
 	// GuestStateFile is the default file name for a vmgs (VM Guest State) file
-	// which combines kernel and initrd and is used to mount DmVerityVhdFile
+	// which contains the kernel and kernel command which mounts DmVerityVhdFile
 	// when booting in the SNP case.
-	GuestStateFile = "kernelinitrd.vmgs"
+	GuestStateFile = "kernel.vmgs"
 	// UVMReferenceInfoFile is the default file name for a COSE_Sign1
 	// reference UVM info, which can be made available to workload containers
 	// and can be used for validation purposes.
@@ -98,8 +98,8 @@ type ConfidentialOptions struct {
 	UVMReferenceInfoFile   string // Filename under `BootFilesPath` for (potentially signed) UVM image reference information.
 	BundleDirectory        string // pod bundle directory
 	DmVerityRootFsVhd      string // The VHD file (bound to the vmgs file via embedded dmverity hash data file) to load.
-	DmVerityHashVhd        string // The VHD file containing the hash tree
 	DmVerityMode           bool   // override to be able to turn off dmverity for debugging
+	DmVerityCreateArgs     string // set dm-verity args when booting with verity in non-SNP mode
 }
 
 // OptionsLCOW are the set of options passed to CreateLCOW() to create a utility vm.
@@ -362,12 +362,6 @@ func makeLCOWVMGSDoc(ctx context.Context, opts *OptionsLCOW, uvm *UtilityVM) (_
 		return nil, fmt.Errorf("the DM Verity VHD file '%s' was not found", dmVerityRootfsTemplatePath)
 	}
 
-	// The root file system comes from the dmverity vhd file which is mounted by the initrd in the vmgs file.
-	dmVerityHashTemplatePath := filepath.Join(opts.BootFilesPath, opts.DmVerityHashVhd)
-	if _, err := os.Stat(dmVerityHashTemplatePath); os.IsNotExist(err) {
-		return nil, fmt.Errorf("the DM Verity Hash file '%s' was not found", dmVerityHashTemplatePath)
-	}
-
 	var processor *hcsschema.Processor2
 	processor, err = fetchProcessor(ctx, opts, uvm)
 	if err != nil {
@@ -394,17 +388,10 @@ func makeLCOWVMGSDoc(ctx context.Context, opts *OptionsLCOW, uvm *UtilityVM) (_
 		}
 	}()
 
-	dmVerityHashFullPath := filepath.Join(opts.BundleDirectory, DefaultDmVerityHashVhd)
-	if err := copyfile.CopyFile(ctx, dmVerityHashTemplatePath, dmVerityHashFullPath, true); err != nil {
-		return nil, fmt.Errorf("failed to copy DM Verity hash template file: %w", err)
-	}
-	defer func() {
-		if err != nil {
-			os.Remove(dmVerityHashFullPath)
-		}
-	}()
-
-	for _, filename := range []string{vmgsFileFullPath, dmVerityRootFsFullPath, dmVerityHashFullPath} {
+	for _, filename := range []string{
+		vmgsFileFullPath,
+		dmVerityRootFsFullPath,
+	} {
 		if err := security.GrantVmGroupAccessWithMask(filename, security.AccessMaskAll); err != nil {
 			return nil, fmt.Errorf("failed to grant VM group access ALL: %w", err)
 		}
@@ -491,16 +478,10 @@ func makeLCOWVMGSDoc(ctx context.Context, opts *OptionsLCOW, uvm *UtilityVM) (_
 							Path:     dmVerityRootFsFullPath,
 							ReadOnly: true,
 						},
-						"1": {
-							Type_:    "VirtualDisk",
-							Path:     dmVerityHashFullPath,
-							ReadOnly: true,
-						},
 					},
 				},
 			}
 			uvm.reservedSCSISlots = append(uvm.reservedSCSISlots, scsi.Slot{Controller: 0, LUN: 0})
-			uvm.reservedSCSISlots = append(uvm.reservedSCSISlots, scsi.Slot{Controller: 0, LUN: 1})
 		}
 	}
 
@@ -738,6 +719,12 @@ func makeLCOWDoc(ctx context.Context, opts *OptionsLCOW, uvm *UtilityVM) (_ *hcs
 			}
 		} else {
 			kernelArgs = "root=/dev/sda ro rootwait init=/init"
+			if opts.DmVerityMode {
+				if len(opts.DmVerityCreateArgs) == 0 {
+					return nil, errors.New("DmVerityCreateArgs must be set when DmVerityMode is true and not booting from a vmgs file.")
+				}
+				kernelArgs = fmt.Sprintf("root=/dev/dm-0 dm-mod.create=%q init=/init", opts.DmVerityCreateArgs)
+			}
 			doc.VirtualMachine.Devices.Scsi[guestrequest.ScsiControllerGuids[0]].Attachments["0"] = hcsschema.Attachment{
 				Type_:    "VirtualDisk",
 				Path:     rootfsFullPath,

pkg/annotations/annotations.go

@@ -266,14 +266,18 @@ const (
 	// GuestStateFile specifies the path of the vmgs file to use if required. Only applies in SNP mode.
 	GuestStateFile = "io.microsoft.virtualmachine.lcow.gueststatefile"
 
-	// DmVerityVhdFile specifies the path of the VHD (with embedded dmverity data) file to use if required. Only applies in SNP mode.
-
+	// DmVerityRootFsVhd specifies the path of the VHD (with embedded dmverity data) file to use if required.
+	// Only applies in SNP mode.
 	DmVerityRootFsVhd = "io.microsoft.virtualmachine.lcow.dmverity-rootfs-vhd"
-	DmVerityHashVhd   = "io.microsoft.virtualmachine.lcow.dmverity-hash-vhd"
 
-	// DmVerityMode
+	// DmVerityMode specifies whether the rootfs is expected to be presented as a standalone SCSI attachment,
+	// in which case the UVM boots with dm-verity.
 	DmVerityMode = "io.microsoft.virtualmachine.lcow.dmverity-mode"
 
+	// DmVerityCreateArgs specifies the `dm-mod.create` parameters to kernel and enables integrity protection of
+	// the rootfs.
+	DmVerityCreateArgs = "io.microsoft.virtualmachine.lcow.dmverity-create-args"
+
 	// UVMSecurityPolicyEnv specifies if confidential containers' related information
 	// should be written to containers' rootfs. The filenames and location are defined
 	// by securitypolicy.PolicyFilename, securitypolicy.HostAMDCertFilename and