HACKING.md

Rules

• commands in bin/ document all actual commands being executed

Tools

Tools that can be used to debug or work with the underlying tech

socat

socat unix-connect:/tmp/qga-t1.sock readline

For the qemu monitor

socat -,echo=0,icanon=0 unix-connect:qemu-monitor.sock

for QMP commands to the agent inside the VM.

guestfish

Part of guestfs-tools, used to create and manipulate VM disks

Provided commands

Commands in bin/commands/

Terminal inside the guest

stty rows 46 cols 189

Collection of commands

# kill build-base-image
poweroff-image.xsh --name @($(ps aux | grep qemu-system-x86_64 | grep -oP -- '--name [^ ]+' | cut -d " " -f2).rstrip())
# build base with logging
build-base-image.xsh --handle d1 --initial_build false --build_nested true --extract_nested true --extract_l1_assets true 2>&1 | tee -a @($(date +"%M%S%H-%a.build.log").rstrip()[1:-1])
# send output of script to virtio socket
./load1.xsh | socat -t 2 stdin,null-eof,escape=0 ./org.fedoraproject.port.0,end-close
socat -v -ddd -t 2 exec:/root/load1.xsh pipe:/dev/vport3p1,wronly,shut-down

# ctrl+a, c, then to add a chardev:
chardev-add socket,id=char0,path=/tmp/vfsd.sock
# beforehand, virtiofsd:
unshare --fork --pid --mount-proc --kill-child=SIGTERM --map-auto --map-root-user --setuid 0 --setgid 0 env -i /usr/lib/virtiofsd --socket-path=/tmp/vfsd.soc
k --shared-dir /mnt --announce-submounts --sandbox none

device_add pcie-root-port,id=myroot,slot=0
device_add vfio-pci,host=01:10.3,id=myid,bus=root
#device_add vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,bus=myroot,addr=01:00

Commands generated by other tools

By guestfish mount:

/usr/bin/qemu-system-x86_64 \
    -global virtio-blk-pci.scsi=off \
    -no-user-config \
    -nodefaults \
    -display none \
    -machine accel=kvm:tcg,graphics=off \
    -cpu max,la57=off \
    -m 1280 \
    -no-reboot \
    -rtc driftfix=slew \
    -no-hpet \
    -global kvm-pit.lost_tick_policy=discard \
    -kernel /var/tmp/.guestfs-0/appliance.d/kernel \
    -initrd /var/tmp/.guestfs-0/appliance.d/initrd \
    -object rng-random,filename=/dev/urandom,id=rng0 \
    -device virtio-rng-pci,rng=rng0 \
    -device virtio-scsi-pci,id=scsi \
    -drive file.file.filename=/tmp/libguestfsfX2stW/overlay1.qcow2,file.driver=qcow2,file.backing.file.locking=off,cache=unsafe,id=hd0,if=none \
    -device scsi-hd,drive=hd0 \
    -drive file=/var/tmp/.guestfs-0/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none,format=raw \
    -device scsi-hd,drive=appliance \
    -device virtio-serial-pci \
    -serial stdio \
    -chardev socket,path=/tmp/libguestfsgGvgKK/guestfsd.sock,id=channel0 \
    -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
    -append "panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=eb98a1dc-2bcf-48e1-83e3-9b804efd7d16 selinux=0 guestfs_verbose=1 TERM=vt220"
qemu-system-x86_64: -no-hpet: warning: -no-hpet is deprecated, use '-machine hpet=off' instead

TODOs

• add in /usr/lib/guestfs/supermin.d/excludefiles -/var/cache/pacman/pkg/* in bootstrap host
• execute libguestfs-test-tool in bootstrapping

Next

• turn the cluster inside-out for invertion of control, i.e. turn it into a library
  • write the python code for the minicluster minicluster
• create the concept of a cluster "project" where files reside, instead of CWD
  • for fstab (instead of /tmp)
  • for kernel and initramfs (instead of cmd)
  • for qcow2 images
  • for downloaded packages (instead of /tmp)
  • for meta-data about the project
    • disk specifications
• create the concept of cluster layout and config
• add inbound networking via different networking modes
• cache packages and reuse them
  • make a package cache image with the commands, new minicluster "arch-cacher"
  • introduce this image in a "base-image" cluster
• build itself inside L1
• move to btrfs
• detect differences between layout spec and actual spec and issue commands

Mid-term

• create a shell bridge which can execute commands inside the vm interactively
  • this bridge is also available as a library, reusable in other python code
• create a connector for ansible to provision such systems (without sshd or the like)

Bigger plans

• an architecture based on command pattern
• the commands are submitted to a daemon (master), who takes care of the actual execution and error handling
• connect clusters on different hardware machines and make them act as one
• all commands are executed by the master process, and client libraries just generate commands to be submitted to it
  • goal: have integrations in various languages: python, js, rust, zig, java, scala, php
• easily wrap different open-source applications in their own appliances and ability to recombine them
• easily write cluster-aware applications in these languages
• generating the cluster artifacts also generates library code allowing the cluster to be steered in that programming language: specific to the concrete layout
  • ability to say in code things like "for all worker servers in python, update the ML models"
• have an UI
  • plug parameters into cluster layouts and spin up customized clusters

Dependencies

pacman -S qemu-base arch-install-scripts libguestfs guestfs-tools

fusermount: option allow_other only allowed if 'user_allow_other' is set in /etc/fuse.conf

python pip: xonsh and xpip autoxsh

Cluster specs

Static top level keys:

• disks
• networking
• machine-types (configurations of disks, cpu, ram, networking, provisioning)

Dynamic:

• boot up a machine of a machine-type
• attach, detach: disks, networking (ram, cpu?)
• interactive commands
• migrate machines across hosts
• generate ansible inventory

Use Cases

Self-Contained minicluster

Goal: the minicluster layout is able to manage itself in the whole lifecycle:

• bootstrapping
• testing
• cleanup
• promoting
• automatically rebuild when new packages arrive
• protocol all changes done to packages

Goal: use the minicluster testing repositories

• same as the minicluster layout, but use the testing repositories
• TODO: evaluate first, should not lead to many breakages
• chain this layout "minicluster-testing" to layout "minicluster", only once "testing" passes, trigger a rebuild/etc of "minicluster"

Repository builder

Goal: given a list of packages, cache all of them, test them, and make a repository for consumption by regular arch installations

Goal: tests various installations and configurations

Goal: builds and uses a "fat image" which does not require re-downloading the packages