Adapt disposable documentation up to 4.3

- Cleared misunderstandings of disposable templates, this word was
  sometimes used interchangeably with disposables;
- Deduplicate content as much as possible;
  - Reference other pages or previous sections;
  - Assume the user has read the previous sections up until the
    current section they are reading;
  - All this deduplication enables easier reading, less clutter, but
    each section doesn't stand by itself, each page does. Some things
    were reinforced on multiple sections, but limited to when
    extremely necessary to learn by reinforcement;
- Structure the files to their distinct use cases:
  - how-to-use-disposables is just the basics to learn what are
    disposables and how to use them, using GUI and CLI alternatives;
  - disposable-customization has everything specific to different
    disposables templates as well their creation and configuration;
  - disposable-implementation assumes the user understand the previous
    pages and contains a more technical description of disposables and
    preloaded disposables as well as their implementation;
- Prepared usage for a GUIVM setup, avoiding mentions to dom0;
- Usage modes has been organized in GUI first and CLI second, grouped by
  origin such as GUIVM and app qube;
- Usage page restricts itself to "default-dvm", it's name is not
  variable like the Whonix disposable template variant and it is the
  most used disposable template, thus we avoid variables replacement
  that the user should think of the value to a definitive value to use
  when learning. Customization page could not benefit much from this
  though, as it is intended to create alternative disposable templates;
- Usage page links to Tails documentation of why using it on a VM is not
  amensiac, so it can be referenced for users that ask to have Tails in
  a VM for anti-forensics purposes;
- Updated the images to R4.3, nobody deserves to see the Qubes version
  of the Xfce application menu anymore when the new app menu rocks. The
  images also had to be updated because it was fullscreen screenshot
  on a large resolution, making it very difficult to read unless
  opening the image in a new tab and zooming in;
- Implementation page shows preloaded disposables alternatives that were
  once used or considered as a comparative for future studies, to answer
  why a different option is not being used and what requirements a
  replacement must meet;
- Content from previous releases have been deleted;
- Updates to Qrexec policy v4;
- Use rST roles;
- Change non-ASCII quotes and em-dashes to ASCII;
- Standardized text writing style, many people have contributed to these
  pages over the years, there is a lot of different writings styles
  which make the text difficult to map. I rewrote a lot of the
  paragraphs to my liking and understanding of the different stages that
  a user may go through when using disposables, and even on paragraphs
  that I didn't completely write, it was modified to follow the same
  standard with the rest of the pages. The usage must be very simple
  with just the basics, we don't want to scare the user's off of using
  disposables, we want to convince them. The customization is a bit more
  advanced, it assumes a lot more knowledge from the user to make
  decisions by themselves such as customizing applications and qube
  settings, which can affect system security. The implementation details
  do not require reading the code, it just exist to explain what is
  being used and why it was chosen.

Fixes: QubesOS/qubes-issues#10282
For: QubesOS/qubes-issues#1512

Author: ben-grande

developer/services/disposablevm-implementation.rst

@@ -2,73 +2,129 @@
 Disposable implementation
 =========================
 
+.. warning::
 
-**Note: The content below applies to Qubes R3.2.**
+      This page is intended for advanced users.
 
-DisposableVM image preparation
-------------------------------
+Disposable behavior
+-------------------
 
 
-DisposableVM is not started like other VMs, by executing equivalent of ``xl create`` - it would be too slow. Instead, DisposableVM are started by restore from a savefile.
+A :term:`disposable template` is an :term:`app qube` with the property ``template_for_dispvms`` enabled.
 
-Preparing a savefile is done by ``/usr/lib/qubes/qubes_prepare_saved_domain.sh`` script. It takes two mandatory arguments, appvm name (APPVM) and the savefile name, and optional path to “prerun” script. The script executes the following steps:
+A :term:`disposable` qube is a qube with the :doc:`DispVM </core-admin:qubes-vm/dispvm` class based on a disposable template. Every disposable type has all of its volumes configured to disable ``save_on_stop``, therefore no changes are saved on shutdown. Unnamed disposables enables the property ``auto_cleanup`` by default, thus automatically removes the qube upon shutdown.
 
-1. APPVM is started by ``qvm-start``
+Named disposables are useful for service qubes, as referencing static names is easier when the qube name is mentioned on Qrexec policies (:file:`qubes.UpdatesProxy` target) or as a property of another qube, such as a disposable :term:`net qube` which is referenced by downstream clients in the ``netvm`` property.
 
-2. xenstore key ``/local/domain/appvm_domain_id/qubes_save_request`` is created
+Unnamed disposables have their names in the format :samp:`disp{1234}`, where :samp:`{1234}` is derived from the ``dispid`` property, a random integer ranging from 0 to 9999 with a fail-safe mechanism to avoid reusing the same value in a short period.
 
-3. if prerun script was specified, copy it to ``qubes_save_script`` xenstore key
+The system and every qube can have the ``default_dispvm`` property. If the qube property is set to the default value, it will use the system's property. This property can only have disposable template as value or an empty value. Qubes which have this property set are allowed to request the creation of a disposable from this property. There are some Qrexec services that which allows execution to this newly created disposable when the destination qube (Qrexec field) uses the ``@dispvm`` tag, most commonly used to open files and URLs, (:file:`qubes.OpenInVM` and :file:`qubes.OpenURL`, respectively).
 
-4. wait for the ``qubes_used_mem`` key to appear
+Preloaded disposables
+---------------------
 
-5. (in APPVM) APPVM boots normally, up to the point in ``/etc/init.d/qubes_core`` script when the presence of ``qubes_save_request`` key is tested. If it exists, then
 
-   1. (in APPVM) if exists, prerun script is retrieved from the respective xenstore key and executed. This preloads filesystem cache with useful applications, so that they will start faster.
+Preloaded disposables are started in the background and kept hidden from the user when not in use. They are interrupted (paused or suspended, as appropriate) and resumed (transparently) when a disposable qube is requested by the user.
 
-   2. (in APPVM) the amount of used memory is stored to ``qubes_used_mem`` xenstore key
+Preloaded disposable stages
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-   3. (in APPVM) busy-waiting for ``qubes_restore_complete`` xenstore key to appear
+There are several stages a disposable goes through while preloading and being used. In short:
 
+- **Preload**: The qube is created and marked as preloaded. Qube is not visible in GUI applications.
 
+  - **Startup**: Begins qube startup, start basic services in it and attempt to interrupt (suspend/pause).
 
-6. when ``qubes_used_mem`` key appears, the domain memory is reduced to this amount, to make the savefile smaller.
+  - **Request**: The qube is removed from the preload list. If startup has not yet reached interrupt, the latter is skipped.
 
-7. APPVM private image is detached
+- **Used**: The qube is marked as used and may be unpaused/resumed (if applicable). Only in this phase, GUI applications treat the qube as any other unnamed disposable and the qube object is returned to the caller if requested.
 
-8. the domain is saved via ``xl save``
+Preloaded disposable's worry-free life-cycle
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-9. the COW file volatile.img (cow for root fs and swap) is packed to ``saved_cows.tar`` archive
 
+There are several events that may trigger the creation or deletion of preloaded disposables. These events happen for different reasons, mostly because:
 
+- There is a gap between the current number of preloaded disposables and the max; or
+- The current number of preloaded disposable is over the max.
 
-The ``qubes_prepare_saved_domain.sh`` script is lowlevel. It is usually called by ``qvm-create-default-dvm`` script, that takes care of creating a special AppVM (named template_name-dvm) to be passed to ``qubes_prepare_saved_domain.sh``, as well as copying the savefile to /dev/shm (the latter action is not done if the ``/var/lib/qubes/dvmdata/dont_use_shm`` file exists).
+The preload creation can also fail for different reasons:
 
-Restoring a DisposableVM from the savefile
-------------------------------------------
+- Qubesd was interrupted mid preload creation, on the next service restart, ``domain-load`` of the disposable template will refresh the incomplete disposables;
+- Service to check if the system is fully operation has failed; and
+- There is not enough memory to preload at the moment.
 
+If there is a gap, it will be capped *event* ually, if it is over the max, it will be deleted as soon as possible. If the volumes are outdated, they will be refreshed. These are common events that trigger changes in preloaded disposables quantity:
 
-Normally, disposable VM is created when qubes rpc request with target *$dispvm* is received. Then, as a part of rpc connection setup, the ``qfile-daemon-dvm`` program is executed; it executes ``/usr/lib/qubes/qubes_restore`` program. It is crucial that this program executes quickly, to make DisposableVM creation overhead bearable for the user. Its main steps are:
+- Setting or deleting the ``preload-dispvm-max`` feature will refill or remove;
+- (Re)starting :file:`qubes-preload-dispvm.service` will refresh;
+- Using a preloaded disposable will refill;
+- Requesting a disposable will refill;
+- Updating the volumes of a template or disposable template will refresh;
+- Changing system's ``default_dispvm`` while system's feature is set to a different value than the disposable template setting will refill or remove;
 
-1. modify the savefile so that the VM name, VM UUID, MAC address and IP address are unique
+Preloaded disposables memory management
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-2. restore the COW files from the ``saved_cows.tar``
 
-3. create the ``/var/run/qubes/fast_block_attach`` file, whose presence tells the ``/etc/xen/scripts/block`` script to bypass some redundant checks and execute as fast as possible.
+At the end of preloading, the qube is paused if it has not been requested yet, but before pausing, on ``domain-pre-pause``, it attempts to retrieve memory from the qube by setting it to use its preferred memory value, which in :doc:`qmemman parlance </developer/services/qmemman>`, is just enough to have the qube running. The memory must be managed before the qube is paused, cause once on the paused state, it is not possible to negotiate with the domain.
 
-4. execute ``xl restore`` in order to restore a domain.
+This process can take a bit of time because it depends on how fast the qube can free up memory. There is a timeout and a threshold in transfer speed. When any of these exit conditions are met, the memory management seizes and the preloaded disposable is paused. Although this process takes some time, we do not worry much about it because it economizes memory on the long run, the biggest problems is that qmemman is synchronous, so only one request can be made at a time, anything that takes too much time on qmemman could prevent ballooning/balancing of other qubes on the system.
 
-5. create the same xenstore keys as normally created when AppVM boots (e.g. ``qubes_ip``)
+Preloaded disposables security
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-6. create the ``qubes_restore_complete`` xenstore key. This allows the boot process in DisposableVM to continue.
 
+As preloaded disposables are started before being used, methods to prevent accidental tampering have been put in place as well as guarantees to prevent reuse:
 
+- The qube has the ``internal`` feature enabled, Qubes GUI applications were patched to hide internal qubes, handling events for ``domain-feature-(set|delete):internal``;
+- When requesting an unnamed disposable, the qube object is only returned to the user once it has finished preloading;
+- The qube is paused as the last stage of preloading, this permits receiving ``domain-unpaused`` events and be notified that the qube was used, marked as such and removed from the preload list to avoid reuse, even without the qube being requested with ``DispVM.from_appvm``;
+- The GUID only connects to the GUI agent on the qube after the preloaded disposable is marked as used, this prevents that an autostarted application such as a terminal appears on the screen before preloading has finished. Enabling a GUI is is controlled by the ``is_preload`` property, that when disabled, allows the GUI connection to initiate. This method delays GUI calls considerably as establishing the connection can take ~2 seconds, research is being done to prevent this delay.
 
-The actual passing of files between AppVM and a DisposableVM is implemented via qubes rpc.
+Alternatives considered
+^^^^^^^^^^^^^^^^^^^^^^^
 
-Validating the DisposableVM savefile
-------------------------------------
 
+Alternatives implementation must meet the following requirements:
 
-DisposableVM savefile contains references to template rootfs and to COW files. The COW files are restored before each DisposableVM start, so they cannot change. On the other hand, if templateVM is started, the template rootfs will change, and it may not be coherent with the COW files.
+- No memory or ``vcpus`` restrictions such as limiting to a few number of ``vcpus`` or assigns memory on request (can be slow).
+- Performant as much as a normal disposable even on long running sessions;
+- Caller transparency, no change necessary for callers, the request must be transparent and the server must find the fastest option. This is to avoid transition burden (API breakage).
 
-Therefore, the check for template rootfs modification time being older than DisposableVM savefile modification time is required. It is done in ``qfilexchgd`` daemon, just before restoring DisposableVM. If necessary, an attempt is made to recreate the DisposableVM savefile, using the last template used (or default template, if run for the first time) and the default prerun script, residing at ``/var/lib/qubes/vm-templates/templatename/dispvm_prerun.sh``. Unfortunately, the prerun script takes a lot of time to execute - therefore, after template rootfs modification, the next DisposableVM creation can be longer by about 2.5 minutes.
+Restoration from savefile
+"""""""""""""""""""""""""
+
+
+**Description**: Disposable template booted and then it was suspend to disk (image dump).
+
+**Evaluation**:
+
+- Used in R3.2, worked at that time, when there was only one disposable template available, see next points of why it can't be used anymore.
+- Incompatible with multiple ``vcpus``.
+- Some memory issues.
+- Savefile creation takes a long time. The disposable qube savefile contains references to template rootfs and :abbr:`CoW (Copy-on-Write)` files, if there is a modification on the template or disposable template, it took longer than 2.5 minutes to generate the next disposable.
+
+Xen domain fork
+"""""""""""""""
+
+
+**Description**: domain forking is the process of creating a domain with an empty memory space and a parent domain specified from which to populate the memory when necessary. For the new domain to be functional the domain state is copied over as part of the fork operation (HVM params, heap allocation etc). This description was sourced from `[Xen-devel] [RFC PATCH for-next 17/18] xen/mem_sharing: VM forking, Tamas K Lengyel <https://lists.xenproject.org/archives/html/xen-devel/2019-09/msg02497.html>`__.
+
+**Evaluation**:
+
+- Shares too much information from the trunk to the forks. This appears to have improved if not totally fixed on Linux 6.14, as mentioned by Andrew Cooper on Qubes OS Summit 2025;
+- Requires changing properties after the fork is done, this includes, but not limited to, ``xid`` of connected qubes, network uplink;
+- Not designed for long running sessions, the initial intention was fuzzing. As fast as the creation can be, the usage may be slower as memory is mapped on request. Xen doesn't have a poper :abbr:`CoW (Copy-on-Write)` support for domain memory, so making a full copy of a domain on fork also has some overhead;
+- Tamas K Lengyiel `VM Forking & Hypervisor-Based Fuzzing with Xen <https://www.youtube.com/watch?v=3MYo8ctD_aU>`__ presentation during the Open Source Summit Europe in 2022, showed impressive results on CPU i5-8350U, an average of time of 0.745 ms per fork (created 1300 VM/s). These fast results were later explained that was due to not initializing the whole VM memory on the fork unless it was requested, as explained on the point above. Still impressive results but current usage is limited to fuzzing.
+
+Preload queue
+"""""""""""""
+
+
+**Description**: Start disposables and queue them in a disposable template feature, unnamed disposables requested will prefer to retrieve disposables from this list.
+
+**Evaluation**:
+
+- Because the qube is running prior to being requested, multiple tools have to be patched to support it to various levels off difficulty. Excluding from backups to allowing removal of disposable templates that only have preloaded disposables to even stranger issues such as deferring net qube change from a preloaded disposable where the old net qube has already been purged from the system.
+- The biggest difference between this queue and the other alternatives is that this solution works and is reliable. A proper solution would be patching upstream Xen to implement :abbr:`CoW (Copy-on-Write)`, but that would involve a lot more work than what the Qubes Team can provide with current resources.