Server Factory QEMU Utils

QEMU utilities toolkit for Server Factory projects. This toolkit provides comprehensive virtualization management capabilities including automated VM image distribution, compression, network configuration, and execution.

Table of Contents

• Overview
• Features
• Requirements
• Quick Start
• Configuration
• Usage
• Networking
• Troubleshooting
• Architecture
• Contributing

Overview

Server Factory QEMU Utils toolkit can be used as part of Server Factory projects or completely independently. The main goals are:

• Image Management: Compression and synchronization of QEMU disk images (matrices of systems used for development and testing)
• Distribution: Publishing and retrieving QEMU images for use across multiple computers (workstations, servers, etc.)
• Network Configuration: Automated bridge and TAP interface management for VM networking
• Cross-Platform Support: Works on both Linux and macOS with appropriate hardware acceleration
• ISO Installation: Support for fresh OS installations from ISO images

Features

Core Capabilities

✅ VM Image Lifecycle Management

• Download, cache, and run pre-configured QEMU disk images
• Automatic image retrieval from remote endpoints
• Local caching with Compressed/Uncompressed directories
• Image compression and publishing to remote servers

✅ ISO-Based Installation

• Install new operating systems from ISO files
• Automated ISO download and caching
• Clean OS installation workflow

✅ Advanced Networking

• Automated bridge creation and management
• TAP interface configuration
• Network interface binding
• Clean network teardown on errors

✅ Platform Support

• macOS with HVF (Hypervisor.framework) acceleration
• Linux with KVM acceleration
• Automatic acceleration detection
• Fallback to software emulation

✅ Storage Management

• Automated qcow2 disk creation
• Configurable disk sizes
• Thin provisioning support

Requirements

Software Dependencies

Required:

• QEMU (qemu-system-x86_64) version 4.0 or higher
  • macOS: brew install qemu
  • Linux: sudo apt install qemu-system-x86 or sudo yum install qemu-kvm
• Bash version 4.0 or higher
• sudo access (for network configuration)

Optional:

• wget or curl for image downloads
• rsync for image synchronization (via image_sync.sh)
• sshpass for automated image publishing (if using SSH password auth)
• tunctl or ip command for TAP interface management (Linux)

Hardware Requirements

• CPU: x86_64 processor with virtualization support
  • Intel: VT-x enabled in BIOS
  • AMD: AMD-V enabled in BIOS
• RAM: Minimum 8GB (VMs configured for 4GB by default)
• Disk: Sufficient space for VM images (typically 10-50GB per image)

Platform-Specific Requirements

macOS:

• macOS 10.13 (High Sierra) or higher
• HVF (Hypervisor.framework) support
• No additional kernel modules required

Linux:

• KVM kernel modules loaded (modprobe kvm, modprobe kvm_intel or modprobe kvm_amd)
• User must be in kvm or libvirt group
• Bridge utilities installed (bridge-utils package)

Quick Start

1. Create Configuration Files

Create three configuration files in the QEMU toolkit directory:

image_location.settings

/path/to/your/qemu/images

image_provider.settings

http://your-server.com

image_sync.sh (for publishing images)

#!/bin/sh
source=$1
where=$2
user="your_user"
password="your_password"
remote="server.com:path/$where"

echo "Synchronizing: $source -> $remote"
sshpass -p "$password" rsync -azP -e 'ssh -p 22' --delete "$source" "$user@$remote"

Make image_sync.sh executable:

chmod +x image_sync.sh

See Examples directory for template files.

2. Directory Structure

The path specified in image_location.settings should contain two subdirectories:

/path/to/your/qemu/images/
├── Compressed/       # .tar.gz archives of QEMU images
└── Uncompressed/     # Extracted QEMU disk images (disk.qcow2)

These directories will be created automatically if they don't exist.

3. Running a VM

Option A: Run an existing image

./run.sh /path/to/working/machines/Centos_8

Option B: Install from ISO

./run.sh /path/to/working/machines/Fedora_Server_35 Fedora-Server-dvd-x86_64-35.iso

Workflow

When you run run.sh:

1. Check if machine exists

   • If yes: Boot the VM immediately
   • If no: Proceed with image acquisition

2. Image Acquisition (if machine doesn't exist)

   • Check Uncompressed/ cache for disk image
   • If not found, check Compressed/ cache for .tar.gz
   • If not found, download from remote server (URL from image_provider.settings)
   • Extract and cache the image
   • Copy to working location

3. Network Setup

   • Create TAP interface (requires sudo)
   • Create or verify bridge exists
   • Bind interface to bridge

4. VM Execution

   • Detect hardware acceleration (KVM on Linux, HVF on macOS)
   • Configure display (SDL on macOS, GTK on Linux)
   • Create disk if needed (for ISO installations)
   • Start QEMU with appropriate parameters

Configuration

image_location.settings

Absolute path to the directory containing your image matrices:

/Volumes/Storage/QEMU/Images

This directory should have Compressed/ and Uncompressed/ subdirectories.

image_provider.settings

Base URL for downloading images:

http://images.example.com

Images will be downloaded from: {base_url}/Images/Qemu/{machine_name}.tar.gz

ISOs will be downloaded from: {base_url}/Images/Iso/{iso_filename}

image_sync.sh

Custom script for publishing images to your remote server. This script receives two arguments:

1. $1 - Source directory (local path with images)
2. $2 - Destination path component (e.g., "Images/Qemu")

Example using rsync over SSH:

#!/bin/sh
source=$1
where=$2
user="admin"
server="images.company.com"
remote_path="/var/www/html/$where"

rsync -azP --delete "$source" "$user@$server:$remote_path"

Example using scp:

#!/bin/sh
source=$1
where=$2

scp -r "$source"/* admin@images.company.com:/var/www/html/$where/

Usage

Running VMs

Run Pre-configured Image

./run.sh /absolute/path/to/machine_name

Example:

./run.sh /Volumes/Storage/QEMU/Working/CentOS_8

The script will:

• Look for machine_name/disk.qcow2
• If not found, search caches and download if needed
• Configure networking
• Boot the VM

Install from ISO

./run.sh /absolute/path/to/new_machine iso_filename.iso

Example:

./run.sh /Volumes/Storage/QEMU/Working/Ubuntu_22 ubuntu-22.04-server-amd64.iso

The script will:

• Download ISO if not in cache
• Create a new 20GB disk
• Boot from ISO
• Allow manual OS installation

Publishing Images

After creating or updating VM images, publish them:

./publish_images.sh

This script:

1. Finds all directories in Uncompressed/
2. Compresses each to .tar.gz using compress.sh
3. Moves archives to Compressed/
4. Executes image_sync.sh to upload to remote server

Image Organization

Working VMs (actively used machines):

/path/to/working/vms/
├── Development_CentOS_8/
│   └── disk.qcow2
├── Test_Ubuntu_22/
│   └── disk.qcow2
└── Production_Fedora_36/
    └── disk.qcow2

Matrix Images (templates for distribution):

/path/to/images/              # From image_location.settings
├── Compressed/
│   ├── CentOS_8.tar.gz
│   ├── Ubuntu_22.tar.gz
│   └── Fedora_36.tar.gz
└── Uncompressed/
    ├── CentOS_8/
    │   └── disk.qcow2
    ├── Ubuntu_22/
    │   └── disk.qcow2
    └── Fedora_36/
        └── disk.qcow2

When you run a machine for the first time, it's copied from Uncompressed/ to your working directory.

Networking

Network Architecture

QEMU VMs use TAP (network tap) interfaces bridged to your host network:

[Host Network] <-> [Bridge: br0] <-> [TAP: tap0] <-> [VM: eth0]

Automatic Network Setup

The run.sh script automatically:

1. Creates a TAP interface (e.g., tap0, tap1, ...)
2. Creates a bridge (br0) if it doesn't exist
3. Binds the TAP to the bridge
4. Configures QEMU to use the TAP interface

Manual Network Management

Create Network

./create_network.sh

This sets up the complete bridge and TAP infrastructure.

Create Bridge

./create_bridge.sh br0 192.168.1.1/24

Parameters:

• Bridge name (e.g., br0)
• IP address with CIDR (e.g., 192.168.1.1/24)

Create TAP Interface

tap_name=$(./create_and_get_tap.sh)
echo "Created TAP: $tap_name"

Returns the created TAP interface name.

Delete Bridge

./delete_bridge.sh br0

Safely removes a bridge and cleans up associated interfaces.

Network Troubleshooting

Problem: VM has no network connectivity

Check bridge exists:

ip link show br0

Check TAP is up:

ip link show tap0

Verify bridge membership:

bridge link show

Problem: Permission denied creating TAP

Ensure you're running with sudo:

sudo ./run.sh /path/to/machine

Problem: Bridge creation fails

Check for conflicting bridges:

ip link show type bridge

Delete old bridges:

sudo ip link delete br0 type bridge

Troubleshooting

Common Issues

QEMU Not Found

Symptom: qemu-system-x86_64: command not found

Solution:

# macOS
brew install qemu

# Ubuntu/Debian
sudo apt install qemu-system-x86

# CentOS/RHEL/Fedora
sudo yum install qemu-kvm

No Hardware Acceleration

Symptom: VM is very slow, warnings about missing KVM/HVF

Solution (Linux):

# Check KVM modules loaded
lsmod | grep kvm

# Load KVM module
sudo modprobe kvm
sudo modprobe kvm_intel  # or kvm_amd

# Add user to kvm group
sudo usermod -a -G kvm $USER

Solution (macOS):

• Ensure macOS 10.13+ (HVF support)
• Check virtualization enabled in System Preferences → Security

Image Download Fails

Symptom: ERROR: Image download failed

Solution:

• Verify image_provider.settings URL is correct
• Check network connectivity: curl -I $(cat image_provider.settings)
• Verify remote image exists at expected path
• Check wget or curl is installed

Disk Space Issues

Symptom: Errors during image extraction or disk creation

Solution:

# Check available space
df -h

# Clean old compressed images
rm /path/to/images/Compressed/old_image.tar.gz

# Reduce disk size in create_disk.sh (default 20GB)

Permission Issues

Symptom: Permission denied errors during network setup

Solution:

# Run with sudo (required for network configuration)
sudo ./run.sh /path/to/machine

# Or configure sudo without password for specific commands
sudo visudo
# Add: your_user ALL=(ALL) NOPASSWD: /usr/bin/ip, /sbin/brctl

Architecture

Script Organization

Qemu/
├── Core Scripts
│   ├── run.sh              - Main entry point for VM execution
│   ├── machine.sh          - QEMU command construction and execution
│   ├── publish_images.sh   - Image publishing pipeline
│   └── compress.sh         - Image compression utility
├── Network Management
│   ├── create_network.sh              - Complete network stack setup
│   ├── create_bridge.sh               - Bridge creation
│   ├── create_and_get_bridge.sh       - Bridge creation with name return
│   ├── create_and_get_tap.sh          - TAP creation with name return
│   ├── bind_interfaces_to_bridge.sh   - Interface binding
│   ├── delete_bridge.sh               - Bridge cleanup
│   ├── qemu-ifup                      - TAP interface up script
│   └── qemu-ifdown                    - TAP interface down script
├── Disk Management
│   └── create_disk.sh                 - qcow2 disk creation
├── Dependencies
│   ├── create_dependencies.sh         - Setup TAP dependencies
│   ├── delete_dependencies.sh         - Cleanup TAP dependencies
│   ├── get_dependencies.sh            - List TAP dependencies
│   └── create_script_path_script.sh   - Generate dependency scripts
├── Utilities
│   ├── get_acceleration.sh            - Detect KVM/HVF support
│   ├── get_display.sh                 - Detect display type
│   ├── get_machine_log_name.sh        - Generate log filenames
│   ├── get_running_machines_count.sh  - Count active VMs
│   ├── is_macos.sh                    - Platform detection
│   ├── fail.sh                        - Error exit
│   └── fail_and_cleanup.sh            - Error exit with cleanup
└── Configuration
    └── Examples/
        ├── image_location.settings    - Image cache path
        ├── image_provider.settings    - Remote server URL
        └── image_sync.sh              - Upload script template

Execution Flow

VM Startup Flow

run.sh
  │
  ├─> Check machine disk exists?
  │   ├─> YES: Go to Network Setup
  │   └─> NO: Continue to Image Acquisition
  │
  ├─> ISO provided?
  │   ├─> YES: Check ISO exists, download if needed
  │   └─> NO: Continue
  │
  ├─> Image Acquisition
  │   ├─> Check Uncompressed cache
  │   ├─> Check Compressed cache
  │   ├─> Download from remote
  │   ├─> Extract to Uncompressed
  │   └─> Copy to working location
  │
  ├─> Network Setup (machine.sh)
  │   ├─> create_and_get_tap.sh → Creates tap0, tap1, etc.
  │   ├─> create_script_path_script.sh → Generates ifup/ifdown scripts
  │   ├─> create_dependencies.sh → Sets up TAP dependencies
  │   └─> Bridge creation (if needed)
  │
  ├─> Disk Setup (machine.sh)
  │   └─> create_disk.sh → Creates qcow2 if doesn't exist
  │
  └─> VM Execution (machine.sh)
      ├─> Detect acceleration (KVM/HVF/none)
      ├─> Detect display (SDL/GTK)
      ├─> Build qemu-system-x86_64 command:
      │   ├─> CPU: qemu64
      │   ├─> RAM: 4096 MB
      │   ├─> SMP: 2 cores
      │   ├─> Display: SDL/GTK with cursor
      │   ├─> USB: Tablet support
      │   ├─> VGA: virtio
      │   ├─> Disk: virtio with qcow2
      │   ├─> Network: TAP interface
      │   └─> CD-ROM: ISO (if provided)
      └─> Execute QEMU
          ├─> SUCCESS: VM running
          └─> FAILURE: fail_and_cleanup.sh → Delete TAP and dependencies

Publishing Flow

publish_images.sh
  │
  ├─> Read image_location.settings
  │
  ├─> For each directory in Uncompressed/:
  │   ├─> compress.sh <directory> → Create .tar.gz
  │   └─> Move .tar.gz to Compressed/
  │
  └─> Execute image_sync.sh Compressed/ "Images/Qemu"
      └─> rsync to remote server

QEMU Command Structure

Example QEMU command generated by machine.sh:

sudo qemu-system-x86_64 \
  -accel hvf \                          # Hardware acceleration (macOS)
  -cpu qemu64 \                         # CPU type
  -m 4096 \                             # RAM in MB
  -smp 2 \                              # CPU cores
  -display sdl,show-cursor=on \         # Display type
  -usb -device usb-tablet \             # USB tablet (better mouse)
  -vga virtio \                         # VirtIO VGA
  -drive file=/path/disk.qcow2,format=qcow2,if=virtio \  # Disk
  -net nic -net tap,ifname=tap0 \       # Network
  -cdrom /path/to/install.iso           # CD-ROM (optional)

Best Practices

Image Management

1. Separate Working from Matrices

   • Keep working VMs in a different location from matrix images
   • Matrices are templates, working VMs are active instances

2. Regular Publishing

   • Publish updated matrices after significant changes
   • Keep remote and local caches synchronized

3. Compression

   • Compressed images significantly reduce storage and transfer time
   • qcow2 files compress well (10-20GB → 2-4GB typical)

Network Configuration

1. Use Separate TAP per VM

   • Each VM gets its own TAP interface (automatic)
   • Prevents network conflicts

2. Bridge Persistence

   • The bridge (br0) persists across VM restarts
   • TAP interfaces are created/destroyed per VM session

3. Firewall Considerations

   • Configure host firewall to allow bridge traffic
   • VMs appear as separate network hosts

Security

1. image_sync.sh Credentials

   • Don't hardcode passwords in image_sync.sh
   • Use SSH keys: rsync -azP -e "ssh -i ~/.ssh/key" ...
   • Or use environment variables

2. Sudo Access

   • Network setup requires sudo
   • Consider sudoers configuration for specific commands only

3. VM Isolation

   • VMs on TAP/bridge can communicate with host network
   • Use firewall rules to restrict if needed

Performance Optimization

Hardware Acceleration

Always use hardware acceleration when available:

• Linux KVM: 10-20x faster than emulation
• macOS HVF: 5-10x faster than emulation

Check acceleration is active:

# Script will show acceleration in use
./get_acceleration.sh

Disk Performance

Use VirtIO drivers in guest OS for best performance:

• Linux guests: VirtIO drivers included in kernel
• Windows guests: Download VirtIO drivers from Fedora project

Network Performance

VirtIO networking provides near-native performance:

• Lower CPU usage vs emulated NIC
• Higher throughput
• Better latency

Resource Allocation

Adjust resources in machine.sh:

# Default: 4GB RAM, 2 CPUs
qemu-system-x86_64 -m 4096 -smp 2 ...

# High-performance: 8GB RAM, 4 CPUs
qemu-system-x86_64 -m 8192 -smp 4 ...

# Minimal: 2GB RAM, 1 CPU
qemu-system-x86_64 -m 2048 -smp 1 ...

Contributing

Contributions are welcome! Please follow these guidelines:

1. Code Style

   • Use consistent indentation (2 spaces)
   • Add comments for complex logic
   • Follow existing naming conventions

2. Testing

   • Test on both Linux and macOS if possible
   • Verify network setup works correctly
   • Check error handling and cleanup

3. Documentation

   • Update README for new features
   • Add inline comments to scripts
   • Document any new dependencies

4. Pull Requests

   • Create feature branch from main
   • Write clear commit messages
   • Reference related issues

Related Projects

• Server Factory Core Framework
• Server Factory Parallels Utils
• Mail Server Factory

License

See LICENSE file for details.

Support

For issues, questions, or contributions:

• Open an issue on GitHub
• Check existing documentation
• Review TOOLKIT_COMPARISON.md for feature details

---

Status: Production Ready (v1.0.0) Platform: Linux, macOS Last Updated: October 10, 2025