[Guide]: "Serial Port Passthrough (/dev/ttyS0) in dockur/windows" #1248

Is your question not already answered in the FAQ?

• I made sure the question is not listed in the FAQ.

Is this a general question and not a technical issue?

• I am sure my question is not about a technical issue.

Question

Description

When attempting to pass through a host serial port (e.g., /dev/ttyS0) to a Windows XP VM running via the dockur/windows Docker image, users might encounter a qemu-system-x86_64: -serial /dev/ttyS0: Could not open '/dev/ttyS0': Device or resource busy error in the Docker logs, even when the devices: mapping is correctly configured in docker-compose.yml.

Additionally, initial attempts to use environment variables like QEMU_ARGS, EXTRA_ARGS, or ARGS to specify -serial /dev/ttyS0 may appear to be ignored or overridden by -serial pty in the final QEMU command.

Root Cause Analysis

Through detailed debugging of the dockur/windows entrypoint scripts (entry.sh, power.sh, config.sh), the following was discovered:

1. Script Execution Order: In entry.sh, the power.sh script is sourced before config.sh.
2. Hardcoded Serial: power.sh (specifically around line 217 in the provided power.sh version) contains the line SERIAL="pty". This hardcodes the SERIAL shell variable to pty.
3. config.sh Behavior: When config.sh is later sourced:
   • The line : "${SERIAL:="mon:stdio"}" does not override SERIAL, because it's already set by power.sh.
   • The SERIAL_OPTS variable is therefore set to -serial pty.
   • However, config.sh constructs the final QEMU ARGS string by appending the $ARGUMENTS environment variable last.
4. QEMU Argument Prioritization: QEMU processes command-line arguments in order. If the same argument appears multiple times (e.g., -serial pty and then -serial /dev/ttyS0), QEMU generally honors the last instance of that argument.

Therefore, the strategy is to leverage the ARGUMENTS environment variable to append our desired serial port configuration (-serial /dev/ttyS0) at the very end of the QEMU command, effectively overriding the default -serial pty.

The "Device or resource busy" error is a separate issue, indicating that the serial port on the host system is actively in use by another process. This needs to be resolved on the host itself.

Solution

The solution involves two key parts:

1. Correct Docker Compose Configuration: Ensuring the device is mapped AND the ARGUMENTS environment variable is used correctly.
2. Host-side Conflict Resolution: Identifying and stopping any processes on the host that are using /dev/ttyS0.

Step 1: Modify docker-compose.yml

Update your docker-compose.yml file as follows. Ensure you remove or comment out any previous attempts to set QEMU_ARGS, EXTRA_ARGS, or SERIAL.

services:
  windows:
    image: dockurr/windows
    container_name: windows
    environment:
      VERSION: "xp"  
      DEBUG: "Y"
      RAM_SIZE: "2G"
      DISK_SIZE: "30G"
      CPU_CORES: "2"
      USERNAME: "your_windows_username" 
      PASSWORD: "your_windows_password" 

      # Previous attempts (comment out or remove):
      # QEMU_ARGS: "-serial /dev/ttyS0" 
      # EXTRA_ARGS: "-serial /dev/ttyS0" 
      # ARGS: "-serial /dev/ttyS0"
      # SERIAL: "/dev/ttyS0" 

      # THIS IS THE KEY TO OVERRIDE THE SERIAL PORT
      ARGUMENTS: "-serial /dev/ttyS0"     
    devices:
      - /dev/kvm
      - /dev/net/tun
      - /dev/ttyS0:/dev/ttyS0 # CRUCIAL: Maps the host device into the container    
    cap_add:
      - NET_ADMIN
    ports:
      - 8006:8006
      - 3389:3389/tcp
      - 3389:3389/udp
    volumes:
      - /mnt/data/your_volume_data:/data # Adjust this path to your host's data storage
    restart: always
    stop_grace_period: 2m

Step 2: Identify and Terminate Conflicting Processes on the Host

After updating your docker-compose.yml and before redeploying, it's crucial to ensure /dev/ttyS0 on your Ubuntu host is not in use.

1. Check if ttyS0 is busy:

   sudo lsof /dev/ttyS0
   # OR
   sudo fuser /dev/ttyS0

   If these commands return any output (e.g., screen, ModemManager, minicom, etc.), it means a process is using the port.

2. Terminate conflicting processes:

   • If lsof or fuser show processes (e.g., screen processes with PIDs), kill them:

     sudo kill <PID_1> <PID_2> ... # Replace with actual PIDs from lsof output
     # OR, if it's multiple 'screen' sessions:
     sudo killall screen

   • If it's a service like ModemManager and you don't need it:

     sudo systemctl stop ModemManager
     sudo systemctl disable ModemManager # To prevent it from starting on boot

   • If you've recently added your user to the dialout group (or the group owning /dev/ttyS0):

     sudo usermod -a -G dialout $USER # Replace 'dialout' with the actual group if different
     sudo systemctl restart docker # Restart Docker service to apply group changes to container daemon
     # You might also need to log out and back in to your SSH session for user group changes to fully take effect.

3. Verify ttyS0 is free:

   sudo lsof /dev/ttyS0

   This command should now return no output.

Step 3: Redeploy and Verify

1. Redeploy your Docker stack:

   docker-compose up -d --force-recreate

   (Or update the stack in Portainer).

2. Check Docker logs:

   docker logs your_container_name

   Look for the Arguments: section. You should now see both -serial pty AND -serial /dev/ttyS0, with -serial /dev/ttyS0 appearing after -serial pty. The critical "Device or resource busy" error should be gone.

3. Test in Windows XP:

   • Open Device Manager in Windows XP (Right-click My Computer -> Properties -> Hardware -> Device Manager -> Ports (COM & LPT)). You will likely see both COM1 and COM2.
   • Use HyperTerminal and configure it to use the second enumerated COM port (usually COM2), as this is typically the one mapped from your host's /dev/ttyS0. Set the baud rate, data bits, parity, etc., to match your device.

By following these steps, you should achieve successful serial communication from your Windows XP VM through your host's /dev/ttyS0.