AuspexAI Worker

The volunteer worker process for AuspexAI — runs on volunteer machines, executes work units dispatched by the coordinator, donates compute to AI safety research.

Status

Open beta — worker v0.2.x, live on the AuspexAI network. First-run enrollment, the signed-heartbeat loop, the assignment pipeline, and the sandboxed runner shipped across Phase 1 (M1–M7, 2026-05). Since then the worker has grown local LLM-inference serving, coordinator-driven model acquisition, RAM-fit routing, and a serve-time GPU-OOM guard (see Since M7 below). Supported targets: Linux x86_64 + ARM64 (ARM64 in production on Jetson-class hardware) and macOS (install script + launchd); Windows/WSL2 remains Phase 2.

What's live:

• auspexai-worker bootstrap — generates an Ed25519 keypair, persists to the OS keyring (libsecret) or an encrypted-file fallback, enrolls anonymously (T0) with the coordinator via POST /api/v0/workers/enroll, records the assigned worker_id in the local SQLite state DB. Idempotent on re-run.
• auspexai-worker status — shows worker_id, tier, pubkey fingerprint, enrollment timestamp, last heartbeat.
• auspexai-worker daemon — runs two threads concurrently: a heartbeat loop posting signed POST /api/v0/workers/{id}/heartbeat with capabilities (OS, arch, RAM total, CPU count, GPU observation + volunteer declaration, declared resource caps), and an assignment poller calling signed GET /api/v0/workers/{id}/assignments and running the M3 gate pipeline on every assignment received. SIGTERM/SIGINT trigger a clean stop on both. --max-ticks=N / --verbose.
• Assignment-handling pipeline (M3):
  • Manifest-pin defense (§5.14): first sighting of an experiment locks its manifest_sha256; a later assignment under a different hash is refused as a manifest-swap attempt.
  • Sensitive-content gate (§5.14 + §5.12): assignments carrying sensitive_content_flags in the payload require an explicit accept <experiment-id> to proceed (default-decline). Note: the M6-era coordinator does NOT yet ship this field on WorkUnitEnvelopeOut; the gate is plumbed and waits for the coordinator-side change.
  • Tenant allow/deny gate (§5.14): tenants on the deny list are always refused; a non-empty allow list restricts acceptance to listed tenants.
  • Refusals call back to the coordinator via POST /api/v0/workers/{id}/assignments/{unit_id}/refuse so the operator console can see the refusal reason and the unit's replication slot is freed for another worker (Option A per Q-W4). Each decision is also written to a local assignment_audit log.
• M4: sandboxed runner + signed result submission. On accept, the daemon spawns auspexai-worker-runner inside a bubblewrap sandbox (Phase 1 permissive policy: --die-with-parent --new-session --dev-bind / / plus env passthrough). The runner reads the work-unit envelope from stdin, executes the tenant executor (LLM-inference tenants reach the worker-served model over a sandbox-local socket, under the manifest's declared generation policy), writes a Result body to $AUSPEXAI_OUTPUT_PATH, exits. Daemon reads the output, signs it with the worker key over a canonical encoding, POSTs to /api/v0/workers/{id}/assignments/{unit_id}/result. Local submitted_results table mirrors the coordinator's ack. Per Q-W4: any failure during dispatch (runner crash, submit error) triggers an explicit refuse to the coordinator. Per Q-W9: one-at-a-time on the poller thread; heartbeat thread runs independently. Per Q-W10: bubblewrap requires unprivileged user namespaces — on AppArmor-restricted hosts (Ubuntu 24.04 default) deploy needs kernel.apparmor_restrict_unprivileged_userns=0 or a worker AppArmor profile. Passthrough mode ([sandbox] use_bubblewrap = false) is the documented escape hatch for CI + dev hosts.
• auspexai-worker abort <unit-id> — reads the runner PID file from the workspace, sends SIGTERM, waits --grace-seconds (default 5), sends SIGKILL if still running. No-op when no workspace / no PID / process already exited; always writes an audit row.
• M3 CLI verbs: queue, peek <unit-id>, accept <coordinator-experiment-id>, refuse <unit-id>, tenant {allow,deny,list} <tenant-id>.
• RFC 9421 HTTP Message Signature signer (ed25519, covered @method+@path+@authority+content-digest, label sig1, created window enforced by the coordinator). Symmetric to the platform's verifier; wire format verified by an inline oracle in the worker's own test suite.
• Keystore backends: libsecret (Secret Service) primary; encrypted-file (ChaCha20-Poly1305, key derived from /etc/machine-id + UID) fallback for headless hosts and containers.
• Local SQLite state at $XDG_STATE_HOME/auspexai-worker/worker.db with a sequential migration framework matching the coordinator's convention. Re-entrant transaction lock so the two daemon threads can write concurrently without colliding on BEGIN.
• systemd --user service unit templates with Phase 1 hardening (PrivateTmp, ProtectHome=read-only, ProtectSystem=strict, NoNewPrivileges, SystemCallFilter=@system-service). Two variants ship: packaging/systemd/auspexai-worker.service for pip/uv installs (ExecStart=%h/.local/bin/auspexai-worker); packaging/systemd-deb/auspexai-worker.service for .deb installs (ExecStart=/opt/auspexai-worker/bin/auspexai-worker, installed by the .deb to /etc/systemd/user/auspexai-worker.service).
• Comprehensive test suite on Python 3.11 + 3.12; ruff check + format-check clean.

Subsequent milestones (per design doc §14):

• M5 ✅ Receipts store on submitted_results + receipts list/show + log CLI (SQLite-as-canonical-store per 2026-05-22 design update).
• M6 ✅ T1 upgrade via OAuth Device Flow (auspexai-worker login) + withdraw flow (auspexai-worker withdraw).
• M7 ✅ SHIPPED 2026-05-23 — Cosign-signed .deb + source tarball + wheel as GitHub release artifacts. dh-virtualenv-built /opt/auspexai-worker/ venv; system-installed user-unit at /etc/systemd/user/; AppArmor profile confining the daemon with a cx -> bwrap_sandbox child profile that grants userns, scoped to bwrap-as-child-of-the-daemon (Q-W10 Phase 2 durable fix); postinst probes the sandbox as nobody before declaring install successful. Release pipeline at .github/workflows/release.yml signs via Sigstore keyless GitHub Actions OIDC (no manual cosign step per release). See AUTHORIZED_SIGNERS.md for the trust roster.

Since M7 (open beta, worker v0.2.x):

• Local LLM-inference serving. With [inference] backend = "ollama" the worker serves a tenant-requested model over a sandbox-local socket (the tenant executor never reaches the network) under the manifest's declared generation policy (greedy / seeded-sampling, applied per request). The model CLI group manages the local model store.
• Fleet-fit RAM routing. The coordinator routes a unit to a worker only when the model fits the worker's usable RAM (HF-derived sizing); the worker refuses to serve what can't fit (non-bypassable BYOM RAM guard).
• GPU-OOM serve guard (v0.2.72). On constrained hosts (≤12 GB usable, e.g. Jetson-class) the worker serves one model at a time (unloading others first) and, on a CUDA out-of-memory at serve, frees all VRAM and retries once; if it still can't serve it refuses with a clear error and surfaces a copy-to-run recovery command on the local dashboard (drop the OS page cache / restart the model server) — informational only, never auto-run. Worker migration 0012 (serve_advisory).
• Volunteer controls. Self pause/resume, login/logout (T1 identity binding), and withdraw.

Full design rationale: Documentation/AuspexAI/v0.1.0/worker_daemon_design.md (ratified into principles doc §5.19 on 2026-05-20).

Install

Supported targets: Ubuntu 24.04+ / Debian 12+ (x86_64 and ARM64) and macOS (install script → ~/.local + launchd). The signed .deb is x86_64; ARM64 (e.g. Jetson) and macOS install from PyPI via pip. Windows/WSL2 remains Phase 2.

Via .deb release artifact (recommended)

Download the .deb and its Cosign signature/cert from the latest release. Verify before installing (the signature anchors at the Maintainer's GitHub OIDC identity via Sigstore — no long-lived signing key to compromise):

# One-time: install cosign if you don't already have it
# (https://docs.sigstore.dev/cosign/installation)

# Verify the .deb
cosign verify-blob \
  --certificate-identity-regexp='^https://github\.com/auspexai/.+/\.github/workflows/.+@.+$' \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
  --signature auspexai-worker_<version>_amd64.deb.sig \
  --certificate auspexai-worker_<version>_amd64.deb.cert \
  auspexai-worker_<version>_amd64.deb

# Install
sudo apt install ./auspexai-worker_<version>_amd64.deb

The postinst will reload the AppArmor profile and probe the sandbox as user nobody. If the probe fails, the install fails with an actionable error pointing at either the sysctl bridge (kernel.apparmor_restrict_unprivileged_userns=0 for lab use) or the AppArmor profile path (durable fix on Ubuntu 24.04+).

After install, each volunteer user runs (as themselves, not root):

systemctl --user enable --now auspexai-worker.service
auspexai-worker status       # confirms enrollment (T0 anonymous)
auspexai-worker login        # binds to GitHub identity (T1)

Tail the daemon: journalctl --user -u auspexai-worker -f.

Open the local dashboard in a browser (since v0.1.4):

http://localhost:7799

Read-only view of your worker's identity, activity log, receipts, and loaded config. When a serve attempt fails on a constrained host (e.g. a CUDA out-of-memory the worker's own unload-and-retry can't clear), the dashboard also surfaces a copy-to-run recovery command (drop the OS page cache / restart the model server) — informational only, never auto-run. Localhost-only by design — bound to 127.0.0.1 and never exposed externally. Disable with [dashboard] enabled = false in ~/.config/auspexai-worker/worker.toml or AUSPEXAI_WORKER_DASHBOARD_ENABLED=0 if you'd rather not have the local HTTP server running.

Container / minimal host caveat

The encrypted-file keystore needs /etc/machine-id for host-specific entropy. Real Ubuntu / Debian / Fedora hosts always have one; minimal containers or stripped embedded images may not. If auspexai-worker bootstrap complains, seed it once:

sudo systemd-machine-id-setup
# or, if that's not available:
cat /proc/sys/kernel/random/uuid | tr -d - | sudo tee /etc/machine-id

Trust roster

The list of identities authorized to sign release artifacts and contribution receipts lives at auspexai/.github/security/AUTHORIZED_SIGNERS.md. Verifiers should match the cosign --certificate-identity-regexp against the entries in that file.

Build the .deb from source

packaging/build-deb.sh            # builds in a podman/docker Ubuntu 24.04 container
packaging/build-deb.sh --test     # also install-tests in a second clean container

Outputs land at /tmp/auspexai-deb-build/auspexai-worker_<version>_amd64.deb.

Scope

The Worker:

• Runs on volunteer machines — Linux x86_64 + ARM64 and macOS (install script + launchd) today; Windows via WSL2 in Phase 2 (per §5.13)
• Generates an Ed25519 keypair on first run, stored in OS-native keystore (libsecret on Linux; Keychain / DPAPI in Phase 2 platforms) — volunteers never paste keys into web forms
• Enrolls anonymously (T0) or upgrades to verified identity via OAuth 2.0 Device Authorization Flow (T1+; M6)
• Executes work units within sandboxed limits (CPU, GPU, RAM, network caps; idle-only mode by default — Phase 2)
• Submits signed results (M4)
• Maintains a local audit log of what it ran, when, and for which manifest hash (M3+)

Worker trust tiers (T0 anonymous through T4 maintainer) govern work eligibility and quorum weight; see the AuspexAI Principles & Scope §6 for the trust model.

Identity binding (T0 → T1)

A fresh install enrolls as T0 anonymous with no user interaction. To bind the worker to a verifiable identity and unlock higher-trust roles (vouching, Approver eligibility, unique work-unit assignments) run:

auspexai-worker login

This launches an OAuth 2.0 Device Authorization Flow against GitHub:

1. The worker prints a short code and a verification URL (github.com/login/device).
2. Open the URL in any browser (any device — phone, laptop, the same machine — your choice), sign in to GitHub, enter the code.
3. GitHub returns an access token to the worker; the worker hands it to the AuspexAI coordinator, which verifies it with GitHub and mints a one-shot binding token; the worker then exchanges that binding token for a T1 upgrade. The access token is never stored on the worker — it's discarded as soon as the binding completes.

The read:user scope is the only scope ever requested. Subsequent worker↔coordinator authentication continues to use the worker's own Ed25519 keypair, not the GitHub token.

Email-fallback for institutional contexts

Phase 1 is GitHub-only for the OAuth IdP. Institutional volunteers, researchers, or workers running in environments where GitHub Device Flow isn't usable can request an alternative binding path by emailing contact@auspexai.network. A formal email-based binding path is on the Phase 2 roadmap (per principles doc §5.10 + §9 Q5); pre-Phase-2 accommodations are case-by-case under the Maintainer's discretion.

Withdrawal

To retire a worker and remove its local state:

auspexai-worker withdraw

The command prints what will be deleted, prompts for explicit confirmation (you must type the word withdraw), then:

1. Calls the coordinator's retire endpoint so the scheduler stops handing it work.
2. Deletes the local SQLite state DB (worker.db) — audit log, manifest pins, receipts, tenant lists, consent rows.
3. Deletes the worker's Ed25519 keypair from the keystore.

Receipts already issued by the coordinator REMAIN in the coordinator's transparency log per Principles & Scope §5.15. They stay signed and verifiable but become unattributed (severable-PII pattern). The CLI prints this verbatim at the confirmation prompt so the volunteer is reminded of the consent terms at the moment of withdrawal.

After withdrawal completes, follow up with your package manager (apt remove auspexai-worker, pipx uninstall auspexai-worker, etc.) to finish the uninstall.

Development

Requires Python 3.11+. Quick start:

uv venv
uv pip install -e ".[dev]"
auspexai-worker status                              # shows "not enrolled"
auspexai-worker bootstrap                           # enrolls against the
                                                    # public coord (default
                                                    # since v0.1.2)
auspexai-worker daemon --max-ticks=3                # run 3 heartbeats then exit
pytest
ruff check src tests
ruff format --check src tests

Running against a coordinator on your own machine (lab mode) — override the default https://coord.auspexai.network:

AUSPEXAI_COORDINATOR_URL=http://127.0.0.1:8080 \
  auspexai-worker bootstrap
AUSPEXAI_COORDINATOR_URL=http://127.0.0.1:8080 \
  auspexai-worker daemon --max-ticks=3

Or persist via TOML at ~/.config/auspexai-worker/worker.toml:

[coordinator]
url = "http://127.0.0.1:8080"

Conduct on the network

Worker conduct on the AuspexAI network is governed by the Volunteer Terms of Participation (forthcoming, Phase 2). Network-level abuse — running malicious workers, abusing reputation systems, attempting to influence quorum — is handled through technical enforcement (revocation, ban, trust-tier demotion), not Code-of-Conduct enforcement.

License

AGPL-3.0 — workers are network-served clients of the AuspexAI coordinator and inherit the platform's copyleft posture.

Governance & policies

• Governance — roles, decision rules, recruitment, conflict of interest
• Code of Conduct — community standards, reporting, escalation pathway
• Contributing — DCO sign-off, PR workflow, RFC requirement for substantial architectural changes
• Research Ethics Policy — what AI safety research can run on the network and how it's reviewed

Watch this repo

The worker is live on the open-beta network; releases ship through the GitHub release pipeline (Cosign-signed .deb + wheel) and PyPI.