-
Notifications
You must be signed in to change notification settings - Fork 0
Troubleshooting
Start with the symptom that most closely matches the failure. Regardless of provider, keep TLS verification enabled, preserve the first relevant log, and do not use broad Docker/WSL resets or prune commands as a first response.
- Quick diagnostics
- Startup reports that the pool controller configuration lock is already held
- Windows no-Go startup prints an HTTP/2 named-pipe diagnostic
- A Docker workload fails with an architecture error
- Docker Sandboxes is unavailable or its preflight fails
- Docker Sandboxes daemon health passes but inventory hangs
- Docker Sandboxes rejects template, policy, or capacity
- Docker Sandboxes creation fails after a runtime-helper prompt
- Docker Sandboxes rejects a staging workspace because SSH-agent forwarding is present
- Docker Hub login succeeds but a private pull is denied in Docker Sandboxes
- An idle runner reports GitHub or Sandbox health warnings
- GitHub Actions runner release resolution reports HTTP 403 or 429
- A scheduled image check or update fails
- A runner is held for diagnostics or an acknowledgement
- Docker image build runs out of space
- Storage keeps growing after updates
- Docker image build fails with TLS certificate errors
- Windows Docker Desktop WSL2 disk is smaller than expected
- Docker Container startup fails
- WSL provider image build fails early
- GitHub runner registration fails
EPAR writes logs under work/logs by default. Start with work/logs/epar-last-error.log, then inspect the matching build log in work/logs/builds/ or instance transcript in work/logs/instances/. Manager events are console-only by default; raw transcripts are file-only unless logging.transcriptSinks includes console.
Long Buildx operations show a bounded console summary with downloaded bytes, completed layers, the active BuildKit step, elapsed time, and growing direct-archive bytes when an export is in progress. The complete raw progress remains in the printed build-log path.
./start --help
./start version
docker version
docker info
docker system dfWithout local Go, use ./start --help; the wrapper selects the containerized toolchain. For Windows WSL2, a WSL-backed Docker daemon, or the WSL provider, also run:
wsl --version
wsl -l -v
docker context ls
docker run --rm ghcr.io/catthehacker/ubuntu:full-latest df -h /Container-visible free space is the relevant value for Docker builds. Windows Explorer or Finder free space does not necessarily equal the free space in a Linux VM backing the daemon.
./start exits with an error like:
ephemeral-action-runner: pool controller configuration lock is already held for "/Users/someone/epar/.local/config.yml" (owner config="/Users/someone/epar/.local/config.yml" provider="docker-sandboxes" prefix="example-pool" pid=12345 startedAt=2026-01-02T03:04:05Z)
This can appear after the startup wrapper rebuilds the native controller because its source digest changed. The rebuild prepares a new binary but does not replace an already-running controller, so the new process still stops at the ownership lock.
The lock prevents two mutating controllers from managing the same pool. Treat the reported config path, provider, prefix, PID, and start time as owner-identification evidence rather than deleting a lock file.
On macOS or Linux, inspect the reported PID, replacing 12345 with the value from the error:
ps -p 12345 -o pid=,ppid=,lstart=,command=On Windows PowerShell:
Get-Process -Id 12345 -ErrorAction SilentlyContinue | Format-List Id, StartTime, PathIf the PID is the intended live EPAR controller, do not start another controller for that pool. Return to its terminal and press Ctrl-C once, or stop it through the service manager that launched it, then wait for Cleanup complete. EPAR can now exit safely. before running ./start again. Use this sequence when activating copied source or a rebuilt controller: stop the old controller cleanly, wait for cleanup, then start the new version.
If no process owns the reported PID, rerun ./start. The operating-system lock is authoritative and is released when its owning process exits; stale descriptive metadata may remain on disk but is replaced on the next successful acquisition. Do not manually delete files under the controller-lock state directory, and do not signal a process solely because its PID matches stale metadata—PID values can be reused.
If the exact EPAR controller is unresponsive, prefer its service manager's graceful stop or a single interrupt and allow time for cleanup. After a confirmed abnormal exit, rerun ./start so normal lifecycle reconciliation can inspect and recover exact owned resources. Avoid force-killing the process, deleting runner records manually, or using broad Docker, Sandbox, WSL, or GitHub cleanup commands as a first response.
Multiple EPAR controllers may run on one machine, but every concurrently active controller must use both a distinct canonical configuration path and a distinct normalized pool.namePrefix. Reusing either the same config path or the same pool prefix intentionally conflicts, including across different project directories or providers. Give each controller its own config and pool identity instead of bypassing the lock.
The Windows no-Go bootstrap prints a line like:
http2: server: error reading preface from client //./pipe/dockerDesktopLinuxEngine: file has already been closed
Docker Desktop can emit this named-pipe transport diagnostic when a client connection closes. If the bootstrap docker build succeeds and the wizard or command continues, it is not an EPAR runner-group or GitHub API failure. The wrapper suppresses only this exact successful-build diagnostic and keeps full Docker stderr when the build fails.
If startup stops, verify the selected engine and context:
docker version
docker info
docker context showResolve a failed command or unhealthy engine; do not treat every named-pipe line as harmless when Docker returned a nonzero exit code.
Docker or Compose exits with exec format error, cannot execute binary file, a platform-mismatch warning, no matching manifest, a QEMU loader error, or exit code 139.
Inspect the runner, daemon, image manifest, and Compose platform setting:
uname -m
docker info --format '{{.OSType}}/{{.Architecture}}'
docker image inspect --format '{{.Os}}/{{.Architecture}}' IMAGE
docker buildx imagetools inspect IMAGE
docker compose configno matching manifest means the image does not publish the requested platform; emulation cannot create a missing manifest. A platform warning alone does not prove failure, and exit code 139 alone does not prove an architecture mismatch. Enabling emulation does not change normal manifest selection, so request the foreign image with docker run --platform ... or the affected Compose service's platform: property. EPAR does not inject DOCKER_DEFAULT_PLATFORM.
For Docker Sandboxes, keep provider.platform native. QEMU/binfmt is disabled by default on Linux, macOS, and Windows with dockerSandboxes.architectureEmulation: native-only, which verifies the guest and private Docker architecture without attempting foreign execution. As of sbx v0.39, QEMU is not fully supported inside Docker Sandboxes, so a passing diagnostic, template build, or image pull is not proof that a foreign-architecture workload will run. Prefer building or publishing a multi-platform image instead of relying on emulation.
If a job requires another architecture, such as X64 on ARM64 or ARM64 on X64, either run EPAR on a machine with the target architecture when using the Docker Sandboxes provider, or run a separate EPAR instance/configuration with provider.type: docker-container. Docker Container uses Docker-in-Docker for isolation and can support QEMU/binfmt inside its private daemon for trusted jobs. Select it in the setup wizard by choosing C. Show compatibility providers, then Docker Container. Keep the target-architecture pool on a distinct runner label and verify the actual workload rather than only pulling the image.
The best-effort and required modes remain available as explicit configuration choices, but they do not make Docker Sandboxes QEMU fully supported on sbx v0.39. Do not add a privileged installer to a Docker Sandboxes workflow: it cannot repair a sandbox kernel that does not expose the required filesystem.
The wizard marks Docker Sandboxes unavailable, or sbx diagnose --output json reports failures.
Check the diagnostic result before editing configuration:
sbx diagnose --output jsonEPAR requires a controller architecture with an available Linux guest template and at least one diagnostic pass with zero failures. Diagnostic warnings and skipped checks remain visible but do not disable the provider. Review the failed item and its hint in the JSON output, fix the prerequisite, then choose R to refresh availability; do not manually force a provider selection or substitute a compatibility provider for a configured Docker Sandboxes pool. Use C. Show compatibility providers only when you deliberately intend to create or maintain a compatibility configuration.
sbx daemon status --json reports running and sbx diagnose -o json passes, but sbx ls --json hangs, returns an empty response slowly, or ends with a runtime-list cancellation. EPAR may repeatedly report context canceled, quarantine instances after the host-trust or inventory deadline, and keep the controller process alive with no ready capacity.
The daemon health and diagnostic commands do not prove that the /sandbox inventory path or a managed sandbox's inner Docker API is responsive. Run sbx ls --json only with an external operating-system timeout; record whether it returns valid JSON, its elapsed time, and whether stderr reports a cancelled runtime listing. Treat a slow or cancelled inventory response as provider state unknown even when daemon status and diagnostics pass.
Preserve the controller output, daemon log, process tree, durable pool lifecycle state, and a before/after count of the daemon's managed Docker-socket descriptors. Do not repeatedly invoke unbounded sbx ls, start a second EPAR controller, use sbx reset, remove a sandbox by prefix, prune Docker state, or manually delete GitHub runners. EPAR intentionally quarantines uncertain resources and keeps them counted against pool.instances until exact ownership and cleanup can be verified.
With the default dockerSandboxes.recoveryMode: exclusive-auto, EPAR keeps the controller alive, acquires a host-global recovery lock, and performs a cold stop, authoritative stopped-state confirmation, configured quiescence interval, sanitized detached start, and repeated inventory verification. During that sequence it lets already-running provider commands drain and gates new provider commands so another lifecycle operation cannot race the daemon restart. It backs off after failed attempts and preserves exact capacity while recovery is unresolved. The recovery path never invokes sbx reset, sbx logout, sbx prune, or wildcard deletion. Set recoveryMode: observe only when an operator deliberately needs to prevent daemon mutation during maintenance.
If automatic recovery is disabled or the controller is not running, stop only the affected controller before a manual daemon interruption. Then run sbx daemon stop, wait until sbx daemon status --json reports stopped, and start with env -u SSH_AUTH_SOCK -u SSH_AUTH_SOCK_GATEWAY -u SSH_AGENT_PID sbx daemon start --detach. Before starting a new EPAR controller, require sbx daemon status --json to report running, sbx diagnose -o json to have no failed checks, and repeated bounded sbx ls --json calls to complete with valid JSON in a stable time. A passing diagnostic report without a responsive inventory is not a healthy Sandbox provider.
Startup reports a template identity/digest mismatch, policy-generation drift, QEMU/binfmt setup failure, or insufficient capacity.
Docker Sandboxes resolves the configured source selector, records the exact OCI identities in a local artifact receipt, verifies the host-global policy fingerprint, and binds the immutable tonistiigi/binfmt release and native-platform manifest into the reusable template evidence. If the desired source, platform, scripts, template inputs, runner inputs, trust inputs, or locked emulation artifact changes, rerun ./start; EPAR builds and imports a replacement and activates it only after exact readback succeeds.
An imported Docker Sandboxes template does not require a matching Docker image. EPAR builds directly to a verified archive, imports that archive, and then removes the transient workspace. If startup reports a missing Docker staging image, the controller is stale; rebuild the native controller and rerun ./start. If direct archive verification or sbx template load fails, use the printed Buildx transcript and archive error; EPAR does not fall back to the memory-heavy Docker load/save path.
Capacity admission accounts for phase-overlapping physical growth on each resolved capacity domain and applies the fixed storage.minimumFree reserve once per domain. The checkout, Docker Engine, Docker Desktop disk, Docker Sandboxes state/cache, WSL distribution backing, and Tart store can be on different filesystems. Docker Sandboxes root and inner-Docker sizes are independent sparse logical maxima and are not added as immediate host usage. Inspect the matching operation with ./start storage status --operation template-build --provider docker-sandboxes --config <path> --project-root <path>: a measured domain with insufficient capacity blocks, while status=unknown and available=unknown mean EPAR could not measure that local domain and will warn but continue enforcing all measurable domains. Use the exact prune preview to reclaim space, or deliberately retry only confirmed insufficiency with --allow-insufficient-storage; that flag does not bypass an unknown measurement, remote Docker-context rejection, or malformed storage topology. A documented-default-assumed warning means EPAR verified a local Docker Desktop context but could only measure the documented default/system location; it never grants cleanup authority. Avoid broad cleanup commands: they can delete stopped containers and intentionally retained resources.
On macOS or Linux, host security asks whether to allow a Docker Sandboxes helper such as mkfs.ext4, mkfs.erofs, or containerd-shim-nerdbox-v1; macOS may say that the helper “is an app downloaded from the Internet.” After a required prompt is denied or blocked, EPAR reports create docker sandbox failed, and sbx may report 500 Internal Server Error: failed to run sandbox container. The runner is neither registered nor marked ready. That exact signature also causes EPAR to append a conditional SSH-agent daemon hint while preserving the original command error; unrelated create failures are returned without that hint.
Docker Sandboxes uses mkfs.ext4 to create an ext4 filesystem inside each sandbox's private Docker disk-image file, mkfs.erofs to construct the read-only template snapshot, and containerd-shim-nerdbox-v1 to launch and manage the sandbox VM. Expected file targets are regular sandbox-owned files beneath the Docker Sandboxes runtime data directory—for example, current macOS releases may use ~/.sbx/run/d/containerd/.../images/<runner>-docker.img and ~/.sbx/run/d/containerd/.../snapshots/<id>/layer.erofs. With the current Homebrew sbx package, the runtime and shim are beneath /opt/homebrew/Caskroom/sbx/<version>/. A formatter must not target a physical device such as /dev/disk*, an EPAR checkout, a home-directory document, or another unrelated path.
If each executable belongs to the Docker Sandboxes installation you intentionally installed and any displayed target is the expected sandbox-owned file, allow the operation through the host's security or application-control prompt, then rerun the same EPAR start or verification command if creation already failed. Do not invoke a formatter or shim yourself, disable host security broadly, or approve a command with an unfamiliar target. The configured private Docker disk is sparse, so its logical maximum does not mean the formatter immediately consumes that amount of physical storage.
If no prompt appeared, or approval still produces the 500 error, preserve the failed runner evidence and inspect the Docker Sandboxes daemon/client logs and sbx diagnose --output json; the same top-level error can also represent a runtime, capacity, or host-policy failure. See Private Filesystem and VM Helper Approval for the provider contract.
Sandbox creation reaches verify dedicated docker sandbox staging workspace and fails with a message that host SSH-agent forwarding is not permitted. Diagnostics may show SSH_AUTH_SOCK=/run/ssh-agent.sock or SSH_AUTH_SOCK_GATEWAY=... inside the guest even though the imported template does not define them.
Docker Sandboxes may forward the host SSH agent when its shared daemon inherits the host's agent environment. EPAR rejects the resulting sandbox because the forwarded socket or gateway could let a workflow use host SSH credentials. This is not evidence that the staging mount is missing or read-only, and deleting only /run/ssh-agent.sock is insufficient when the forwarding gateway remains configured.
In the default recoveryMode: exclusive-auto, EPAR treats the known immediate create-stage signature as a bounded admission incident and may perform one stop-wait-start recovery using the existing host-global gates, then retry lifecycle reconciliation. The recovery is limited to one daemon restart per incident, including controller reconciliation retries; recoveryMode: observe never probes or mutates the daemon. Coordinate the interruption with every process using the shared Docker Sandboxes daemon. If the automatic attempt has already been used or manual recovery is required, stop it and restart it with all forwarding variables removed before retrying EPAR:
sbx daemon stop
env -u SSH_AUTH_SOCK -u SSH_AUTH_SOCK_GATEWAY -u SSH_AGENT_PID sbx daemon start --detachEPAR strips these variables from Docker Sandboxes commands it launches, so a stopped daemon auto-started through those commands is sanitized, but an already-running daemon retains the environment with which another shell or tool started it. Do not disable this admission check, use sbx reset or sbx logout, or forward an agent into a reusable runner template. If the failed creation predates the immutable-receipt fix, preserve its reported sandbox UUID and use exact provider cleanup; never delete a same-name resource by prefix alone.
A workflow's Docker login step reports Login Succeeded, but a later pull of a private Docker Hub image fails with insufficient_scope: authorization failed, pull access denied, or an equivalent authorization response. The same workflow and credentials may succeed with Docker Container or a GitHub-hosted runner. Using docker --config /home/agent/.docker pull ... produces the same denial.
First verify only metadata, never credential contents: the listener should run as agent with HOME=/home/agent and DOCKER_CONFIG=/home/agent/.docker, its control-plane environment should explicitly use http://gateway.docker.internal:3128, and a post-login config should be owned by agent with restrictive permissions. The unconditional job-start hook never propagates that gateway proxy to workflow steps. On a Windows overlay runner it sets only HTTPS_PROXY/https_proxy to the raw-tunnel workflow listener at http://127.0.0.1:3130, after proving the authenticated EPAR relay is live; plain HTTP and other protocols remain direct. The private Docker daemon separately uses the TLS-terminating listener at http://127.0.0.1:3129. Otherwise the hook clears all proxy variables. If the GITHUB_ENV boundary or relay validation fails, the job stops before user steps. If Docker reaches the registry and returns an authorization response, do not investigate CA copying unless an x509 or TLS error is also present.
Inspect the host Docker Sandboxes daemon log for a message that the proxy is overriding a client-supplied registry credential with a host credential. When that message is present, the workflow credential was written correctly but dockerd used the credential-injecting forward path. An explicit guest config path cannot bypass that path.
On a Windows controller with image.hostTrustMode: overlay, /run/epar/egress-relay-active must be a root-owned regular marker, docker info must report http://127.0.0.1:3129 as its HTTPS proxy and an empty HTTP proxy, and sbx policy log <sandbox-name> must contain a fresh allowed transparent record for the controller relay port, normally rendered as localhost:<port>. A fresh Docker Hub forward record during activation is an admission failure. NoProxy=* is only the pre-activation bootstrap contract or the runtime contract for configurations that do not require the Windows relay; a Windows overlay job never falls back to that route. In either mode /etc/docker/daemon.json must be root-owned and non-symlinked. Stop the controller, let exact cleanup finish, rebuild, and use a newly created runner after any contract mismatch.
If host-trust relay activation fails, start with work/logs/epar-last-error.log. The error includes a fixed stage such as private-dockerd-contract, registry-tls-proof, or guest-bridge-health; use that stage to select the matching guest and Docker Sandboxes diagnostics. These stage messages are intentionally redacted and do not include the relay token or configuration payload.
If a transparent Docker Sandboxes connection presents Norton Web/Mail Shield Untrusted Root or another antivirus-generated untrusted issuer, do not add that issuer to EPAR's trust overlay. It is a synthetic error certificate indicating that the inspector rejected Docker Sandboxes' upstream path, not a missing ordinary inspection root. Current Windows overlay runners avoid that path by making the native controller host establish the public TCP connection. Workflow clients retain end-to-end TLS through the raw listener and validate the normal host-approved inspection chain with /opt/epar/trust/ca-bundle.pem. The Docker listener terminates daemon TLS with a root-only per-sandbox ephemeral authority and independently verifies the upstream host-approved chain, working around Docker Engine's stalled HelloRetryRequest without exposing registry credentials to the host relay. Failure of either listener, daemon restart/readback, Registry TLS proof, or fresh relay policy evidence blocks registration.
Do not set DOCKER_SANDBOXES_NO_PROXY expecting it to disable credential injection. That host variable only excludes destinations from an optional upstream proxy used after traffic reaches the mandatory Sandbox proxy. Replacing docker/login-action with docker login, combining login and pull in one shell step, or changing DOCKER_CONFIG also leaves an old daemon's forward route unchanged.
Keep the host sbx login identity intentionally different from the workflow identity when proving this fix. A successful private pull together with a fresh transparent relay-port entry and no fresh Docker Hub forward entry proves that the guest credential is authoritative. Changing the host login to match the workflow can diagnose the old interception behavior, but it is a shared-identity workaround rather than the fix.
EPAR does not query or mutate host-global sbx secrets. After creating its exact sandbox, provider admission rejects any nonempty authentication capability actually attached to that sandbox without exposing credential metadata, then passes the Docker Sandboxes gateway proxy only to registration and the Actions listener. The hook's GITHUB_ENV validation and local relay substitution are scoped to the current runner's _runner_file_commands directory; they do not mutate unrelated sandboxes or sbx login. A root-capable workflow can still deliberately reconnect a client to Docker Sandboxes' gateway proxy, and v0.37.1 has no documented per-sandbox switch that disables the interceptor. Arbitrary nested images also need their own CA installation because the host bundle path does not exist inside them. Use a least-privilege host sbx account and choose Docker Container if that residual capability is outside the trust boundary. See Docker Hub Credentials and Transparent Egress.
A GitHub 429/5xx response or an sbx command timeout makes runner health temporarily unknown; it does not prove that the Actions listener stopped. EPAR preserves uncertain local capacity, but a runner whose host-trust transport or lease cannot be maintained is quarantined and its exact GitHub registration is fenced after immutable name-and-ID verification so it cannot accept new work with an expired lease. Cleanup for an inactive listener requires two consecutive guest probes that successfully execute and explicitly report the process stopped. Review the instance guest transcript when warnings repeat; do not delete the runner merely because one API or Sandbox inspection failed.
networkBaseline: open is a sandbox-scoped public-egress compatibility rule with EPAR host-alias deny guardrails. It does not alter the host-global policy. If a required service is blocked, use a narrow additionalAllow hostname rule; do not allow host.docker.internal, gateway.docker.internal, kubernetes.docker.internal, or host.containers.internal through the Open-policy guardrails.
Plain ./start intentionally keeps fail-fast startup behavior for a human-attended invocation. For an unattended host, start the configured controller with --external-outage-retry=continuous or a bounded duration such as --external-outage-retry=4h. Use ./start status --no-github to inspect the local incident stage, attempt, next retry, and deadline even while GitHub is unavailable.
Only typed transient failures such as HTTP 408, 429, 5xx, rate-limited 403 responses, DNS failures, connection resets/refusals, and timeouts are retried. Correct authentication, ordinary authorization, certificate trust, missing image manifests, storage, local Docker or Sandbox readiness, ownership, configuration, platform, and custom-script failures instead of expecting outage supervision to mask them. EPAR never queries GitHub Status and never silently changes a provider, source image, registry, credential, or trust policy.
A GitHub App installation token can be rejected with HTTP 401 before EPAR's cached expiry. For an installation-authenticated GitHub request, EPAR invalidates that rejected cached token, obtains the current installation token, and repeats the logical request once. A second 401 remains terminal because it indicates a persistent authentication or installation problem; EPAR does not place repeated 401 responses into the external-outage retry loop. Verify github.appId, the private key, the organization installation, and Actions runner permissions when a 401 persists.
The public actions/runner Releases API has a separate unauthenticated rate limit. An EPAR GitHub App installation token is not used for that public repository because it may not have permission to read it. On a 403, 429, timeout, or transient server response, EPAR prints the HTTP status and any safe rate-limit, retry, and request-ID headers, then uses the reviewed exact release in third_party/actions-runner-release.lock.json. This is an explicit metadata-only fallback: the selected version must match image.runnerVersion when that value is pinned, the expected asset name and canonical GitHub URL are checked, and the downloaded package must still match the lock's SHA-256 before it is installed.
If EPAR says the checked-in fallback is unusable, restore that file from the EPAR release or update to a release with a refreshed lock; do not replace the digest with an unverified value. Authentication, malformed successful API responses, selector mismatches, missing assets, and other non-transient API failures remain errors rather than using the fallback.
Docker Desktop can supply a stored ghcr.io credential to Buildx even when the requested Catthehacker image is public. A stale, revoked, wrong-account, or insufficiently scoped credential can then fail authorization instead of using anonymous public access. When the failed operation is authorization-shaped, EPAR performs one separate read-only anonymous OCI descriptor probe using the configured build-trust roots. If that exact built-in source is anonymously readable, the error explains the credential conflict. EPAR does not retry the failed pull anonymously, run docker logout or tart logout, or alter Docker Desktop's credential store.
For public-only GHCR use, remove or refresh the stored credential yourself. For private GHCR use, sign in again with a valid credential that has package pull access. Removing a host credential may also remove access to private packages, so EPAR never performs that action automatically. If the anonymous diagnostic probe cannot prove public access, EPAR preserves the original failure without speculating about the credential.
Run ./start status to see the last successful remote check, next check or retry, pending immutable identity, deferred reason, and last error. A failed scheduled check or build keeps the previous exactly verified generation available and retries with bounded backoff; a missing artifact or changed local configuration still fails closed. Use ./start image update to retry an immediate remote check, or correct local input errors and rerun ./start.
An instance is retained, quarantined, or shown as requiring an acknowledgement after a provisioning, policy, or runtime failure.
Preserve the instance and inspect work/logs/instances/<instance>.guest.log, the matching runner diagnostics, and controller output before acknowledging or removing it. EPAR deliberately keeps uncertain ownership, failed cleanup, and unverified remote state inside the strict pool.instances cap instead of creating a replacement storm.
If an incident requires stopping new work immediately, stop the controller with Ctrl-C or the service manager that launched it. This prevents replacement; it does not erase retained evidence. Use the configured EPAR cleanup command only after identifying the exact affected pool. Do not use a broad docker system prune, WSL unregister, or reset as an incident-disable switch.
Set EPAR_DISABLE_DOCKER_SANDBOXES=1 before starting EPAR when Docker Sandboxes admission must remain disabled during an incident or compatibility investigation. This fails the provider closed without changing configuration or deleting evidence.
After reviewing retained Docker Sandboxes diagnostics, acknowledge that review only for the exact configured pool:
./start cleanup --acknowledge-failed-diagnosticsstart or image build reports No space left on device or E: You don't have enough free space in /var/cache/apt/archives/..
The temporary guest or Docker writable layer is full; this does not necessarily mean the host OS drive is full. Inspect the active Docker daemon:
docker run --rm ghcr.io/catthehacker/ubuntu:full-latest df -h /
docker system df
docker system df -vIncrease the relevant Docker/VM data-disk allocation or deliberately remove unneeded data after reviewing it. Docker prune commands can remove stopped containers, unused images, build cache, networks, and volumes; they are not a safe generic fix.
Old EPAR images, Docker Sandboxes templates, staging archives, or no-Go controller files remain after an update or an interrupted start.
Run ./start once more. Startup reconciles incomplete exact-owned work and retires an unreferenced superseded generation after its replacement passes readback. It does not delete shared images, prefix-only historical resources, active containers, active sandboxes, or resources referenced by another configuration.
Inspect the exact classification before removing anything manually:
./start storage status
./start storage prune
./start storage prune --legacyUse normal storage prune --execute only for exact catalog-owned resources. Legacy prefix-era entries require the plan hash printed by storage prune --legacy; they are not removed automatically. Do not use broad Docker prune/reset commands or VHDX compaction as a substitute for this review.
After .local/state, .local/cache, a Docker image, or a Docker Sandboxes template was removed manually, startup reports a missing project-local receipt/evidence path or discovers stale external ownership.
Current EPAR treats a genuinely missing cataloged receipt or compact evidence file as unavailable cache rather than trusted ownership evidence. It skips reuse of that external template and reacquires or rebuilds the configured immutable artifact; malformed JSON, digest disagreement, redirected paths, and non-regular evidence still fail closed as integrity errors.
First stop other controllers for the same configuration and start once more. If recovery remains blocked, preview an exact reset:
./start storage status --config .local/config.yml
./start storage reset --config .local/config.ymlReview every exact target and shared-resource line. Execute only with the new preview hash printed by the command. The config, GitHub App key, certificates, and custom scripts are preserved, while the next start performs a cold acquisition. See Generated files and recovery before deleting project or host data manually.
After switching between Docker Desktop, OrbStack, or another local Docker daemon, Buildx output names the previous context or socket, reports a node-level connection error despite a successful docker buildx inspect exit status, or reports that the expected buildx_buildkit_epar-... control container does not exist on the current daemon.
Current EPAR controllers record the Docker Engine identity with each config-scoped Buildx receipt. When valid ownership metadata proves that the stale definition belongs to the current project and configuration, EPAR automatically removes only that exact client-side definition while preserving daemon-local BuildKit state, recreates it against the active backend, bootstraps it, and verifies its image, configuration, registry trust, and CA bundle before continuing. Missing control containers and node errors embedded in otherwise successful Buildx output receive the same bounded recovery. No manual docker rm, docker buildx rm, or broad prune is normally required.
EPAR never adopts or removes a same-name builder without valid exact ownership metadata. If automatic recovery reports that the definition remained present, or the recreated builder fails verification twice, preserve the error report and inspect .local/storage/buildx/<config-id>/metadata.json together with docker context show, docker info, and docker buildx inspect <builder>; repair the real daemon or ownership problem rather than deleting shared Docker state. A context that changes repeatedly during one reconciliation is detected and retried once, but the Docker target must remain stable for the image build itself. An unavailable daemon, failed registry access, or failure to produce a required verified artifact can still block startup.
HTTPS access fails with curl: (60), certificate verification failed, or an unknown issuer during an EPAR build or job.
Do not disable certificate verification. First identify which trust boundary failed.
For a no-Go native-controller build, ./start automatically reads host system roots, excludes explicitly distrusted certificates, validates the short-lived feed in an offline container, and mounts only the resulting CA bundle into the Go compiler container. Runner CA inheritance remains independent. If the build still reports an unknown issuer, inspect work/logs/epar-native-controller-build.log: the wrapper prints the requested host, presented certificate subject and issuer, SHA-256 fingerprint, validity, and verification result, and on Windows it lists matching roots from LocalMachine\Root and CurrentUser\Root. A remaining failure means the expected issuer was absent, distrusted, malformed, expired, or not the certificate actually presented; EPAR never disables TLS verification or retries insecurely.
For an EPAR Buildx failure, leave image.hostTrustMode unchanged. EPAR automatically supplies host system roots to its config-owned builder and prints the full build transcript path before docker buildx build. The console and error report include a bounded redacted tail. Inspect the underlying x509 line together with the registry host, builder identity, and active trust generation:
docker buildx ls
Get-ChildItem .local/storage/buildx -Recurse -Filter metadata.json | Get-Content
Get-ChildItem .local/storage/buildkit -Recurse -Filter buildkitd.toml | Get-ContentThe owned metadata records the Docker backend identity, exact registry set, configuration digest, certificate bundle, and trust generation. Rerunning the same command reconciles that exact builder and preserves its daemon-local BuildKit state; EPAR never changes Docker's shared/default builder. If the source-image docker pull itself fails before Buildx starts, configure the authorized CA in the host daemon because builder trust cannot repair host-daemon trust.
Configure runner overlay only when jobs inside an ephemeral runner must inherit host roots:
image:
hostTrustMode: overlay
hostTrustScopes: [system, user]Use [system] on Linux. Overlay mode collects the current host roots, validates them before registration, and combines them with Ubuntu roots and any image.trustedCaCertificatePaths; it is root-anchor inheritance rather than exact Windows/macOS TLS-policy emulation. It requires runner.ephemeral: true. Omitted or disabled mode remains valid for Docker Sandboxes; the unconditional job-start preparation hook still runs and accepts the explicit disabled marker while clearing workflow proxy variables.
Use the normal host entry point so EPAR can inspect the real Windows certificate stores or macOS Keychain:
./start
./start image build --replaceUse ./start image build --replace on every supported shell, including native Windows PowerShell. The wrapper uses a native-host trust feed while compiling the native controller; the resulting native controller reads host trust directly for start, image build, pool up, and pool verify, even when runner overlay is disabled. Direct scripts/run-with-docker.* calls are wrapper-development diagnostics. A bare Linux toolchain container is not a replacement for the project-local native-controller path.
docker run --rm ghcr.io/catthehacker/ubuntu:full-latest df -h / shows much less capacity than Windows Explorer.
Docker Desktop stores Linux container data in a WSL-backed virtual disk. Older WSL2 installations can have a smaller default VHD maximum than newer ones, but the reported container filesystem is the evidence that matters for image pulls and builds. Inspect Docker usage first, then change Docker Desktop/WSL storage using the product's supported settings. See Microsoft WSL disk-space guidance and Docker Desktop WSL guidance.
Docker Container requires a host Docker runtime that permits privileged Linux containers:
docker run --rm --privileged alpine:3.20 trueIf nested Docker operations fail with overlay-mount errors, retain the default inner storage driver:
EPAR_DOCKERD_STORAGE_DRIVER=vfs
Use overlay2 or auto only in a derived image after proving it works on the exact host runtime.
The WSL image build fails before import, during import with 0xffffffff, or before systemd is ready.
The default WSL build obtains a Docker source image before importing it into WSL. Verify Docker and WSL first:
docker version
docker pull ghcr.io/catthehacker/ubuntu:full-latest
wsl -l -vFor Wsl/Service/CreateInstance/E_UNEXPECTED, Catastrophic failure, or import exit 0xffffffff, stop EPAR, save work in other distros, then run wsl --shutdown. This stops every running WSL distro, including any Docker backend using WSL. Restart the affected Docker host runtime, verify a normal distro command returns 0, then rerun ./start; a matching cached source rootfs is reused. If it persists, update WSL, shut it down again, reboot, and consult Microsoft's WSL troubleshooting guidance.
If a guest exists but systemd does not become ready, inspect work/logs/builds/<image>.wsl-build.log and work/logs/builds/<temporary-distro>.guest.log. Do not unregister a distro until you have identified the exact EPAR-owned target and accepted that unregistration is irreversible.
EPAR cannot request a registration token, add a runner to a group, or observe the runner online.
Verify GitHub App organization self-hosted-runner read/write permission and a readable private key:
github:
appId: 123456
organization: your-org
privateKeyPath: .local/github-app.pemThen inspect runner-group policy and the first registration error. A strict policy can intentionally block a group that is default, overly broad, or public-repository enabled. See Runner Group Security.
For a confirmed stale EPAR resource, run the configured cleanup command:
./start cleanupCleanup is bounded by the configured pool and durable exact lifecycle identities; it does not authorize a broad prefix deletion, wildcard, Docker prune, or removal of unknown/shared resources. Keep pool.namePrefix unique per controller and organization.
Generated from the main repository docs at 7e156a9. Edit README.md and docs/; the wiki copy is overwritten by automation.