Nix

This is the nix driven monorepo, organized as small modules and composed via flake.nix.

Layout

• flake.nix: flake inputs and system definition.
• pkgs/: repo-local runnable source trees; each package owns its own flake and is aggregated into top-level flake outputs such as .#example-hello-rust, .#cloudflare-apps, .#cloudflare-apps-deploy, or package-local child-flake entrypoints such as ./pkgs/cloudflare-apps/llmug-hello#wrangler-deploy.
• pkgs/examples/: example packages used as reference implementations and test beds for package patterns.
• lib/ext/: standalone derivation definitions consumed by overlays and helper scripts.
• hosts/<host>/default.nix: host-specific system definition and module imports.
• users/pvl/default.nix: Base user + Home Manager module builder for pvl.
• lib/*.nix: single-topic NixOS modules imported directly by hosts.
• lib/flake/: flake support helpers such as linting and the custom flake tree helper.
• overlays/: custom overlays used by the system.
• hosts/nixbot.nix: deploy mapping (plain Nix attrset).
• data/secrets/default.nix: agenix recipients map for *.age files.

Services

The repo has three service models, each documented in detail under docs/.

Native Services (docs/services.md)

The default model for turning repo packages into system services. Stick to the simplest native Linux patterns:

• Package lives in a package-local default.nix under pkgs/.
• Package-local flake.nix exports a nixosModules entry.
• Host enables the service with services.<name>.enable = true.
• The module defines plain systemd.services and, when scheduled, systemd.timers.
• No repo-specific service abstraction — NixOS modules plus systemd units are the framework.

Reference example: pkgs/examples/hello-rust/flake.nix.

Podman Compose (docs/podman-compose.md)

Container workloads run as rootless Podman compose stacks managed by a shared NixOS module:

• Shared Podman base config lives in lib/podman.nix.
• Shared compose lifecycle logic lives in lib/podman-compose/default.nix with shared shell logic in lib/podman-compose/helper.sh.
• Deploy-time user-manager orchestration lives in lib/systemd-user-manager/default.nix with shared shell logic in lib/systemd-user-manager/helper.sh (documented in docs/systemd-user-manager.md).
• Hosts declare stacks under services.podman-compose.<stack> in a host service module, commonly hosts/<host>/services.nix or hosts/<host>/services/default.nix.
• Compose content can be a Nix attrset, inline YAML, a file path, or a staged directory tree.
• Lifecycle tags (bootTag, recreateTag, imageTag) drive deploy-time actions such as restart, force-recreate, and image pull.
• exposedPorts metadata auto-derives firewall rules, nginx reverse-proxy config, and Cloudflare Tunnel ingress.
• Secrets are injected via file-backed envSecrets.

Incus Guests (docs/incus-vms.md)

VM and container guests run under Incus with a shared declarative lifecycle module:

• Parent host orchestration lives in hosts/<parent-host>/incus.nix.
• Shared lifecycle logic lives in lib/incus/default.nix.
• Reusable guest bootstrap lives in the selected machine profile under lib/profiles/.
• Base image builds live in lib/images/incus-lxc-base.nix and lib/images/incus-vm-base.nix.
• Guests are declared under services.incus-manager.instances.<name>.
• Machines can use the shared default image or point at per-machine image overrides.
• Lifecycle tags (bootTag, recreateTag, imageTag) control guest stop/start, delete/recreate, and declared image re-import.
• Config-hash changes and non-disk device changes trigger automatic guest recreate; disk devices sync in place.
• After bootstrap, guests become normal nixbot deploy targets.

Guiding Principle

Prefer the native operating model of each tool — systemd for services, Podman for container workloads, Incus for guests — and define consistent patterns and naming conventions on top rather than inventing repo-specific abstractions.

Hosts (docs/hosts.md)

This repo manages NixOS hosts agnostic of where they run. A host can be a physical machine, a VM on any hosted provider (GCP, AWS, Hetzner, etc), an Incus container on a local server, laptop, edge device or anything that boots NixOS. The repo does not encode provider-specific logic at the host level — all hosts are first-class citizens regardless of their backing infrastructure.

• Each host is a directory under hosts/<host-name>/ with a default.nix entry point that imports the appropriate profile and host-specific modules.
• Profiles under lib/profiles/ provide layered baselines (core.nix for headless, all.nix for desktop, lxc.nix for Incus guests).
• Device modules under lib/devices/ encode physical hardware quirks.
• hosts/default.nix registers every host into nixosConfigurations.
• hosts/nixbot.nix maps deploy targets, SSH routing, and ordering.
• The same deploy flow, secret model, and module composition apply whether the target is a laptop, a cloud VM, or a nested Incus container.

GitHub Actions Deploy

Workflow: .github/workflows/nixbot.yaml.

• Push to master: trigger build-only run.
• Manual (workflow_dispatch): set hosts and optionally deploy. Host selectors accept exact names, all, globs, and ! exclusions such as all,!pvl-a1.

The workflow is intentionally thin: it only SSHes into the configured CI host via the packaged nixbot entrypoint with --ci-trigger.

Security note: deploy does not SCP/upload a script to CI host at runtime. The CI host forced-command key is restricted directly to the packaged nixbot command from pkgs/tools/nixbot, so CI/local trigger only invokes that allowed command.

Deployment

High-level architecture:

• GitHub Actions connects to the configured CI host using a restricted ingress key and forced command (ssh-gate).
• CI host runs the packaged nixbot command directly from the Nix store to build/deploy selected NixOS hosts.
• Deploy SSH key material is stored as age-encrypted secrets in data/secrets/*.age, with bootstrap and rotation rules documented in deployment docs.

Deployment-specific architecture, key model, bootstrap flow, rotation procedure, and operational notes are documented in:

• docs/deployment.md
• docs/nixbot-security-trust-model.md

Primary files for deployment are:

• hosts/nixbot.nix (deploy target mapping/defaults)
• pkgs/tools/nixbot/ (canonical packaged nixbot source)
• nixbot (packaged deployment entrypoint)
• pkgs/tools/nixbot/flake.nix (packaged nixbot application wrapper)
• pkgs/tools/nixbot/nixos-module.nix (package-owned NixOS service module)
• hosts/common/all.nix and hosts/common/ci.nix (host-side nixbot policy)
• nixbot runs in a cached nix shell toolchain with pinned tools (nix, age, cloudflared, coreutils, git, jq, nixos-rebuild-ng, openssh, opentofu, procps) so deploy runs use consistent command sets everywhere.
• The packaged nixbot wrapper ships that same runtime toolchain directly and executes the packaged nixbot entrypoint without depending on repo-relative flake discovery.

Deploy Actions

nixbot supports multiple top-level actions:

• deps: enter the pinned nixbot runtime shell, verify the required toolchain, and exit
• check-deps: verify the required toolchain is already available in the current environment and exit
• --list-hosts: list selected deploy hosts using the same ordered host block shown in the run banner, then exit
• run (default full workflow): build/deploy flow with optional TF phases
• deploy: host build and deploy only
• build: host build only
• dev-build: local host build only, writing result-dev/<host> links into the current repo checkout as temporary GC roots
• tf: all Terraform phases (tf-dns, tf-platform, tf-apps)
• tf-dns: Cloudflare DNS only
• tf-platform: Cloudflare platform resources only
• tf-apps: Cloudflare Workers/package deployments only

OpenTofu

Infrastructure managed outside NixOS modules lives in tf/.

• tf/: Terraform/OpenTofu project container.
• tf/cloudflare-dns/: pre-deploy Cloudflare DNS OpenTofu project.
• tf/cloudflare-platform/: Cloudflare platform OpenTofu project for non-app resources.
• tf/cloudflare-apps/: post-build Cloudflare Workers/package OpenTofu project.
• tf/modules/cloudflare/: Cloudflare module implementation shared by the phase-specific projects.
• tf/README.md: Terraform project layout docs.
• nixbot tf-dns|tf-platform|tf-apps: runs the phase-specific OpenTofu projects locally or through the CI host-trigger path used by nixbot. Project discovery is suffix-based, so future tf/<provider>-dns, tf/<provider>-platform, and tf/<provider>-apps projects participate automatically.
• .github/workflows/nixbot.yaml: can dispatch the same CI host-based build/deploy flow and the standard Terraform phase actions only; it does not expose per-project tf/<project> actions.
• Terraform credentials can be stored as repo-managed age secrets under data/secrets/cloudflare/*.key.age; the phase-specific OpenTofu actions decrypt them on demand using the existing CI host age key.

Deploy Ordering

• nixbot run runs Cloudflare in phases: tf-dns, tf-platform, host build/deploy, then tf-apps.
• nixbot tf runs the Terraform phases only: tf-dns, tf-platform, then tf-apps.
• hosts/nixbot.nix may declare per-host deps = [ ... ]; for build/deploy ordering.
• All selected hosts are built before deploy starts.
• Build parallelism: NIXBOT_BUILD_JOBS / --build-jobs. When build jobs is greater than 1, nixbot disables Nix's flake eval cache for per-host build and output-path evaluation commands to avoid parallel SQLite cache contention.
• Deploy parallelism: NIXBOT_JOBS / --deploy-jobs (default: 16) for deploy work and rollback execution.
• Verification parallelism: NIXBOT_VERIFY_JOBS / --verify-jobs (default: 16) for rollback snapshots and health checks.
• NIXBOT_CI_FIRST / --ci-first prioritizes the CI host first for both build ordering and deploy waves when selected.
• Deploy derives dependency waves from deps.

Package Conventions

Package and child-flake conventions are documented in docs/flake-package.md.

Linting

• nix fmt: format the repo.
• nix run .#lint: run the standard lint and check flow.
• nix run .#lint -- fix: apply safe auto-fixes, then re-run lint.
• ./scripts/git-install-hooks.sh: install the repo Git hooks.
• The Git hook runs on pre-push, not pre-commit, and only lints the changed scope.
• Full lint workflow details live in docs/linting.md.

Contributing

Before pushing upstream, install the repo hooks once:

./scripts/git-install-hooks.sh

Use these checks locally before you push:

nix fmt
nix run .#lint

The Git hook is intentionally on pre-push, not pre-commit. That keeps commit-time iteration fast and avoids running the full lint flow on every local commit.

The pre-push hook is smart diff-based linting. It computes the push base and only runs the root diff checks and package checks for what changed, using the same nix run .#lint -- --diff --base <base> flow that CI and manual local usage rely on.

Lint workflow details and package-level commands are documented in docs/linting.md.