Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions ansible/roles/hyrule_cloud/defaults/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,9 @@ hyrule_cloud_required_env_keys:
- HCP_LAUNCH_PROOF_REAL_XCPNG
- HYRULE_MAX_PAID_ACTIVE_VMS
- HYRULE_NETWORK_PROXY_TOKEN
- HYRULE_CUSTOMER_IPV6_SUPERNET
- HYRULE_CUSTOMER_IPV6_GATEWAY
- HYRULE_CUSTOMER_IPV6_DNS

# Base apt packages on the target. uv is installed separately via the
# upstream script (same idiom as hyrule_web / hyrule_mcp).
Expand Down
11 changes: 11 additions & 0 deletions ansible/roles/hyrule_cloud/tasks/health.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,17 @@
- name: Flush handlers (reload systemd if the unit changed) before restart
ansible.builtin.meta: flush_handlers

- name: Run hyrule-cloud database migrations
ansible.builtin.command:
cmd: /usr/local/bin/uv run alembic upgrade head

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Load the rendered env before running migrations

In production apply this new task runs outside systemd, so it does not inherit EnvironmentFile=/opt/hyrule-cloud/.env that the existing service ExecStartPre uses. The role only supplies HYRULE_DATABASE_URL via that rendered env, and uv run only loads env files when passed --env-file (checked uv help run), so Alembic will fail before the restart/health check whenever it reads the app DB URL. Please invoke Alembic with the rendered env file or keep relying on the systemd pre-start migration.

Useful? React with 👍 / 👎.

chdir: "{{ hyrule_cloud_install_dir }}"
become: true
become_user: "{{ hyrule_cloud_app_user }}"
environment:
PYTHONPATH: "{{ hyrule_cloud_install_dir }}"
changed_when: false
when: hyrule_cloud_pyproject.stat.exists

- name: Restart hyrule-cloud (deterministic on every apply)
ansible.builtin.systemd:
name: "{{ hyrule_cloud_service_name }}"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ HYRULE_IP_PREFIX_PEPPER={{ .Data.data.ip_prefix_pepper }}
HYRULE_NETWORK_PROXY_URL=http://[2a0c:b641:b50:2::e0]:8450
HYRULE_NETWORK_PROXY_TOKEN={{ or .Data.data.network_proxy_token "" }}
HYRULE_NETWORK_PROXY_HEALTH_TTL_SECONDS=15
HYRULE_CUSTOMER_IPV6_SUPERNET={{ or .Data.data.customer_ipv6_supernet "2a0c:b641:b51::/48" }}
HYRULE_CUSTOMER_IPV6_GATEWAY={{ or .Data.data.customer_ipv6_gateway "2a0c:b641:b51::1" }}
HYRULE_CUSTOMER_IPV6_DNS={{ or .Data.data.customer_ipv6_dns "2a0c:b641:b51::1" }}

# DNS (RFC 2136 dynamic updates to Knot DNS)
HYRULE_DNS_SERVER=2a0c:b641:b50:2::10
Expand Down
3 changes: 3 additions & 0 deletions configs/hyrule-cloud.env.j2
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,9 @@ HYRULE_DEPLOY_DOMAIN=deploy.hyrule.host
HYRULE_NETWORK_PROXY_URL=http://[2a0c:b641:b50:2::e0]:8450
HYRULE_NETWORK_PROXY_TOKEN={{ network_proxy_token | default('') }}
HYRULE_NETWORK_PROXY_HEALTH_TTL_SECONDS=15
HYRULE_CUSTOMER_IPV6_SUPERNET={{ hyrule_customer_ipv6_supernet | default('2a0c:b641:b51::/48') }}
HYRULE_CUSTOMER_IPV6_GATEWAY={{ hyrule_customer_ipv6_gateway | default('2a0c:b641:b51::1') }}
HYRULE_CUSTOMER_IPV6_DNS={{ hyrule_customer_ipv6_dns | default('2a0c:b641:b51::1') }}

# DNS (RFC 2136 dynamic updates to Knot DNS)
HYRULE_DNS_SERVER=2a0c:b641:b50:2::10
Expand Down
24 changes: 24 additions & 0 deletions docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,30 @@ Internet (IPv4 + IPv6)
xenbr-mgmt (link-local): dom0, rtr enX0, xoa (+ 10.0.0.x for XOA→XAPI)
```

### Customer VM IPv6 provisioning

Customer VMs live on the shared XCP-NG `xenbr-vm` L2 segment behind `rtr`
`enX3`. `rtr` owns the customer aggregate as `2a0c:b641:b51::1/48` in
FRR and advertises `2a0c:b641:b51::/48` by BGP.

There is intentionally no RA/DHCPv6 service on `enX3` for paid VMs. The
customer design is one `/64` per VM, and RA is link-wide on the shared L2, so
it cannot hand a different `/64` to each VM. Hyrule allocates a deterministic
per-VM `/64` from `2a0c:b641:b51::/48` and passes Debian netplan v2
`networkConfig` through XO:

- prefix index `0` (`2a0c:b641:b51::/64`) is reserved for the router, legacy
shared addresses, and static canaries such as `ci-pr`;
- paid VM prefixes start at index `1`, with the VM address set to `::2` inside
its allocated `/64`;
- the default route points at `2a0c:b641:b51::1` with `on-link: true` because
the gateway address is outside each VM-specific `/64`;
- DNS points at `2a0c:b641:b51::1`.

This is a tactical allocator in `hyrule-cloud`, not the long-term source of
truth. Proper IPAM deployment and Hyrule integration are tracked in
<https://github.com/AS215932/network-operations/issues/346>.

### Routers (WireGuard mesh, iBGP + OSPF6)

| Router | Location | OS | Underlay address | Loopback | Router-ID |
Expand Down
18 changes: 18 additions & 0 deletions docs/rtr-vrf.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,24 @@ filter allows proto 89 on both the VRF master (`overlay`) and WG slaves
must time out, while DNS to rtr on the customer gateway and public HTTPS egress
must still work.

## Customer VM addressing

`enX3` carries the customer aggregate `2a0c:b641:b51::/48`; FRR assigns
`2a0c:b641:b51::1/48` to the interface and advertises the aggregate. Paid VMs
use one `/64` each from that `/48`.

Do not enable `IPv6SendRA`, radvd, dnsmasq, Kea, or another link-wide
RA/DHCPv6 service on `enX3` for this design. The customer VMs share one L2
segment, and RA would advertise the same prefix information to every VM.
Instead, `hyrule-cloud` allocates a per-VM `/64` and passes static Debian
netplan `networkConfig` to XO. The guest address is `::2` inside its assigned
`/64`; the default route uses `2a0c:b641:b51::1` with `on-link: true`.

`2a0c:b641:b51::/64` remains reserved for router/legacy/static use, including
the existing `ci-pr` address. Proper IPAM is tracked separately in
<https://github.com/AS215932/network-operations/issues/346>; until then the
Hyrule allocator is the tactical source of VM prefix assignments.

## IPv4 DNAT VRF leak

The infra subnet `10.0.2.0/24` lives on `enX2` (overlay VRF). External
Expand Down
3 changes: 3 additions & 0 deletions tests/iac/mock_inventory.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,3 +81,6 @@ hyrule_cloud_required_env_keys:
- HYRULE_DNS_TSIG_KEY
- HYRULE_DATABASE_URL
- HYRULE_NETWORK_PROXY_TOKEN
- HYRULE_CUSTOMER_IPV6_SUPERNET
- HYRULE_CUSTOMER_IPV6_GATEWAY
- HYRULE_CUSTOMER_IPV6_DNS
3 changes: 3 additions & 0 deletions tests/iac/test_mock_render.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,9 @@ def test_hyrule_cloud_vault_template_contains_secret_keys(self):
"tsig_secret",
"db_password",
"network_proxy_token",
"customer_ipv6_supernet",
"customer_ipv6_gateway",
"customer_ipv6_dns",
):
self.assertIn(f".Data.data.{key}", rendered)

Expand Down
15 changes: 15 additions & 0 deletions tests/iac/test_vault_and_runner_contracts.py
Original file line number Diff line number Diff line change
Expand Up @@ -465,6 +465,21 @@ def test_app_roles_restart_deterministically_on_apply(self):
self.assertIsNotNone(restart_task, role)
self.assertEqual(restart_task["ansible.builtin.systemd"]["state"], "restarted")

def test_hyrule_cloud_runs_migrations_before_restart(self):
health_tasks = yaml.safe_load((REPO / "ansible/roles/hyrule_cloud/tasks/health.yml").read_text())
task_names = [task["name"] for task in health_tasks]
migration_task = _task_by_name(health_tasks, "Run hyrule-cloud database migrations")

self.assertIsNotNone(migration_task)
self.assertLess(
task_names.index("Run hyrule-cloud database migrations"),
task_names.index("Restart hyrule-cloud (deterministic on every apply)"),
)
command = migration_task["ansible.builtin.command"]
self.assertEqual(command["cmd"], "/usr/local/bin/uv run alembic upgrade head")
self.assertEqual(command["chdir"], "{{ hyrule_cloud_install_dir }}")
self.assertEqual(migration_task["environment"]["PYTHONPATH"], "{{ hyrule_cloud_install_dir }}")

def test_noc_action_signing_secret_has_no_empty_fallback(self):
vault_template = (REPO / "ansible/roles/vault_agent/templates/noc-agent.env.ctmpl.j2").read_text()
noc_env = (REPO / "configs/noc-agent.env.j2").read_text()
Expand Down
Loading