Skip to content

Ship cross-target staticlib bundles in release CI so builders don't recompile runtime/stdlib/ui per target #1083

Description

@proggeramlug

Problem

Every Perry build worker that cross-compiles to a target other than its
host (Linux worker → ios/macos/tvos/android/windows; the macOS worker
→ ios/tvos; etc.) currently has to build libperry_runtime.a,
libperry_stdlib.a, and libperry_ui_*.a from source for each target
on every perry update
. The release tarballs ship host-only artifacts.

For the linux x86_64 host bundle:

perry-linux-x86_64.tar.gz (144 MB)
  perry                    ← CLI
  libperry_runtime.a       ← LINUX HOST only
  libperry_stdlib.a        ← LINUX HOST only
  libperry_ui_gtk4.a       ← LINUX HOST only

If a worker also handles Android, iOS, macOS, tvOS, Windows — it has to
build each target's libs itself via cargo build --release -p perry-runtime -p perry-stdlib -p perry-ui-<target> --target <triple>
plus environment plumbing for the cross toolchain (CC, CXX, CFLAGS,
SDKROOT, AR, BINDGEN_EXTRA_CLANG_ARGS, and so on).

Cost we keep paying

Over the past three days a single project (PerryTS/builder linux worker

  • hub) hit, recovered from, and re-hit:
  • aws-lc-sys jitterentropy includes macOS-only CoreServices/ CoreServices.h when compiled against the iOS / tvOS sysroot. Fix
    was a systemd drop-in on the worker setting
    AWS_LC_SYS_NO_JITTER_ENTROPY=1. Without it every Apple cross-target
    rebuild fails non-fatally and leaves a stale .a in place, then
    every user iOS/macOS build link-fails with
    _perry_scene_will_connect / __perry_user_main undefined.
  • Windows .lib rebuild needs a Windows host in the perry pipeline.
    Today's setup SSHes from the Linux worker into a small Azure
    Windows VM, runs cargo build there, scps the resulting .libs
    back. The VM disk is 30 GB and fills up mid-build at least once per
    update cycle, hanging the SSH session forever (had to pkill -f perryadmin@4.207 four times in three days). Each hang stalls the
    whole perry-update flow.
  • oboe-sys (Android C++ audio binding) needs the NDK's clang++
    wrapper. cc-rs's default lookup is aarch64-linux-android-clang++
    (no API version) which the NDK doesn't provide. Worker-side env fix
    set CXX_aarch64_linux_android to an absolute path; without it
    every Android user build with this transitive dep fails.
  • alsa-sys (Linux audio) needs libasound2-dev in the build
    container. Not in the worker's Dockerfile before today; every Linux
    native user build with this dep failed.
  • minimp3-sys on Windows MSVC target wants lib.exe. We pass
    AR_x86_64_pc_windows_msvc=llvm-lib + CC=clang-cl now, but the
    whole class of "cc-rs hardcodes a host-OS tool name" problems
    recurs whenever an npm-published native dep makes a new assumption.

These are all worker-side environment-fragility problems. The perry
project itself produces the correct .as in CI for tests — that's
already proven. We're paying the cost of re-deriving the build env
across all consumers' workers, every release, forever.

Proposal

Have release CI emit per-target staticlib bundles alongside the host
bundle
, one tarball per Rust target triple:

perry-linux-x86_64.tar.gz               (unchanged — host)
perry-macos-x86_64.tar.gz               (unchanged — host)
...

# NEW (one per cross-compile target Perry officially supports):
perry-cross-aarch64-apple-ios.tar.gz
perry-cross-aarch64-apple-darwin.tar.gz
perry-cross-aarch64-apple-tvos.tar.gz
perry-cross-aarch64-linux-android.tar.gz
perry-cross-x86_64-pc-windows-msvc.tar.gz

Each cross-target bundle contains exactly what a consumer worker needs
to LINK against without re-deriving:

perry-cross-aarch64-apple-ios.tar.gz:
  libperry_runtime.a
  libperry_runtime_gameloop.a   (when applicable — currently swapped manually)
  libperry_stdlib.a
  libperry_ui_ios.a
  manifest.json                  ← versions + sha256 of each file

Build each on the obvious CI runner (Apple targets on macos-latest with
real Xcode, Windows MSVC on windows-latest, Android on ubuntu-latest +
NDK, Linux on ubuntu-latest). All of these are runners that GitHub
Actions already provides for free, and each one is the correct host
for that toolchain — no Linux→Apple env-wrangling, no Linux→Windows
Azure-VM dance.

Worker-side simplification

The Perry-builder-linux run_perry_update function today is ~350 LOC
of cross-compile orchestration plus an SSH-to-Azure step. With shipped
prebuilts it becomes:

async fn run_perry_update(perry_binary: &str) -> Result<...> {
    let version = latest_release_version().await?;
    let triple = host_triple();

    download_and_extract(
        format!(\"perry-{triple}.tar.gz\"),
        // overwrites perry CLI + host libs
    ).await?;

    for cross in cross_targets_we_support() {
        download_and_extract(
            format!(\"perry-cross-{cross}.tar.gz\"),
            // overwrites that target's .a's
        ).await?;
    }
}

A 5-25-min build flow becomes a 30-second download. Every fragility
the worker hits today (AWS_LC, Azure disk, NDK lookup, alsa headers)
just stops being its problem.

Acceptance criteria

A clean Linux machine with only perry (no Rust toolchain, no Apple
SDK sysroot, no Android NDK, no Azure VM, no lib.exe)
can:

  1. perry compile src/main.ts --target ios → produces working iOS bundle
  2. perry compile src/main.ts --target macos → produces working macOS bundle
  3. perry compile src/main.ts --target tvos → produces working tvOS bundle
  4. perry compile src/main.ts --target android → produces working APK
  5. perry compile src/main.ts --target windows → produces working .exe

All by linking against the prebuilt cross-target .as shipped in the
release. Today only target #5 needs a Windows VM and only target #4
needs the NDK; with this change neither is needed.

Out of scope

  • Optimizing the host bundle itself.
  • npm-published native crate prebuilts (separate issue at the
    bloomengine engine level — that's a different layer and a different
    set of consumers).
  • The Perry CLI runtime version mismatch story between a worker's
    perry binary and its cross-libs — both come from the same release
    tarball so they're version-locked by construction.

Concrete pain point evidence

The three-day saga that produced this issue lives in PerryTS/builder
git history and in the running session at the time, but TL;DR:

  • 5 Apple cross-compile recompile cycles, each 5-25 min, each
    preceded by SSH-into-Azure hang fixes;
  • bloom-jump (a real user game) failed every iOS/macOS build for 48h
    because the worker's libperry_runtime_ios.a was a stale ABI
    against the new compiler — caused by perry-stdlib iOS rebuild
    failing non-fatally inside the worker;
  • 3 separate Azure-disk-full incidents during Windows .lib rebuild;
  • One full hub OOM crash whose root cause was unrelated but whose
    recovery was complicated by the worker being mid-cross-compile.

None of this would happen if the release CI shipped the libs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions