Skip to content

build/pysimlin: CFFI extension rebuild silently reuses a stale libsimlin.a (no static-lib dependency tracking) #682

Description

@bpowers

Problem

The pysimlin CFFI extension can silently relink/reuse a .so built against a stale native static library after a Rust-only change to simlin-engine/libsimlin, masking the engine change from Python tests.

The documented dev rebuild flow (src/pysimlin/DEVELOPMENT.md) is, in two steps:

  1. Rebuild libsimlin: cd src/libsimlin && cargo build --release --features mimalloc
  2. Rebuild the extension: cd src/pysimlin && rm -f simlin/_clib.abi3.so && uv run python setup.py build_ext --inplace

The footgun is that setuptools/cffi dependency tracking only covers the generated C source, not the static archive it links. In src/pysimlin/simlin/_ffi_build.py, the native lib is passed via:

ffibuilder.set_source(
    "simlin._clib",
    ...,
    extra_objects=[_get_library_path()],   # -> <repo>/target/release/libsimlin.a
    ...,
)

extra_objects entries are linked but are not part of setuptools' source-newer-than-target staleness check. So after a Rust-only change rebuilds target/release/libsimlin.a, build_ext --inplace can report the C source is "already up-to-date" and re-copy a .so from the build cache (build/temp.../...) that is still linked against the old libsimlin.a. Deleting simlin/_clib.abi3.so does not help, because the cached compiled object under build/temp.../ is what gets reused.

How it was discovered

Observed today (2026-06-02) while validating an engine behavior change end-to-end through pysimlin. After rebuilding libsimlin.a and running the documented build_ext --inplace rebuild, the Python tests kept exercising the old engine behavior. Only build_ext --force (which forces a recompile/relink) actually picked up the engine change.

Why it matters

  • Correctness / false confidence: a silent stale-lib reuse makes pysimlin tests pass against the OLD engine, so a developer (or reviewer) can conclude an engine change is validated through Python when it is not. The failure is invisible -- no error, just stale results.
  • CI risk: if build caches (build/temp.../) are ever reused across runs in CI, the same stale-lib reuse could mask engine regressions there too.
  • Developer experience: the documented flow looks correct but doesn't reliably do what it claims; the make dev target (build-lib + uv sync --extra dev) builds the lib but never invokes build_ext at all, so it shares the same blind spot.

Components affected

  • src/pysimlin/simlin/_ffi_build.py (_get_library_path, set_source(..., extra_objects=[...]))
  • src/pysimlin/setup.py (cffi cffi_modules build hook)
  • src/pysimlin/Makefile (dev / build targets)
  • src/pysimlin/DEVELOPMENT.md (documented rebuild flow)
  • Possibly scripts/dev-init.sh

Possible approaches

  1. Make the extension build depend on the static lib's mtime. A custom build_ext subclass that compares libsimlin.a mtime against the built .so/object and forces a relink when the archive is newer. This is the principled fix -- it tracks the real dependency rather than papering over it.
  2. Always force in the documented flow / Makefile. Change DEVELOPMENT.md and the Makefile to pass build_ext --force (and add a build_ext --inplace --force step to make dev). Simpler, but blunt (always recompiles).
  3. Have scripts/dev-init.sh / a Makefile dev target own the rebuild so a single command reliably rebuilds both the lib and the extension with correct invalidation.

Approach 1 is preferred since it makes the dependency correct for all entry points (manual, Makefile, CI) rather than relying on every caller remembering --force.

Context

Identified during manual validation of an engine behavior change through pysimlin on 2026-06-02.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ciCI, build pipeline, test hygiene

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions