Skip to content

test: add codegen fidelity tests for generated structs - #347

Open
knasher wants to merge 1 commit into
atrium-rs:update/lexicon-5c3b7c9c8from
knasher:test/codegen-fidelity
Open

test: add codegen fidelity tests for generated structs#347
knasher wants to merge 1 commit into
atrium-rs:update/lexicon-5c3b7c9c8from
knasher:test/codegen-fidelity

Conversation

@knasher

@knasher knasher commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Stacked on #346 — based on update/lexicon-5c3b7c9c8 so the diff stays at two files rather than pulling in the whole regeneration. Happy to retarget to main once #346 lands (GitHub should do it automatically). The tests reference types that only exist after the regen, which is why they can't be based on main today.

Why

cargo build doesn't compile the test target, so it can't catch a regeneration breaking hand-written fixtures or silently dropping a field. #346's own catch-up commit ("New required fields in test fixtures across atrium-api, bsky-sdk and bsky-cli") is exactly the class of breakage that stays invisible until someone runs the tests — so it seemed worth having a gate that fails loudly at regen time.

What it asserts

Each case checks three invariants against a lexicon-shaped payload:

  1. Typed coverage — every field is reachable as a typed struct field.
  2. No leakageObject::extra_data stays empty.
  3. Round-trip stability — re-serializing yields the original payload.

Invariant 2 is the one that does the real work. Round-trip alone isn't sufficient, because extra_data round-trips unknown fields too — a dropped field falls through to extra_data rather than failing to compile, so the payload still survives a round trip and the regression passes unnoticed. A field is only proven typed when it's both reachable as a struct field and absent from extra_data.

Coverage

  • chat.bsky.convo.defs#messageView / #messageInput replyTo, including the recursive union variant resolving rather than falling into Union::Unknown
  • tools.ozone.report.queryActivities
  • app.bsky.actor.contentVisibilityDeclaration

The last one guards the camelCase record NSID case fixed in #345. Referring to the record through <ContentVisibilityDeclaration as Collection>::Record turns a missing Collection impl back into a compile error, and the assertions catch the Collection impl and the KnownRecord variant disagreeing on the NSID or the record type — the specific mismatch that left downstream crates with a KnownRecord arm they couldn't satisfy.

Notes

  • All cases are feature-gated to their namespace: 3 tests run under default features, 5 under --all-features.
  • Adds serde_json as an atrium-api dev-dependency.
  • cargo fmt-no-gen --check clean; cargo clippy -p atrium-api --all-features --all-targets clean under -D warnings.

🤖 Generated with Claude Code

@sugyan

sugyan commented Aug 19, 2026

Copy link
Copy Markdown
Member

Thanks — I verified this before replying, and the extra_data-as-completeness-check
idea is genuinely good. I dropped sortDirection from the generated
queryActivities params and the round-trip assertion sailed straight through while
assert_fully_typed caught it. That's the non-obvious part and you got it right.

My hesitation is about where the tests live. The regressions these catch are all
codegen bugs, but the input is a payload pinned to today's lexicons — upstream's
moving target. When Bluesky adds a required field these fail for a reason unrelated
to codegen, and I can't tell the two apart from the failure. Regen is exactly when
this gate fires, so an ambiguous signal there is a real cost.

The version I'd want drives atrium-codegen from a fixture lexicon checked into the
repo, so the input never moves. That's a bigger piece of work and wants lexicon/
in CI first, so I'd rather do it separately.

That said, not all of this churns. What actually caught the #345 regression is
payload-free: removing the Collection impl fails at the use, and mangling the
NSID fails on assert_eq!(ContentVisibilityDeclaration::NSID, NSID). Naming the
record through <ContentVisibilityDeclaration as Collection>::Record is doing real
work and costs nothing to maintain. It's the deserialize / extra_data / round-trip
layer on top that carries the lexicon-shaped payload.

Would you be up for trimming to that core? Keep the NSID and associated-type
assertions guarding #345, drop the chat and ozone modules. Near-zero maintenance,
and the extra_data idea stays available for the codegen-side version later.

One nit either way: serde_json doesn't need adding as a dev-dependency — it's
already an unconditional [dependencies] entry (atrium-api/Cargo.toml:24), and
Cargo makes normal deps available to test targets.

`cargo build` does not compile the test target, so nothing catches codegen
dropping a `Collection` impl. atrium-rs#345 fixed exactly that: `app.bsky.actor.
contentVisibilityDeclaration` is the first record whose NSID leaf is
camelCase, and the NSID was rebuilt from the snake_case'd file stem, so the
record never matched its own schema and got no `Collection` impl.
`KnownRecord` is keyed on schema ids, so it *did* gain the variant, leaving
downstream crates with an arm they could not satisfy.

Two assertions, neither coupled to lexicon contents:

- Naming `ContentVisibilityDeclaration::NSID` requires the `Collection`
  impl to exist; comparing it against the literal requires it to carry the
  NSID verbatim rather than a mangled form.
- A never-called function taking `<ContentVisibilityDeclaration as
  Collection>::Record` and returning the corresponding `KnownRecord`
  variant asserts at compile time that the two halves agree on the record
  type. Drifting the associated type fails to build here while the NSID
  assertions still pass, so this catches a case the runtime checks cannot.

Verified by reintroducing the atrium-rs#345 breakage against the current tree:
deleting the `Collection` impl fails the test target to compile, as does
pointing its `Record` at a different record type.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@knasher
knasher force-pushed the test/codegen-fidelity branch from 832c7f1 to 6596489 Compare August 20, 2026 22:15
@knasher

knasher commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Trimmed, and dropped the dev-dependency — you're right that it's already an
unconditional [dependencies] entry, so Cargo.toml is untouched now.

On placement: agreed, and the case I couldn't answer for is a required field
landing in messageView. Deserialization fails and the error says nothing about
whether codegen is at fault — arriving exactly at regen, which is when you least
want to be triaging it.

I went slightly further than you asked, because the actor test still had a
payload layer on top. hideFromAlgorithmicRecommendations is small, but it's the
same category of thing, so I replaced the deserialize / extra_data /
round-trip part with a compile-time assertion:

#[allow(dead_code)]
fn collection_record_is_the_known_record_payload(
    record: <ContentVisibilityDeclaration as Collection>::Record,
) -> KnownRecord {
    KnownRecord::AppBskyActorContentVisibilityDeclaration(Box::new(record))
}

That keeps the half of the KnownRecord agreement check that was doing real
work, with no payload at all. It also isn't redundant with the NSID assertions:
if the impl exists and the NSID is correct but type Record drifts, both
assert_eq!s pass and only this fails.

I reintroduced both breakages against the current tree rather than assuming:

  • deleting the Collection impl — the test target fails to compile
  • pointing its Record at profile::Recordmismatched types, while the
    NSID assertions stay green

Result is 39 lines in one file, no payload, chat and ozone gone.

Happy to leave extra_data for the codegen-driven version — driving
atrium-codegen from a checked-in fixture lexicon is the right home for it, and
it wants lexicon/ in CI first as you say. If a hand would be useful on that
once the CI piece exists, let me know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants