[FEATURE] Mark schema fields and models as deprecated and propagate through docs, PySpark, and the changelog

[sethfitz]

Summary

Add a single, authoritative way to mark a Pydantic schema field or model as deprecated, and propagate that one annotation to every generated artifact: a deprecation notice in the Markdown reference, an opt-in strict validation mode in the generated PySpark checks, and a changelog entry. The marking mechanisms are native Python — Pydantic's Field(deprecated=...) for fields and PEP 702's @deprecated for models — no custom marker.

Motivation

We retire fields and whole feature types as the schema evolves, but today "this is deprecated" lives only in a maintainer's head or a scattered prose note. Consumers reading the Markdown reference, data producers validating against the generated PySpark checks, and anyone tracking releases through the changelog each get no consistent signal. One annotation on the model should light up all three surfaces, so a deprecation is announced once and shows up everywhere it matters.

Marking mechanism

Two native Python mechanisms, both reducing to a free-text deprecation message:

• Fields — Pydantic's Field(deprecated=...) (available since Pydantic 2.7) accepts True (generic message "deprecated"), a message string, or a PEP 702 deprecated(...) marker. The marker also works directly in the field's Annotated metadata (Annotated[str, deprecated("...")]), so the same deprecated(...) idiom spans fields and models. All forms normalize at extraction to FieldInfo.deprecation_message (str | None). A bare deprecated=True is allowed: less signal, still signal.
• Models and other classes — PEP 702's @deprecated("message") decorator. On Python 3.13+ this is warnings.deprecated; on our 3.10 floor it is typing_extensions.deprecated. It sets __deprecated__ = "message" on the decorated class, and Pydantic already honors it (a deprecated model warns on instantiation). Extraction reads getattr(cls, "__deprecated__", None).

old_field: str | None = Field(
    default=None,
    deprecated="Use `new_field` instead. Deprecated in v1.18.0.",
)

@deprecated("Use `NewFeature` instead. Deprecated in v1.18.0.")
class OldFeature(Feature):
    ...

typing_extensions.deprecated(...) is the backport of the standard library's @deprecated (PEP 702; warnings.deprecated from Python 3.13). It marks a class or function deprecated for both static type checkers and the runtime, carrying a message.

The message is free text. Extraction reads it straight off FieldInfo / the class object.

Scope: fields and models are both in scope. Enum members and bare type aliases (Annotated[...], NewType) have no clean stdlib decorator to hang a message on, so deprecating those would need a project-specific convention — a later extension, not this issue.

What flows where

1. Markdown reference (deprecation notice)

The feature reference renders each field as a table row with italic annotations appended to the Description cell (e.g. *Minimum length: 1*), and renders the model's own description at the top of the page. Both are natural homes for a deprecation notice:

• Extraction carries a deprecated: str | None on FieldSpec (from FieldInfo.deprecation_message) and on the model spec (from the class's __deprecated__).
• Field-level: the renderer appends a **Deprecated:** <message> note to the field's Description cell, reusing the existing constraint-note appender, and optionally tags the Type column with (deprecated) alongside (optional).
• Model-level: a **Deprecated:** <message> banner at the top of the feature page.

This is the lowest-risk of the three surfaces and the natural MVP.

2. PySpark validation (opt-in strict mode via a CLI flag)

Deprecation is a validation mode, not a new severity. The generated PySpark validation is strictly pass/fail today — every check returns an error string or null, and explain_errors unpivots violations into (field, check, message) rows — and a deprecation should reuse that channel rather than add a severity tier. A deprecated field that is populated becomes a hard error, but only when the operator opts in.

Default is ignore, and that is deliberate: during a deprecation window real published data legitimately still carries the deprecated field, so erroring by default would fail every ordinary validation run. Strict mode is opt-in — a data producer runs overture-validate in strict mode to find every row still populating a deprecated column, then cleans the data up before the field is removed.

This also gives the deprecation message its runtime home: the deprecation check's error string is the field's deprecation_message, so "Use `new_field` instead." reaches the producer at the point of the violation.

Wiring, end to end:

• Generate one deprecation check per deprecated field. For each FieldSpec.deprecated, the pyspark codegen emits a Check that errors when the field is present and non-null, carrying the deprecation message as its error string.
• Keep them off the default path. Emit these into a separate builder on ModelValidation — a deprecation_checks: Callable[[], list[Check]] alongside the existing checks — so the default check list is byte-for-byte unchanged and empty for models with no deprecated fields. (Alternative: tag each Check with a category and filter at runtime; a separate builder is cleaner and leaves the Check dataclass untouched.)
• Toggle at validation time. validate_model gains a keyword parameter, deprecated: Literal["ignore", "error"] = "ignore". In "error" mode it concatenates validation.deprecation_checks() into the evaluated set; everything downstream (evaluate_checks, explain_errors, the (field, check, message) contract) is unchanged, because a deprecation check is just a normal check.
• Expose the flag. overture-validate gains --deprecated [ignore|error] (default ignore), wired straight to validate_model(..., deprecated=...), next to the existing --skip-schema-check / --suppress options.

Generated conformance tests should cover both modes for a model with a deprecated field: a populated deprecated field is a violation under error and clean under ignore.

3. Changelog (towncrier)

Manual path (trivial). Add a deprecation type — one more [[tool.towncrier.type]] block in the root pyproject.toml — and let a PR that deprecates a field or model ship a <pr>.deprecation.md fragment, exactly like any other change. This alone gives deprecations their own changelog section.

Automatic path (the harder, valuable part). The goal is to detect when a deprecation is introduced, without a human remembering to write the fragment. A codegen deprecation-manifest target emits a small structured list (not the full Markdown) of every deprecated element — field path or model name, plus its message; think of it as a "summary" generation target simpler than the reference. Run in diff mode against the last released tag, the target compares that manifest to the current one and emits each newly-appeared entry directly as a deprecation fragment — detection and scaffolding fold into the one target, and the fragment body is the message already authored on the annotation.

Proposed phasing

1. Extraction carrier (FieldSpec.deprecated from FieldInfo.deprecation_message; model-spec deprecated from __deprecated__) + Markdown notice for fields and models. Self-contained, shippable, delivers the reference-doc value.
2. deprecation towncrier type + contributor convention (manual path).
3. PySpark: per-field deprecation checks in a separate ModelValidation.deprecation_checks bucket → validate_model(deprecated=...) parameter → overture-validate --deprecated flag. Reuses the existing error channel; no severity tier.
4. Stretch: a deprecation-manifest codegen target that, in diff mode against the last release, emits deprecation fragments directly.

Open questions

• Trigger condition (PySpark). A deprecation check fires when a deprecated field is present and non-null — "still populating it" — not on mere schema presence. Confirm that is the intended trigger.
• towncrier automation shape. Is the manifest-plus-diff the right mechanism, and where does it live — a codegen target driven by a small tool at release time? The manual deprecation-type path can ship first regardless.
• Enum members / type aliases. These have no native @deprecated hook. Defer, or define a project convention now? (Fields and models need neither.)

Integration points

Concrete anchors for an implementer (paths under packages/):

• Extraction (source of truth): overture-schema-codegen/.../codegen/extraction/model_extraction.py reads FieldInfo at line ~159 and builds FieldSpec at ~175-183 (description at ~179). Read field_info.deprecation_message for fields here; read getattr(model_class, "__deprecated__", None) where the model spec is built. Nothing in the codegen or pyspark trees reads deprecation today.
• Carrier: overture-schema-codegen/.../codegen/extraction/specs.py, FieldSpec at ~122-135 — add deprecated: str | None. The model/record spec (sibling RecordSpec at ~139) carries the model-level message. (EnumMemberSpec, NewTypeSpec carry description but no deprecation — the later-extension case.)
• Markdown: overture-schema-codegen/.../codegen/markdown/renderer.py, _field_template_context (~203) building _FieldRow (~122); append via _annotate_constraint_notes (~218). Template: codegen/markdown/templates/feature.md.jinja2 field loop (~11-12) for fields, page header (~2-4) for the model-level banner.
• PySpark (opt-in strict mode): generate per-field deprecation checks in codegen/pyspark/renderer.py (render_model_module); carry them in a new ModelValidation.deprecation_checks builder in overture-schema-pyspark/.../pyspark/check.py (ModelValidation ~56-61), leaving the Check dataclass (~22-51) untouched. Add a deprecated: Literal["ignore","error"] = "ignore" param to validate_model (.../pyspark/validate.py ~298) and concatenate deprecation_checks() when "error" — evaluate_checks (~125) and explain_errors (~187) are unchanged. Wire a --deprecated option into validate_cli (.../pyspark/cli.py, alongside --skip-schema-check). _registry.py picks the extra builder up automatically via MODEL_VALIDATION.
• towncrier: root pyproject.toml [tool.towncrier] block (~88) — add a [[tool.towncrier.type]] for deprecation. The deprecation-manifest target (stretch) is a new codegen output format alongside markdown; in diff mode it emits deprecation fragments directly.