Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

49 changes: 41 additions & 8 deletions docs/specs/editions.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,39 @@ released through June 2026).
Editions can be used to constrain your minimum required vortex reader, since latest version over vortex across all
editions is the earliest version of vortex required to read that file.

## What an edition contains

Every member of an edition is recorded with a **component kind**: `array`, `layout`, or `aggregate`. The kind is
recorded because ids are unique only within a kind — a layout named `vortex.flat` and an array encoding named
`vortex.flat` are different members — and because membership is resolved one kind at a time. The writer holds a
separate id set per kind and enforces each where that kind is written:

| Kind | Written | Enforced at |
| --- | --- | --- |
| `array` | every serialized array | array serialization context |
| `layout` | the footer's layout tree | layout serialization context |
| `aggregate` | zone maps in zoned layouts | the layout writer context |

**Writing a component outside the enabled editions fails the write, for every kind.** A zone map is only an
optimization, so a forbidden aggregate could in principle be dropped instead — but a file that silently prunes
worse than the writer was configured for is a bug you find in a benchmark six months later, not an error you can
act on. Violations surface at write time or not at all.

Aggregates are checked against the set the write would actually record: an aggregate that a column's dtype cannot
hold is not written, so it is not a violation either.

**A kind with no declared members is unrestricted.** An edition that declares no layouts makes no promise about
layouts, so the writer leaves them alone rather than forbidding all of them; declaring the first member of a kind is
what arms its filter. `core2026.08.0` declares the aggregates the default writer records in zone maps — `min`,
`max`, `bounded_min`, `bounded_max`, `nan_count`, `null_count` — so that filter is armed by default. `sum` is not
among them: a zone sum prunes nothing, so the writer records none. File-level statistics still carry a sum, which
this filter does not govern. A
session that registers components outside `core`, such as the spatial extension types, enables its own edition
family alongside `core`, and the writer may emit the union.

## Resolving an unknown-encoding error

If a read failed with an unknown encoding ID and pointed you here, the reader met an encoding
If a read failed with an unknown encoding ID and pointed you here, the reader met an array encoding
it does not support. Find the encoding ID in the [registry](#edition-registry) below:

1. **The ID is listed under an edition.** The file is newer than your Vortex build. Upgrade to
Expand All @@ -39,15 +69,15 @@ someone else's read error later.

The enabled editions are stored on the writer's Vortex session. Registering an edition makes
its declaration available to the session; enabling it separately allows the writer to emit its
encodings. Enabling another edition from the same family replaces the earlier selection.
array encodings. Enabling another edition from the same family replaces the earlier selection.

Two knobs exist when the default is not what you want:

- **Pin an older edition** when files must stay readable by deployments running older Vortex.
- **Opt in to additional edition families.** Editions come in independently versioned,
additive families — `core` today, with families for more specialised encoding groups (for
example spatial encodings) possible later. A writer targets at most one edition per family
and may emit any encoding in their union; each encoding belongs to exactly one family.
and may emit any encoding in their union; each member belongs to exactly one family.

Lower-level sessions without an enabled-editions store opt out of editions entirely and can write
custom or experimental encodings. A raw `with_allow_encodings` writer policy is another explicit
Expand All @@ -56,12 +86,15 @@ encodings can read the files.

## How editions change

A published edition is frozen — its encoding list never grows or shrinks. New encodings are
A published edition is frozen — its member list never grows or shrinks. New members are
staged in a **draft** edition and become guaranteed only when that draft is frozen as the next
edition; each encoding's registry entry records the edition it joined in. In the future an
encoding may be *deprecated*, meaning writers stop emitting it — but readers keep decoding it
indefinitely, so deprecation never invalidates existing files.
edition; each member's registry entry records its component kind and the edition it joined in.
Declaring the first member of a kind arms that kind's write-time filter, so a kind gains
enforcement at the edition that first declares one.
In the future an encoding may be *deprecated*, meaning writers stop emitting it — but readers
keep decoding it indefinitely, so deprecation never invalidates existing files.

## Edition registry

Coming soon..
Coming soon.. It will list each edition's members with their component kind, the edition they
joined in, and the Vortex release required to read them.
2 changes: 1 addition & 1 deletion encodings/parquet-variant/src/vtable.rs
Original file line number Diff line number Diff line change
Expand Up @@ -412,7 +412,7 @@ mod tests {
.read(|map| map.keys().copied().collect::<Vec<_>>());
for id in ids {
editions
.declare_inclusion(EditionInclusion::new(&id, TEST_EDITION))
.declare_inclusion(EditionInclusion::array(&id, TEST_EDITION))
.map_err(|error| vortex_err!("{error}"))?;
}
session
Expand Down
173 changes: 128 additions & 45 deletions vortex-edition/src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,19 +1,24 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright the Vortex contributors

//! Definitions of Vortex *editions*: named, frozen sets of encodings that a writer may put in
//! a file, carrying a forever read-compatibility guarantee.
//! Definitions of Vortex *editions*: named, frozen sets of components that a writer may put
//! in a file, carrying a forever read-compatibility guarantee.
//!
//! Editions live on the session, like encodings do: [`EditionSession`] holds the registered
//! editions and [`EnabledEditions`] selects which of them a writer may emit. Declarations
//! are plain constants — an [`EditionId`] plus an [`Edition`] record, and one
//! [`EditionInclusion`] per encoding stating that it is a member of an edition *and every
//! [`EditionInclusion`] per member stating that it is a member of an edition *and every
//! later edition of the same family*. Any crate can register declarations into a session,
//! so inclusions can live next to the encoding they describe.
//! so inclusions can live next to the component they describe.
//!
//! Every membership is typed by a [`ComponentKind`], and members are resolved one kind at a
//! time with [`EditionSessionExt::enabled_component_ids`]: the file writer restricts the
//! arrays, layouts, and aggregates it writes from three separate id sets, never one untyped
//! set.
//!
//! An edition is a **draft** until its [`Edition::min_vortex_version`] is recorded —
//! recording it is the act of freezing. The per-edition encoding sets are computed from the
//! registered declarations by [`EditionSession::encodings_in`], and correctness is enforced
//! recording it is the act of freezing. The per-edition member sets are computed from the
//! registered declarations by [`EditionSession::components_in`], and correctness is enforced
//! by unit tests: [`EditionSession::validate`] checks a whole registry, and
//! [`test_harness::validate_edition`] validates one edition's constraints — call it once in
//! the `#[cfg(test)]` module of each edition definition.
Expand Down Expand Up @@ -109,15 +114,42 @@ impl Display for EditionId {
}
}

/// An edition: a named set of encodings with a read-compatibility guarantee, registered with
/// The kind of component an edition membership covers.
///
/// Ids are unique per kind, not globally: a layout named `vortex.flat` and an array named
/// `vortex.flat` are different members. Every membership records its kind, and the writer
/// resolves one kind at a time, so the set restricting written arrays never restricts
/// written layouts. Further kinds (scalar functions, say) can be added the same way.
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum ComponentKind {
/// An array encoding, e.g. `vortex.alp`, registered in the session's array registry.
Array,
/// A layout encoding, e.g. `vortex.flat`, registered in the session's layout registry.
Layout,
/// An aggregate function, e.g. `vortex.min`, written into zone maps and registered in
/// the session's aggregate function registry.
Aggregate,
}

impl Display for ComponentKind {
fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
f.write_str(match self {
Self::Array => "array",
Self::Layout => "layout",
Self::Aggregate => "aggregate",
})
}
}

/// An edition: a named set of components with a read-compatibility guarantee, registered with
/// [`EditionSession::declare_edition`]. The set itself is computed from the registered
/// [`EditionInclusion`]s by [`EditionSession::encodings_in`].
/// [`EditionInclusion`]s by [`EditionSession::components_in`].
#[derive(Clone, Copy, Debug)]
pub struct Edition {
/// The edition identifier. Also carries the freeze date: `core2026.07.0` freezes in
/// 2026-07.
pub id: EditionId,
/// The minimum Vortex version whose reader supports every encoding in this edition.
/// The minimum Vortex version whose reader supports every member of this edition.
///
/// Recording this is the act of freezing: an edition with `None` is a **draft** — being
/// assembled, carrying no guarantee, free to change, never the default write target.
Expand All @@ -133,83 +165,133 @@ impl Edition {
}
}

/// Declares that an encoding is a member of an edition — and of every later edition of the
/// Declares that a component is a member of an edition — and of every later edition of the
/// same family. Registered with [`EditionSession::declare_inclusion`].
#[derive(Clone, Copy, Debug)]
pub struct EditionInclusion {
/// The interned encoding id, e.g. `vortex.alp`. Globally unique across everything an
/// edition can cover: when layout encodings join editions, their ids must be distinct
/// from array encoding ids.
pub encoding_id: Id,
/// The first edition this encoding is a member of.
/// What the membership covers. Ids are unique per kind, so this is part of the
/// member's identity, not a label.
pub kind: ComponentKind,
/// The interned component id, e.g. `vortex.alp`.
pub component_id: Id,
/// The first edition this component is a member of.
pub since: EditionId,
/// The earliest Vortex release able to read and execute this encoding, recorded from
/// The earliest Vortex release able to read and execute this component, recorded from
/// evidence (e.g. compat-fixture history). `None` until recorded.
pub required_vortex_release: Option<&'static str>,
}

/// A source of an encoding id for edition declarations.
/// A source of a component id for edition declarations.
///
/// Implemented for raw id strings (`"vortex.alp"`) and interned [`Id`]s here; encoding
/// vtables implement it where they are defined, so a declaration can name the vtable
/// (`&Primitive`) instead of spelling its id.
pub trait AsEncodingId: Debug + Send + Sync {
/// The interned encoding id.
fn encoding_id(&self) -> Id;
/// (`&Primitive`) instead of spelling its id. The id alone does not say what kind of
/// component it names — [`EditionMember`] pairs it with a [`ComponentKind`].
pub trait AsComponentId: Debug + Send + Sync {
/// The interned component id.
fn component_id(&self) -> Id;
}

impl AsEncodingId for str {
impl AsComponentId for str {
#[expect(
clippy::disallowed_methods,
reason = "interning a dynamic encoding id at declaration time"
reason = "interning a dynamic component id at declaration time"
)]
fn encoding_id(&self) -> Id {
fn component_id(&self) -> Id {
Id::new(self)
}
}

impl AsEncodingId for Id {
fn encoding_id(&self) -> Id {
impl AsComponentId for Id {
fn component_id(&self) -> Id {
*self
}
}

// `str` is unsized and cannot be a trait object, so declaration blocks (slices of
// `&dyn AsEncodingId`) name encodings as `&"vortex.alp"` through this impl.
impl AsEncodingId for &'static str {
fn encoding_id(&self) -> Id {
(**self).encoding_id()
// `str` is unsized and cannot be a trait object, so declaration blocks name components as
// `&"vortex.alp"` through this impl.
impl AsComponentId for &'static str {
fn component_id(&self) -> Id {
(**self).component_id()
}
}

/// Declares an edition together with the encodings that join the family at it, in one
/// block. Registered with [`EditionSession::declare`], which derives each encoding's
/// A component that joins an edition, named by id string or vtable and tagged with the kind
/// of registry it belongs to. Built with the per-kind constructors, so a declaration reads
/// as `EditionMember::array(&"vortex.alp")`.
#[derive(Clone, Copy, Debug)]
pub struct EditionMember {
/// What kind of component this is.
pub kind: ComponentKind,
/// The component, named by id string or by vtable.
pub component: &'static dyn AsComponentId,
}

impl EditionMember {
/// An array encoding member, e.g. `vortex.alp`.
pub const fn array(component: &'static dyn AsComponentId) -> Self {
Self {
kind: ComponentKind::Array,
component,
}
}

/// A layout member, e.g. `vortex.flat`.
pub const fn layout(component: &'static dyn AsComponentId) -> Self {
Self {
kind: ComponentKind::Layout,
component,
}
}

/// An aggregate function member, e.g. `vortex.min`.
pub const fn aggregate(component: &'static dyn AsComponentId) -> Self {
Self {
kind: ComponentKind::Aggregate,
component,
}
}
}

/// Declares an edition together with the components that join the family at it, in one
/// block. Registered with [`EditionSession::declare`], which derives each member's
/// membership (`since` = the declared edition) from the block structure.
#[derive(Clone, Copy, Debug)]
pub struct EditionDeclaration {
/// The edition being declared.
pub edition: Edition,
/// The encodings that join the family at this edition, named by id string or by
/// vtable. Members of earlier editions are inherited and never restated.
pub added: &'static [&'static dyn AsEncodingId],
/// The components that join the family at this edition, each tagged with its
/// [`ComponentKind`]. Members of earlier editions are inherited and never restated.
pub added: &'static [EditionMember],
}

impl EditionInclusion {
/// Declare that an encoding is a member of `since` and every later edition of the same
/// family. The encoding can be named by id string or by vtable.
pub fn new<E: AsEncodingId + ?Sized>(encoding: &E, since: EditionId) -> Self {
/// Declare that a component of `kind` is a member of `since` and every later edition of
/// the same family. The component can be named by id string or by vtable.
pub fn new<C: AsComponentId + ?Sized>(
kind: ComponentKind,
component: &C,
since: EditionId,
) -> Self {
Self {
encoding_id: encoding.encoding_id(),
kind,
component_id: component.component_id(),
since,
required_vortex_release: None,
}
}

/// Validate the declaration's form: a lowercase `namespace.name` encoding id and, if
/// Declare that an array encoding is a member of `since` and every later edition of the
/// same family.
pub fn array<C: AsComponentId + ?Sized>(encoding: &C, since: EditionId) -> Self {
Self::new(ComponentKind::Array, encoding, since)
}

/// Validate the declaration's form: a lowercase `namespace.name` component id and, if
/// recorded, a well-formed `major.minor.patch` release. Checked for every declared
/// inclusion by [`EditionSession::validate`].
pub fn validate(&self) -> Result<(), EditionError> {
let id = self.encoding_id.as_str();
let id = self.component_id.as_str();
let well_formed = !id.starts_with('.')
&& !id.ends_with('.')
&& id.contains('.')
Expand All @@ -218,15 +300,16 @@ impl EditionInclusion {
.all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || "._-".contains(c));
if !well_formed {
return Err(EditionError::new(format!(
"invalid encoding id {id:?}: expected lowercase `namespace.name`, e.g. \
`vortex.alp`"
"invalid {} id {id:?}: expected lowercase `namespace.name`, e.g. `vortex.alp`",
self.kind
)));
}
if let Some(release) = self.required_vortex_release
&& parse_release(release).is_none()
{
return Err(EditionError::new(format!(
"encoding {id} declares malformed required_vortex_release {release:?}"
"{} {id} declares malformed required_vortex_release {release:?}",
self.kind
)));
}
Ok(())
Expand Down
Loading
Loading