api: plan for narwhals.stable.v2

[MarcoGorelli]

I think we can aim for narwhals.stable.v2 some time in 2025

The main things I'd like to achieve are:

• remove "interchange" level of support, and instead only have:
  • "full": must support Series and have a defined row order (pandas, polars.DataFrame, pyarrow.Table, modin.DataFrame, cudf.DataFrame)
  • "lazy": must support collect, adding/removing/subsetting columns, filtering rows. Initially, at least, row-order-dependent operations (such as nw.cum_sum) will be left out. This will include Polars.LazyFrame, DuckDBPyRelation, pyspark.DataFrame, ibis.Table, dask.DataFrame. Later we can add an order_by argument to these functions which will become required when working with nw.Lazyframe
• make the breaking changes listed in breaking changes

Candidate changes:

• rework from_native. It will get simpler once we remove interchange-level things, but it's still fairly complicated. Not sure how much I like it to be completely honest, but I also haven't come up with anything better

Usage of narwhals.stable.v1 should remain unaffected

Tentative date: some time in 2025

---

Why remove support for the dataframe interchange protocol?

• the project has received no updates in over a year, issues and requests go unanswered, and it has no governance model in place so nobody is taking responsibility for it
• only a handful of dataframe libraries implement the protocol, and for most of them, we already provide (or are working to provide) native support in Narwhals. The only exception is Vaex, which itself seems to be deprecated (no releases in ~1.5 years), but we could support that too if there was demand
• from a Narwhals usage perspective, it adds a lot of complexity to have 3 layers of support. It's much simpler to just have: full, and lazy.

---

Outstanding items

Breaking changes:

• deprecate order-dependent expressions being passed to nw.LazyFrame
• deprecate length-changing order-dependent expressions (Expr.sort, Expr.gather_every, Expr.head, Expr.tail, Expr.sample)
• introduce restrictions on when length-changing expressions can be used in lazy API
• deprecate eager_or_interchange_only from from_native
• all expressions passed to group_by(...).agg should aggregate
• sort out [Bug]: Pyspark reduction does not preserve names root names with nw.all() #1780 (to make sure that addressing it doesn't require further API breaks)
• does nw.LazyFrame really need .clone? feat!: require at least one expression be passed to lazyframe select and lazyframe.with_columns, remove lazyframe.clone #2206
• native_namespace -> backend in IO functions Deprecate native_namespace in favour of backend in IO functions #1888
• deprecateLazyFrame.gather_every
• should pd.Categorical get mapped to nw.Enum? Feat: nw.Enum support for pandas #2192
• time zones in duckdb api: check whether we should get the connection time zone for duckdb/pyspark #2165
• empty sum: 0 or null?
• expression expansion in group-by api: formalise expression expansion in group-by #2225
• keep argument in mode feat: add required keep argument to Expr.mode #1793
• LazyFrame.with_row_index: allow? require that order_by be specified? should it be nw.row_index instead? [Enh]: Allow LazyFrame.with_row_index(..., order_by=...) #2307
• horizontal concatenation disallow concat(how='horizontal') for lazyframes #2340

[MarcoGorelli]

Later we can add an order_by argument to these functions which will become required when working with nw.Lazyframe

One more comment on this - I think this would be much simpler to implement as

nw.col('sales').cum_sum().over(order_by='date')

as opposed to

nw.col('sales').cum_sum(order_by='date')

Because then, we just have a single over context, and it's straightforward to also allow

nw.col('sales').cum_sum().over('store', order_by='date')

whereas nw.col('sales').cum_sum(order_by='date').over('store') would require us to fuse two window expressions together. Not saying that can't be done, but I'd prefer to keep things simple

---

Having said that, nw.col('sales').rolling_mean(2).over('store', order_by='date') would also require us to fuse overs. So, we may need to get #1848 and then look into fusing overs

[MarcoGorelli]

Some more rules for order-dependent operations:

1. Any order-dependent function must immediately be followed by an over, in which order_by (and optionally partition_by) are specified, e.g.:
   • nw.col('a').cum_sum().over('store', order_by='date') is allowed
   • nw.col('a').cum_sum().abs().over('store', order_by='date') is not
2. Order-dependent functions can not be called on CHANGES_LENGTH expressions. nw.col('a').drop_nulls().cum_sum().over('store', order_by='date') is not allowed

This should allow for a fairly straightforward translation of Narwhals API to PySpark / SQL:

• cum_sum just applies sum, and with .over('store', order_by='date') it becomes sum(a) over (partition by store order by date rows between unbounded preceding and current row)
• rolling_sum(2) also just applies sum, but we store the metadata 2 somewhere, and with .over('store', order_by='date') it becomes case when count(a) over (partition by store order by date rows between 1 preceding and current row) >= 2 then sum(a) over (partition by store order by date rows between 1 preceding and current row) else null end

---

Note that all of the above only applies to LazyFrames. DataFrames are pretty free from restrictions

---

Some operations which we should aim for. df denotes DataFrame (eager), lf LazyFrame (lazy):

• eager chain of order-dependent functions without over df.select(nw.col('a').shift(1).rolling_mean(2)): allowed
• eager chain of order-dependent functions with order-by over df.select(nw.col('a').shift(1).rolling_mean(2).over(order_by='i')): allowed. Not allowed if there's a partition_by in over, e.g. .over('b', order_by='i'), as pandas doesn't allow non-elementary group-bys
• lazy chain of order-dependent operations lf.select(nw.col('a').shift(1).rolling_mean(2).over(order_by='i')): not allowed
• lazy order-dependent operation in combination with elementwise operation lf.select(nw.col('a').abs().rolling_mean(2).over(order_by='i')) allowed
• lazy order-dependent operation in combination with n-ary operation of literals or elementwise lf.select((nw.col('a')+1).rolling_mean(2).over(order_by='i')) allowed
• n-ary operation inside over lf.select((nw.col('a').abs().rolling_mean(2)+1).over(order_by='i')) are not allowed. Instead, users can do lf.select((nw.col('a').abs().rolling_mean(2)).over(order_by='i'))+1

[dangotbanned]

Have you considered moving some of the instance attributes into ClassVar(s)?
Example of proposing for nw.Schema.

Here, I'm thinking more in terms of things like:

narwhals/narwhals/dataframe.py

Lines 73 to 77 in 4a2ca52

	def _from_compliant_dataframe(self: Self, df: Any) -> Self:
	# construct, preserving properties
	return self.__class__( # type: ignore[call-arg]
	df, level=self._level
	)

Instead you'd have:

class BaseFrame(Generic[_FrameT]):
    _compliant_frame: Any
    _level: ClassVar[Literal["full", "lazy"]]

    @classmethod
    def _from_compliant_dataframe(cls, df: Any):
        # NOTE: `_level` would already be accessible in `__init__`
        return cls(df)

Looking at _stableify, you might be able to benefit from __init_subclass__.

Achieves the same goal, but you get a context that is evaluated once per-class (e.g. enforce some error handling at declaration-time)

_stableify

narwhals/narwhals/stable/v1/__init__.py

Lines 1085 to 1122 in 4a2ca52

	@overload
	def _stableify(obj: NwDataFrame[IntoFrameT]) -> DataFrame[IntoFrameT]: ...
	@overload
	def _stableify(obj: NwLazyFrame[IntoFrameT]) -> LazyFrame[IntoFrameT]: ...
	@overload
	def _stableify(obj: NwSeries[Any]) -> Series: ...
	@overload
	def _stableify(obj: NwExpr) -> Expr: ...
	@overload
	def _stableify(obj: Any) -> Any: ...
	def _stableify(
	obj: NwDataFrame[IntoFrameT] | NwLazyFrame[IntoFrameT] | NwSeries[Any] | NwExpr | Any,
	) -> DataFrame[IntoFrameT] | LazyFrame[IntoFrameT] | Series | Expr | Any:
	if isinstance(obj, NwDataFrame):
	return DataFrame(
	obj._compliant_frame._change_version(Version.V1),
	level=obj._level,
	)
	if isinstance(obj, NwLazyFrame):
	return LazyFrame(
	obj._compliant_frame._change_version(Version.V1),
	level=obj._level,
	)
	if isinstance(obj, NwSeries):
	return Series(
	obj._compliant_series._change_version(Version.V1),
	level=obj._level,
	)
	if isinstance(obj, NwExpr):
	return Expr(
	obj._to_compliant_expr,
	is_order_dependent=obj._is_order_dependent,
	changes_length=obj._changes_length,
	aggregates=obj._aggregates,
	)
	return obj

Short example of what that could look like:

# nw.dataframe.DataFrame / BaseFrame
class NwDataFrame(Generic[IntoFrameT]):
    def __init_subclass__(
        cls, *args: Any, version: Version = Version.MAIN, **kwds: Any
    ) -> None:
        super().__init_subclass__(*args, **kwds)
        cls._version: Version = version


class DataFrame(NwDataFrame[IntoDataFrameT], version=Version.V1): ...

I'm just picking out version & level as two common arguments.
This could apply anywhere that some attribute is fixed for all instances of a class.

Maybe there is good reason to be able to configure this on the instance-level - but if not - you'd end up with less context that needs passing around 🙂

[MarcoGorelli]

Thanks for looking into this! As far as I understood, this would not be user-facing, right? If so, definitely on-board with refactoring

[dangotbanned]

Thanks for looking into this! As far as I understood, this would not be user-facing, right? If so, definitely on-board with refactoring

Great @MarcoGorelli

Yeah I don't think any of this would be user-facing.

Making changes like this for v2 would impact internals and probably extension authors.
To what extent though depends on what things you might want to move around

[MarcoGorelli]

that's fine then, the extensions mechanism is still marked as experimental (and I don't know of anyone who's actually tried extending Narwhals yet)

[dangotbanned]

Achieves the same goal, but you get a context that is evaluated once per-class (e.g. enforce some error handling at declaration-time

One attribute that wouldn't work with this is backend_version: tuple[int, ...].
AFAICT, for that you'd want something more like a write-once attribute, per-implementation

narwhals/narwhals/utils.py

Lines 387 to 398 in f79a9ea

	MIN_VERSIONS: dict[Implementation, tuple[int, ...]] = {
	Implementation.PANDAS: (0, 25, 3),
	Implementation.MODIN: (0, 25, 3),
	Implementation.CUDF: (24, 10),
	Implementation.PYARROW: (11,),
	Implementation.PYSPARK: (3, 5),
	Implementation.POLARS: (0, 20, 3),
	Implementation.DASK: (2024, 8),
	Implementation.DUCKDB: (1,),
	Implementation.IBIS: (6,),
	Implementation.SQLFRAME: (3, 14, 2),
	}

You can't know what version is installed (if any) at class-definition time.
However, you would know that the version cannot change after it has been seen once.

So for something like (#2007 (comment)), you can be sure backend_version is known after the first ArrowSeries is instantiated.

PandasLike... would probably need the most thought on if it could support this

[dangotbanned]

Using __init_subclass__ for Version in #2324 (comment)