use protocols instead of optional 3rd party dependencies in public type annotations

[MarcoGorelli]

This was brought up here: https://discord.com/channels/1235257048170762310/1272835922924273694/1439943293860446292

is there any particular reason to use numpdy ndarrays concrete types instead of Protocols?
This is a issue I already talked about in Polars here:
pola-rs/polars#24436

If numpy is not installed in the venv, the LSP will complain each time I'm updating narwhals/polars about unknown numpy types, and I have to manually delete them to fix it.

I think it's a valid request

Dan's comment:

Yeah I think using Protocols more would help your use-case

I've been adding some for solving the same issue in narwhals._native.py - but hadn't considered numpy.

I think the main challenge with ndarray is that we'd need to define something that only matches that - but doesn't match the 8474930 other things that based their API on it 😂

TBH I'd be ok with accidentally matching something non-numpy, if it's not supported there'll be a runtime error anyway

[dangotbanned]

Oooh yeah I forgot about this one

TBH I'd be ok with accidentally matching something non-numpy, if it's not supported there'll be a runtime error anyway

Would picking a subset of array-api work?

If not, the main overlap I think we'd want to avoid is with pd.Series.
It has a lot of members that would match np.ndarray by name, but the signatures are usually incompatible

[MarcoGorelli]

Would picking a subset of array-api work?

yeah, sure

[dangotbanned]

TBH I'd be ok with accidentally matching something non-numpy, if it's not supported there'll be a runtime error anyway

Would picking a subset of array-api work?

Sorry for not getting back to you on this!

TL;DR: it doesn't look like the typing will help our use case

Array API

The standard in array_api_stubs/_draft/array_object.py specifies mainly:

• operators
• properties like: dtype, shape, size

Export is possible via __dlpack__ and the main functionality is exposed by the untyped __array_namespace__.

The numpy implementation (array_api_strict/_array_object.py) has these, but also some notes on pre-3.12 compatibility.
The typing appears to be a little better.
But the main aspect of it (__array_namespace__ returns here) - would effectively be typed as Any because it lacks protocols like we have for plugins.

Why does any of this matter?

Assuming we add some protocols to fill in the gaps, I know at-least polars doesn't support __dlpack__ but pyarrow does.
So it might just be simpler for now, to just do something like narwhals/_native.py for explicitly np.ndarray and revisit the more general thing later

[dangotbanned]

Mentioned this in discord comment

pyarrow example from (#2572)

In these two Protocol defs, note how *DataFrame references *Series:

narwhals/narwhals/_plan/arrow/typing.py

Lines 60 to 67 in ef8f7d5

	class NativeArrowSeries(NativeSeries, Protocol):
	@property
	def chunks(self) -> list[Any]: ...
	class NativeArrowDataFrame(NativeDataFrame, Protocol):
	def column(self, *args: Any, **kwds: Any) -> NativeArrowSeries: ...
	@property
	def columns(self) -> Sequence[NativeArrowSeries]: ...

When defining an @overload:

• Use the *Native protocol for the parameter type
• Use the concrete class in the return type

That detail avoids overlapping overloads.
They can happen when a missing dependency on a parameter type becomes an Unknown:

narwhals/narwhals/_plan/dataframe.py

Lines 386 to 405 in ef8f7d5

	@overload
	@classmethod
	def from_native(
	cls: type[DataFrame[Any, Any]], native: NativeArrowDataFrame, /
	) -> DataFrame[pa.Table, pa.ChunkedArray[Any]]: ...
	@overload
	@classmethod
	def from_native(
	cls: type[DataFrame[Any, Any]], native: NativeDataFrameT, /
	) -> DataFrame[NativeDataFrameT]: ...
	@classmethod
	def from_native(
	cls: type[DataFrame[Any, Any]], native: NativeDataFrameT, /
	) -> DataFrame[Any, Any]:
	if is_pyarrow_table(native):
	from narwhals._plan import arrow as _arrow
	return cls(_arrow.DataFrame.from_native(native, cls._version))
	raise NotImplementedError(type(native))

And then here's a typing test to show the NativeArrowDataFrame -> NativeArrowSeries relationship avoids a false positive for pa.RecordBatch:

narwhals/tests/plan/compliant_test.py

Lines 817 to 826 in ef8f7d5

	def test_dataframe_from_native_overloads() -> None:
	"""Ensure we can reveal the `NativeSeries` **without** a dependency."""
	data: dict[str, Any] = {}
	native_good = pa.table(data)
	result_good = nwp.DataFrame.from_native(native_good)
	assert_type(result_good, "nwp.DataFrame[pa.Table, pa.ChunkedArray[Any]]")
	native_bad = native_good.to_batches()[0]
	nwp.DataFrame.from_native(native_bad) # type: ignore[call-overload]
	assert_type(native_bad, "pa.RecordBatch")