Skip to content

Commit 7a33cfa

Browse files
paddyroddyzen-xuTallJimbotekumarapre-commit-ci[bot]
committed
apacheGH-32609: [Python] Add typing to arrow using stubs (#3)
* Initial commit * init project * complete most of the annotations * fix FixedSizeBufferWriter init annotation * bump 10.0.1.2 * complete parquet core annotations * bump 10.0.1.3 * re-export modules * fix: add return type for foreign_buffer * fix output_stream and read_message annotations * ci: add release job * pre-commit specify flake8 version to 5.0.4 * flake8 ignore F821 for private files * optimize annotations * bump 10.0.1.4 * if param supports IOBase, it should also support NativeFile * bump 10.0.1.5 * pre-commit adds mypy lint * bump 10.0.1.6 * fix ci name * Remove version restrictions for Python. * release 10.0.1.7 * update poetry ci * Fix stubs for Table factory methods The main problem was that these were annotated as instance methods rather than static/class methods, but I've added some detail, too. * update pre-commit * update * fix: make fs.FileSystem.from_uri and hdfs.HadoopFileSystem.from_uri as classmethod * fix: fix read_metadata and read_schema wrong annotations (#11) * fix: typo S3FileSystem schema -> scheme (#12) * bump version 10.0.1.8 (#13) * . (#16) * make DataType hashable (#22) * pa.table support recordbatch (#20) * RecordBatchStreamReader supports next (#18) * add RecordBatch.to_pylist (#23) * precise return types for to_pandas (#25) * bump version 10.0.1.9 (#26) * [pre-commit.ci] pre-commit autoupdate (#27) * [pre-commit.ci] pre-commit autoupdate (#28) * Fix types in FlightDescriptor class (#29) * Fix types in FlightDescriptor class * Add argument types * chore: update pre-commit config (#30) * build: use `pixi` to manage project (#31) * chore: add taplo config (#32) * chore: update LICENSE date (#33) * doc: add CODE_OF_CONDUCT.md (#34) * [pre-commit.ci] pre-commit autoupdate (#38) * [pre-commit.ci] pre-commit autoupdate (#39) updates: - [github.com/astral-sh/ruff-pre-commit: v0.5.7 → v0.6.1](astral-sh/ruff-pre-commit@v0.5.7...v0.6.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (apache#48) updates: - [github.com/astral-sh/ruff-pre-commit: v0.6.1 → v0.6.2](astral-sh/ruff-pre-commit@v0.6.1...v0.6.2) - [github.com/pre-commit/mirrors-mypy: v1.11.1 → v1.11.2](pre-commit/mirrors-mypy@v1.11.1...v1.11.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * refactor: rewrite type annotations by hand. (#35) * chore: restart * update ruff config * build: add extra dependencies * update mypy config * feat: add util.pyi * feat: add types.pyi * feat: impl lib.pyi * update * feat: add acero.pyi * feat: add compute.pyi * add benchmark.pyi * add cffi * feat: add csv.pyi * disable isort single line * reformat * update compute.pyi * add _auzurefs.pyi * add _cuda.pyi * add _dataset.pyi * rename _stub_typing.pyi -> _stubs_typing.pyi * add _dataset_orc.pyi * add pyarrow-stubs/_dataset_parquet_encryption.pyi * add _dataset_parquet.pyi * add _feather.pyi * feat: add _flight.pyi * add _fs.pyi * add _gcsfs.pyi * add _hdfs.pyi * add _json.pyi * add _orc.pyi * add _parquet_encryption.pyi * add _parquet.pyi * update * add _parquet.pyi * add _s3fs.pyi * add _substrait.pyi * update * update * add parquet/core.pyi * add parquet/encryption.pyi * add BufferProtocol * impl _filesystemdataset_write * add dataset.pyi * add feather.pyi * add flight.pyi * add fs.pyi * add gandiva.pyi * add json.pyi * add orc.pyi * add pandas_compat.pyi * add substrait.pyi * update util.pyi * add interchange * add __lib_pxi * update __lib_pxi * update * update * add types.pyi * feat: add scalar.pyi * update types.pyi * update types.pyi * update scalar.pyi * update * update * update * update * update * update * feat: impl array * feat: add builder.pyi * add scipy * add tensor.pyi * feat: impl NativeFile * update io.pyi * complete io.pyi * add ipc.pyi * mv benchmark.pyi into __lib_pxi * add table.pyi * do re-export in lib.pyi * fix io.pyi * update * optimize scalar.pyi * optimize indices * complete ipc.pyi * update * fix NullableIterable * fix string array * ignore overload-overlap error * fix _Tabular.__getitem__ * remove additional_dependencies * remove check-mypy.sh (apache#49) * release 20240828 (apache#50) * fix release tag (apache#51) * ci: install hatch by pip (apache#52) * ci: fix hatch keyring (apache#53) * ci: use Release environment (apache#54) * remove Scalar generic type var _IsValid (apache#56) * remove Scalar generic type var _IsValid * make Array, Scalar, Types generic type var as covariant type (apache#57) * remove Field generic type var _Nullable (apache#58) * remove Field generic type var _Nullable * fix: pa.dictionary and pa.schema annotation (apache#59) * fix pa.dictionary annotation * fix: schema annotation * release new version (apache#60) * [pre-commit.ci] pre-commit autoupdate (apache#62) * release: 2024.9.3 (apache#63) use new date release format %Y.%m.%d * support pyarrow compute funcs (apache#61) * update compute.pyi * impl Aggregation funcs * impl arithmetic * imit bit-wise functions * imit rounding functions * optimize annotation * impl logarithmic functions * update * impl comparisons funcs * impl logical funcs * impl string predicates and transforms * impl string padding * impl string trimming * impl string splitting and component extraction * impl string joining and slicing * impl Containment tests * impl Categorizations * impl Structural transforms * impl Conversions * impl Temporal component extraction * impl random, Timezone handling * impl Array-wise functions * fix timestamp scalar * support build array with list of scalar (apache#64) * release 2024.9.4 (apache#65) * Version follows the version of pyarrow (apache#66) * import parquet.core into parquet __init__.py (apache#67) Update __init__.pyi * release 17.1 (apache#69) * fix: add missing submodule benchmark, csv and cuda (apache#71) * release 17.2 (apache#72) * fix: from_pylist covariance (apache#73) * [pre-commit.ci] pre-commit autoupdate (apache#74) * Fix return type for middleware factory's start_call (apache#75) It can return None if middleware is not needed for a given call. * release 17.3 (apache#76) * fix: add missing return type in FlightDescriptor static methods (apache#80) * Support Tabular filter with Expression (apache#81) support Tabular filter with Expression * Support compute functions to accept Expression as parameter (apache#82) * fix: Fix the return value of Expression comparison (apache#83) * release 17.4 (apache#84) * fix: fix the array return type (apache#89) * a few type improvements, mostly flight related (apache#90) * FlightError.extra_info -> bytes * annotate FlightStreamReader.cancel return * BasicAuth serialize/deserialize * RecordBatchFileReader.schema * actually str | bytes * add_type_to_Field (apache#87) * add_type_to_Field * Field.type should return the covariant DataType --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * Support fsspec.AbstractFileSystem (apache#88) * supported_filesystem * fixes * remove unused import --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * release 17.5 (apache#91) * [pre-commit.ci] pre-commit autoupdate (apache#95) * fix: parquet not accepting NativeFile (apache#98) * feat: support pa.Buffer buffer protocol (apache#99) * feat: Support `compute` functions to accept ChunkedArray. (apache#100) * release 17.6 (apache#101) * [pre-commit.ci] pre-commit autoupdate (apache#102) * working towards making return signatures only have one type (mean and exp) (apache#105) * group_by_returns_TableGroupBy * return_single_type_for_mean_exp * revert table.pyi * compute.mean does not support BinaryScalar or BinaryArray --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * a table group_by was returing Self but should return TableGroupBy (apache#104) group_by_returns_TableGroupBy * [pre-commit.ci] pre-commit autoupdate (apache#106) updates: - [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](pre-commit/pre-commit-hooks@v4.6.0...v5.0.0) - [github.com/astral-sh/ruff-pre-commit: v0.6.7 → v0.6.9](astral-sh/ruff-pre-commit@v0.6.7...v0.6.9) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: RecordBatch missing `from_arrays` and `from_pandas` (apache#108) * release 17.7 (apache#109) * fix_combine_chunks (apache#110) * make Self backward compatible (apache#115) * fix: update ConvertOptions (apache#114) * add type property to Array (apache#112) * add type property to Array * Array.type should return covariant --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * release 17.8 (apache#117) * Add include_columns parameter in ConvertOptions (apache#118) * add list[str] overload to rename_columns (apache#119) * release 17.9 (apache#120) * [pre-commit.ci] pre-commit autoupdate (apache#124) updates: - [github.com/astral-sh/ruff-pre-commit: v0.6.9 → v0.7.0](astral-sh/ruff-pre-commit@v0.6.9...v0.7.0) - [github.com/pre-commit/mirrors-mypy: v1.11.2 → v1.12.1](pre-commit/mirrors-mypy@v1.11.2...v1.12.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * improve type annotations for parquet writer (apache#125) Add support for per-field compression specification Add missing none compression value. * Add missing return type for Schema.serialize (apache#123) * Add `Schema.field(int)` (apache#122) * Change various io related functions to support `StrPath` as a path input (apache#121) * Change various io related functions to support StrPath as a path input * fmt * Added StrPath | IO for feather types * fix type hint for sort_by (apache#130) sort_by takes str or list[tuple(name, order)] as its argument where str is a field name not a sort order * metadata on a schema can be passed as str (apache#128) For details see https://github.com/apache/arrow/blob/apache-arrow-17.0.0/python/pyarrow/types.pxi\#L2053-L2056 * Correct typevars for DictionaryType, MapType, RunEncodedType (apache#126) Correct type hints for Dictionary, RunEndEncoded and Map Signed-off-by: Jonas Dedden <university@jonas-dedden.de> Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * Add some more StrPath io parts that were overlooked. (apache#131) * Add some more StrPath io parts that were overlooked. Additionally, add the utility typealias `SingleOrList` that can be used in places where we want a concise type declaration but the there is a large union of types. * write_dataset(base_dir = ) can also take Path * Support ChunkedArray in add/append methods in Table (apache#129) * Add missing partitioning typing case (apache#132) This should now support the examples in the docstring for partitioning. * fix: typo 'permissive' instead of 'premissive' (apache#133) * release 17.10 (apache#134) * fix incorrect type hints for compute.sort_indices (apache#135) * disallow passing `names` as an argument to table when using dictionaries (apache#137) * [pre-commit.ci] pre-commit autoupdate (apache#138) updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.0 → v0.7.1](astral-sh/ruff-pre-commit@v0.7.0...v0.7.1) - [github.com/pre-commit/mirrors-mypy: v1.12.1 → v1.13.0](pre-commit/mirrors-mypy@v1.12.1...v1.13.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add missing type for FlightEndpoint (apache#136) * release 17.11 (apache#139) * [pre-commit.ci] pre-commit autoupdate (apache#140) updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.1 → v0.7.2](astral-sh/ruff-pre-commit@v0.7.1...v0.7.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (apache#142) updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.2 → v0.7.3](astral-sh/ruff-pre-commit@v0.7.2...v0.7.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * chore: Create FUNDING.yml (apache#143) Create FUNDING.yml * fix: `read_schema` should return Schema (apache#145) fix: read_schema should return Schema * release 17.12 (apache#146) * [pre-commit.ci] pre-commit autoupdate (apache#147) updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.3 → v0.7.4](astral-sh/ruff-pre-commit@v0.7.3...v0.7.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: `to_table` argument `columns` can be a dict of expressions (apache#149) * [pre-commit.ci] pre-commit autoupdate (apache#148) * [pre-commit.ci] pre-commit autoupdate updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.4 → v0.8.1](astral-sh/ruff-pre-commit@v0.7.4...v0.8.1) * ruff: ignore PYI063 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * release 17.13 (apache#151) * fix: FileSystem metadata value should be str (apache#152) * fix: FileSystemHandler metadata value should be str (apache#153) * [pre-commit.ci] pre-commit autoupdate (apache#154) updates: - [github.com/astral-sh/ruff-pre-commit: v0.8.1 → v0.8.2](astral-sh/ruff-pre-commit@v0.8.1...v0.8.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * improve coverage for pyarrow.struct typehint (apache#157) * fix: ipc typing (apache#159) * release 17.14 (apache#160) * fix: add missing param 'nbytes' to NativeFile.read (apache#163) * release 17.15 (apache#164) * [pre-commit.ci] pre-commit autoupdate (apache#161) updates: - [github.com/astral-sh/ruff-pre-commit: v0.8.2 → v0.8.3](astral-sh/ruff-pre-commit@v0.8.2...v0.8.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add 'None' as a valid argument for partitioning to the various parquet reading functions (apache#166) * [pre-commit.ci] pre-commit autoupdate (apache#165) updates: - [github.com/astral-sh/ruff-pre-commit: v0.8.3 → v0.8.6](astral-sh/ruff-pre-commit@v0.8.3...v0.8.6) - [github.com/pre-commit/mirrors-mypy: v1.13.0 → v1.14.1](pre-commit/mirrors-mypy@v1.13.0...v1.14.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: should use Collection[Array] instead list[Array] (apache#170) "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance Consider using "Sequence" instead, which is covariant * fix: update type hints for path_or_paths and source parameters in ParquetDataset and read_table (apache#171) * [pre-commit.ci] pre-commit autoupdate (apache#167) updates: - [github.com/astral-sh/ruff-pre-commit: v0.8.6 → v0.9.1](astral-sh/ruff-pre-commit@v0.8.6...v0.9.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * release 17.16 (apache#172) * Fixed pa.fixed_shape_tensor (apache#175) * [pre-commit.ci] pre-commit autoupdate (apache#173) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.1 → v0.9.4](astral-sh/ruff-pre-commit@v0.9.1...v0.9.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: Preserve generic in `ChunkedArray.type` (apache#177) * release 17.17 (apache#178) * [pre-commit.ci] pre-commit autoupdate (apache#176) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.4 → v0.9.6](astral-sh/ruff-pre-commit@v0.9.4...v0.9.6) - [github.com/pre-commit/mirrors-mypy: v1.14.1 → v1.15.0](pre-commit/mirrors-mypy@v1.14.1...v1.15.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: support to construct ListArray with primitive type (apache#179) * fix: Avoid `chunked_array` overlapping overloads (apache#183) * fix: Add placeholder annotations to `pc.if_else` (apache#182) * fix: Widen `Array` to `Array | ChunkedArray` (apache#181) * fix: add `pc.fill_null` (apache#185) - https://arrow.apache.org/docs/python/generated/pyarrow.compute.fill_null.html - https://github.com/narwhals-dev/narwhals/blob/05e47b27ebe27b24196cee5956d07748d65a62ee/narwhals/_arrow/series.py#L675 * fix: Allow Table.from_arrays to take a list containing a mix of Array and ChunkedArray (apache#187) Update table.pyi * release 17.18 (apache#188) * [pre-commit.ci] pre-commit autoupdate (apache#180) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.6 → v0.9.10](astral-sh/ruff-pre-commit@v0.9.6...v0.9.10) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: from_arrays for both Table and RecordBatch (apache#189) * fix: resolve some `pa.compute` overlaps (apache#184) * fix: resolve overlapping `compute.(add|divide)` * fix: copy from non-cloned signature * fix: resolve overlapping `compute.exp` * fix: resolve overlapping `compute.power` * fix: resolve overlapping `compute.equal` * fix: resolve overlapping `compute.and_` * fix: Include `Array` in `chunked_array` overload (apache#190) narwhals-dev/narwhals@0237f7a * release 17.19 (apache#191) * Add Scalar, Array and Type classes for Json & Uuid (apache#194) * Add Scalar, Array and Type classes for Json & Uuid * Formatting fixes * [pre-commit.ci] pre-commit autoupdate (apache#192) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.10 → v0.11.2](astral-sh/ruff-pre-commit@v0.9.10...v0.11.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Revert "Add Scalar, Array and Type classes for Json & Uuid" (apache#195) Revert "Add Scalar, Array and Type classes for Json & Uuid (apache#194)" * fix: Add missing `pc.equal` overload (apache#196) * feat: support pyarrow 19.0 (apache#198) * build: upgrade pyarrow min version to 19.0 * feat: support pyarrow 19.0 * omit mypy bool8 override error * fix: reexport new types (apache#199) * feat: override new patterns for func repeat and nulls (apache#200) * fix: reexport decimal64 array and decimal128 array * feat: override new patterns for func `repeat` and `nulls` * release: 19.1 (apache#201) * fix: Allow `Iterable[Table]` in `concat_tables` (apache#203) https://arrow.apache.org/docs/python/generated/pyarrow.concat_tables.html > tables : iterable of pyarrow.Table objects * fix: Allow `ChunkedArray[BooleanScalar]` in `pc.invert` (apache#204) Fixes https://github.com/narwhals-dev/narwhals/blob/caabc0efdef54f117c83888926860e3972ef69d5/narwhals/_arrow/series.py#L298-L299 * feat: Fully spec `TableGroupBy.aggregate` (apache#197) - https://arrow.apache.org/docs/python/compute.html#grouped-aggregations - https://arrow.apache.org/docs/python/generated/pyarrow.TableGroupBy.html#pyarrow.TableGroupBy.aggregate - https://github.com/apache/arrow/blob/34a984c842db42b409a1359e6e2cf167a2365a48/python/pyarrow/table.pxi#L6578-L6604 * fix: Add missing return type to `ChunkedArray.filter` (apache#205) * fix: Add relaxed final overload to logical functions (apache#206) Covers all of `pc.(and_ | and_kleene | and_not | and_not_kleene | or_ | or_kleene | xor)` Resolves: - https://github.com/narwhals-dev/narwhals/blob/caabc0efdef54f117c83888926860e3972ef69d5/narwhals/_arrow/series.py#L219-L233 - https://github.com/narwhals-dev/narwhals/blob/caabc0efdef54f117c83888926860e3972ef69d5/narwhals/_arrow/series.py#L662 * fix: Allow `ChunkedArray` in `Table.set_column` (apache#211) Also being more consistent with `ArrayOrChunkedArray[Any]` everywhere Discovered in - https://github.com/vega/vega-datasets/blob/343b7101391a81190ba24e1e8d62a381d2fef3bd/scripts/species.py#L798-L799 * chore: Ignore `fsspec` `[import-untyped]` (apache#210) ```py _fs.pyi:18: error: Skipping analyzing "fsspec": module is installed, but missing library stubs or py.typed marker [import-untyped] _fs.pyi:18: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports Found 1 error in 1 file (checked 64 source files) ``` - fsspec/filesystem_spec#625 - fsspec/filesystem_spec#1676 * feat: Convert `types.is_*` into `TypeIs` guards (apache#215) * chore: Add `types.__all__` * feat: Convert `types._is_*` into `TypeIs` guards I've been using this for a little while, but makes more sense to live in the stubs https://github.com/narwhals-dev/narwhals/blob/16427440e6d74939c403083b52ce3fb0af7d63c7/narwhals/_arrow/utils.py#L44-L67 * fix: Resolve `bit_wise_and` overlaps (apache#214) Fixes 3 errors: ```py compute.pyi:608:5 - error: Overload 1 for "bit_wise_and" overlaps overload 4 and returns an incompatible type (reportOverlappingOverload) compute.pyi:608:5 - error: Overload 1 for "bit_wise_and" overlaps overload 5 and returns an incompatible type (reportOverlappingOverload) compute.pyi:620:5 - error: Overload 3 for "bit_wise_and" will never be used because its parameters overlap overload 1 (reportOverlappingOverload) ``` * fix: Resolve `list_*` overlapping overloads (apache#213) * fix: Resolve `list_value_length` overlaps * fix: Resolve `list_element` overlaps * fix: Resolve `list_(flatten|slice|parent_indices)` overlaps An improvement, but still not that accurate * fix: Include `VarianceOptions` in `TableGroupBy.aggregate` (apache#212) - Follow-up to apache#197 - Noticed while writing up (narwhals-dev/narwhals#2385) - We already use it for `std`, `var` in https://github.com/narwhals-dev/narwhals/blob/16427440e6d74939c403083b52ce3fb0af7d63c7/narwhals/_arrow/group_by.py#L81-L82 * [pre-commit.ci] pre-commit autoupdate (apache#202) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.2 → v0.11.5](astral-sh/ruff-pre-commit@v0.11.2...v0.11.5) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: Resolve `Scalar.as_py` warnings for `DictionaryType` (apache#207) > scalar.pyi:75:20 - warning: TypeVar "_AsPyTypeK" appears only once in generic function signature > Use "object" instead (reportInvalidTypeVarUse) > scalar.pyi:85:20 - warning: TypeVar "_AsPyTypeK" appears only once in generic function signature > Use "object" instead (reportInvalidTypeVarUse) Instead just using `int`, which should be all that is possible from: https://github.com/zen-xu/pyarrow-stubs/blob/02552b81161d19d4aa71d8656b028eefac84612b/pyarrow-stubs/__lib_pxi/types.pyi#L154-L164 https://github.com/zen-xu/pyarrow-stubs/blob/02552b81161d19d4aa71d8656b028eefac84612b/pyarrow-stubs/__lib_pxi/types.pyi#L63-L70 * fix: Add default to `pc.sort_indices` (apache#216) * fix: Add default to `pc.sort_indices` Fixes narwhals-dev/narwhals#2390 (comment) Default is specified in https://arrow.apache.org/docs/python/generated/pyarrow.compute.sort_indices.html * refactor: Reuse some aliases * fix: Allow `list_size` with `Field` in `pa.list_` (apache#218) Closes apache#217 * allow `Table` or `RecordBatch` for dataset (apache#222) allow source argument pyarrow.dataset.dataset() to be RecordBatch | Table * refactor: Simplify `types` overloads (apache#219) * fix: `binary` overlap * fix: Simplify list constructors, `_Ordered` * refactor: Use `_Tz` default * fix: iter ChunkedArray should return scalar value (apache#224) * release: 19.2 (apache#225) * fix: Add missing `DictionaryArray` methods/properties (apache#226) - https://arrow.apache.org/docs/python/generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray.dictionary - https://arrow.apache.org/docs/python/generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray.indices - https://arrow.apache.org/docs/python/generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray.dictionary_decode - https://arrow.apache.org/docs/python/generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray.dictionary_encode - https://github.com/narwhals-dev/narwhals/blob/c23e56c56630761f0fbc58b575a1c987e57d58d5/narwhals/_arrow/series.py#L787-L798 - https://github.com/narwhals-dev/narwhals/blob/c23e56c56630761f0fbc58b575a1c987e57d58d5/narwhals/_arrow/series_cat.py#L14-L18 * chore: use pyright as static type checker (apache#227) * use pyright as static type checker * make pyright happy * fix: fix pyright action (apache#229) fix github ci * fix: Match runtime behavior of `(Table|RecordBatch).select` (apache#221) * fix: Match runtime behavior of `(Table|RecordBatch).select` - https://github.com/MarcoGorelli/narwhals/blob/5b02b592183b8d39e2d32e0aedd6c234bb22d405/narwhals/_arrow/dataframe.py#L305-L307 - https://github.com/MarcoGorelli/narwhals/blob/5b02b592183b8d39e2d32e0aedd6c234bb22d405/narwhals/_arrow/dataframe.py#L285-L294 Following up on what I thought was a simple stub issue, but we're both *too strict* and *too permissive* in different ways {placeholder} - https://github.com/apache/arrow/blob/d2ddee62329eb711572b4d71d6380673d7f7edd1/python/pyarrow/table.pxi#L4367-L4374 - https://github.com/apache/arrow/blob/d2ddee62329eb711572b4d71d6380673d7f7edd1/python/pyarrow/table.pxi#L1721-L1739 * update select * update select --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * [pre-commit.ci] pre-commit autoupdate (apache#220) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.5 → v0.11.8](astral-sh/ruff-pre-commit@v0.11.5...v0.11.8) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * feat: narrow scalar when type is given (apache#230) * rename Uint -> UInt * feat: narrow scalar when type is given * release 19.3 (apache#231) * chore: pyright use strict mode (apache#233) * fix types * update array.pyi * update scalar.pyi * update * update array * update array * optimize chunked_array * optimizer iterchunks * update * update pyproject.toml * fix: pa.nulls accept type rather than types (apache#234) * [pre-commit.ci] pre-commit autoupdate (apache#232) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.8 → v0.11.9](astral-sh/ruff-pre-commit@v0.11.8...v0.11.9) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * release 19.4 (apache#235) * lint(pyright): disable reportUnknownMemberType (apache#239) * [pre-commit.ci] pre-commit autoupdate (apache#236) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.9 → v0.11.13](astral-sh/ruff-pre-commit@v0.11.9...v0.11.13) - [github.com/RobertCraigie/pyright-python: v1.1.400 → v1.1.401](RobertCraigie/pyright-python@v1.1.400...v1.1.401) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * feat: support pyarrow 20.0 (apache#240) * [pre-commit.ci] pre-commit autoupdate (apache#241) updates: - [github.com/RobertCraigie/pyright-python: v1.1.401 → v1.1.402](RobertCraigie/pyright-python@v1.1.401...v1.1.402) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * support docstring (apache#242) * doc: complete tensor doc * doc: complete table doc * doc: complete scalar doc * doc: complete orc doc * doc: complete memory doc * doc: complete lib doc * doc: complete json doc * doc: complete hdfs doc * doc: complete gcsfs doc * doc: complete fs doc * doc: complete flight doc * doc: complete dataset doc * doc: complete dataset parquet doc * doc: complete dataset parquet encryption doc * doc: complete cuda doc * doc: complete csv doc * doc: complete azurefs doc * doc: complete core doc * doc: complete interchange doc * doc: complete array doc * doc: complete builder doc * doc: complete device doc * doc: complete io doc * doc: complete ipc doc * doc: complete types doc * mark deprecated apis * doc: complete _compute doc * doc: complete compute doc * doc: update compute doc * lint code * release 20.0.0.20250618 (apache#243) * fix: make ParquetFileFormat constructor args optional (apache#244) * fix: Field.remove_metadata should return Self (apache#246) * [pre-commit.ci] pre-commit autoupdate (apache#245) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.13 → v0.12.0](astral-sh/ruff-pre-commit@v0.11.13...v0.12.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * release 20.0.0.20250627 (apache#247) * fix: chunked_array with type should be specified (apache#250) * [pre-commit.ci] pre-commit autoupdate (apache#248) updates: - [github.com/astral-sh/ruff-pre-commit: v0.12.0 → v0.12.3](astral-sh/ruff-pre-commit@v0.12.0...v0.12.3) - [github.com/RobertCraigie/pyright-python: v1.1.402 → v1.1.403](RobertCraigie/pyright-python@v1.1.402...v1.1.403) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * release 20.0.0.20250715 (apache#251) * fix: The type parameter of array should be covariant (apache#253) * release 20.0.0.20250716 (apache#254) * Add py.typed file to signify that the library is typed See the relevant PEP https://peps.python.org/pep-0561 * Prepare `pyarrow-stubs` for history merging MINOR: [Python] Prepare `pyarrow-stubs` for history merging Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * Add `ty` configuration and suppress error codes * One line per rule * Add licence header from original repo for all `.pyi` files * Revert "Add licence header from original repo for all `.pyi` files" * Prepare for licence merging * Exclude `stubs` from `rat` test * Add Apache licence clause to `py.typed` * Reduce list * Resolve merge conflict --------- Signed-off-by: Jonas Dedden <university@jonas-dedden.de> Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> Co-authored-by: Jim Bosch <talljimbo@gmail.com> Co-authored-by: Oliver Mannion <125105+tekumara@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eugene Toder <eltoder@users.noreply.github.com> Co-authored-by: fvankrieken <fvankrieken@planning.nyc.gov> Co-authored-by: Ilia Ablamonov <ilia@flamefork.ru> Co-authored-by: Mathias Beguin <mathias.beguin@hotmail.com> Co-authored-by: Dylan Scott <dylan.scott@gmail.com> Co-authored-by: deanm0000 <37878412+deanm0000@users.noreply.github.com> Co-authored-by: Jan Moravec <moravecj@post.cz> Co-authored-by: Marius van Niekerk <marius.v.niekerk@gmail.com> Co-authored-by: Jonas Dedden <university@jonas-dedden.de> Co-authored-by: Fábio D. Batista <fabio@atelie.dev.br> Co-authored-by: ben-freist <93315290+ben-freist@users.noreply.github.com> Co-authored-by: Jiahao Yuan <kahojyun@icloud.com> Co-authored-by: Pim de Haan <pimdehaan@gmail.com> Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com> Co-authored-by: Tom Crasset <25140344+tcrasset@users.noreply.github.com> Co-authored-by: Tom McTiernan <tmct@users.noreply.github.com> Co-authored-by: Rok Mihevc <rok@mihevc.org>
1 parent 2cf52b4 commit 7a33cfa

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

68 files changed

+40577
-0
lines changed

dev/release/rat_exclude_files.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,7 @@ python/pyarrow/tests/__init__.py
9595
python/pyarrow/vendored/*
9696
python/pyarrow/src/arrow/python/vendored/*
9797
python/requirements*.txt
98+
python/stubs/*
9899
pax_global_header
99100
MANIFEST.in
100101
__init__.pxd

python/py.typed

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.

python/pyproject.toml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,3 +97,24 @@ version_file = 'pyarrow/_generated_version.py'
9797
version_scheme = 'guess-next-dev'
9898
git_describe_command = 'git describe --dirty --tags --long --match "apache-arrow-[0-9]*.*"'
9999
fallback_version = '22.0.0a0'
100+
101+
[tool.ty.rules]
102+
invalid-argument-type = "ignore"
103+
invalid-assignment = "ignore"
104+
invalid-context-manager = "ignore"
105+
invalid-return-type = "ignore"
106+
invalid-type-form = "ignore"
107+
no-matching-overload = "ignore"
108+
non-subscriptable = "ignore"
109+
not-iterable = "ignore"
110+
possibly-unbound-attribute = "ignore"
111+
possibly-unbound-import = "ignore"
112+
too-many-positional-arguments = "ignore"
113+
unknown-argument = "ignore"
114+
unresolved-attribute = "ignore"
115+
unresolved-global = "ignore"
116+
unresolved-import = "ignore"
117+
unresolved-reference = "ignore"
118+
unsupported-operator = "ignore"
119+
missing-argument = "ignore"
120+
call-non-callable = "ignore"

python/stubs/LICENSE

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
BSD 2-Clause License
2+
3+
Copyright (c) 2024, ZhengYu, Xu
4+
5+
Redistribution and use in source and binary forms, with or without
6+
modification, are permitted provided that the following conditions are met:
7+
8+
1. Redistributions of source code must retain the above copyright notice, this
9+
list of conditions and the following disclaimer.
10+
11+
2. Redistributions in binary form must reproduce the above copyright notice,
12+
this list of conditions and the following disclaimer in the documentation
13+
and/or other materials provided with the distribution.
14+
15+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
16+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
17+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
18+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
19+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
20+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
21+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
22+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
23+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
24+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

0 commit comments

Comments
 (0)