ARROW-317: Add Slice, Copy methods to Buffer #177

wesm · 2016-10-17T14:35:17Z

There's also a little bit of naming cleanup in bit-util.h, pardon the diff noise.

Change-Id: I63fca0131fd094b0bdbd7790410c918bbe1f50b6

Change-Id: Ia6f232bf020a0249c19453a2c39eea51b52f86cf

wesm · 2016-10-17T17:47:07Z

rebased

Change-Id: I4b8cfb4272ce0dea04974bada20e3e83697d7ceb

wesm · 2016-10-17T20:36:05Z

Taking forever to get Travis builds lately -- I just sent an email to support about it

wesm · 2016-10-18T01:17:55Z

Well I only had to wait like 7 hours for the build. +1

…threads I am hitting this problem at another location too. Author: Deepak Majeti <deepak.majeti@hpe.com> Closes apache#177 from majetideepak/PARQUET-739 and squashes the following commits: 85a73f1 [Deepak Majeti] PARQUET-739

…threads I am hitting this problem at another location too. Author: Deepak Majeti <deepak.majeti@hpe.com> Closes apache#177 from majetideepak/PARQUET-739 and squashes the following commits: 85a73f1 [Deepak Majeti] PARQUET-739 Change-Id: I745c246ba928a7cc5c29a4535e5b01f1bd391a82

@throws

* Flight SQL Ratification Based On Community Feedback #7 (apache#98) * Remove scope from 'hamcrest' dependency on java/pom.xml * Use flight top-level module on parent pom.xml instead of declaring each one * Avoid using getStatement inside StatementContext methods * Make StatementContext.getQuery() return String * Minor fixes on pom.xml * Move 'os-maven-plugin' to parent pom.xml * Update protobuf generation on pom.xml files * Use ClassLoader#getResource to get network.properties on TestFlightSql * Bind to any ephemeral port on TestFlightSql * Move JDBC-Arrow type default conversion from JdbcToArrowConfig to JdbcToArrowUtils * Micro-optimization: initialize ArrayList with the right size * Fix null-check on PreparedStatement#setParameters * Avoid wrapping vector into a ImmutableList and then into an ArrayList on FlightSqlExample#getTablesRoot * Remove null-check on VectorSchemaRoot on FlightSqlClient#setParameters() * Remove the need of separate cache for ResultSets * Add missing 'final' modifiers * Fix maven build from different directories (apache#114) * Add CrossReference methods to SqlProducer * Clean-up: remove boilerplate code by replacing with tools provided by Arrow Flight JDBC Adapter * Fix checkstyle and dependency management errors * Ignore broken tests * Fix broken tests for CreatePreparedStatement * Fix Schema generation not setting an unknown column type to nullable * Change FlightSqlProducer from abstract class to interface * Refactor tests due to creation of new column of primaryKey * Nit: fix checkstyle * Fix leaking connections on connection pool * [WIP] FlightSQL Ratification based on Community Comments (apache#73) * Move FlightSql examples to their own subpackage * Fix checkstyle issues * fix: change Status use to CallStatus * Remove unnecessary overhead of wrapping nullable objects into Optionals for the sole purpose of null-checking * Replace Guava's Preconditions with the ones provided by Apache * Fix typo in FlightSql.proto * Fix ordering of schema for FlightSql.proto * Explain why reserved range of IDs for GetSqlInfo is not entirely in use * Add comment to CommandGetTables to explain the encoding of table_schema * Remove redundat information on schemas * Fixed Javadoc on some methods, added Thread interrupt to executeUpdate methods, and updated Signal exceptions to CallStatus with description * Replace int32 with uint32 for GetSqlInfo name representation * Replace AssertionError with StatusRuntimeException for whenever attempting to unpack an invalid protobuf message * add comment to FlightSql.proto to update_rule and delete_rule * Replace inconsistent exception handling with CallStatus predetermined exceptions * correct comment to CreatePreparedStatement on FlightSql.proto * Remove unused dependencies * fix: change Status use to CallStatus on FlightSqlProducer * Changed from if not null check to Objects requireNonNull on Flight SQL Client * Remove Nullable annotation * Changed from checkNotNull to Objects#requireNotNull with description on Flight SQL Example * Add CallOptions to every RPC call by the client * Fix Maven dependency problems and checkstyle violations * Replace generic Collections with Lists when order matters in an RPC call * Fix Javadoc for FlightSqlClient * Add description to StatusRuntimeExceptions * Add descriptions to Exceptions * Correct update_rule and delete_rule description on FlighSql.proto * Verify wheter Root is empty before sending request to server * Add call options to PreparedStatement * Replace constant checking of whether client is open with #checkOpen * Add CallOptions to #close for PreparedStatement * Refactor PreparedStatement usages of CallOptions * Fix broken tests * Fix FlightSql.proto documentation * Update documentation for format/FlightSql.proto Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> * Fix checkstyle violations * Require non null tables for GetExportedKeys and GetImportedKeys * Not storing CallOptions in PreparedStatement * Update documentation comments for protobuf * Replace IntVector for UInt1Vector for delete_rule and update_rule * Fix protobuf for FlightSQL * Fix bug with empty metadata * Update update_rule and delete_rule documentation on proto * Remove explicit dependency on JDBC's DatabaseMetaData on UpdateDeleteRules * Use MessageOptions instead of FieldOptions on proto * Add missing JavaDoc about 'options' parameter * Fix CommandGetSqlInfo documentation * Add @throws to FlightSqlClient#checkOpen JavaDoc Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br> Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com> Co-authored-by: Rafael Telles <rafael@telles.dev> Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * Fix Minor POM Issues to allow Maven build (apache#161) * Fix minor POM Issues to allow Maven Build * Revert Flight SQL POM State * Remove unused imports Co-authored-by: Rafael Telles <rafael@telles.dev> * Ignore broken tests * Refactor tests due to creation of new column of primaryKey * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * [FlightRPC] Flight SQL POC Add extensions in the Apache Arrow project’s Arrow Flight modules to provide a standard way for clients and servers to communicate with SQL-like semantics. Do not pull to master. A message to the mailing list will accompany this and another proposal in the coming days for discussion. * Update FlightSqlProducer to conform to new design. Rename files for SQL -> Sql. Correct compilation errors in client code, but design needs to be updated. Tests do not yet compile. * Correct the dense_union type for schema return of SQL info. Correct some additional SQL -> Sql file renames. Reduce the test compilation problems (still more to do). * Additional CR changes. Note - FlightSqlExample is not functional and needs to be updated. * Add support for primary and foreign keys. * Fix broken Maven build * Add Hamcrest as a root-level dependency * Create test for GetCatalogs * Update tests for GetTables * Fix test for GetTables * Enable support for includeSchema in GetTables * Fix checkstyle and dependency management errors * Remove boilerplate code for creating a new Schema by reusing default converter from JdbcToArrowConfig#DEFAULT_JDBC_TO_ARROW_TYPE_CONVERTER * WIP: Working on fixing data consistency issue where catalog is null in some parts and "" in others * Implement FlightSqlExample's GetPrimaryKey command * WIP: Add support for GetCatalogs: GetFlightInfoCatalogs * WIP: Add support for GetCatalogs: GetStreamCatalogs * WIP: Add support for GetTableTypes: getFlightInfoTableTypes * WIP: Add support for GetTableTypes: getStreamTableTypes * Fix rebase conflicts * Minor refactor: apply DRY principle to repeated methods * Ignore broken tests * Fix broken tests for CreatePreparedStatement * Fix Schema generation not setting an unknown column type to nullable * Change FlightSqlProducer from abstract class to interface * Update FlightSql protobuf to allow nullable values as parameters for nullable fields * Start separating commandForeignKey into two new commands * Fix CheckStyle issues * Change empty string to null value * Implement CommandGetImportedKeys * Fix checkstyle violations * Minor refactor: remove unused methods * Implement FlightSqlClient.executeUpdate * Refactor the code to not use string when getting from cache * Nit: fix checkstyle * Fix conflicts between tests for creating a new statement and checking its schema * Minor refactor: remove unused fields @ FlightSqlUtils * Fix wrong StreamListener usages and multiple instances of RootAllocators * Fix checkstyle * Modify execute preparedStatement flow * Fix leaking connections on connection pool * Deal with query with parameter in the preparedStatement * Fix rebase issues with FlightSql.proto * Fix missing code on adapter/jdbc/JdbcToArrowUtils.java due to rebase issue * [WIP] FlightSQL Ratification based on Community Comments (apache#73) * Move FlightSql examples to their own subpackage * Fix checkstyle issues * fix: change Status use to CallStatus * Remove unnecessary overhead of wrapping nullable objects into Optionals for the sole purpose of null-checking * Replace Guava's Preconditions with the ones provided by Apache * Fix typo in FlightSql.proto * Fix ordering of schema for FlightSql.proto * Explain why reserved range of IDs for GetSqlInfo is not entirely in use * Add comment to CommandGetTables to explain the encoding of table_schema * Remove redundat information on schemas * Fixed Javadoc on some methods, added Thread interrupt to executeUpdate methods, and updated Signal exceptions to CallStatus with description * Replace int32 with uint32 for GetSqlInfo name representation * Replace AssertionError with StatusRuntimeException for whenever attempting to unpack an invalid protobuf message * add comment to FlightSql.proto to update_rule and delete_rule * Replace inconsistent exception handling with CallStatus predetermined exceptions * correct comment to CreatePreparedStatement on FlightSql.proto * Remove unused dependencies * fix: change Status use to CallStatus on FlightSqlProducer * Changed from if not null check to Objects requireNonNull on Flight SQL Client * Remove Nullable annotation * Changed from checkNotNull to Objects#requireNotNull with description on Flight SQL Example * Add CallOptions to every RPC call by the client * Fix Maven dependency problems and checkstyle violations * Replace generic Collections with Lists when order matters in an RPC call * Fix Javadoc for FlightSqlClient * Add description to StatusRuntimeExceptions * Add descriptions to Exceptions * Correct update_rule and delete_rule description on FlighSql.proto * Verify wheter Root is empty before sending request to server * Add call options to PreparedStatement * Replace constant checking of whether client is open with #checkOpen * Add CallOptions to #close for PreparedStatement * Refactor PreparedStatement usages of CallOptions * Fix broken tests * Fix FlightSql.proto documentation * Update documentation for format/FlightSql.proto Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> * Fix checkstyle violations * Require non null tables for GetExportedKeys and GetImportedKeys * Not storing CallOptions in PreparedStatement * Update documentation comments for protobuf * Replace IntVector for UInt1Vector for delete_rule and update_rule * Fix protobuf for FlightSQL * Fix bug with empty metadata * Update update_rule and delete_rule documentation on proto * Remove explicit dependency on JDBC's DatabaseMetaData on UpdateDeleteRules * Use MessageOptions instead of FieldOptions on proto * Add missing JavaDoc about 'options' parameter * Fix CommandGetSqlInfo documentation * Add @throws to FlightSqlClient#checkOpen JavaDoc Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br> Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com> Co-authored-by: Rafael Telles <rafael@telles.dev> Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> * FlightSQL Ratification based on Community Comments (round 2) (apache#85) * Remove unused client_execution_handler from Protobuf * Update documentation on CommandGetPrimaryKeys * Update documentation on CommandGetImportedKeys and CommandGetExportedKeys * Change exception type on FlightSqlClient#executeUpdate * Add @return to FlightSqlClient#executeUpdate JavaDoc * Switch order of key_name and key_sequence on CommandGetTableKeys documentation * Update JavaDoc for FlIghtSqlClient#clearParameters * Add private constructor to FlightSqlProducer.SqlInfo * Update JavaDoc for FlIghtSqlClient#clearParameters * Fix wrong CommandGetPrimaryKeys documentation on Proto file * Fix order of key_name and key_sequence * Split CommandStatementQuery in 2 messages, one for Command and other for Ticket * Fix leaking Connections on FlightSqlExample * Remove unused variable of cache * Flight SQL Ratification Based On Community Feedback #5 (apache#91) * Delegate GetSchemaImportedKeys * Remove schema retrieval methods for catalog functions and delegate to constants * Add IPC encapsulation to Schema serialization * Fix checkstyle violations * Update javadoc for FlightSqlClient * Update documentation for FlightSql.proto Co-authored-by: Abner Eduardo Ferreira <abenaru@protonmail.ch> * Flight SQL Ratification Based On Community Feedback #6 (apache#94) * Refactored FlightSql Statement Constant names * Defined non-nullable parameters for FlightSql proto * Resolved minimal checkstyle issues * Added further documentation for catalog and schema * Refactored FlightSql proto comments to include more information * Added Field/FieldType notNullable methods * Refactored FlightSqlClient and FlightSqlExample to leverage Field notNullable method * Removed opaque query warning from FlightSql proto * Added the optional tag for the returned schema of getTables to proto * Flight SQL Ratification Based On Community Feedback #7 (apache#98) * Remove scope from 'hamcrest' dependency on java/pom.xml * Use flight top-level module on parent pom.xml instead of declaring each one * Avoid using getStatement inside StatementContext methods * Make StatementContext.getQuery() return String * Minor fixes on pom.xml * Move 'os-maven-plugin' to parent pom.xml * Update protobuf generation on pom.xml files * Use ClassLoader#getResource to get network.properties on TestFlightSql * Bind to any ephemeral port on TestFlightSql * Move JDBC-Arrow type default conversion from JdbcToArrowConfig to JdbcToArrowUtils * Micro-optimization: initialize ArrayList with the right size * Fix null-check on PreparedStatement#setParameters * Avoid wrapping vector into a ImmutableList and then into an ArrayList on FlightSqlExample#getTablesRoot * Remove null-check on VectorSchemaRoot on FlightSqlClient#setParameters() * Remove the need of separate cache for ResultSets * Add missing 'final' modifiers * Fix missing generated sources on built flight-sql jar (apache#101) * Bump protobuf version * Flight SQL Ratification Based On Community Feedback #8 (apache#113) * Change scope of arrow-memory-netty to test for flight-sql * Remove unused dependency arrow-memory-netty * Update common-pool2 and common-dbcp2 dependencies * Remove 'executions' from parent pom.xml for plugin protobuf-maven-plugin * Adjust protobuf-maven-plugin settings on pom.xml files * Move dep.protobuf.version and dep.grpc.version to top pom.xml * Remove <url> from arrow-flight's pom.xml * Fix maven build from different directories (apache#114) * Add test cases for bitshifting operations required for filtering out some SqlInfo data * Fix Schema serialization and deserialization on Flight SQL methods * [FlightSQL] Add missing method for creating bitmask from GetSqlInfo option enum (apache#148) * Add util method for creating bitmask from multiple protobuf enums for FlightSql GetSqlInfo enum * Add test cases for utility method for creating bitmask from protobuf options * Make changes regarding to reviews Co-authored-by: Rafael Telles <rafael@telles.dev> * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * WIP: Refactor SqlInfoProvider * WIP * WIP * WIP * Fix rebase issues * Restore FlightSql.proto state * Fix checkstyle * Fix rebase colateral * Restore FlightSql.proto Co-authored-by: Jose Almeida <almeidajcr90@gmail.com> Co-authored-by: Abner Eduardo Ferreira <abenaru@protonmail.ch> Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br> Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com> Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> Co-authored-by: Vinicius Fraga <62815192+vfraga@users.noreply.github.com> Co-authored-by: Ryan Nicholson <rnicholson@dremio.com> Co-authored-by: Kyle Porter <kporter@dremio.com>

@throws

* Flight SQL Ratification Based On Community Feedback #7 (apache#98) * Remove scope from 'hamcrest' dependency on java/pom.xml * Use flight top-level module on parent pom.xml instead of declaring each one * Avoid using getStatement inside StatementContext methods * Make StatementContext.getQuery() return String * Minor fixes on pom.xml * Move 'os-maven-plugin' to parent pom.xml * Update protobuf generation on pom.xml files * Use ClassLoader#getResource to get network.properties on TestFlightSql * Bind to any ephemeral port on TestFlightSql * Move JDBC-Arrow type default conversion from JdbcToArrowConfig to JdbcToArrowUtils * Micro-optimization: initialize ArrayList with the right size * Fix null-check on PreparedStatement#setParameters * Avoid wrapping vector into a ImmutableList and then into an ArrayList on FlightSqlExample#getTablesRoot * Remove null-check on VectorSchemaRoot on FlightSqlClient#setParameters() * Remove the need of separate cache for ResultSets * Add missing 'final' modifiers * Fix maven build from different directories (apache#114) * Add CrossReference methods to SqlProducer * Clean-up: remove boilerplate code by replacing with tools provided by Arrow Flight JDBC Adapter * Fix checkstyle and dependency management errors * Ignore broken tests * Fix broken tests for CreatePreparedStatement * Fix Schema generation not setting an unknown column type to nullable * Change FlightSqlProducer from abstract class to interface * Refactor tests due to creation of new column of primaryKey * Nit: fix checkstyle * Fix leaking connections on connection pool * [WIP] FlightSQL Ratification based on Community Comments (apache#73) * Move FlightSql examples to their own subpackage * Fix checkstyle issues * fix: change Status use to CallStatus * Remove unnecessary overhead of wrapping nullable objects into Optionals for the sole purpose of null-checking * Replace Guava's Preconditions with the ones provided by Apache * Fix typo in FlightSql.proto * Fix ordering of schema for FlightSql.proto * Explain why reserved range of IDs for GetSqlInfo is not entirely in use * Add comment to CommandGetTables to explain the encoding of table_schema * Remove redundat information on schemas * Fixed Javadoc on some methods, added Thread interrupt to executeUpdate methods, and updated Signal exceptions to CallStatus with description * Replace int32 with uint32 for GetSqlInfo name representation * Replace AssertionError with StatusRuntimeException for whenever attempting to unpack an invalid protobuf message * add comment to FlightSql.proto to update_rule and delete_rule * Replace inconsistent exception handling with CallStatus predetermined exceptions * correct comment to CreatePreparedStatement on FlightSql.proto * Remove unused dependencies * fix: change Status use to CallStatus on FlightSqlProducer * Changed from if not null check to Objects requireNonNull on Flight SQL Client * Remove Nullable annotation * Changed from checkNotNull to Objects#requireNotNull with description on Flight SQL Example * Add CallOptions to every RPC call by the client * Fix Maven dependency problems and checkstyle violations * Replace generic Collections with Lists when order matters in an RPC call * Fix Javadoc for FlightSqlClient * Add description to StatusRuntimeExceptions * Add descriptions to Exceptions * Correct update_rule and delete_rule description on FlighSql.proto * Verify wheter Root is empty before sending request to server * Add call options to PreparedStatement * Replace constant checking of whether client is open with #checkOpen * Add CallOptions to #close for PreparedStatement * Refactor PreparedStatement usages of CallOptions * Fix broken tests * Fix FlightSql.proto documentation * Update documentation for format/FlightSql.proto Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> * Fix checkstyle violations * Require non null tables for GetExportedKeys and GetImportedKeys * Not storing CallOptions in PreparedStatement * Update documentation comments for protobuf * Replace IntVector for UInt1Vector for delete_rule and update_rule * Fix protobuf for FlightSQL * Fix bug with empty metadata * Update update_rule and delete_rule documentation on proto * Remove explicit dependency on JDBC's DatabaseMetaData on UpdateDeleteRules * Use MessageOptions instead of FieldOptions on proto * Add missing JavaDoc about 'options' parameter * Fix CommandGetSqlInfo documentation * Add @throws to FlightSqlClient#checkOpen JavaDoc Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br> Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com> Co-authored-by: Rafael Telles <rafael@telles.dev> Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * Fix Minor POM Issues to allow Maven build (apache#161) * Fix minor POM Issues to allow Maven Build * Revert Flight SQL POM State * Remove unused imports Co-authored-by: Rafael Telles <rafael@telles.dev> * Ignore broken tests * Refactor tests due to creation of new column of primaryKey * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * [FlightRPC] Flight SQL POC Add extensions in the Apache Arrow project’s Arrow Flight modules to provide a standard way for clients and servers to communicate with SQL-like semantics. Do not pull to master. A message to the mailing list will accompany this and another proposal in the coming days for discussion. * Update FlightSqlProducer to conform to new design. Rename files for SQL -> Sql. Correct compilation errors in client code, but design needs to be updated. Tests do not yet compile. * Correct the dense_union type for schema return of SQL info. Correct some additional SQL -> Sql file renames. Reduce the test compilation problems (still more to do). * Additional CR changes. Note - FlightSqlExample is not functional and needs to be updated. * Add support for primary and foreign keys. * Fix broken Maven build * Add Hamcrest as a root-level dependency * Create test for GetCatalogs * Update tests for GetTables * Fix test for GetTables * Enable support for includeSchema in GetTables * Fix checkstyle and dependency management errors * Remove boilerplate code for creating a new Schema by reusing default converter from JdbcToArrowConfig#DEFAULT_JDBC_TO_ARROW_TYPE_CONVERTER * WIP: Working on fixing data consistency issue where catalog is null in some parts and "" in others * Implement FlightSqlExample's GetPrimaryKey command * WIP: Add support for GetCatalogs: GetFlightInfoCatalogs * WIP: Add support for GetCatalogs: GetStreamCatalogs * WIP: Add support for GetTableTypes: getFlightInfoTableTypes * WIP: Add support for GetTableTypes: getStreamTableTypes * Fix rebase conflicts * Minor refactor: apply DRY principle to repeated methods * Ignore broken tests * Fix broken tests for CreatePreparedStatement * Fix Schema generation not setting an unknown column type to nullable * Change FlightSqlProducer from abstract class to interface * Update FlightSql protobuf to allow nullable values as parameters for nullable fields * Start separating commandForeignKey into two new commands * Fix CheckStyle issues * Change empty string to null value * Implement CommandGetImportedKeys * Fix checkstyle violations * Minor refactor: remove unused methods * Implement FlightSqlClient.executeUpdate * Refactor the code to not use string when getting from cache * Nit: fix checkstyle * Fix conflicts between tests for creating a new statement and checking its schema * Minor refactor: remove unused fields @ FlightSqlUtils * Fix wrong StreamListener usages and multiple instances of RootAllocators * Fix checkstyle * Modify execute preparedStatement flow * Fix leaking connections on connection pool * Deal with query with parameter in the preparedStatement * Fix rebase issues with FlightSql.proto * Fix missing code on adapter/jdbc/JdbcToArrowUtils.java due to rebase issue * [WIP] FlightSQL Ratification based on Community Comments (apache#73) * Move FlightSql examples to their own subpackage * Fix checkstyle issues * fix: change Status use to CallStatus * Remove unnecessary overhead of wrapping nullable objects into Optionals for the sole purpose of null-checking * Replace Guava's Preconditions with the ones provided by Apache * Fix typo in FlightSql.proto * Fix ordering of schema for FlightSql.proto * Explain why reserved range of IDs for GetSqlInfo is not entirely in use * Add comment to CommandGetTables to explain the encoding of table_schema * Remove redundat information on schemas * Fixed Javadoc on some methods, added Thread interrupt to executeUpdate methods, and updated Signal exceptions to CallStatus with description * Replace int32 with uint32 for GetSqlInfo name representation * Replace AssertionError with StatusRuntimeException for whenever attempting to unpack an invalid protobuf message * add comment to FlightSql.proto to update_rule and delete_rule * Replace inconsistent exception handling with CallStatus predetermined exceptions * correct comment to CreatePreparedStatement on FlightSql.proto * Remove unused dependencies * fix: change Status use to CallStatus on FlightSqlProducer * Changed from if not null check to Objects requireNonNull on Flight SQL Client * Remove Nullable annotation * Changed from checkNotNull to Objects#requireNotNull with description on Flight SQL Example * Add CallOptions to every RPC call by the client * Fix Maven dependency problems and checkstyle violations * Replace generic Collections with Lists when order matters in an RPC call * Fix Javadoc for FlightSqlClient * Add description to StatusRuntimeExceptions * Add descriptions to Exceptions * Correct update_rule and delete_rule description on FlighSql.proto * Verify wheter Root is empty before sending request to server * Add call options to PreparedStatement * Replace constant checking of whether client is open with #checkOpen * Add CallOptions to #close for PreparedStatement * Refactor PreparedStatement usages of CallOptions * Fix broken tests * Fix FlightSql.proto documentation * Update documentation for format/FlightSql.proto Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> * Fix checkstyle violations * Require non null tables for GetExportedKeys and GetImportedKeys * Not storing CallOptions in PreparedStatement * Update documentation comments for protobuf * Replace IntVector for UInt1Vector for delete_rule and update_rule * Fix protobuf for FlightSQL * Fix bug with empty metadata * Update update_rule and delete_rule documentation on proto * Remove explicit dependency on JDBC's DatabaseMetaData on UpdateDeleteRules * Use MessageOptions instead of FieldOptions on proto * Add missing JavaDoc about 'options' parameter * Fix CommandGetSqlInfo documentation * Add @throws to FlightSqlClient#checkOpen JavaDoc Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br> Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com> Co-authored-by: Rafael Telles <rafael@telles.dev> Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> * FlightSQL Ratification based on Community Comments (round 2) (apache#85) * Remove unused client_execution_handler from Protobuf * Update documentation on CommandGetPrimaryKeys * Update documentation on CommandGetImportedKeys and CommandGetExportedKeys * Change exception type on FlightSqlClient#executeUpdate * Add @return to FlightSqlClient#executeUpdate JavaDoc * Switch order of key_name and key_sequence on CommandGetTableKeys documentation * Update JavaDoc for FlIghtSqlClient#clearParameters * Add private constructor to FlightSqlProducer.SqlInfo * Update JavaDoc for FlIghtSqlClient#clearParameters * Fix wrong CommandGetPrimaryKeys documentation on Proto file * Fix order of key_name and key_sequence * Split CommandStatementQuery in 2 messages, one for Command and other for Ticket * Fix leaking Connections on FlightSqlExample * Remove unused variable of cache * Flight SQL Ratification Based On Community Feedback #5 (apache#91) * Delegate GetSchemaImportedKeys * Remove schema retrieval methods for catalog functions and delegate to constants * Add IPC encapsulation to Schema serialization * Fix checkstyle violations * Update javadoc for FlightSqlClient * Update documentation for FlightSql.proto Co-authored-by: Abner Eduardo Ferreira <abenaru@protonmail.ch> * Flight SQL Ratification Based On Community Feedback #6 (apache#94) * Refactored FlightSql Statement Constant names * Defined non-nullable parameters for FlightSql proto * Resolved minimal checkstyle issues * Added further documentation for catalog and schema * Refactored FlightSql proto comments to include more information * Added Field/FieldType notNullable methods * Refactored FlightSqlClient and FlightSqlExample to leverage Field notNullable method * Removed opaque query warning from FlightSql proto * Added the optional tag for the returned schema of getTables to proto * Flight SQL Ratification Based On Community Feedback #7 (apache#98) * Remove scope from 'hamcrest' dependency on java/pom.xml * Use flight top-level module on parent pom.xml instead of declaring each one * Avoid using getStatement inside StatementContext methods * Make StatementContext.getQuery() return String * Minor fixes on pom.xml * Move 'os-maven-plugin' to parent pom.xml * Update protobuf generation on pom.xml files * Use ClassLoader#getResource to get network.properties on TestFlightSql * Bind to any ephemeral port on TestFlightSql * Move JDBC-Arrow type default conversion from JdbcToArrowConfig to JdbcToArrowUtils * Micro-optimization: initialize ArrayList with the right size * Fix null-check on PreparedStatement#setParameters * Avoid wrapping vector into a ImmutableList and then into an ArrayList on FlightSqlExample#getTablesRoot * Remove null-check on VectorSchemaRoot on FlightSqlClient#setParameters() * Remove the need of separate cache for ResultSets * Add missing 'final' modifiers * Fix missing generated sources on built flight-sql jar (apache#101) * Bump protobuf version * Flight SQL Ratification Based On Community Feedback #8 (apache#113) * Change scope of arrow-memory-netty to test for flight-sql * Remove unused dependency arrow-memory-netty * Update common-pool2 and common-dbcp2 dependencies * Remove 'executions' from parent pom.xml for plugin protobuf-maven-plugin * Adjust protobuf-maven-plugin settings on pom.xml files * Move dep.protobuf.version and dep.grpc.version to top pom.xml * Remove <url> from arrow-flight's pom.xml * Fix maven build from different directories (apache#114) * Add test cases for bitshifting operations required for filtering out some SqlInfo data * Fix Schema serialization and deserialization on Flight SQL methods * [FlightSQL] Add missing method for creating bitmask from GetSqlInfo option enum (apache#148) * Add util method for creating bitmask from multiple protobuf enums for FlightSql GetSqlInfo enum * Add test cases for utility method for creating bitmask from protobuf options * Make changes regarding to reviews Co-authored-by: Rafael Telles <rafael@telles.dev> * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * WIP: Refactor SqlInfoProvider * WIP * WIP * WIP * Fix rebase issues * Restore FlightSql.proto state * Fix checkstyle * Fix rebase colateral * Restore FlightSql.proto Co-authored-by: Jose Almeida <almeidajcr90@gmail.com> Co-authored-by: Abner Eduardo Ferreira <abenaru@protonmail.ch> Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br> Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com> Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> Co-authored-by: Vinicius Fraga <62815192+vfraga@users.noreply.github.com> Co-authored-by: Ryan Nicholson <rnicholson@dremio.com> Co-authored-by: Kyle Porter <kporter@dremio.com>

@throws

* Flight SQL Ratification Based On Community Feedback #7 (apache#98) * Remove scope from 'hamcrest' dependency on java/pom.xml * Use flight top-level module on parent pom.xml instead of declaring each one * Avoid using getStatement inside StatementContext methods * Make StatementContext.getQuery() return String * Minor fixes on pom.xml * Move 'os-maven-plugin' to parent pom.xml * Update protobuf generation on pom.xml files * Use ClassLoader#getResource to get network.properties on TestFlightSql * Bind to any ephemeral port on TestFlightSql * Move JDBC-Arrow type default conversion from JdbcToArrowConfig to JdbcToArrowUtils * Micro-optimization: initialize ArrayList with the right size * Fix null-check on PreparedStatement#setParameters * Avoid wrapping vector into a ImmutableList and then into an ArrayList on FlightSqlExample#getTablesRoot * Remove null-check on VectorSchemaRoot on FlightSqlClient#setParameters() * Remove the need of separate cache for ResultSets * Add missing 'final' modifiers * Fix maven build from different directories (apache#114) * Add CrossReference methods to SqlProducer * Clean-up: remove boilerplate code by replacing with tools provided by Arrow Flight JDBC Adapter * Fix checkstyle and dependency management errors * Ignore broken tests * Fix broken tests for CreatePreparedStatement * Fix Schema generation not setting an unknown column type to nullable * Change FlightSqlProducer from abstract class to interface * Refactor tests due to creation of new column of primaryKey * Nit: fix checkstyle * Fix leaking connections on connection pool * [WIP] FlightSQL Ratification based on Community Comments (apache#73) * Move FlightSql examples to their own subpackage * Fix checkstyle issues * fix: change Status use to CallStatus * Remove unnecessary overhead of wrapping nullable objects into Optionals for the sole purpose of null-checking * Replace Guava's Preconditions with the ones provided by Apache * Fix typo in FlightSql.proto * Fix ordering of schema for FlightSql.proto * Explain why reserved range of IDs for GetSqlInfo is not entirely in use * Add comment to CommandGetTables to explain the encoding of table_schema * Remove redundat information on schemas * Fixed Javadoc on some methods, added Thread interrupt to executeUpdate methods, and updated Signal exceptions to CallStatus with description * Replace int32 with uint32 for GetSqlInfo name representation * Replace AssertionError with StatusRuntimeException for whenever attempting to unpack an invalid protobuf message * add comment to FlightSql.proto to update_rule and delete_rule * Replace inconsistent exception handling with CallStatus predetermined exceptions * correct comment to CreatePreparedStatement on FlightSql.proto * Remove unused dependencies * fix: change Status use to CallStatus on FlightSqlProducer * Changed from if not null check to Objects requireNonNull on Flight SQL Client * Remove Nullable annotation * Changed from checkNotNull to Objects#requireNotNull with description on Flight SQL Example * Add CallOptions to every RPC call by the client * Fix Maven dependency problems and checkstyle violations * Replace generic Collections with Lists when order matters in an RPC call * Fix Javadoc for FlightSqlClient * Add description to StatusRuntimeExceptions * Add descriptions to Exceptions * Correct update_rule and delete_rule description on FlighSql.proto * Verify wheter Root is empty before sending request to server * Add call options to PreparedStatement * Replace constant checking of whether client is open with #checkOpen * Add CallOptions to #close for PreparedStatement * Refactor PreparedStatement usages of CallOptions * Fix broken tests * Fix FlightSql.proto documentation * Update documentation for format/FlightSql.proto Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> * Fix checkstyle violations * Require non null tables for GetExportedKeys and GetImportedKeys * Not storing CallOptions in PreparedStatement * Update documentation comments for protobuf * Replace IntVector for UInt1Vector for delete_rule and update_rule * Fix protobuf for FlightSQL * Fix bug with empty metadata * Update update_rule and delete_rule documentation on proto * Remove explicit dependency on JDBC's DatabaseMetaData on UpdateDeleteRules * Use MessageOptions instead of FieldOptions on proto * Add missing JavaDoc about 'options' parameter * Fix CommandGetSqlInfo documentation * Add @throws to FlightSqlClient#checkOpen JavaDoc Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br> Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com> Co-authored-by: Rafael Telles <rafael@telles.dev> Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * Fix Minor POM Issues to allow Maven build (apache#161) * Fix minor POM Issues to allow Maven Build * Revert Flight SQL POM State * Remove unused imports Co-authored-by: Rafael Telles <rafael@telles.dev> * Ignore broken tests * Refactor tests due to creation of new column of primaryKey * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * [FlightRPC] Flight SQL POC Add extensions in the Apache Arrow project’s Arrow Flight modules to provide a standard way for clients and servers to communicate with SQL-like semantics. Do not pull to master. A message to the mailing list will accompany this and another proposal in the coming days for discussion. * Update FlightSqlProducer to conform to new design. Rename files for SQL -> Sql. Correct compilation errors in client code, but design needs to be updated. Tests do not yet compile. * Correct the dense_union type for schema return of SQL info. Correct some additional SQL -> Sql file renames. Reduce the test compilation problems (still more to do). * Additional CR changes. Note - FlightSqlExample is not functional and needs to be updated. * Add support for primary and foreign keys. * Fix broken Maven build * Add Hamcrest as a root-level dependency * Create test for GetCatalogs * Update tests for GetTables * Fix test for GetTables * Enable support for includeSchema in GetTables * Fix checkstyle and dependency management errors * Remove boilerplate code for creating a new Schema by reusing default converter from JdbcToArrowConfig#DEFAULT_JDBC_TO_ARROW_TYPE_CONVERTER * WIP: Working on fixing data consistency issue where catalog is null in some parts and "" in others * Implement FlightSqlExample's GetPrimaryKey command * WIP: Add support for GetCatalogs: GetFlightInfoCatalogs * WIP: Add support for GetCatalogs: GetStreamCatalogs * WIP: Add support for GetTableTypes: getFlightInfoTableTypes * WIP: Add support for GetTableTypes: getStreamTableTypes * Fix rebase conflicts * Minor refactor: apply DRY principle to repeated methods * Ignore broken tests * Fix broken tests for CreatePreparedStatement * Fix Schema generation not setting an unknown column type to nullable * Change FlightSqlProducer from abstract class to interface * Update FlightSql protobuf to allow nullable values as parameters for nullable fields * Start separating commandForeignKey into two new commands * Fix CheckStyle issues * Change empty string to null value * Implement CommandGetImportedKeys * Fix checkstyle violations * Minor refactor: remove unused methods * Implement FlightSqlClient.executeUpdate * Refactor the code to not use string when getting from cache * Nit: fix checkstyle * Fix conflicts between tests for creating a new statement and checking its schema * Minor refactor: remove unused fields @ FlightSqlUtils * Fix wrong StreamListener usages and multiple instances of RootAllocators * Fix checkstyle * Modify execute preparedStatement flow * Fix leaking connections on connection pool * Deal with query with parameter in the preparedStatement * Fix rebase issues with FlightSql.proto * Fix missing code on adapter/jdbc/JdbcToArrowUtils.java due to rebase issue * [WIP] FlightSQL Ratification based on Community Comments (apache#73) * Move FlightSql examples to their own subpackage * Fix checkstyle issues * fix: change Status use to CallStatus * Remove unnecessary overhead of wrapping nullable objects into Optionals for the sole purpose of null-checking * Replace Guava's Preconditions with the ones provided by Apache * Fix typo in FlightSql.proto * Fix ordering of schema for FlightSql.proto * Explain why reserved range of IDs for GetSqlInfo is not entirely in use * Add comment to CommandGetTables to explain the encoding of table_schema * Remove redundat information on schemas * Fixed Javadoc on some methods, added Thread interrupt to executeUpdate methods, and updated Signal exceptions to CallStatus with description * Replace int32 with uint32 for GetSqlInfo name representation * Replace AssertionError with StatusRuntimeException for whenever attempting to unpack an invalid protobuf message * add comment to FlightSql.proto to update_rule and delete_rule * Replace inconsistent exception handling with CallStatus predetermined exceptions * correct comment to CreatePreparedStatement on FlightSql.proto * Remove unused dependencies * fix: change Status use to CallStatus on FlightSqlProducer * Changed from if not null check to Objects requireNonNull on Flight SQL Client * Remove Nullable annotation * Changed from checkNotNull to Objects#requireNotNull with description on Flight SQL Example * Add CallOptions to every RPC call by the client * Fix Maven dependency problems and checkstyle violations * Replace generic Collections with Lists when order matters in an RPC call * Fix Javadoc for FlightSqlClient * Add description to StatusRuntimeExceptions * Add descriptions to Exceptions * Correct update_rule and delete_rule description on FlighSql.proto * Verify wheter Root is empty before sending request to server * Add call options to PreparedStatement * Replace constant checking of whether client is open with #checkOpen * Add CallOptions to #close for PreparedStatement * Refactor PreparedStatement usages of CallOptions * Fix broken tests * Fix FlightSql.proto documentation * Update documentation for format/FlightSql.proto Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> * Fix checkstyle violations * Require non null tables for GetExportedKeys and GetImportedKeys * Not storing CallOptions in PreparedStatement * Update documentation comments for protobuf * Replace IntVector for UInt1Vector for delete_rule and update_rule * Fix protobuf for FlightSQL * Fix bug with empty metadata * Update update_rule and delete_rule documentation on proto * Remove explicit dependency on JDBC's DatabaseMetaData on UpdateDeleteRules * Use MessageOptions instead of FieldOptions on proto * Add missing JavaDoc about 'options' parameter * Fix CommandGetSqlInfo documentation * Add @throws to FlightSqlClient#checkOpen JavaDoc Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br> Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com> Co-authored-by: Rafael Telles <rafael@telles.dev> Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> * FlightSQL Ratification based on Community Comments (round 2) (apache#85) * Remove unused client_execution_handler from Protobuf * Update documentation on CommandGetPrimaryKeys * Update documentation on CommandGetImportedKeys and CommandGetExportedKeys * Change exception type on FlightSqlClient#executeUpdate * Add @return to FlightSqlClient#executeUpdate JavaDoc * Switch order of key_name and key_sequence on CommandGetTableKeys documentation * Update JavaDoc for FlIghtSqlClient#clearParameters * Add private constructor to FlightSqlProducer.SqlInfo * Update JavaDoc for FlIghtSqlClient#clearParameters * Fix wrong CommandGetPrimaryKeys documentation on Proto file * Fix order of key_name and key_sequence * Split CommandStatementQuery in 2 messages, one for Command and other for Ticket * Fix leaking Connections on FlightSqlExample * Remove unused variable of cache * Flight SQL Ratification Based On Community Feedback #5 (apache#91) * Delegate GetSchemaImportedKeys * Remove schema retrieval methods for catalog functions and delegate to constants * Add IPC encapsulation to Schema serialization * Fix checkstyle violations * Update javadoc for FlightSqlClient * Update documentation for FlightSql.proto Co-authored-by: Abner Eduardo Ferreira <abenaru@protonmail.ch> * Flight SQL Ratification Based On Community Feedback #6 (apache#94) * Refactored FlightSql Statement Constant names * Defined non-nullable parameters for FlightSql proto * Resolved minimal checkstyle issues * Added further documentation for catalog and schema * Refactored FlightSql proto comments to include more information * Added Field/FieldType notNullable methods * Refactored FlightSqlClient and FlightSqlExample to leverage Field notNullable method * Removed opaque query warning from FlightSql proto * Added the optional tag for the returned schema of getTables to proto * Flight SQL Ratification Based On Community Feedback #7 (apache#98) * Remove scope from 'hamcrest' dependency on java/pom.xml * Use flight top-level module on parent pom.xml instead of declaring each one * Avoid using getStatement inside StatementContext methods * Make StatementContext.getQuery() return String * Minor fixes on pom.xml * Move 'os-maven-plugin' to parent pom.xml * Update protobuf generation on pom.xml files * Use ClassLoader#getResource to get network.properties on TestFlightSql * Bind to any ephemeral port on TestFlightSql * Move JDBC-Arrow type default conversion from JdbcToArrowConfig to JdbcToArrowUtils * Micro-optimization: initialize ArrayList with the right size * Fix null-check on PreparedStatement#setParameters * Avoid wrapping vector into a ImmutableList and then into an ArrayList on FlightSqlExample#getTablesRoot * Remove null-check on VectorSchemaRoot on FlightSqlClient#setParameters() * Remove the need of separate cache for ResultSets * Add missing 'final' modifiers * Fix missing generated sources on built flight-sql jar (apache#101) * Bump protobuf version * Flight SQL Ratification Based On Community Feedback #8 (apache#113) * Change scope of arrow-memory-netty to test for flight-sql * Remove unused dependency arrow-memory-netty * Update common-pool2 and common-dbcp2 dependencies * Remove 'executions' from parent pom.xml for plugin protobuf-maven-plugin * Adjust protobuf-maven-plugin settings on pom.xml files * Move dep.protobuf.version and dep.grpc.version to top pom.xml * Remove <url> from arrow-flight's pom.xml * Fix maven build from different directories (apache#114) * Add test cases for bitshifting operations required for filtering out some SqlInfo data * Fix Schema serialization and deserialization on Flight SQL methods * [FlightSQL] Add missing method for creating bitmask from GetSqlInfo option enum (apache#148) * Add util method for creating bitmask from multiple protobuf enums for FlightSql GetSqlInfo enum * Add test cases for utility method for creating bitmask from protobuf options * Make changes regarding to reviews Co-authored-by: Rafael Telles <rafael@telles.dev> * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * Create a module for keeping all Arrow Flight-related submodules * Fix checkstyle violations in Arrow Flight JDBC Driver * WIP: Refactor SqlInfoProvider * WIP * WIP * WIP * Fix rebase issues * Restore FlightSql.proto state * Fix checkstyle * Fix rebase colateral * Restore FlightSql.proto Co-authored-by: Jose Almeida <almeidajcr90@gmail.com> Co-authored-by: Abner Eduardo Ferreira <abenaru@protonmail.ch> Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br> Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com> Co-authored-by: kylep-dremio <38920967+kylep-dremio@users.noreply.github.com> Co-authored-by: Vinicius Fraga <62815192+vfraga@users.noreply.github.com> Co-authored-by: Ryan Nicholson <rnicholson@dremio.com> Co-authored-by: Kyle Porter <kporter@dremio.com>

* Initial commit * init project * complete most of the annotations * fix FixedSizeBufferWriter init annotation * bump 10.0.1.2 * complete parquet core annotations * bump 10.0.1.3 * re-export modules * fix: add return type for foreign_buffer * fix output_stream and read_message annotations * ci: add release job * pre-commit specify flake8 version to 5.0.4 * flake8 ignore F821 for private files * optimize annotations * bump 10.0.1.4 * if param supports IOBase, it should also support NativeFile * bump 10.0.1.5 * pre-commit adds mypy lint * bump 10.0.1.6 * fix ci name * Remove version restrictions for Python. * release 10.0.1.7 * update poetry ci * Fix stubs for Table factory methods The main problem was that these were annotated as instance methods rather than static/class methods, but I've added some detail, too. * update pre-commit * update * fix: make fs.FileSystem.from_uri and hdfs.HadoopFileSystem.from_uri as classmethod * fix: fix read_metadata and read_schema wrong annotations (#11) * fix: typo S3FileSystem schema -> scheme (#12) * bump version 10.0.1.8 (#13) * . (#16) * make DataType hashable (#22) * pa.table support recordbatch (#20) * RecordBatchStreamReader supports next (#18) * add RecordBatch.to_pylist (#23) * precise return types for to_pandas (#25) * bump version 10.0.1.9 (#26) * [pre-commit.ci] pre-commit autoupdate (#27) * [pre-commit.ci] pre-commit autoupdate (#28) * Fix types in FlightDescriptor class (#29) * Fix types in FlightDescriptor class * Add argument types * chore: update pre-commit config (#30) * build: use `pixi` to manage project (#31) * chore: add taplo config (#32) * chore: update LICENSE date (#33) * doc: add CODE_OF_CONDUCT.md (#34) * [pre-commit.ci] pre-commit autoupdate (#38) * [pre-commit.ci] pre-commit autoupdate (#39) updates: - [github.com/astral-sh/ruff-pre-commit: v0.5.7 → v0.6.1](astral-sh/ruff-pre-commit@v0.5.7...v0.6.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (apache#48) updates: - [github.com/astral-sh/ruff-pre-commit: v0.6.1 → v0.6.2](astral-sh/ruff-pre-commit@v0.6.1...v0.6.2) - [github.com/pre-commit/mirrors-mypy: v1.11.1 → v1.11.2](pre-commit/mirrors-mypy@v1.11.1...v1.11.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * refactor: rewrite type annotations by hand. (#35) * chore: restart * update ruff config * build: add extra dependencies * update mypy config * feat: add util.pyi * feat: add types.pyi * feat: impl lib.pyi * update * feat: add acero.pyi * feat: add compute.pyi * add benchmark.pyi * add cffi * feat: add csv.pyi * disable isort single line * reformat * update compute.pyi * add _auzurefs.pyi * add _cuda.pyi * add _dataset.pyi * rename _stub_typing.pyi -> _stubs_typing.pyi * add _dataset_orc.pyi * add pyarrow-stubs/_dataset_parquet_encryption.pyi * add _dataset_parquet.pyi * add _feather.pyi * feat: add _flight.pyi * add _fs.pyi * add _gcsfs.pyi * add _hdfs.pyi * add _json.pyi * add _orc.pyi * add _parquet_encryption.pyi * add _parquet.pyi * update * add _parquet.pyi * add _s3fs.pyi * add _substrait.pyi * update * update * add parquet/core.pyi * add parquet/encryption.pyi * add BufferProtocol * impl _filesystemdataset_write * add dataset.pyi * add feather.pyi * add flight.pyi * add fs.pyi * add gandiva.pyi * add json.pyi * add orc.pyi * add pandas_compat.pyi * add substrait.pyi * update util.pyi * add interchange * add __lib_pxi * update __lib_pxi * update * update * add types.pyi * feat: add scalar.pyi * update types.pyi * update types.pyi * update scalar.pyi * update * update * update * update * update * update * feat: impl array * feat: add builder.pyi * add scipy * add tensor.pyi * feat: impl NativeFile * update io.pyi * complete io.pyi * add ipc.pyi * mv benchmark.pyi into __lib_pxi * add table.pyi * do re-export in lib.pyi * fix io.pyi * update * optimize scalar.pyi * optimize indices * complete ipc.pyi * update * fix NullableIterable * fix string array * ignore overload-overlap error * fix _Tabular.__getitem__ * remove additional_dependencies * remove check-mypy.sh (apache#49) * release 20240828 (apache#50) * fix release tag (apache#51) * ci: install hatch by pip (apache#52) * ci: fix hatch keyring (apache#53) * ci: use Release environment (apache#54) * remove Scalar generic type var _IsValid (apache#56) * remove Scalar generic type var _IsValid * make Array, Scalar, Types generic type var as covariant type (apache#57) * remove Field generic type var _Nullable (apache#58) * remove Field generic type var _Nullable * fix: pa.dictionary and pa.schema annotation (apache#59) * fix pa.dictionary annotation * fix: schema annotation * release new version (apache#60) * [pre-commit.ci] pre-commit autoupdate (apache#62) * release: 2024.9.3 (apache#63) use new date release format %Y.%m.%d * support pyarrow compute funcs (apache#61) * update compute.pyi * impl Aggregation funcs * impl arithmetic * imit bit-wise functions * imit rounding functions * optimize annotation * impl logarithmic functions * update * impl comparisons funcs * impl logical funcs * impl string predicates and transforms * impl string padding * impl string trimming * impl string splitting and component extraction * impl string joining and slicing * impl Containment tests * impl Categorizations * impl Structural transforms * impl Conversions * impl Temporal component extraction * impl random, Timezone handling * impl Array-wise functions * fix timestamp scalar * support build array with list of scalar (apache#64) * release 2024.9.4 (apache#65) * Version follows the version of pyarrow (apache#66) * import parquet.core into parquet __init__.py (apache#67) Update __init__.pyi * release 17.1 (apache#69) * fix: add missing submodule benchmark, csv and cuda (apache#71) * release 17.2 (apache#72) * fix: from_pylist covariance (apache#73) * [pre-commit.ci] pre-commit autoupdate (apache#74) * Fix return type for middleware factory's start_call (apache#75) It can return None if middleware is not needed for a given call. * release 17.3 (apache#76) * fix: add missing return type in FlightDescriptor static methods (apache#80) * Support Tabular filter with Expression (apache#81) support Tabular filter with Expression * Support compute functions to accept Expression as parameter (apache#82) * fix: Fix the return value of Expression comparison (apache#83) * release 17.4 (apache#84) * fix: fix the array return type (apache#89) * a few type improvements, mostly flight related (apache#90) * FlightError.extra_info -> bytes * annotate FlightStreamReader.cancel return * BasicAuth serialize/deserialize * RecordBatchFileReader.schema * actually str | bytes * add_type_to_Field (apache#87) * add_type_to_Field * Field.type should return the covariant DataType --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * Support fsspec.AbstractFileSystem (apache#88) * supported_filesystem * fixes * remove unused import --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * release 17.5 (apache#91) * [pre-commit.ci] pre-commit autoupdate (apache#95) * fix: parquet not accepting NativeFile (apache#98) * feat: support pa.Buffer buffer protocol (apache#99) * feat: Support `compute` functions to accept ChunkedArray. (apache#100) * release 17.6 (apache#101) * [pre-commit.ci] pre-commit autoupdate (apache#102) * working towards making return signatures only have one type (mean and exp) (apache#105) * group_by_returns_TableGroupBy * return_single_type_for_mean_exp * revert table.pyi * compute.mean does not support BinaryScalar or BinaryArray --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * a table group_by was returing Self but should return TableGroupBy (apache#104) group_by_returns_TableGroupBy * [pre-commit.ci] pre-commit autoupdate (apache#106) updates: - [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](pre-commit/pre-commit-hooks@v4.6.0...v5.0.0) - [github.com/astral-sh/ruff-pre-commit: v0.6.7 → v0.6.9](astral-sh/ruff-pre-commit@v0.6.7...v0.6.9) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: RecordBatch missing `from_arrays` and `from_pandas` (apache#108) * release 17.7 (apache#109) * fix_combine_chunks (apache#110) * make Self backward compatible (apache#115) * fix: update ConvertOptions (apache#114) * add type property to Array (apache#112) * add type property to Array * Array.type should return covariant --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * release 17.8 (apache#117) * Add include_columns parameter in ConvertOptions (apache#118) * add list[str] overload to rename_columns (apache#119) * release 17.9 (apache#120) * [pre-commit.ci] pre-commit autoupdate (apache#124) updates: - [github.com/astral-sh/ruff-pre-commit: v0.6.9 → v0.7.0](astral-sh/ruff-pre-commit@v0.6.9...v0.7.0) - [github.com/pre-commit/mirrors-mypy: v1.11.2 → v1.12.1](pre-commit/mirrors-mypy@v1.11.2...v1.12.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * improve type annotations for parquet writer (apache#125) Add support for per-field compression specification Add missing none compression value. * Add missing return type for Schema.serialize (apache#123) * Add `Schema.field(int)` (apache#122) * Change various io related functions to support `StrPath` as a path input (apache#121) * Change various io related functions to support StrPath as a path input * fmt * Added StrPath | IO for feather types * fix type hint for sort_by (apache#130) sort_by takes str or list[tuple(name, order)] as its argument where str is a field name not a sort order * metadata on a schema can be passed as str (apache#128) For details see https://github.com/apache/arrow/blob/apache-arrow-17.0.0/python/pyarrow/types.pxi\#L2053-L2056 * Correct typevars for DictionaryType, MapType, RunEncodedType (apache#126) Correct type hints for Dictionary, RunEndEncoded and Map Signed-off-by: Jonas Dedden <university@jonas-dedden.de> Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * Add some more StrPath io parts that were overlooked. (apache#131) * Add some more StrPath io parts that were overlooked. Additionally, add the utility typealias `SingleOrList` that can be used in places where we want a concise type declaration but the there is a large union of types. * write_dataset(base_dir = ) can also take Path * Support ChunkedArray in add/append methods in Table (apache#129) * Add missing partitioning typing case (apache#132) This should now support the examples in the docstring for partitioning. * fix: typo 'permissive' instead of 'premissive' (apache#133) * release 17.10 (apache#134) * fix incorrect type hints for compute.sort_indices (apache#135) * disallow passing `names` as an argument to table when using dictionaries (apache#137) * [pre-commit.ci] pre-commit autoupdate (apache#138) updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.0 → v0.7.1](astral-sh/ruff-pre-commit@v0.7.0...v0.7.1) - [github.com/pre-commit/mirrors-mypy: v1.12.1 → v1.13.0](pre-commit/mirrors-mypy@v1.12.1...v1.13.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add missing type for FlightEndpoint (apache#136) * release 17.11 (apache#139) * [pre-commit.ci] pre-commit autoupdate (apache#140) updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.1 → v0.7.2](astral-sh/ruff-pre-commit@v0.7.1...v0.7.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (apache#142) updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.2 → v0.7.3](astral-sh/ruff-pre-commit@v0.7.2...v0.7.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * chore: Create FUNDING.yml (apache#143) Create FUNDING.yml * fix: `read_schema` should return Schema (apache#145) fix: read_schema should return Schema * release 17.12 (apache#146) * [pre-commit.ci] pre-commit autoupdate (apache#147) updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.3 → v0.7.4](astral-sh/ruff-pre-commit@v0.7.3...v0.7.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: `to_table` argument `columns` can be a dict of expressions (apache#149) * [pre-commit.ci] pre-commit autoupdate (apache#148) * [pre-commit.ci] pre-commit autoupdate updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.4 → v0.8.1](astral-sh/ruff-pre-commit@v0.7.4...v0.8.1) * ruff: ignore PYI063 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * release 17.13 (apache#151) * fix: FileSystem metadata value should be str (apache#152) * fix: FileSystemHandler metadata value should be str (apache#153) * [pre-commit.ci] pre-commit autoupdate (apache#154) updates: - [github.com/astral-sh/ruff-pre-commit: v0.8.1 → v0.8.2](astral-sh/ruff-pre-commit@v0.8.1...v0.8.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * improve coverage for pyarrow.struct typehint (apache#157) * fix: ipc typing (apache#159) * release 17.14 (apache#160) * fix: add missing param 'nbytes' to NativeFile.read (apache#163) * release 17.15 (apache#164) * [pre-commit.ci] pre-commit autoupdate (apache#161) updates: - [github.com/astral-sh/ruff-pre-commit: v0.8.2 → v0.8.3](astral-sh/ruff-pre-commit@v0.8.2...v0.8.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add 'None' as a valid argument for partitioning to the various parquet reading functions (apache#166) * [pre-commit.ci] pre-commit autoupdate (apache#165) updates: - [github.com/astral-sh/ruff-pre-commit: v0.8.3 → v0.8.6](astral-sh/ruff-pre-commit@v0.8.3...v0.8.6) - [github.com/pre-commit/mirrors-mypy: v1.13.0 → v1.14.1](pre-commit/mirrors-mypy@v1.13.0...v1.14.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: should use Collection[Array] instead list[Array] (apache#170) "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance Consider using "Sequence" instead, which is covariant * fix: update type hints for path_or_paths and source parameters in ParquetDataset and read_table (apache#171) * [pre-commit.ci] pre-commit autoupdate (apache#167) updates: - [github.com/astral-sh/ruff-pre-commit: v0.8.6 → v0.9.1](astral-sh/ruff-pre-commit@v0.8.6...v0.9.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * release 17.16 (apache#172) * Fixed pa.fixed_shape_tensor (apache#175) * [pre-commit.ci] pre-commit autoupdate (apache#173) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.1 → v0.9.4](astral-sh/ruff-pre-commit@v0.9.1...v0.9.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: Preserve generic in `ChunkedArray.type` (apache#177) * release 17.17 (apache#178) * [pre-commit.ci] pre-commit autoupdate (apache#176) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.4 → v0.9.6](astral-sh/ruff-pre-commit@v0.9.4...v0.9.6) - [github.com/pre-commit/mirrors-mypy: v1.14.1 → v1.15.0](pre-commit/mirrors-mypy@v1.14.1...v1.15.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: support to construct ListArray with primitive type (apache#179) * fix: Avoid `chunked_array` overlapping overloads (apache#183) * fix: Add placeholder annotations to `pc.if_else` (apache#182) * fix: Widen `Array` to `Array | ChunkedArray` (apache#181) * fix: add `pc.fill_null` (apache#185) - https://arrow.apache.org/docs/python/generated/pyarrow.compute.fill_null.html - https://github.com/narwhals-dev/narwhals/blob/05e47b27ebe27b24196cee5956d07748d65a62ee/narwhals/_arrow/series.py#L675 * fix: Allow Table.from_arrays to take a list containing a mix of Array and ChunkedArray (apache#187) Update table.pyi * release 17.18 (apache#188) * [pre-commit.ci] pre-commit autoupdate (apache#180) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.6 → v0.9.10](astral-sh/ruff-pre-commit@v0.9.6...v0.9.10) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: from_arrays for both Table and RecordBatch (apache#189) * fix: resolve some `pa.compute` overlaps (apache#184) * fix: resolve overlapping `compute.(add|divide)` * fix: copy from non-cloned signature * fix: resolve overlapping `compute.exp` * fix: resolve overlapping `compute.power` * fix: resolve overlapping `compute.equal` * fix: resolve overlapping `compute.and_` * fix: Include `Array` in `chunked_array` overload (apache#190) narwhals-dev/narwhals@0237f7a * release 17.19 (apache#191) * Add Scalar, Array and Type classes for Json & Uuid (apache#194) * Add Scalar, Array and Type classes for Json & Uuid * Formatting fixes * [pre-commit.ci] pre-commit autoupdate (apache#192) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.10 → v0.11.2](astral-sh/ruff-pre-commit@v0.9.10...v0.11.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Revert "Add Scalar, Array and Type classes for Json & Uuid" (apache#195) Revert "Add Scalar, Array and Type classes for Json & Uuid (apache#194)" This reverts commit 8f77909. * fix: Add missing `pc.equal` overload (apache#196) * feat: support pyarrow 19.0 (apache#198) * build: upgrade pyarrow min version to 19.0 * feat: support pyarrow 19.0 * omit mypy bool8 override error * fix: reexport new types (apache#199) * feat: override new patterns for func repeat and nulls (apache#200) * fix: reexport decimal64 array and decimal128 array * feat: override new patterns for func `repeat` and `nulls` * release: 19.1 (apache#201) * fix: Allow `Iterable[Table]` in `concat_tables` (apache#203) https://arrow.apache.org/docs/python/generated/pyarrow.concat_tables.html > tables : iterable of pyarrow.Table objects * fix: Allow `ChunkedArray[BooleanScalar]` in `pc.invert` (apache#204) Fixes https://github.com/narwhals-dev/narwhals/blob/caabc0efdef54f117c83888926860e3972ef69d5/narwhals/_arrow/series.py#L298-L299 * feat: Fully spec `TableGroupBy.aggregate` (apache#197) ## Related - https://arrow.apache.org/docs/python/compute.html#grouped-aggregations - https://arrow.apache.org/docs/python/generated/pyarrow.TableGroupBy.html#pyarrow.TableGroupBy.aggregate - https://github.com/apache/arrow/blob/34a984c842db42b409a1359e6e2cf167a2365a48/python/pyarrow/table.pxi#L6578-L6604 * fix: Add missing return type to `ChunkedArray.filter` (apache#205) * fix: Add relaxed final overload to logical functions (apache#206) Covers all of `pc.(and_ | and_kleene | and_not | and_not_kleene | or_ | or_kleene | xor)` Resolves: - https://github.com/narwhals-dev/narwhals/blob/caabc0efdef54f117c83888926860e3972ef69d5/narwhals/_arrow/series.py#L219-L233 - https://github.com/narwhals-dev/narwhals/blob/caabc0efdef54f117c83888926860e3972ef69d5/narwhals/_arrow/series.py#L662 * fix: Allow `ChunkedArray` in `Table.set_column` (apache#211) Also being more consistent with `ArrayOrChunkedArray[Any]` everywhere Discovered in - https://github.com/vega/vega-datasets/blob/343b7101391a81190ba24e1e8d62a381d2fef3bd/scripts/species.py#L798-L799 * chore: Ignore `fsspec` `[import-untyped]` (apache#210) ```py _fs.pyi:18: error: Skipping analyzing "fsspec": module is installed, but missing library stubs or py.typed marker [import-untyped] _fs.pyi:18: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports Found 1 error in 1 file (checked 64 source files) ``` - fsspec/filesystem_spec#625 - fsspec/filesystem_spec#1676 * feat: Convert `types.is_*` into `TypeIs` guards (apache#215) * chore: Add `types.__all__` * feat: Convert `types._is_*` into `TypeIs` guards I've been using this for a little while, but makes more sense to live in the stubs https://github.com/narwhals-dev/narwhals/blob/16427440e6d74939c403083b52ce3fb0af7d63c7/narwhals/_arrow/utils.py#L44-L67 * fix: Resolve `bit_wise_and` overlaps (apache#214) Fixes 3 errors: ```py compute.pyi:608:5 - error: Overload 1 for "bit_wise_and" overlaps overload 4 and returns an incompatible type (reportOverlappingOverload) compute.pyi:608:5 - error: Overload 1 for "bit_wise_and" overlaps overload 5 and returns an incompatible type (reportOverlappingOverload) compute.pyi:620:5 - error: Overload 3 for "bit_wise_and" will never be used because its parameters overlap overload 1 (reportOverlappingOverload) ``` * fix: Resolve `list_*` overlapping overloads (apache#213) * fix: Resolve `list_value_length` overlaps * fix: Resolve `list_element` overlaps * fix: Resolve `list_(flatten|slice|parent_indices)` overlaps An improvement, but still not that accurate * fix: Include `VarianceOptions` in `TableGroupBy.aggregate` (apache#212) - Follow-up to apache#197 - Noticed while writing up (narwhals-dev/narwhals#2385) - We already use it for `std`, `var` in https://github.com/narwhals-dev/narwhals/blob/16427440e6d74939c403083b52ce3fb0af7d63c7/narwhals/_arrow/group_by.py#L81-L82 * [pre-commit.ci] pre-commit autoupdate (apache#202) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.2 → v0.11.5](astral-sh/ruff-pre-commit@v0.11.2...v0.11.5) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: Resolve `Scalar.as_py` warnings for `DictionaryType` (apache#207) > scalar.pyi:75:20 - warning: TypeVar "_AsPyTypeK" appears only once in generic function signature > Use "object" instead (reportInvalidTypeVarUse) > scalar.pyi:85:20 - warning: TypeVar "_AsPyTypeK" appears only once in generic function signature > Use "object" instead (reportInvalidTypeVarUse) Instead just using `int`, which should be all that is possible from: https://github.com/zen-xu/pyarrow-stubs/blob/02552b81161d19d4aa71d8656b028eefac84612b/pyarrow-stubs/__lib_pxi/types.pyi#L154-L164 https://github.com/zen-xu/pyarrow-stubs/blob/02552b81161d19d4aa71d8656b028eefac84612b/pyarrow-stubs/__lib_pxi/types.pyi#L63-L70 * fix: Add default to `pc.sort_indices` (apache#216) * fix: Add default to `pc.sort_indices` Fixes narwhals-dev/narwhals#2390 (comment) Default is specified in https://arrow.apache.org/docs/python/generated/pyarrow.compute.sort_indices.html * refactor: Reuse some aliases * fix: Allow `list_size` with `Field` in `pa.list_` (apache#218) Closes apache#217 * allow `Table` or `RecordBatch` for dataset (apache#222) allow source argument pyarrow.dataset.dataset() to be RecordBatch | Table * refactor: Simplify `types` overloads (apache#219) * fix: `binary` overlap * fix: Simplify list constructors, `_Ordered` * refactor: Use `_Tz` default * fix: iter ChunkedArray should return scalar value (apache#224) * release: 19.2 (apache#225) * fix: Add missing `DictionaryArray` methods/properties (apache#226) ## Docs - https://arrow.apache.org/docs/python/generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray.dictionary - https://arrow.apache.org/docs/python/generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray.indices - https://arrow.apache.org/docs/python/generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray.dictionary_decode - https://arrow.apache.org/docs/python/generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray.dictionary_encode ## Fixes - https://github.com/narwhals-dev/narwhals/blob/c23e56c56630761f0fbc58b575a1c987e57d58d5/narwhals/_arrow/series.py#L787-L798 - https://github.com/narwhals-dev/narwhals/blob/c23e56c56630761f0fbc58b575a1c987e57d58d5/narwhals/_arrow/series_cat.py#L14-L18 * chore: use pyright as static type checker (apache#227) * use pyright as static type checker * make pyright happy * fix: fix pyright action (apache#229) fix github ci * fix: Match runtime behavior of `(Table|RecordBatch).select` (apache#221) * fix: Match runtime behavior of `(Table|RecordBatch).select` ## Resolves - https://github.com/MarcoGorelli/narwhals/blob/5b02b592183b8d39e2d32e0aedd6c234bb22d405/narwhals/_arrow/dataframe.py#L305-L307 - https://github.com/MarcoGorelli/narwhals/blob/5b02b592183b8d39e2d32e0aedd6c234bb22d405/narwhals/_arrow/dataframe.py#L285-L294 ##Description Following up on what I thought was a simple stub issue, but we're both *too strict* and *too permissive* in different ways ##Examples {placeholder} ##Related - https://github.com/apache/arrow/blob/d2ddee62329eb711572b4d71d6380673d7f7edd1/python/pyarrow/table.pxi#L4367-L4374 - https://github.com/apache/arrow/blob/d2ddee62329eb711572b4d71d6380673d7f7edd1/python/pyarrow/table.pxi#L1721-L1739 * update select * update select --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * [pre-commit.ci] pre-commit autoupdate (apache#220) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.5 → v0.11.8](astral-sh/ruff-pre-commit@v0.11.5...v0.11.8) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * feat: narrow scalar when type is given (apache#230) * rename Uint -> UInt * feat: narrow scalar when type is given * release 19.3 (apache#231) * chore: pyright use strict mode (apache#233) * fix types * update array.pyi * update scalar.pyi * update * update array * update array * optimize chunked_array * optimizer iterchunks * update * update pyproject.toml * fix: pa.nulls accept type rather than types (apache#234) * [pre-commit.ci] pre-commit autoupdate (apache#232) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.8 → v0.11.9](astral-sh/ruff-pre-commit@v0.11.8...v0.11.9) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * release 19.4 (apache#235) * lint(pyright): disable reportUnknownMemberType (apache#239) * [pre-commit.ci] pre-commit autoupdate (apache#236) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.9 → v0.11.13](astral-sh/ruff-pre-commit@v0.11.9...v0.11.13) - [github.com/RobertCraigie/pyright-python: v1.1.400 → v1.1.401](RobertCraigie/pyright-python@v1.1.400...v1.1.401) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * feat: support pyarrow 20.0 (apache#240) * [pre-commit.ci] pre-commit autoupdate (apache#241) updates: - [github.com/RobertCraigie/pyright-python: v1.1.401 → v1.1.402](RobertCraigie/pyright-python@v1.1.401...v1.1.402) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * support docstring (apache#242) * doc: complete tensor doc * doc: complete table doc * doc: complete scalar doc * doc: complete orc doc * doc: complete memory doc * doc: complete lib doc * doc: complete json doc * doc: complete hdfs doc * doc: complete gcsfs doc * doc: complete fs doc * doc: complete flight doc * doc: complete dataset doc * doc: complete dataset parquet doc * doc: complete dataset parquet encryption doc * doc: complete cuda doc * doc: complete csv doc * doc: complete azurefs doc * doc: complete core doc * doc: complete interchange doc * doc: complete array doc * doc: complete builder doc * doc: complete device doc * doc: complete io doc * doc: complete ipc doc * doc: complete types doc * mark deprecated apis * doc: complete _compute doc * doc: complete compute doc * doc: update compute doc * lint code * release 20.0.0.20250618 (apache#243) * fix: make ParquetFileFormat constructor args optional (apache#244) * fix: Field.remove_metadata should return Self (apache#246) * [pre-commit.ci] pre-commit autoupdate (apache#245) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.13 → v0.12.0](astral-sh/ruff-pre-commit@v0.11.13...v0.12.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * release 20.0.0.20250627 (apache#247) * fix: chunked_array with type should be specified (apache#250) * [pre-commit.ci] pre-commit autoupdate (apache#248) updates: - [github.com/astral-sh/ruff-pre-commit: v0.12.0 → v0.12.3](astral-sh/ruff-pre-commit@v0.12.0...v0.12.3) - [github.com/RobertCraigie/pyright-python: v1.1.402 → v1.1.403](RobertCraigie/pyright-python@v1.1.402...v1.1.403) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * release 20.0.0.20250715 (apache#251) * fix: The type parameter of array should be covariant (apache#253) * release 20.0.0.20250716 (apache#254) * Add py.typed file to signify that the library is typed See the relevant PEP https://peps.python.org/pep-0561 * Prepare `pyarrow-stubs` for history merging MINOR: [Python] Prepare `pyarrow-stubs` for history merging Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * Add `ty` configuration and suppress error codes * One line per rule * Add licence header from original repo for all `.pyi` files * Revert "Add licence header from original repo for all `.pyi` files" This reverts commit 1631f39. * Prepare for licence merging * Exclude `stubs` from `rat` test * Add Apache licence clause to `py.typed` * Reduce list * Resolve merge conflict --------- Signed-off-by: Jonas Dedden <university@jonas-dedden.de> Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> Co-authored-by: Jim Bosch <talljimbo@gmail.com> Co-authored-by: Oliver Mannion <125105+tekumara@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eugene Toder <eltoder@users.noreply.github.com> Co-authored-by: fvankrieken <fvankrieken@planning.nyc.gov> Co-authored-by: Ilia Ablamonov <ilia@flamefork.ru> Co-authored-by: Mathias Beguin <mathias.beguin@hotmail.com> Co-authored-by: Dylan Scott <dylan.scott@gmail.com> Co-authored-by: deanm0000 <37878412+deanm0000@users.noreply.github.com> Co-authored-by: Jan Moravec <moravecj@post.cz> Co-authored-by: Marius van Niekerk <marius.v.niekerk@gmail.com> Co-authored-by: Jonas Dedden <university@jonas-dedden.de> Co-authored-by: Fábio D. Batista <fabio@atelie.dev.br> Co-authored-by: ben-freist <93315290+ben-freist@users.noreply.github.com> Co-authored-by: Jiahao Yuan <kahojyun@icloud.com> Co-authored-by: Pim de Haan <pimdehaan@gmail.com> Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com> Co-authored-by: Tom Crasset <25140344+tcrasset@users.noreply.github.com> Co-authored-by: Tom McTiernan <tmct@users.noreply.github.com> Co-authored-by: Rok Mihevc <rok@mihevc.org>

* Initial commit * init project * complete most of the annotations * fix FixedSizeBufferWriter init annotation * bump 10.0.1.2 * complete parquet core annotations * bump 10.0.1.3 * re-export modules * fix: add return type for foreign_buffer * fix output_stream and read_message annotations * ci: add release job * pre-commit specify flake8 version to 5.0.4 * flake8 ignore F821 for private files * optimize annotations * bump 10.0.1.4 * if param supports IOBase, it should also support NativeFile * bump 10.0.1.5 * pre-commit adds mypy lint * bump 10.0.1.6 * fix ci name * Remove version restrictions for Python. * release 10.0.1.7 * update poetry ci * Fix stubs for Table factory methods The main problem was that these were annotated as instance methods rather than static/class methods, but I've added some detail, too. * update pre-commit * update * fix: make fs.FileSystem.from_uri and hdfs.HadoopFileSystem.from_uri as classmethod * fix: fix read_metadata and read_schema wrong annotations (#11) * fix: typo S3FileSystem schema -> scheme (#12) * bump version 10.0.1.8 (#13) * . (#16) * make DataType hashable (#22) * pa.table support recordbatch (#20) * RecordBatchStreamReader supports next (#18) * add RecordBatch.to_pylist (#23) * precise return types for to_pandas (#25) * bump version 10.0.1.9 (#26) * [pre-commit.ci] pre-commit autoupdate (#27) * [pre-commit.ci] pre-commit autoupdate (#28) * Fix types in FlightDescriptor class (#29) * Fix types in FlightDescriptor class * Add argument types * chore: update pre-commit config (#30) * build: use `pixi` to manage project (#31) * chore: add taplo config (#32) * chore: update LICENSE date (#33) * doc: add CODE_OF_CONDUCT.md (#34) * [pre-commit.ci] pre-commit autoupdate (#38) * [pre-commit.ci] pre-commit autoupdate (#39) updates: - [github.com/astral-sh/ruff-pre-commit: v0.5.7 → v0.6.1](astral-sh/ruff-pre-commit@v0.5.7...v0.6.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (apache#48) updates: - [github.com/astral-sh/ruff-pre-commit: v0.6.1 → v0.6.2](astral-sh/ruff-pre-commit@v0.6.1...v0.6.2) - [github.com/pre-commit/mirrors-mypy: v1.11.1 → v1.11.2](pre-commit/mirrors-mypy@v1.11.1...v1.11.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * refactor: rewrite type annotations by hand. (#35) * chore: restart * update ruff config * build: add extra dependencies * update mypy config * feat: add util.pyi * feat: add types.pyi * feat: impl lib.pyi * update * feat: add acero.pyi * feat: add compute.pyi * add benchmark.pyi * add cffi * feat: add csv.pyi * disable isort single line * reformat * update compute.pyi * add _auzurefs.pyi * add _cuda.pyi * add _dataset.pyi * rename _stub_typing.pyi -> _stubs_typing.pyi * add _dataset_orc.pyi * add pyarrow-stubs/_dataset_parquet_encryption.pyi * add _dataset_parquet.pyi * add _feather.pyi * feat: add _flight.pyi * add _fs.pyi * add _gcsfs.pyi * add _hdfs.pyi * add _json.pyi * add _orc.pyi * add _parquet_encryption.pyi * add _parquet.pyi * update * add _parquet.pyi * add _s3fs.pyi * add _substrait.pyi * update * update * add parquet/core.pyi * add parquet/encryption.pyi * add BufferProtocol * impl _filesystemdataset_write * add dataset.pyi * add feather.pyi * add flight.pyi * add fs.pyi * add gandiva.pyi * add json.pyi * add orc.pyi * add pandas_compat.pyi * add substrait.pyi * update util.pyi * add interchange * add __lib_pxi * update __lib_pxi * update * update * add types.pyi * feat: add scalar.pyi * update types.pyi * update types.pyi * update scalar.pyi * update * update * update * update * update * update * feat: impl array * feat: add builder.pyi * add scipy * add tensor.pyi * feat: impl NativeFile * update io.pyi * complete io.pyi * add ipc.pyi * mv benchmark.pyi into __lib_pxi * add table.pyi * do re-export in lib.pyi * fix io.pyi * update * optimize scalar.pyi * optimize indices * complete ipc.pyi * update * fix NullableIterable * fix string array * ignore overload-overlap error * fix _Tabular.__getitem__ * remove additional_dependencies * remove check-mypy.sh (apache#49) * release 20240828 (apache#50) * fix release tag (apache#51) * ci: install hatch by pip (apache#52) * ci: fix hatch keyring (apache#53) * ci: use Release environment (apache#54) * remove Scalar generic type var _IsValid (apache#56) * remove Scalar generic type var _IsValid * make Array, Scalar, Types generic type var as covariant type (apache#57) * remove Field generic type var _Nullable (apache#58) * remove Field generic type var _Nullable * fix: pa.dictionary and pa.schema annotation (apache#59) * fix pa.dictionary annotation * fix: schema annotation * release new version (apache#60) * [pre-commit.ci] pre-commit autoupdate (apache#62) * release: 2024.9.3 (apache#63) use new date release format %Y.%m.%d * support pyarrow compute funcs (apache#61) * update compute.pyi * impl Aggregation funcs * impl arithmetic * imit bit-wise functions * imit rounding functions * optimize annotation * impl logarithmic functions * update * impl comparisons funcs * impl logical funcs * impl string predicates and transforms * impl string padding * impl string trimming * impl string splitting and component extraction * impl string joining and slicing * impl Containment tests * impl Categorizations * impl Structural transforms * impl Conversions * impl Temporal component extraction * impl random, Timezone handling * impl Array-wise functions * fix timestamp scalar * support build array with list of scalar (apache#64) * release 2024.9.4 (apache#65) * Version follows the version of pyarrow (apache#66) * import parquet.core into parquet __init__.py (apache#67) Update __init__.pyi * release 17.1 (apache#69) * fix: add missing submodule benchmark, csv and cuda (apache#71) * release 17.2 (apache#72) * fix: from_pylist covariance (apache#73) * [pre-commit.ci] pre-commit autoupdate (apache#74) * Fix return type for middleware factory's start_call (apache#75) It can return None if middleware is not needed for a given call. * release 17.3 (apache#76) * fix: add missing return type in FlightDescriptor static methods (apache#80) * Support Tabular filter with Expression (apache#81) support Tabular filter with Expression * Support compute functions to accept Expression as parameter (apache#82) * fix: Fix the return value of Expression comparison (apache#83) * release 17.4 (apache#84) * fix: fix the array return type (apache#89) * a few type improvements, mostly flight related (apache#90) * FlightError.extra_info -> bytes * annotate FlightStreamReader.cancel return * BasicAuth serialize/deserialize * RecordBatchFileReader.schema * actually str | bytes * add_type_to_Field (apache#87) * add_type_to_Field * Field.type should return the covariant DataType --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * Support fsspec.AbstractFileSystem (apache#88) * supported_filesystem * fixes * remove unused import --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * release 17.5 (apache#91) * [pre-commit.ci] pre-commit autoupdate (apache#95) * fix: parquet not accepting NativeFile (apache#98) * feat: support pa.Buffer buffer protocol (apache#99) * feat: Support `compute` functions to accept ChunkedArray. (apache#100) * release 17.6 (apache#101) * [pre-commit.ci] pre-commit autoupdate (apache#102) * working towards making return signatures only have one type (mean and exp) (apache#105) * group_by_returns_TableGroupBy * return_single_type_for_mean_exp * revert table.pyi * compute.mean does not support BinaryScalar or BinaryArray --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * a table group_by was returing Self but should return TableGroupBy (apache#104) group_by_returns_TableGroupBy * [pre-commit.ci] pre-commit autoupdate (apache#106) updates: - [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](pre-commit/pre-commit-hooks@v4.6.0...v5.0.0) - [github.com/astral-sh/ruff-pre-commit: v0.6.7 → v0.6.9](astral-sh/ruff-pre-commit@v0.6.7...v0.6.9) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: RecordBatch missing `from_arrays` and `from_pandas` (apache#108) * release 17.7 (apache#109) * fix_combine_chunks (apache#110) * make Self backward compatible (apache#115) * fix: update ConvertOptions (apache#114) * add type property to Array (apache#112) * add type property to Array * Array.type should return covariant --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * release 17.8 (apache#117) * Add include_columns parameter in ConvertOptions (apache#118) * add list[str] overload to rename_columns (apache#119) * release 17.9 (apache#120) * [pre-commit.ci] pre-commit autoupdate (apache#124) updates: - [github.com/astral-sh/ruff-pre-commit: v0.6.9 → v0.7.0](astral-sh/ruff-pre-commit@v0.6.9...v0.7.0) - [github.com/pre-commit/mirrors-mypy: v1.11.2 → v1.12.1](pre-commit/mirrors-mypy@v1.11.2...v1.12.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * improve type annotations for parquet writer (apache#125) Add support for per-field compression specification Add missing none compression value. * Add missing return type for Schema.serialize (apache#123) * Add `Schema.field(int)` (apache#122) * Change various io related functions to support `StrPath` as a path input (apache#121) * Change various io related functions to support StrPath as a path input * fmt * Added StrPath | IO for feather types * fix type hint for sort_by (apache#130) sort_by takes str or list[tuple(name, order)] as its argument where str is a field name not a sort order * metadata on a schema can be passed as str (apache#128) For details see https://github.com/apache/arrow/blob/apache-arrow-17.0.0/python/pyarrow/types.pxi\#L2053-L2056 * Correct typevars for DictionaryType, MapType, RunEncodedType (apache#126) Correct type hints for Dictionary, RunEndEncoded and Map Signed-off-by: Jonas Dedden <university@jonas-dedden.de> Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * Add some more StrPath io parts that were overlooked. (apache#131) * Add some more StrPath io parts that were overlooked. Additionally, add the utility typealias `SingleOrList` that can be used in places where we want a concise type declaration but the there is a large union of types. * write_dataset(base_dir = ) can also take Path * Support ChunkedArray in add/append methods in Table (apache#129) * Add missing partitioning typing case (apache#132) This should now support the examples in the docstring for partitioning. * fix: typo 'permissive' instead of 'premissive' (apache#133) * release 17.10 (apache#134) * fix incorrect type hints for compute.sort_indices (apache#135) * disallow passing `names` as an argument to table when using dictionaries (apache#137) * [pre-commit.ci] pre-commit autoupdate (apache#138) updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.0 → v0.7.1](astral-sh/ruff-pre-commit@v0.7.0...v0.7.1) - [github.com/pre-commit/mirrors-mypy: v1.12.1 → v1.13.0](pre-commit/mirrors-mypy@v1.12.1...v1.13.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add missing type for FlightEndpoint (apache#136) * release 17.11 (apache#139) * [pre-commit.ci] pre-commit autoupdate (apache#140) updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.1 → v0.7.2](astral-sh/ruff-pre-commit@v0.7.1...v0.7.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * [pre-commit.ci] pre-commit autoupdate (apache#142) updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.2 → v0.7.3](astral-sh/ruff-pre-commit@v0.7.2...v0.7.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * chore: Create FUNDING.yml (apache#143) Create FUNDING.yml * fix: `read_schema` should return Schema (apache#145) fix: read_schema should return Schema * release 17.12 (apache#146) * [pre-commit.ci] pre-commit autoupdate (apache#147) updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.3 → v0.7.4](astral-sh/ruff-pre-commit@v0.7.3...v0.7.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: `to_table` argument `columns` can be a dict of expressions (apache#149) * [pre-commit.ci] pre-commit autoupdate (apache#148) * [pre-commit.ci] pre-commit autoupdate updates: - [github.com/astral-sh/ruff-pre-commit: v0.7.4 → v0.8.1](astral-sh/ruff-pre-commit@v0.7.4...v0.8.1) * ruff: ignore PYI063 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * release 17.13 (apache#151) * fix: FileSystem metadata value should be str (apache#152) * fix: FileSystemHandler metadata value should be str (apache#153) * [pre-commit.ci] pre-commit autoupdate (apache#154) updates: - [github.com/astral-sh/ruff-pre-commit: v0.8.1 → v0.8.2](astral-sh/ruff-pre-commit@v0.8.1...v0.8.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * improve coverage for pyarrow.struct typehint (apache#157) * fix: ipc typing (apache#159) * release 17.14 (apache#160) * fix: add missing param 'nbytes' to NativeFile.read (apache#163) * release 17.15 (apache#164) * [pre-commit.ci] pre-commit autoupdate (apache#161) updates: - [github.com/astral-sh/ruff-pre-commit: v0.8.2 → v0.8.3](astral-sh/ruff-pre-commit@v0.8.2...v0.8.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add 'None' as a valid argument for partitioning to the various parquet reading functions (apache#166) * [pre-commit.ci] pre-commit autoupdate (apache#165) updates: - [github.com/astral-sh/ruff-pre-commit: v0.8.3 → v0.8.6](astral-sh/ruff-pre-commit@v0.8.3...v0.8.6) - [github.com/pre-commit/mirrors-mypy: v1.13.0 → v1.14.1](pre-commit/mirrors-mypy@v1.13.0...v1.14.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: should use Collection[Array] instead list[Array] (apache#170) "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance Consider using "Sequence" instead, which is covariant * fix: update type hints for path_or_paths and source parameters in ParquetDataset and read_table (apache#171) * [pre-commit.ci] pre-commit autoupdate (apache#167) updates: - [github.com/astral-sh/ruff-pre-commit: v0.8.6 → v0.9.1](astral-sh/ruff-pre-commit@v0.8.6...v0.9.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * release 17.16 (apache#172) * Fixed pa.fixed_shape_tensor (apache#175) * [pre-commit.ci] pre-commit autoupdate (apache#173) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.1 → v0.9.4](astral-sh/ruff-pre-commit@v0.9.1...v0.9.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: Preserve generic in `ChunkedArray.type` (apache#177) * release 17.17 (apache#178) * [pre-commit.ci] pre-commit autoupdate (apache#176) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.4 → v0.9.6](astral-sh/ruff-pre-commit@v0.9.4...v0.9.6) - [github.com/pre-commit/mirrors-mypy: v1.14.1 → v1.15.0](pre-commit/mirrors-mypy@v1.14.1...v1.15.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: support to construct ListArray with primitive type (apache#179) * fix: Avoid `chunked_array` overlapping overloads (apache#183) * fix: Add placeholder annotations to `pc.if_else` (apache#182) * fix: Widen `Array` to `Array | ChunkedArray` (apache#181) * fix: add `pc.fill_null` (apache#185) - https://arrow.apache.org/docs/python/generated/pyarrow.compute.fill_null.html - https://github.com/narwhals-dev/narwhals/blob/05e47b27ebe27b24196cee5956d07748d65a62ee/narwhals/_arrow/series.py#L675 * fix: Allow Table.from_arrays to take a list containing a mix of Array and ChunkedArray (apache#187) Update table.pyi * release 17.18 (apache#188) * [pre-commit.ci] pre-commit autoupdate (apache#180) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.6 → v0.9.10](astral-sh/ruff-pre-commit@v0.9.6...v0.9.10) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: from_arrays for both Table and RecordBatch (apache#189) * fix: resolve some `pa.compute` overlaps (apache#184) * fix: resolve overlapping `compute.(add|divide)` * fix: copy from non-cloned signature * fix: resolve overlapping `compute.exp` * fix: resolve overlapping `compute.power` * fix: resolve overlapping `compute.equal` * fix: resolve overlapping `compute.and_` * fix: Include `Array` in `chunked_array` overload (apache#190) narwhals-dev/narwhals@0237f7a * release 17.19 (apache#191) * Add Scalar, Array and Type classes for Json & Uuid (apache#194) * Add Scalar, Array and Type classes for Json & Uuid * Formatting fixes * [pre-commit.ci] pre-commit autoupdate (apache#192) updates: - [github.com/astral-sh/ruff-pre-commit: v0.9.10 → v0.11.2](astral-sh/ruff-pre-commit@v0.9.10...v0.11.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Revert "Add Scalar, Array and Type classes for Json & Uuid" (apache#195) Revert "Add Scalar, Array and Type classes for Json & Uuid (apache#194)" * fix: Add missing `pc.equal` overload (apache#196) * feat: support pyarrow 19.0 (apache#198) * build: upgrade pyarrow min version to 19.0 * feat: support pyarrow 19.0 * omit mypy bool8 override error * fix: reexport new types (apache#199) * feat: override new patterns for func repeat and nulls (apache#200) * fix: reexport decimal64 array and decimal128 array * feat: override new patterns for func `repeat` and `nulls` * release: 19.1 (apache#201) * fix: Allow `Iterable[Table]` in `concat_tables` (apache#203) https://arrow.apache.org/docs/python/generated/pyarrow.concat_tables.html > tables : iterable of pyarrow.Table objects * fix: Allow `ChunkedArray[BooleanScalar]` in `pc.invert` (apache#204) Fixes https://github.com/narwhals-dev/narwhals/blob/caabc0efdef54f117c83888926860e3972ef69d5/narwhals/_arrow/series.py#L298-L299 * feat: Fully spec `TableGroupBy.aggregate` (apache#197) - https://arrow.apache.org/docs/python/compute.html#grouped-aggregations - https://arrow.apache.org/docs/python/generated/pyarrow.TableGroupBy.html#pyarrow.TableGroupBy.aggregate - https://github.com/apache/arrow/blob/34a984c842db42b409a1359e6e2cf167a2365a48/python/pyarrow/table.pxi#L6578-L6604 * fix: Add missing return type to `ChunkedArray.filter` (apache#205) * fix: Add relaxed final overload to logical functions (apache#206) Covers all of `pc.(and_ | and_kleene | and_not | and_not_kleene | or_ | or_kleene | xor)` Resolves: - https://github.com/narwhals-dev/narwhals/blob/caabc0efdef54f117c83888926860e3972ef69d5/narwhals/_arrow/series.py#L219-L233 - https://github.com/narwhals-dev/narwhals/blob/caabc0efdef54f117c83888926860e3972ef69d5/narwhals/_arrow/series.py#L662 * fix: Allow `ChunkedArray` in `Table.set_column` (apache#211) Also being more consistent with `ArrayOrChunkedArray[Any]` everywhere Discovered in - https://github.com/vega/vega-datasets/blob/343b7101391a81190ba24e1e8d62a381d2fef3bd/scripts/species.py#L798-L799 * chore: Ignore `fsspec` `[import-untyped]` (apache#210) ```py _fs.pyi:18: error: Skipping analyzing "fsspec": module is installed, but missing library stubs or py.typed marker [import-untyped] _fs.pyi:18: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports Found 1 error in 1 file (checked 64 source files) ``` - fsspec/filesystem_spec#625 - fsspec/filesystem_spec#1676 * feat: Convert `types.is_*` into `TypeIs` guards (apache#215) * chore: Add `types.__all__` * feat: Convert `types._is_*` into `TypeIs` guards I've been using this for a little while, but makes more sense to live in the stubs https://github.com/narwhals-dev/narwhals/blob/16427440e6d74939c403083b52ce3fb0af7d63c7/narwhals/_arrow/utils.py#L44-L67 * fix: Resolve `bit_wise_and` overlaps (apache#214) Fixes 3 errors: ```py compute.pyi:608:5 - error: Overload 1 for "bit_wise_and" overlaps overload 4 and returns an incompatible type (reportOverlappingOverload) compute.pyi:608:5 - error: Overload 1 for "bit_wise_and" overlaps overload 5 and returns an incompatible type (reportOverlappingOverload) compute.pyi:620:5 - error: Overload 3 for "bit_wise_and" will never be used because its parameters overlap overload 1 (reportOverlappingOverload) ``` * fix: Resolve `list_*` overlapping overloads (apache#213) * fix: Resolve `list_value_length` overlaps * fix: Resolve `list_element` overlaps * fix: Resolve `list_(flatten|slice|parent_indices)` overlaps An improvement, but still not that accurate * fix: Include `VarianceOptions` in `TableGroupBy.aggregate` (apache#212) - Follow-up to apache#197 - Noticed while writing up (narwhals-dev/narwhals#2385) - We already use it for `std`, `var` in https://github.com/narwhals-dev/narwhals/blob/16427440e6d74939c403083b52ce3fb0af7d63c7/narwhals/_arrow/group_by.py#L81-L82 * [pre-commit.ci] pre-commit autoupdate (apache#202) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.2 → v0.11.5](astral-sh/ruff-pre-commit@v0.11.2...v0.11.5) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * fix: Resolve `Scalar.as_py` warnings for `DictionaryType` (apache#207) > scalar.pyi:75:20 - warning: TypeVar "_AsPyTypeK" appears only once in generic function signature > Use "object" instead (reportInvalidTypeVarUse) > scalar.pyi:85:20 - warning: TypeVar "_AsPyTypeK" appears only once in generic function signature > Use "object" instead (reportInvalidTypeVarUse) Instead just using `int`, which should be all that is possible from: https://github.com/zen-xu/pyarrow-stubs/blob/02552b81161d19d4aa71d8656b028eefac84612b/pyarrow-stubs/__lib_pxi/types.pyi#L154-L164 https://github.com/zen-xu/pyarrow-stubs/blob/02552b81161d19d4aa71d8656b028eefac84612b/pyarrow-stubs/__lib_pxi/types.pyi#L63-L70 * fix: Add default to `pc.sort_indices` (apache#216) * fix: Add default to `pc.sort_indices` Fixes narwhals-dev/narwhals#2390 (comment) Default is specified in https://arrow.apache.org/docs/python/generated/pyarrow.compute.sort_indices.html * refactor: Reuse some aliases * fix: Allow `list_size` with `Field` in `pa.list_` (apache#218) Closes apache#217 * allow `Table` or `RecordBatch` for dataset (apache#222) allow source argument pyarrow.dataset.dataset() to be RecordBatch | Table * refactor: Simplify `types` overloads (apache#219) * fix: `binary` overlap * fix: Simplify list constructors, `_Ordered` * refactor: Use `_Tz` default * fix: iter ChunkedArray should return scalar value (apache#224) * release: 19.2 (apache#225) * fix: Add missing `DictionaryArray` methods/properties (apache#226) - https://arrow.apache.org/docs/python/generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray.dictionary - https://arrow.apache.org/docs/python/generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray.indices - https://arrow.apache.org/docs/python/generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray.dictionary_decode - https://arrow.apache.org/docs/python/generated/pyarrow.DictionaryArray.html#pyarrow.DictionaryArray.dictionary_encode - https://github.com/narwhals-dev/narwhals/blob/c23e56c56630761f0fbc58b575a1c987e57d58d5/narwhals/_arrow/series.py#L787-L798 - https://github.com/narwhals-dev/narwhals/blob/c23e56c56630761f0fbc58b575a1c987e57d58d5/narwhals/_arrow/series_cat.py#L14-L18 * chore: use pyright as static type checker (apache#227) * use pyright as static type checker * make pyright happy * fix: fix pyright action (apache#229) fix github ci * fix: Match runtime behavior of `(Table|RecordBatch).select` (apache#221) * fix: Match runtime behavior of `(Table|RecordBatch).select` - https://github.com/MarcoGorelli/narwhals/blob/5b02b592183b8d39e2d32e0aedd6c234bb22d405/narwhals/_arrow/dataframe.py#L305-L307 - https://github.com/MarcoGorelli/narwhals/blob/5b02b592183b8d39e2d32e0aedd6c234bb22d405/narwhals/_arrow/dataframe.py#L285-L294 Following up on what I thought was a simple stub issue, but we're both *too strict* and *too permissive* in different ways {placeholder} - https://github.com/apache/arrow/blob/d2ddee62329eb711572b4d71d6380673d7f7edd1/python/pyarrow/table.pxi#L4367-L4374 - https://github.com/apache/arrow/blob/d2ddee62329eb711572b4d71d6380673d7f7edd1/python/pyarrow/table.pxi#L1721-L1739 * update select * update select --------- Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * [pre-commit.ci] pre-commit autoupdate (apache#220) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.5 → v0.11.8](astral-sh/ruff-pre-commit@v0.11.5...v0.11.8) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * feat: narrow scalar when type is given (apache#230) * rename Uint -> UInt * feat: narrow scalar when type is given * release 19.3 (apache#231) * chore: pyright use strict mode (apache#233) * fix types * update array.pyi * update scalar.pyi * update * update array * update array * optimize chunked_array * optimizer iterchunks * update * update pyproject.toml * fix: pa.nulls accept type rather than types (apache#234) * [pre-commit.ci] pre-commit autoupdate (apache#232) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.8 → v0.11.9](astral-sh/ruff-pre-commit@v0.11.8...v0.11.9) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * release 19.4 (apache#235) * lint(pyright): disable reportUnknownMemberType (apache#239) * [pre-commit.ci] pre-commit autoupdate (apache#236) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.9 → v0.11.13](astral-sh/ruff-pre-commit@v0.11.9...v0.11.13) - [github.com/RobertCraigie/pyright-python: v1.1.400 → v1.1.401](RobertCraigie/pyright-python@v1.1.400...v1.1.401) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * feat: support pyarrow 20.0 (apache#240) * [pre-commit.ci] pre-commit autoupdate (apache#241) updates: - [github.com/RobertCraigie/pyright-python: v1.1.401 → v1.1.402](RobertCraigie/pyright-python@v1.1.401...v1.1.402) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * support docstring (apache#242) * doc: complete tensor doc * doc: complete table doc * doc: complete scalar doc * doc: complete orc doc * doc: complete memory doc * doc: complete lib doc * doc: complete json doc * doc: complete hdfs doc * doc: complete gcsfs doc * doc: complete fs doc * doc: complete flight doc * doc: complete dataset doc * doc: complete dataset parquet doc * doc: complete dataset parquet encryption doc * doc: complete cuda doc * doc: complete csv doc * doc: complete azurefs doc * doc: complete core doc * doc: complete interchange doc * doc: complete array doc * doc: complete builder doc * doc: complete device doc * doc: complete io doc * doc: complete ipc doc * doc: complete types doc * mark deprecated apis * doc: complete _compute doc * doc: complete compute doc * doc: update compute doc * lint code * release 20.0.0.20250618 (apache#243) * fix: make ParquetFileFormat constructor args optional (apache#244) * fix: Field.remove_metadata should return Self (apache#246) * [pre-commit.ci] pre-commit autoupdate (apache#245) updates: - [github.com/astral-sh/ruff-pre-commit: v0.11.13 → v0.12.0](astral-sh/ruff-pre-commit@v0.11.13...v0.12.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * release 20.0.0.20250627 (apache#247) * fix: chunked_array with type should be specified (apache#250) * [pre-commit.ci] pre-commit autoupdate (apache#248) updates: - [github.com/astral-sh/ruff-pre-commit: v0.12.0 → v0.12.3](astral-sh/ruff-pre-commit@v0.12.0...v0.12.3) - [github.com/RobertCraigie/pyright-python: v1.1.402 → v1.1.403](RobertCraigie/pyright-python@v1.1.402...v1.1.403) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * release 20.0.0.20250715 (apache#251) * fix: The type parameter of array should be covariant (apache#253) * release 20.0.0.20250716 (apache#254) * Add py.typed file to signify that the library is typed See the relevant PEP https://peps.python.org/pep-0561 * Prepare `pyarrow-stubs` for history merging MINOR: [Python] Prepare `pyarrow-stubs` for history merging Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> * Add `ty` configuration and suppress error codes * One line per rule * Add licence header from original repo for all `.pyi` files * Revert "Add licence header from original repo for all `.pyi` files" * Prepare for licence merging * Exclude `stubs` from `rat` test * Add Apache licence clause to `py.typed` * Reduce list * Resolve merge conflict --------- Signed-off-by: Jonas Dedden <university@jonas-dedden.de> Co-authored-by: ZhengYu, Xu <zen-xu@outlook.com> Co-authored-by: Jim Bosch <talljimbo@gmail.com> Co-authored-by: Oliver Mannion <125105+tekumara@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Eugene Toder <eltoder@users.noreply.github.com> Co-authored-by: fvankrieken <fvankrieken@planning.nyc.gov> Co-authored-by: Ilia Ablamonov <ilia@flamefork.ru> Co-authored-by: Mathias Beguin <mathias.beguin@hotmail.com> Co-authored-by: Dylan Scott <dylan.scott@gmail.com> Co-authored-by: deanm0000 <37878412+deanm0000@users.noreply.github.com> Co-authored-by: Jan Moravec <moravecj@post.cz> Co-authored-by: Marius van Niekerk <marius.v.niekerk@gmail.com> Co-authored-by: Jonas Dedden <university@jonas-dedden.de> Co-authored-by: Fábio D. Batista <fabio@atelie.dev.br> Co-authored-by: ben-freist <93315290+ben-freist@users.noreply.github.com> Co-authored-by: Jiahao Yuan <kahojyun@icloud.com> Co-authored-by: Pim de Haan <pimdehaan@gmail.com> Co-authored-by: Dan Redding <125183946+dangotbanned@users.noreply.github.com> Co-authored-by: Tom Crasset <25140344+tcrasset@users.noreply.github.com> Co-authored-by: Tom McTiernan <tmct@users.noreply.github.com> Co-authored-by: Rok Mihevc <rok@mihevc.org>

wesm added 2 commits October 17, 2016 13:46

Use more conforming names in bit-util.h

cb9519d

Change-Id: I63fca0131fd094b0bdbd7790410c918bbe1f50b6

Add Slice, Copy methods to Buffer

3ab4e7a

Change-Id: Ia6f232bf020a0249c19453a2c39eea51b52f86cf

wesm force-pushed the ARROW-317 branch from 58e4875 to 3ab4e7a Compare October 17, 2016 17:46

xhochy approved these changes Oct 17, 2016

View reviewed changes

Fix up pyarrow usage of BitUtil

0666b22

Change-Id: I4b8cfb4272ce0dea04974bada20e3e83697d7ceb

asfgit closed this in 676c32c Oct 18, 2016

wesm deleted the ARROW-317 branch October 18, 2016 01:19

github-actions bot mentioned this pull request Nov 26, 2024

GH-36924: [Java] support offset/length and filter in scan option #36967

Open

paddyroddy pushed a commit to rok/arrow that referenced this pull request Jul 19, 2025

fix: Preserve generic in ChunkedArray.type (apache#177)

5cad279

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ARROW-317: Add Slice, Copy methods to Buffer #177

ARROW-317: Add Slice, Copy methods to Buffer #177

Uh oh!

wesm commented Oct 17, 2016

Uh oh!

wesm commented Oct 17, 2016

Uh oh!

wesm commented Oct 17, 2016

Uh oh!

wesm commented Oct 18, 2016

Uh oh!

Uh oh!

ARROW-317: Add Slice, Copy methods to Buffer #177

ARROW-317: Add Slice, Copy methods to Buffer #177

Uh oh!

Conversation

wesm commented Oct 17, 2016

Uh oh!

wesm commented Oct 17, 2016

Uh oh!

wesm commented Oct 17, 2016

Uh oh!

wesm commented Oct 18, 2016

Uh oh!

Uh oh!