Use uv project manager and add CI tests

[wengh]

Changes

• Migrate from Poetry specific pyproject.toml to the standard format. And switch to uv project manager
  • lock files are committed to ensure that the environemnt is reproducible
• Add ci.yml to test with combinations of different python (3.9, 3.13) and pyspark (3.5.6, >=4.0.0) versions on Github PRs
• Fix tests to make them compatible with pyspark 3.5.6, and to skip write tests if HF_TOKEN is missing

Testing

• See CI results

[Copilot]

Pull Request Overview

Migrates the project from Poetry to the UV build system by adopting PEP 621 metadata and restructuring dependencies.

• Consolidate metadata under a PEP 621 [project] table instead of [tool.poetry]
• Define core and development dependencies using [project.dependencies] and [dependency-groups]
• Configure uv_build as the PEP 517 build backend and pin the development Python version

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
pyproject.toml	Replace Poetry fields with PEP 621 [project], add dependency-groups, and switch to uv_build backend
.python-version	Pin the Python interpreter version to match the project’s minimum requirement

[Copilot]

Pull Request Overview

This PR integrates the uv project manager for dependency management and adds CI tests to ensure cross-version compatibility with Spark and Python.

• Updates test files to improve compatibility with PySpark 3.x by replacing the use of toArrow().
• Migrates project metadata to the new PEP 621 format and switches to uv_build.
• Introduces a GitHub Actions workflow that runs tests on multiple Python versions and package combinations.

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_huggingface_writer.py	Updates test logic and fixture for environment-based token and converts DataFrame using pyarrow for Spark 3.x compatibility.
tests/test_huggingface.py	Adds import for pyspark_huggingface.
pyproject.toml	Migrates project configuration to PEP 621 and defines dependency groups using uv_build.
.python-version	Specifies the default Python version for local development.
.github/workflows/ci.yml	Introduces a CI workflow with a matrix for Python and package versions.

Comments suppressed due to low confidence (1)

tests/test_huggingface_writer.py:134

• Collecting all rows from the DataFrame into memory for conversion to a PyArrow table may impact performance if the test data scale increases. Consider using a method that processes data in batches or limiting the test dataset size.

arrow_table = pa.Table.from_pylist([row.asDict() for row in df.collect()], schema=to_arrow_schema(df.schema))

[Copilot]

The .python-version file statically specifies Python 3.9, yet the CI workflow includes Python 3.13. It would be helpful to document the intended supported Python versions or update .python-version accordingly for consistency.

[lhoestq]

nice !