PySpark Data Sources

This repository showcases custom Spark data sources built using the new Python Data Source API introduced in Apache Spark 4.0. For an in-depth understanding of the API, please refer to the API source code. Note this repo is demo only and please be aware that it is not intended for production use. Contributions and feedback are welcome to help improve the examples.

Installation

pip install pyspark-data-sources

Usage

Make sure you have pyspark >= 4.0.0 installed.

pip install pyspark

Or use Databricks Runtime 15.4 LTS or above versions, or Databricks Serverless.

from pyspark_datasources.fake import FakeDataSource

# Register the data source
spark.dataSource.register(FakeDataSource)

spark.read.format("fake").load().show()

# For streaming data generation
spark.readStream.format("fake").load().writeStream.format("console").start()

Example Data Sources

Data Source	Short Name	Description	Dependencies
GithubDataSource	github	Read pull requests from a Github repository	None
FakeDataSource	fake	Generate fake data using the Faker library	faker
StockDataSource	stock	Read stock data from Alpha Vantage	None
GoogleSheetsDataSource	googlesheets	Read table from public Google Sheets	None
KaggleDataSource	kaggle	Read datasets from Kaggle	kagglehub, pandas
SimpleJsonDataSource	simplejson	Write JSON data to Databricks DBFS	databricks-sdk

See more here: https://allisonwang-db.github.io/pyspark-data-sources/.

Official Data Sources

For production use, consider these official data source implementations built with the Python Data Source API:

Data Source	Repository	Description	Features
HuggingFace Datasets	@huggingface/pyspark_huggingface	Production-ready Spark Data Source for 🤗 Hugging Face Datasets	• Stream datasets as Spark DataFrames • Select subsets/splits with filters • Authentication support • Save DataFrames to Hugging Face

Contributing

We welcome and appreciate any contributions to enhance and expand the custom data sources.:

• Add New Data Sources: Want to add a new data source using the Python Data Source API? Submit a pull request or open an issue.
• Suggest Enhancements: If you have ideas to improve a data source or the API, we'd love to hear them!
• Report Bugs: Found something that doesn't work as expected? Let us know by opening an issue.

Development

Environment Setup

poetry install
poetry env activate

Build Docs

mkdocs serve