QuackFS is a FUSE-based filesystem that uses PostgreSQL and S3 as a backend for persistence. Its storage model is based on Differential Storage, meaning that it only stores the differences between write operations, which allows for time-travel, zero-copy sharing and efficient versioning.
QuackFS is specialized for DuckDB database files and their associated Write-Ahead Log (WAL) files. It is not intended as a general-purpose filesystem. Checkout the architecture diagram for more details.
This project was created for learning purposes and to explore the idea of Differential Storage. It was greatly inspired by this blog post by MotherDuck 🦆. Do check it out to have a better understanding of what I've implemented here.
Warning
This project is not production-ready. It is a proof of concept and should not be used in production environments.
- Docker
- VSCode, Cursor or Github Codespaces
- Install Dev Containers extension
It should work with any other IDE that supports Dev Containers, but I've only tested it with the above.
After opening the project in your IDE and having installed the Dev Containers extension, you can open the command palette (Ctrl+Shift+P or Cmd+Shift+P) and search and select Dev Containers: Reopen in Container. This will start the container, which can take a couple minutes to build.
All the commands listed below should be run inside the container that was created in the previous step. If you're using VSCode, you can open the integrated terminal and it will be running inside the container already. Alternatively, you can run docker exec -it -w /workspaces/quackfs quackfs zsh on your favorite terminal and follow from there.
To run the QuackFS process (storage backend), you can use the following command:
$ make runIn another terminal you can run DuckDB CLI to open/create a database in the FUSE mountpoint (default is /tmp/fuse):
$ duckdb /tmp/fuse/db.duckdbAlternatively, you can run our "load" test script (make sure make run is running in another terminal), because this script will run against the Differential Storage implementation:
$ make loadTo run all the tests, run following command:
$ make testFor other commands, check the Makefile.
This project is currently in development. Some of planned features are:
- Use PostgreSQL for metadata and data persistence
- Use S3 for data storage instead of Postgres
- Time travel: be able to query the database from old versions
- Creating new databases from a specific point in time (sharing data with zero copy)
- Merging of snapshot layers
- Garbage collection of snapshot layers
- Add proper database indexing