ai-storage

🚀2.3x faster than MinIO for 4KB object payloads. RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.

High performance distributed S3 compatible object storage focused on speed and designed to be an open alternative to MinIO and RustFS.

Part of the sai3 project that delivers multi-protocol storage access for AI/ML workflows, supporting Pytorch, Tensorflow and Jax. This project provides a CLI, along with Rust and Python libraries for AI/ML storage workflows. Supporting S3, File, Azure Blob and GCS using the latest Rust SDKs.

Hybrid Storage with freely combine

We will look at SQL and NoSQL databases and decide the ideal database to be used based on the format of data. Then you will implement storing and querying data in different databases such as MySQL, MongoDB, and Cassandra.

You will look at the different formats for storing, transmitting and organizing very large collections of data. You will learn how to choose the right data formats as well as look at compression.

High-performance AI data storage pipeline utilizing Shelby's global object storage and Aptos blockchain for verifiable data provenance. Optimized for low-latency AI workloads.

You will look at several case studies, examining everything from AI being used to manipulate elections, to AI displaying racial and sexist prejudices. Implement a simple sentiment classifier to differentiate between positive and negative words and sentences. You'll observe how this works in many cases, and display the problematic biases and huma…

Open source, cloud native Parallel NFS (pNFS) on AWS, Azure and GCP — quick deploy into your own cloud account. Free Community Edition.

GitOps-managed secure software-defined storage platform: multi-backend SDS, multi-plane CNI, DPU/SR-IOV offload, Vault KMS, and staged catalog-to-deployment orchestration.

Open Cloud-native Lustre Community Edition AWS, Azure and GCP — quick deploy into your own cloud account. Free Community Edition

Discover the fundamentals of Apache Spark, including its architecture, transformations and actions. Learn why the design choices of RDD were made and how this enhances the Hadoop and MapReduce construct.

A technical publication defining storage architecture, data movement, tiering, governance, and infrastructure requirements for AI data pipelines and AI factories.

WEKA (WekaIO, Inc.) builds NeuralMesh, a software-defined, high-performance data platform for AI, machine learning, and HPC workloads. Its container-native storage system delivers a unified namespace across NVMe SSD and object storage, running on bare metal and across AWS, Azure, GCP, and OCI.

MinIO is a high-performance, S3-compatible object storage system built for large-scale AI, analytics, and cloud-native data infrastructure. Its API is a drop-in implementation of the Amazon S3 REST API (bucket and object operations, multipart uploads, versioning, object locking / WORM retention, lifecycle policies, replication, and encryption)…