Data deduplication engine, supporting optional compression and public key encryption.
-
Updated
Aug 25, 2022 - Rust
#
data-deduplication
Here are 26 public repositories matching this topic...
Official Repository of "LLM × DATA" Survey Paper
data-transformation data-acquisition data-deduplication data-filtering vlm data-selection data-synthesis data-provenance llm data-mixing
-
Updated
Nov 2, 2025
🚢 Data Toolkit for Sailor Language Models
-
Updated
Feb 24, 2025 - Python
Self-contained C# library for data deduplication using Sqlite
compression storage nuget dedupe sqlite-database data-deduplication chunk compress deduplication chunk-data duplicate-data chunk-key
-
Updated
Apr 7, 2023 - C#
Fast and efficient content-defined chunking for data deduplication. Java implementation of FastCDC as library.
-
Updated
Sep 21, 2023 - Java
Optimal distributed data deduplication and supervised learning pipeline using Apache Spark
distributed-systems data-science spark hadoop data-deduplication data-engineering data-cleaning deduplication
-
Updated
Aug 19, 2020 - Scala
A JAVA project that splits data using hashing techniques and removes duplicate blocks to save cloud storage. This project also uses the CloudSim framework for cloud storage simulation.
-
Updated
Jan 6, 2021 - Java
General deduping engine for JDBC sources with output to JDBC/csv targets
-
Updated
Dec 21, 2020 - Kotlin
A pure-JS, content-addressed, copy-on-write virtual filesystem for the browser, featuring: deduplication, filesystem universes (snapshots), events, and optional asynchronous sync.
browser localstorage pure-javascript data-deduplication indexeddb leightweight content-addressable-storage copy-on-write delta-sync webxdc virtual-filesystem
-
Updated
Dec 2, 2025 - JavaScript
RepoCapsule is a Python toolkit for turning GitHub, local, and other text/code sources into clean JSONL corpora for LLM pre-training, fine-tuning, or RAG. It provides structure-aware chunking, robust Unicode decoding, pluggable quality/safety screening, and optional dataset card + deduplication support.
python data-deduplication dataset-creation data-pipelines repository-mining jsonl github-repos rag text-preprocessing quality-filtering code-mining llm llm-training llm-datasets
-
Updated
Dec 7, 2025 - Python
PolyDeDupe: Multi-Lingual Data Deduplication
-
Updated
Dec 8, 2025 - Python
Enterprise-grade SaaS platform for importing, cleaning, and managing large-scale mailing lists with advanced deduplication and enrichment.
-
Updated
Nov 24, 2025 - TypeScript
Этот проект представляет собой мощный инструмент для поиска и анализа дублирующихся файлов в указанной директории. Программа позволяет эффективно выявлять одинаковые файлы на основе их содержимого, используя алгоритм хеширования SHA-256. Она поддерживает настройку параметров, таких как минимальный размер файла для проверки и игнорирование определен
python hashing productivity multithreading data-deduplication file-system sha256 file-management system-utility cli-tool dev-tools file-deduplication file-comparison disk-cleanup command-line-utility duplicate-file-finder
-
Updated
Feb 14, 2025 - Python
-
Updated
Jan 7, 2023 - Python
A calculator for storage and transmission of deduplicated data. Output: charts and tables
data-deduplication deduplication deduplication-calculator storage-deduplication-calculator network-deduplication-calculator
-
Updated
Oct 12, 2025
Fellow is a package for creating people that can be unified by their shared values via a singleton list on the class
-
Updated
Nov 22, 2025 - TypeScript
Automated business record matching using fuzzy algorithms (RapidFuzz) and browser automation (Playwright)
python record-linkage pandas fuzzy-matching data-deduplication data-cleaning web-automation playwright rapidfuzz
-
Updated
Nov 18, 2025 - Python
ETL workflow for stock data processing using Mage and PostgreSQL
python etl docker-compose postgresql data-deduplication data-engineering stock-market data-processing financial-data data-modeling data-cleaning data-aggregation api-integration alpha-vantage mage-ai
-
Updated
Jan 17, 2025 - Python
The HR Roster Change Detection Pipeline is an automated solution for processing HR roster data. Leveraging Apache Airflow and PostgreSQL, it enables seamless data ingestion, deduplication, and change detection, streamlining HR operations.
python open-source automation etl postgresql data-deduplication data-engineering data-pipelines apache-airflow roster-management workforce-analytics scalable-solutions hr-technology delta-detection hr-data-processing
-
Updated
Dec 4, 2024 - Python
fast dataset merging and deduplication tool
data-transformation data-deduplication data-integration batch-processing performance-analytics data-optimization data-merging large-scale-data dataset-processing memory-efficient-processing
-
Updated
Nov 10, 2025 - JavaScript
Improve this page
Add a description, image, and links to the data-deduplication topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the data-deduplication topic, visit your repo's landing page and select "manage topics."