Scalable master data management, identity resolution, entity resolution, and deduplication using ML

RocketMQ消息幂等去重消费者，支持使用MySQL或者Redis做幂等表，开箱即用

Accelerating the deduplication and collapsing process for reads with Unique Molecular Identifiers (UMI). Heavily optimized for scalability and orders of magnitude faster than a previous tool.

Java DSL for (online) deduplication

WInte.r is a Java framework for end-to-end data integration. The WInte.r framework implements well-known methods for data pre-processing, schema matching, identity resolution, data fusion, and result evaluation.

A general purpose deduplication framework

🖇️Record Linkage tool used by https://cidacs.bahia.fiocruz.br/

PRIMAT - Private Matching Toolbox

Mirror of https://bitbucket.org/resteorts/smered

A java based database driven backup tool with multi storage support and other nice things

Deduplication of EndNote and Zotero RIS files

This is a prototype of our DSC'18 paper.

Data bus based on Apache Kafka and consisting of separate components [copied from own private repos]

Spring Boot-based quiz leaderboard system that processes distributed quiz events, eliminates duplicate submissions, aggregates participant scores, and generates accurate leaderboards using idempotent event processing.

Distributed image deduplication service with block-level storage and hash-based compression

Client-side lossless image deduplication engine using block-level content-addressable storage

Plug-and-play Spring Boot idempotency library using Redis. Deduplication, TTL, metrics, logging.

Resilient Spring Boot data pipeline for distributed API polling, O(1) event deduplication, and leaderboard aggregation.

Project for helping brother in finding duplicates in his photos directory.

A UI application for File Deduplication using Hashing