#

text-deduplication

Here are 3 public repositories matching this topic...

tomventa / dedupe

Tool to deduplicate file contents

dedupe deduplication text-deduplication file-content-deduplication

Updated Jan 26, 2026
Go

SeanWong17 / Semantic-Text-Deduplicator

一个基于 Transformer 模型（如BERT）和 FAISS 索引的高性能文本去重工具，专为处理大规模语料库中的语义重复问题而设计。

nlp transformer semantic-search faiss text-deduplication

Updated Aug 12, 2025
Python

Jihad-41 / fast-website-content-crawler

high-performance website content extractor

data-extraction content-analysis content-aggregation website-scraping competitive-research site-structure-analysis bulk-url-crawler text-deduplication parallel-crawling web-data-insights

Updated Nov 11, 2025
Python

Improve this page

Add a description, image, and links to the text-deduplication topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the text-deduplication topic, visit your repo's landing page and select "manage topics."