Knowledge & Data · Software component

Semantic Deduplicator

Software componentKnowledge & DataKnowledge & Dataarc:SemanticDeduplicator

A deduplicator that compares document embeddings and removes documents whose semantic similarity exceeds a configured threshold.

Responsibility. Removes documents that paraphrase already retained content.

Also known as: Semantic deduplication, Embedding-similarity deduplication

Variant of Content Deduplicator abstract

When to choose. Choose when duplicates express the same information in different words (rewrites, summaries, translations); most expensive level.

Direct neighbourhood (hover for relationship types)

Relationships

invokes dependency

Text Embedding Service Ch6.4

is orchestrated by control

Cascading Deduplicator Ch6.4

alternative to variability

Design guidance

SHOULD use semantic similarity rather than exact matching when comparing documents for contradictions.

Quantitative guidance

As stated by the sources; verify before use.

Conservative threshold 0.98 vs aggressive 0.85; 2-3 min for ~4,200 documents, catching the last ~20% of duplicates (Ch6.4).

Classification

Patterns: Embedding similarity matching
Quality attributes: Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)
Risks mitigated: Paraphrased or translated duplicatesContradictory rewritten guidance

Sources

Ch6.4: T. Nguyen, "Data Quality Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.4. ISBN: 9798244538229.