Knowledge & Data · Software component
Exact Hash Deduplicator
Software componentKnowledge & DataKnowledge & Dataarc:ExactHashDeduplicator
A content deduplicator that fingerprints content with a cryptographic hash and discards items whose fingerprint has already been seen.
Responsibility. Removes byte-identical duplicates via constant-time fingerprint lookup.
Also known as: Content hashing deduplication, Exact deduplication, Exact matching, Hash-based deduplication, Exact deduplication stage, MD5 hash deduplicator
Variant of Content Deduplicator abstract
When to choose. Choose when duplicates are identical after cleaning and pairwise similarity comparison would be computationally prohibitive.
Relationships
reads dependency
writes dependency
receives data from dynamic
sends data to dynamic
is orchestrated by control
alternative to variability
Design guidance
- SHOULD use collision-resistant hashes so false duplicate detection is negligible.
- SHOULD keep the first occurrence of each hash and discard the rest.
Quantitative guidance
As stated by the sources; verify before use.
- SHA-256 produces 64-character fingerprints enabling O(1) duplicate detection, scaling to millions of documents (Ch6.3A).
- Exact deduplication of 50M documents: 14.2 hours CPU vs 42 minutes GPU (20.3x) (Ch6.3B).
- Catches 30-40% of duplicates in typical enterprise corpora; processes 10,000 documents in under a second (Ch6.4).
- MD5 fingerprints are 16 bytes, so hashes fit in GPU memory even for petabyte datasets; billions of hashes compute in minutes (Ch7.5).
Classification
- Patterns
- SHA-256 content fingerprintingSet-lookup duplicate detectionCryptographic hashing
- Technologies
- SHA-256NVIDIA NeMo Curator
- Quality attributes
- Performance efficiency (ISO/IEC 25010)
- Risks mitigated
- Verbatim duplicate documentsMemorization of repeated press releases, disclaimers and cloned sites
Sources
- Ch6.3A: T. Nguyen, "ETL Pipeline Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.3A. ISBN: 9798244538229.
- Ch6.3B: T. Nguyen, "ETL Worked Example - Load Phase & Pipeline Integration," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.3B. ISBN: 9798244538229.
- Ch6.4: T. Nguyen, "Data Quality Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.4. ISBN: 9798244538229.
- Ch7.5: T. Nguyen, "NeMo Curator, Riva Speech AI & Multimodal Integration," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 7.5. ISBN: 9798244538229.