Agentic AI Reference Architecture

Knowledge & Data · Software component

Cascading Deduplicator

Software componentKnowledge & DataKnowledge & Dataarc:CascadingDeduplicator

A deduplicator that runs exact, fuzzy and semantic deduplication sequentially so each more expensive level only processes survivors of the cheaper ones.

Responsibility. Sequences deduplication levels from cheapest to most expensive.

Also known as: Three-level deduplication, Multi-level deduplication pipeline

Variant of Content Deduplicator abstract

When to choose. Choose when a corpus contains exact, near and semantic duplicates together (typical in production); orders cheap levels first to minimise cost.

alternative to; orchestratesis orchestrated byspecializesorchestratesreceives data fromorchestratesSemantic Deduplicator: alternative to; orchestratesSemantic DeduplicatorIngestion Pipeline Orchestrator: is orchestrated byIngestion Pipeline Orche…Content Deduplicator: specializesContent DeduplicatorExact Hash Deduplicator: orchestratesExact Hash DeduplicatorData Format Normalizer: receives data fromData Format NormalizerFuzzy Text Deduplicator: orchestratesFuzzy Text Deduplicator
Direct neighbourhood (hover for relationship types)

Relationships

receives data from dynamic

is orchestrated by control

orchestrates control

alternative to variability

Design guidance

Quantitative guidance

As stated by the sources; verify before use.

Classification

Patterns
Cost-ordered cascade
Quality attributes
Performance efficiency (ISO/IEC 25010)Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)
Risks mitigated
Systematic duplicates left by single-level approaches

Sources

  1. Ch6.4: T. Nguyen, "Data Quality Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.4. ISBN: 9798244538229.