Observability & Evaluation · Software component
Failure Case Curator
Software componentObservability & EvaluationObservability & Evaluationarc:FailureCaseCurator
An evaluation-flywheel component that collects production failures and user-reported issues, validates their representativeness, and converts them into difficulty-calibrated regression test cases.
Responsibility. Turns verified production failures into new benchmark test cases.
Also known as: Real issue collection, Test set augmentation, Benchmark evolution, Test Set Backfill
Relationships
reads dependency
writes dependency
receives data from dynamic
Design guidance
- SHOULD distinguish genuine capability gaps from one-off edge cases before adding a case.
- SHOULD reconstruct the exact pre-fix scenario and add unrelated data so agents must separate signal from noise.
Quantitative guidance
As stated by the sources; verify before use.
- Flywheel adopters report 8-15% annual improvement versus 2-3% with static benchmarks; 5-15% per iteration cycle (Ch3.3).
- Multi-hop QA flywheels: 8-12% annual improvement versus typical 2-3% (Ch3.3).
- DCA-Bench: 221 real-world test cases collected from dataset platforms (Ch3.3).
Classification
- Patterns
- Evaluation flywheelClosed-loop benchmark evolutionClosed-loop test set augmentation
- Technologies
- NVIDIA NeMoGalileo
- Quality attributes
- Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)Maintainability (ISO/IEC 25010)
- Risks mitigated
- Static benchmarks diverging from production reality
Sources
- Ch3.3: T. Nguyen, "Web Navigation and Interaction Benchmarks," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.3. ISBN: 9798244538229.
- Ch3.8: T. Nguyen, "Action Accuracy Metrics," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.8. ISBN: 9798244538229.
- Ref8.03: "Agent Evaluation Frameworks and Metrics," unpublished reference note (03-Agent-Evaluation-Frameworks.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note