Observability & Evaluation · Software component
Retrieval Quality Evaluator
Software componentObservability & EvaluationObservability & Evaluationarc:RetrievalQualityEvaluator
An evaluation component that measures retrieval precision and recall over time against human-labeled relevance judgments for representative queries.
Responsibility. Measures retrieval precision and recall against labeled relevance judgments.
Also known as: Retrieval drift monitor, Precision/recall tracker, Retrieval analytics
Relationships
reads dependency
emits telemetry to dynamic
triggers dynamic
evaluates assurance
monitors assurance
Design guidance
- SHOULD maintain human-labeled relevance judgments for representative queries and alert on sudden precision or recall drops.
- SHOULD measure retrieval on representative queries to tune hybrid fusion weights and justify embedding model upgrades.
- SHOULD track retrieval ranking quality with MRR, NDCG and Precision@K (Ref6.01).
Quantitative guidance
As stated by the sources; verify before use.
- Targets: top-1 relevance > 0.85, top-5 average > 0.70, diversity > 0.6, >= 80% of response supported by retrieval; alert when top-1 relevance < 0.75 (Ref8.04).
Classification
- Patterns
- Golden-query relevance judgmentsRetrieval drift detection
- Quality attributes
- Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)Maintainability (ISO/IEC 25010)
- Risks mitigated
- Retrieval drift after embedding-model or similarity-function changesSilent degradation from knowledge base noise
Sources
- Ch5.8: T. Nguyen, "Semantic Memory," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 5.8. ISBN: 9798244538229.
- Ch6.1: T. Nguyen, "Embeddings and RAG Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.1. ISBN: 9798244538229.
- Ch6.4: T. Nguyen, "Data Quality Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.4. ISBN: 9798244538229.
- Ch6.5: T. Nguyen, "Production RAG Systems," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.5. ISBN: 9798244538229.
- Ref6.01: S. Schürch, "How to Make Your LLM More Accurate with RAG & Fine-Tuning," Towards Data Science, Mar. 11, 2025. [Online]. Available: https://towardsdatascience.com/how-to-make-your-llm-more-accurate-with-rag-fine-tuning/
- Ref8.04: "Data Quality and Drift Detection for Agent Systems," unpublished reference note (04-Data-Quality-Drift-Detection.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note