Observability & Evaluation · Software component
RAG Evaluator
Software componentObservability & EvaluationObservability & Evaluationarc:RAGEvaluator
An automated evaluation component that scores retrieval-augmented answers on faithfulness, answer relevance, context precision, and context recall.
Responsibility. Measures retrieval and grounding quality of RAG outputs.
Also known as: Retrieval Augmented Generation Assessment, RAGAS evaluator
Relationships
invokes dependency
is invoked by dependency
reads dependency
evaluates assurance
- Answer Synthesizer abstract Ch3.3 Ch6.4 +2
- Retriever abstract Ch3.3
Design guidance
- SHOULD prioritize recall over precision in multi-hop retrieval, since missing documents are more damaging than irrelevant ones.
- SHOULD be the starting evaluator for RAG-based workflows.
- SHOULD monitor accuracy across both retrieval and fine-tuned generation in hybrid systems (Ref6.01).
Classification
- Patterns
- Precision@K / Recall@K / F1@KFaithfulness and groundedness scoringLLM-as-a-judge
- Technologies
- RAGAS
- Quality attributes
- Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)
- Risks mitigated
- Semantic noise from dense retrievalMissing answer-bearing documentsUngrounded responsesIrrelevant retrieved context
Sources
- Ch3.3: T. Nguyen, "Web Navigation and Interaction Benchmarks," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.3. ISBN: 9798244538229.
- Ch6.4: T. Nguyen, "Data Quality Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.4. ISBN: 9798244538229.
- Ch6.5: T. Nguyen, "Production RAG Systems," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.5. ISBN: 9798244538229.
- Ref3.01: NVIDIA, "Agent Evaluation in NVIDIA NeMo Agent Toolkit," NVIDIA NeMo Agent Toolkit Documentation, v1.8. Accessed: Sep. 27, 2026. [Online]. Available: https://docs.nvidia.com/nemo/agent-toolkit/latest/improve-workflows/evaluate.html
- Ref6.01: S. Schürch, "How to Make Your LLM More Accurate with RAG & Fine-Tuning," Towards Data Science, Mar. 11, 2025. [Online]. Available: https://towardsdatascience.com/how-to-make-your-llm-more-accurate-with-rag-fine-tuning/