Agentic AI Reference Architecture

Observability & Evaluation · Software component

Evaluator Calibrator

Software componentObservability & EvaluationObservability & Evaluationarc:EvaluatorCalibrator

An evaluation component that compares automated and LLM-judge reasoning scores with expert ratings on gold-standard data to set scoring thresholds and confirm agreement.

Responsibility. Calibrates automated reasoning evaluators against expert judgment.

evaluatesreceives data fromevaluatesevaluatesevaluatesreadswritesLLM Judge: evaluatesLLM JudgeHuman Evaluator: receives data fromHuman EvaluatorReasoning Consistency Checker: evaluatesReasoning Consistency Ch…Entailment Step Validator: evaluatesEntailment Step ValidatorToken Uncertainty Scorer: evaluatesToken Uncertainty ScorerReference Reasoning Dataset: readsReference Reasoning Data…Reasoning Quality Threshold Configuration: writesReasoning Quality Thresh…
Direct neighbourhood (hover for relationship types)

Relationships

reads dependency

writes dependency

receives data from dynamic

evaluates assurance

Classification

Patterns
Statistical sampling to validate automated-human correlation
Quality attributes
Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)
Risks mitigated
Automated metrics optimising proxies that do not reflect reasoning qualityUnreliable automated detection

Sources

  1. Ch3.9: T. Nguyen, "Reasoning Quality," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.9. ISBN: 9798244538229.
  2. Ch3.10: T. Nguyen, "Efficiency Metrics," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.10. ISBN: 9798244538229.