Agentic AI Reference Architecture

Observability & Evaluation · Software component

Reasoning Quality Scorer

Software componentObservability & EvaluationObservability & EvaluationVariation point (abstract)arc:ReasoningQualityScorer

An evaluation component that combines per-dimension reasoning scores (intra-step correctness, inter-step consistency, informativeness, relevancy) into an overall reasoning quality result.

Responsibility. Aggregates dimension scores into an overall reasoning quality assessment.

Also known as: Composite reasoning quality score, Reasoning quality aggregation

is orchestrated by; is invoked byemits telemetry toevaluatessends data toreceives data fromreceives data fromis specialized byreceives data fromreceives data fromis specialized byis specialized byEvaluation Harness: is orchestrated by; is invoked byEvaluation HarnessMetrics Collector: emits telemetry toMetrics CollectorReasoning Engine: evaluatesReasoning EngineOversight Gate: sends data toOversight GateReasoning Consistency Checker: receives data fromReasoning Consistency Ch…Entailment Step Validator: receives data fromEntailment Step ValidatorReasoning Path Quality Classifier: is specialized byReasoning Path Quality C…Goal Alignment Checker: receives data fromGoal Alignment CheckerInformation Gain Scorer: receives data fromInformation Gain ScorerMinimum Aggregation Quality Scorer: is specialized byMinimum Aggregation Qual…Weighted Aggregation Quality Scorer: is specialized byWeighted Aggregation Qua…
Direct neighbourhood (hover for relationship types)

Variants

VariantWhen to choose
Minimum Aggregation Quality ScorerChoose when agents must not receive full credit for correct answers reached through flawed reasoning and the bottleneck dimension should be surfaced.
Reasoning Path Quality Classifier—
Weighted Aggregation Quality ScorerChoose when dimensions carry different importance, e.g., high-stakes medical reasoning weighting correctness over efficiency or consumer chatbots weighting relevancy and informativeness.

Relationships

is invoked by dependency

emits telemetry to dynamic

receives data from dynamic

sends data to dynamic

is orchestrated by control

evaluates assurance

Design guidance

Quantitative guidance

As stated by the sources; verify before use.

Classification

Risks mitigated
Monolithic metric trap

Sources

  1. Ch3.9: T. Nguyen, "Reasoning Quality," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.9. ISBN: 9798244538229.
  2. Ch5.1: T. Nguyen, "Chain-of-Thought (CoT) Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 5.1. ISBN: 9798244538229.
  3. Ch5.3: T. Nguyen, "Self-Consistency Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 5.3. ISBN: 9798244538229.
  4. Ref1.01: NVIDIA, "NVIDIA NeMo Agent Toolkit overview," NVIDIA NeMo Agent Toolkit Documentation, v1.8. Accessed: Sep. 26, 2026. [Online]. Available: https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html