Agentic AI Reference Architecture

Observability & Evaluation · Software component

Agent Version Experimenter

Software componentObservability & EvaluationObservability & EvaluationVariation point (abstract)arc:AgentVersionExperimenter

An abstract production experimentation component that compares a candidate agent version with the current production version on real traffic before full deployment.

Responsibility. Compares candidate and production agent versions on production traffic.

emits telemetry tois specialized byis specialized byMetrics Collector: emits telemetry toMetrics CollectorA/B Test Traffic Splitter: is specialized byA/B Test Traffic SplitterShadow Test Runner: is specialized byShadow Test Runner
Direct neighbourhood (hover for relationship types)

Variants

VariantWhen to choose
A/B Test Traffic SplitterChoose when comparing versions on live outcome metrics with users actually served by the new version, controlling for temporal and population effects.
Shadow Test RunnerChoose when a new version must be assessed on production inputs before committing to deployment, without executing its actions.

Relationships

emits telemetry to dynamic

Design guidance

Classification

Technologies
NVIDIA NeMoGalileo
Quality attributes
Safety (ISO/IEC 25010 | NIST AI RMF: safe)Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)
Risks mitigated
Regressions reaching all users

Sources

  1. Ch3.3: T. Nguyen, "Web Navigation and Interaction Benchmarks," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.3. ISBN: 9798244538229.