Observability & Evaluation · Software component

Shadow Test Runner

Software componentObservability & EvaluationObservability & Evaluationarc:ShadowTestRunner

A version experimenter that runs a new agent version silently alongside production, logging what it would have done without executing those actions.

Responsibility. Executes candidate versions side-effect-free on mirrored production inputs.

Also known as: Shadow testing, Shadow deployment

Variant of Agent Version Experimenter abstract

When to choose. Choose when a new version must be assessed on production inputs before committing to deployment, without executing its actions.

Direct neighbourhood (hover for relationship types)

Relationships

invokes dependency

Agent Controller abstract Ch3.3

writes dependency

Trace Store Ch3.3

alternative to variability

A/B Test Traffic Splitter Ch3.3

Design guidance

SHOULD compare shadow outputs with production outputs and ground truth before deployment.

Classification

Patterns: Shadow mode
Quality attributes: Safety (ISO/IEC 25010 | NIST AI RMF: safe)
Risks mitigated: Side effects from untested versions

Sources

Ch3.3: T. Nguyen, "Web Navigation and Interaction Benchmarks," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.3. ISBN: 9798244538229.