Observability & Evaluation · Data artifact
A/B Test Configuration
Data artifactObservability & EvaluationObservability & Evaluationarc:ABTestConfig
A configuration specifying variant traffic allocation, experiment metrics, statistical design parameters and automatic rollback thresholds for an online experiment.
Responsibility. Declares allocation and guardrail thresholds for an online agent experiment.
Also known as: Canary rollout plan
Relationships
configures structural
is written by dependency
- Rollout Manager abstract Ch3.1A Ch4.2
Quantitative guidance
As stated by the sources; verify before use.
- Rollback if treatment TSR >5% below control, error rate >10%, or satisfaction drops >0.5 on a 5-point scale (Ch3.1A).
- Canary rollback tolerances: success rate 3% below stable, p95 latency +2 s, or quality score -0.1; per-metric quality tolerance 3% (Ch4.2).
Sources
- Ch3.1A: T. Nguyen, "Implement Evaluation Pipelines and Task Benchmarks," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.1A. ISBN: 9798244538229.
- Ch4.2: T. Nguyen, "Deployment and Scaling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.2. ISBN: 9798244538229.
- Ch6.5: T. Nguyen, "Production RAG Systems," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.5. ISBN: 9798244538229.