Observability & Evaluation · Data artifact
Evaluation Protocol
Data artifactObservability & EvaluationObservability & Evaluationarc:EvaluationProtocol
A prospectively documented specification of controlled-comparison conditions, trial count and seeds, sample-size requirements, statistical tests, and statistical and practical significance thresholds for an evaluation.
Responsibility. Fixes comparison conditions and decision criteria before an experiment runs.
Also known as: Pre-registered evaluation plan, Controlled comparison methodology
Relationships
configures structural
Design guidance
- MUST hold all non-tested variables constant (dataset, model and hyperparameters, retrieval corpus, retry and error policies, prompts and few-shot examples).
- SHOULD run multiple trials with different random seeds and paired examples for each condition.
- MUST specify significance thresholds, practical improvement thresholds and tests before experiments begin.
Quantitative guidance
As stated by the sources; verify before use.
- Practical significance may be ~0.5% at millions of monthly interactions but 5-10% in healthcare workflows (Ch3.2).
Classification
- Risks mitigated
- Confounded comparisonsRetrospective cherry-picking of statistical tests
Sources
- Ch3.2: T. Nguyen, "Compare Agent Performance Across Tasks and Datasets," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.2. ISBN: 9798244538229.