Observability & Evaluation · Data artifact
Evaluation Baseline
Data artifactObservability & EvaluationObservability & Evaluationarc:EvaluationBaseline
The recorded metrics of the current production agent on the evaluation dataset, with acceptable ranges, serving as the comparison target for every change.
Responsibility. Defines the reference performance that candidate agent versions are compared against.
Also known as: Production baseline, Baseline metrics, Control condition, Baseline performance distribution, Efficiency baseline, Hallucination baseline, FP16 accuracy baseline
Relationships
is read by dependency
- Agent Hyperparameter Optimizer Ch3.10
- Alert Manager Ch3.10
- Alignment Drift Monitor Ch9.5
- Canary Rollout Controller Ch4.1
- Continuous Compliance Monitor Ref9.06
- Evaluation Harness Ch3.7 Ch7.3 +1
- Rollout Manager abstract Ch3.10
- Online Evaluator Ref8.03
- Quality Drift Detector Ch3.10 Ch10.5
- Regression Gate Ch3.1A Ch3.1B +1
- Statistical Comparator Ref10.05
is written by dependency
is produced by lifecycle
Design guidance
- MUST compare candidates to the production baseline rather than judging absolute metrics in isolation.
- SHOULD be version-controlled alongside the regression thresholds.
- SHOULD be measured multiple times with identical random seeds where possible to establish baseline variability.
- SHOULD be recalibrated continuously (e.g., quarterly) from production data stratified by query type.
Quantitative guidance
As stated by the sources; verify before use.
- Three baseline runs of 87%, 89% and 88% imply about +/-2% stochastic fluctuation; a 1% drop is within noise, a 5% drop signals contribution (Ch3.7).
- Loan baseline 8,500 tokens/application verified a 62% reduction to 3,200 tokens (Ch3.10).
Classification
- Patterns
- Repeated measurement with fixed seeds
- Quality attributes
- Reliability (ISO/IEC 25010 | NIST AI RMF: valid and reliable)Maintainability (ISO/IEC 25010)
- Risks mitigated
- Mistaking stochastic fluctuation for component contributionConfiguration driftUnverifiable optimisation claimsStale baselines missing drift
Sources
- Ch3.1A: T. Nguyen, "Implement Evaluation Pipelines and Task Benchmarks," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.1A. ISBN: 9798244538229.
- Ch3.1B: T. Nguyen, "Implement Evaluation Pipelines and Task Benchmarks - Guided Practice," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.1B. ISBN: 9798244538229.
- Ch3.7: T. Nguyen, "Tool Usage Auditing," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.7. ISBN: 9798244538229.
- Ch3.10: T. Nguyen, "Efficiency Metrics," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.10. ISBN: 9798244538229.
- Ch4.1: T. Nguyen, "Introduction to AI Agent Deployment and Scaling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.1. ISBN: 9798244538229.
- Ch7.3: T. Nguyen, "NeMo Agent Toolkit Profiling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 7.3. ISBN: 9798244538229.
- Ch7.4: T. Nguyen, "TensorRT-LLM Fundamentals and Quantization," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 7.4. ISBN: 9798244538229.
- Ch9.5: T. Nguyen, "Constitutional AI," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.5. ISBN: 9798244538229.
- Ch10.3: T. Nguyen, "RLHF Methodology," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.3. ISBN: 9798244538229.
- Ch10.5: T. Nguyen, "Human-over-the-Loop," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.5. ISBN: 9798244538229.
- Ref3.10: "Powering the Next Generation of AI Agents," unpublished reference note (10-Powering-Next-Generation-AI-Agents.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
- Ref8.03: "Agent Evaluation Frameworks and Metrics," unpublished reference note (03-Agent-Evaluation-Frameworks.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
- Ref9.06: "Auditing and Compliance Monitoring for AI Systems," unpublished reference note (references/Chapter 9 - Safety, Ethics, and Compliance/06-Auditing-Compliance-Monitoring.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
- Ref10.05: "The Data Flywheel: Continuous Improvement Loop," unpublished reference note (05-Data-Flywheel-Continuous-Improvement.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note