Observability & Evaluation · Software component
Production Quality Monitor
Software componentObservability & EvaluationObservability & EvaluationVariation point (abstract)arc:ProductionQualityMonitor
An abstract monitoring component that computes model-quality, drift and data-quality metrics over production predictions against a reference baseline and raises alerts on anomalies, compensating for silent failures and delayed ground truth.
Responsibility. Computes production model-quality and drift metrics and alerts on anomalies.
Also known as: ML monitoring, Model monitoring job
Variants
| Variant | When to choose |
|---|---|
| Batch Quality Monitor | Choose for non-real-time applications, cost-sensitive monitoring and historical analysis. |
| Streaming Quality Monitor | Choose for critical or high-volume applications that need real-time alerts. |
Relationships
invokes dependency
writes dependency
sends data to dynamic
triggers dynamic
is evaluated by assurance
monitors assurance
Design guidance
- SHOULD select primary, proxy (when ground truth is delayed), leading-indicator and business metrics, and track performance by segment.
- SHOULD define absolute, relative and statistical-significance alert thresholds with severity levels, and tune them to reduce false alerts.
- MAY combine real-time basic metrics with batch deep analysis (hybrid approach).
- SHOULD monitor agent decision correctness, tool calling accuracy, interaction success and goal achievement for agentic systems.
Classification
- Patterns
- Proxy metrics under delayed ground truthReference-baseline comparison
- Technologies
- Evidently AIArizeMonte CarloDatadogPrometheus + Grafana
- Quality attributes
- Reliability (ISO/IEC 25010 | NIST AI RMF: valid and reliable)Maintainability (ISO/IEC 25010)
- Risks mitigated
- Silent model failuresConcept driftData distribution shiftDelayed ground truth
- Frameworks & regulations
- NIST AI RMF: MEASUREISO/IEC 42001 Annex A: monitoring controls
Sources
- Ch9.8: T. Nguyen, "Standards and Frameworks for AI Governance," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.8. ISBN: 9798244538229.
- Ref8.02: "Machine Learning Monitoring in Production," unpublished reference note (02-ML-Monitoring-Production.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note