Observability & Evaluation · Data artifact
Operational Runbook
Data artifactObservability & EvaluationObservability & Evaluationarc:OperationalRunbook
A documented diagnosis-and-mitigation procedure for a recurring operational issue, such as high latency or high error rate, that branches by bottleneck or error type to specific checks and actions.
Responsibility. Prescribes diagnosis steps and mitigations for a known issue type.
Also known as: Runbook, Documented procedures, AI incident response procedure, Incident response plan
Relationships
configures structural
is read by dependency
Design guidance
- SHOULD maintain runbooks for common issues that start with a short diagnosis (about 2 minutes) identifying the bottleneck step or error type.
- SHOULD specify who is notified, who investigates, immediate actions, affected-party communication and recovery steps.
- SHOULD document the incident response plan and exercise it regularly with the on-call team.
Quantitative guidance
As stated by the sources; verify before use.
- High-latency runbook triggers at > 10 s response latency; high-error-rate runbook at > 2% error rate (Ref8.06).
Classification
- Patterns
- Runbook-driven incident response
- Technologies
- LangSmith
- Quality attributes
- Reliability (ISO/IEC 25010 | NIST AI RMF: valid and reliable)Maintainability (ISO/IEC 25010)
- Risks mitigated
- Slow, ad-hoc incident diagnosis
- Frameworks & regulations
- NIST AI RMF: MANAGEISO/IEC 42001 Annex A: incident management controls
Sources
- Ch9.8: T. Nguyen, "Standards and Frameworks for AI Governance," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.8. ISBN: 9798244538229.
- Ref8.06: "Error Troubleshooting and Incident Response for Agent Systems," unpublished reference note (06-Error-Troubleshooting-Incident-Response.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
- Ref9.08: "Safety Incident Response for AI Systems," unpublished reference note (08-Safety-Incident-Response.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
- Ref9.10: "Chapter 9 Summary: Safety, Ethics, and Compliance," unpublished reference note (10-Chapter-9-Summary.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note