Observability & Evaluation · Software component

Metrics Dashboard

Software componentObservability & EvaluationObservability & EvaluationVariation point (abstract)arc:MetricsDashboard

Variants

Relationships

Design guidance

• SHOULD organise dashboards around operational questions (SLA attainment, error concentration, capacity, cost trend) rather than technical components.
• SHOULD include LLM-specific views: latency, token efficiency, cost, vector query time, GPU utilization and MCP health.
• SHOULD correlate replica count with request rate, utilization with targets, and P95 latency with SLOs to validate autoscaling.
• SHOULD show request rate, P50/P95/P99 latency, per-GPU utilization, error rate vs SLA line, tokens/s, CPU/memory/network and cost panels.
• SHOULD provide audience-specific executive, operations and optimization dashboards.
• SHOULD correlate GPU hardware metrics with application-level latency on the same view to separate GPU saturation from other causes.
• SHOULD place safety and infrastructure panels separately, each with thresholds matching its own SLO.

Quantitative guidance

As stated by the sources; verify before use.

• Typical production NIM dashboard has 7 panels (Ch7.2).
• Infrastructure panel thresholds 0.1% green / 0.2% yellow (2x burn) / 0.5% red (5x burn); safety panel 1% green / 2% yellow / 3% red (Ch8.2B).
• Klarna tracks 40+ performance metrics in real-time dashboards (Ch10.1).

Classification

Patterns

Question-oriented dashboardsTask lifecycle dashboardAgent dependency graphPre/post-deployment comparisonCorrelated replica-count / request-rate / utilization / P95 panels

Technologies

GrafanaDatadogNVIDIA GPU Dashboard (Grafana ID 12239)LangSmithDataDogNew Relic

Quality attributes

Maintainability (ISO/IEC 25010)

Risks mitigated

Slow incident root-cause analysisUndetected scaling oscillationMis-tuned scaling thresholds

Sources

1. Ch4.1: T. Nguyen, "Introduction to AI Agent Deployment and Scaling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.1. ISBN: 9798244538229.
2. Ch4.3: T. Nguyen, "Container Orchestration and Edge Deployment," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.3. ISBN: 9798244538229.
3. Ch4.5: T. Nguyen, "NVIDIA NIM and Triton Inference Server," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.5. ISBN: 9798244538229.
4. Ch4.7: T. Nguyen, "Scaling Strategies," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.7. ISBN: 9798244538229.
5. Ch6.2B: T. Nguyen, "Production Vector Database Deployment," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.2B. ISBN: 9798244538229.
6. Ch6.4: T. Nguyen, "Data Quality Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.4. ISBN: 9798244538229.
7. Ch6.5: T. Nguyen, "Production RAG Systems," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.5. ISBN: 9798244538229.
8. Ch7.2: T. Nguyen, "Performance Optimization and Production Monitoring," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 7.2. ISBN: 9798244538229.
9. Ch7.3: T. Nguyen, "NeMo Agent Toolkit Profiling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 7.3. ISBN: 9798244538229.
10. Ch8.1: T. Nguyen, "Latency Metrics," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 8.1. ISBN: 9798244538229.
11. Ch8.2A: T. Nguyen, "Error Rates and Reliability," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 8.2A. ISBN: 9798244538229.
12. Ch8.2B: T. Nguyen, "NeMo Guardrails Integration," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 8.2B. ISBN: 9798244538229.
13. Ch8.3: T. Nguyen, "Token Economics and Architecture," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 8.3. ISBN: 9798244538229.
14. Ch9.2: T. Nguyen, "Action Constraints and Permission Models," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.2. ISBN: 9798244538229.
15. Ch9.4: T. Nguyen, "Fairness and Bias Mitigation," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.4. ISBN: 9798244538229.
16. Ch10.1: T. Nguyen, "Conversational UI," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.1. ISBN: 9798244538229.
17. Ch10.5: T. Nguyen, "Human-over-the-Loop," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.5. ISBN: 9798244538229.
18. Ref4.05: NVIDIA, "Setting up Prometheus," NVIDIA GPU Telemetry Documentation. Accessed: Sep. 27, 2026. [Online]. Available: https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/kube-prometheus.html
19. Ref7.15: "Advanced Agentic AI Optimization Techniques," unpublished reference note (15-Advanced-Agentic-Optimization.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
20. Ref7.16: "Production Monitoring and Operations for Agentic AI," unpublished reference note (16-Production-Monitoring-Operations.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
21. Ref7.18: "Chapter 7 Summary: NVIDIA Platform Implementation," unpublished reference note (18-Chapter-7-Summary.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
22. Ref8.01: LangChain, "LangSmith observability: AI agent observability platform," LangChain. Accessed: Sep. 27, 2026. [Online]. Available: https://www.langchain.com/langsmith/observability
23. Ref8.04: "Data Quality and Drift Detection for Agent Systems," unpublished reference note (04-Data-Quality-Drift-Detection.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
24. Ref8.07: "Agent Health Checks and Diagnostics," unpublished reference note (07-Agent-Health-Checks-Diagnostics.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
25. Ref8.09: "Chapter 8 Summary: Run, Monitor, and Maintain," unpublished reference note (09-Chapter-8-Summary.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
26. Ref9.06: "Auditing and Compliance Monitoring for AI Systems," unpublished reference note (references/Chapter 9 - Safety, Ethics, and Compliance/06-Auditing-Compliance-Monitoring.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
27. Ref9.07: "Risk Assessment and Management for AI Systems," unpublished reference note (references/Chapter 9 - Safety, Ethics, and Compliance/07-Risk-Assessment-Management.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
28. Ref9.09: "Compliance Automation and Tools," unpublished reference note (09-Compliance-Automation-Tools.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
29. Ref9.10: "Chapter 9 Summary: Safety, Ethics, and Compliance," unpublished reference note (10-Chapter-9-Summary.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note