Part 8 — Reliability & Cost Management

5 chapters · 10.9 study hours allocated in the Study Plan · 0 slide decks · 2 videos · 29 code example files

On this page

1. Chapters
2. Chapter summaries
   1. 8.1. Latency Fundamentals
   2. 8.2A. Error Taxonomy and SLO
   3. 8.2B. Circuit Breakers and NeMo Integration
   4. 8.3. Token Economics
   5. 8.4. Success Metrics
   6. Additional worked examples
   7. Labs

Chapters

Rating tags show which certification knowledge maps rate the chapter H (highly relevant) in at least one item: NV NCP-AAI · AWS AIP-C01 · DBX Databricks GenAI Engineer · GCP Professional ML Engineer · MS AI-102. See Certifications.

Ch.	Title	Hours	Slides	Quiz	Videos	Figures	Code	H-rated for
8.1	Latency Fundamentals	2.2	—	Quiz	0	5	8	NV AWS GCP
8.2A	Error Taxonomy and SLO	3.5	—	Quiz	2	10	8	NV AWS GCP
8.2B	Circuit Breakers and NeMo Integration	1.2	—	Quiz	0	11	5	NV AWS DBX GCP MS
8.3	Token Economics	3.2	—	Quiz	0	10	8	NV AWS GCP
8.4	Success Metrics	0.8	—	Quiz	0	7	—	NV AWS GCP MS

Notes. The Videos column counts the videos shown under each chapter summary below, out of the unique direct links in Part_08_YoutubeVideos.md (“3 of 5”). A video is left out when its link is dead, embedding is disabled, or YouTube’s title does not match the entry; see the link check. Chapters can also list search suggestions instead of links.

† Linked by chapter-family number, not an exact ID match: the deck, quiz, or figure set is numbered differently from this chapter in the source files (for example a quiz or deck numbered 6.2 for chapters 6.2A and 6.2B).

‡ A combined deck that covers more than one chapter.

A chapter that is missing from a certification’s mapping file shows no tag for that certification: the NVIDIA file omits 4.1 and 10.6, and the other four omit 1.8, 9.16, and 9.17.

Chapter summaries

Summaries are excerpted from Study_Plan.md, which also lists each chapter’s key concepts and self-check questions.

8.1. Latency Fundamentals

Agent latency monitoring requires simultaneous tracking of end-to-end metrics and granular per-step measurements to distinguish between average performance that masks outliers and percentile-based metrics revealing true user experience. From diagnosis through distributed tracing to GPU-level observability, this chapter provides the comprehensive measurement framework necessary for production optimization.

No videos are shown for this chapter: its list has only search suggestions, or its links failed the link check.

Code examples (8 files)

• agent_tracing_code_04_instrumented_agent_execution.py
• inventory_caching_code_05_caching_optimization.py
• metrics_collection_code_08_gpu_metrics_collector.py
• nvidia_dcgm_monitoring_code_06_gpu_metrics_initialization.py
• opentelemetry_imports_code_01_imports_setup.py
• otlp_exporter_setup_code_03_exporter_configuration.py
• prometheus_metrics_code_07_prometheus_gauge_setup.py
• tracer_provider_config_code_02_tracer_initialization.py

8.2A. Error Taxonomy and SLO

This chapter provides a systematic framework for categorizing AI agent failures into three tiers (planning, execution, verification) and using Service Level Objectives (SLOs) with error budgets and burn rate metrics to make reliability-velocity tradeoffs explicit and measurable. It also covers multi-agent coordination failures and distributed tracing techniques for diagnosing invisible failure patterns in concurrent systems.

Videos (2)

Dynatrace Tips & Tricks #15 - SLO Error Budget Burn Rate Based Alerting · Dynatrace

09 - Multi-Threaded Index Concurrency Control (CMU Databases Systems / Fall 2019) · CMU Database Group

Code examples (8 files)

• agent_execution_with_dependency_tracking_code_04_agent_execution_with_dependency_tracking.py
• burn_rate_alerting_configuration_code_02_burn_rate_alerting_configuration.py
• dependency_graph_with_cycle_detection_code_06_dependency_graph_with_cycle_detection.py
• optimistic_locking_with_retry_code_08_optimistic_locking_with_retry.py
• ordered_execution_with_barriers_code_07_ordered_execution_with_barriers.py
• research_orchestrator_instrumentation_code_03_research_orchestrator_instrumentation.py
• shared_workflow_state_tracing_code_05_shared_workflow_state_tracing.py
• tracer_configuration_initialization_code_01_tracer_configuration_initialization.py

8.2B. Circuit Breakers and NeMo Integration

This chapter addresses how circuit breakers prevent cascading failures in distributed systems through fast-fail behavior, and how to categorize production errors into safety violations versus infrastructure failures for proper team escalation and monitoring. The practical focus includes implementing a three-state circuit breaker automaton and designing separate monitoring pipelines that distinguish NeMo Guardrails safety blocks from execution exceptions.

No videos are shown for this chapter: its list has only search suggestions, or its links failed the link check.

Code examples (5 files)

• complete_solution_04_full_circuit_breaker.py
• moderate_hints_02_call_method_structure.py
• nemo_guardrails_integration_05_error_classification.py
• scaffolded_code_01_circuit_breaker_setup.py
• strong_hints_03_half_open_and_closed_logic.py

8.3. Token Economics

Token economics fundamentally shape LLM cost optimization strategies through asymmetric pricing, where output tokens cost 4-5× more than input tokens due to computational differences between single-pass encoding and iterative decoding. This chapter establishes a three-tier monitoring architecture and demonstrates how systematic multi-faceted optimizations can achieve significant cost reductions while maintaining quality metrics.

No videos are shown for this chapter: its list has only search suggestions, or its links failed the link check.

Code examples (8 files)

• feature_level_analysis_code_03_feature_level_analysis.py
• model_routing_complexity_code_08_model_routing_complexity.py
• output_constraint_code_06_output_constraint.py
• output_format_guidance_code_07_output_format_guidance.py
• prompt_caching_implementation_code_04_prompt_caching_implementation.py
• rag_context_retrieval_code_05_rag_context_retrieval.py
• request_level_tracking_code_01_request_level_tracking.py
• token_tracking_infrastructure_code_02_token_tracking_infrastructure.py

8.4. Success Metrics

This chapter explores multi-dimensional measurement of AI agent success through balanced scorecards that track task completion, user satisfaction, efficiency, and safety metrics simultaneously. Rather than optimizing for single metrics in isolation, production systems must measure across complementary dimensions to prevent optimization pathologies that degrade unmeasured but equally important success factors.

Additional worked examples

From more_examples/part_08/:

• custom_dashboard.py
• grafana_dashboard.json

Labs

Chapter 8.2B has the project’s first lab written to the lab template: Build a Circuit Breaker for a Flaky Downstream Tool (status: draft). See Labs.