Agentic AI Reference Architecture

Observability & Evaluation · Human role

Agent Developer

Human roleObservability & EvaluationObservability & Evaluationarc:AgentDeveloper

An engineer who builds agents and investigates their failures through trace-first forensic debugging, then applies targeted fixes to prompts, tool descriptions and state logic.

Responsibility. Diagnoses agent failures from traces and drives targeted remediation.

Also known as: Debugging engineer, Agent engineer, Reward engineer, Utility function designer, ML team

receives data from; receives escalation frominvokesevaluatesinvokestriggersinvokesinvokessends data toinvokesinvokesreceives data frominvokesreceives data fromreceives data frominvokesAlert Manager: receives data from; receives escalation fromAlert ManagerEvaluation Harness: invokesEvaluation HarnessReasoning Engine: evaluatesReasoning EngineTrace Collector: invokesTrace CollectorContinuous Integration Runner: triggersContinuous Integration R…Execution Profiler: invokesExecution ProfilerGPU System Profiler: invokesGPU System ProfilerUtility Function Specification: sends data toUtility Function Specifi…Bottleneck Analyzer: invokesBottleneck AnalyzerWorkflow Execution Debugger: invokesWorkflow Execution Debug…Deployment Notifier: receives data fromDeployment NotifierTrace Visualizer: invokesTrace VisualizerOptimization Recommender: receives data fromOptimization RecommenderPerformance Trend Analyzer: receives data fromPerformance Trend AnalyzerTrace Pattern Miner: invokesTrace Pattern Miner
Direct neighbourhood (hover for relationship types)

Relationships

invokes dependency

receives data from dynamic

receives escalation from dynamic

sends data to dynamic

triggers dynamic

evaluates assurance

Design guidance

Classification

Patterns
Trace-first debuggingFive-step trace investigation workflowWeekly trace review sessions
Quality attributes
Maintainability (ISO/IEC 25010)
Risks mitigated
Fixing symptoms rather than root causes

Sources

  1. Ch3.6: T. Nguyen, "Trace Analysis and Execution Debugging," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.6. ISBN: 9798244538229.
  2. Ch4.1: T. Nguyen, "Introduction to AI Agent Deployment and Scaling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.1. ISBN: 9798244538229.
  3. Ch4.2: T. Nguyen, "Deployment and Scaling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.2. ISBN: 9798244538229.
  4. Ch4.4: T. Nguyen, "Performance Profiling and Optimization," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.4. ISBN: 9798244538229.
  5. Ch5.10: T. Nguyen, "Utility-Based Decision Making Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 5.10. ISBN: 9798244538229.
  6. Ch7.3: T. Nguyen, "NeMo Agent Toolkit Profiling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 7.3. ISBN: 9798244538229.
  7. Ch8.2B: T. Nguyen, "NeMo Guardrails Integration," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 8.2B. ISBN: 9798244538229.
  8. Ch10.5: T. Nguyen, "Human-over-the-Loop," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.5. ISBN: 9798244538229.
  9. Ref8.01: LangChain, "LangSmith observability: AI agent observability platform," LangChain. Accessed: Sep. 27, 2026. [Online]. Available: https://www.langchain.com/langsmith/observability