Agentic AI Reference Architecture

Observability & Evaluation · Data store

Trace Store

Data storeObservability & EvaluationObservability & Evaluationarc:TraceStore

A store of captured agent execution traces that supports technical-view explanations, debugging, and audit investigation.

Responsibility. Persists traces for analysis.

Also known as: Production query logs, Tool call log, Execution traces, Shadow run log, Observability backend, Centralized trace storage, Tool audit logs, Persistent reasoning trace storage, Tracing workspace, Reasoning trace log

is written by; is read byis written by; is read byreceives telemetry from; is written byis read byis written byis read byis read byis read byis read byis read byis audited byis read byis read bysends data toreceives telemetry fromis read byis read byis read byEvaluation Harness: is written by; is read byEvaluation HarnessEvaluation Trace Sampler: is written by; is read byEvaluation Trace SamplerTrace Exporter: receives telemetry from; is written byTrace ExporterMetrics Collector: is read byMetrics CollectorTrace Collector: is written byTrace CollectorLLM Judge: is read byLLM JudgeExplanation Presenter: is read byExplanation PresenterData Curator: is read byData CuratorOnline Evaluator: is read byOnline EvaluatorToken Cost Meter: is read byToken Cost MeterCompliance Officer: is audited byCompliance OfficerExecution Profiler: is read byExecution ProfilerCitation Verifier: is read byCitation VerifierEvaluation Dataset: sends data toEvaluation DatasetProactive Agent: receives telemetry fromProactive AgentSLO Monitor: is read bySLO MonitorEvaluation Failure Analyzer: is read byEvaluation Failure Analy…Behavioral Signal Tracker: is read byBehavioral Signal Tracker+39 more (see relationships)
Direct neighbourhood (hover for relationship types)

Relationships

is configured by structural

is read by dependency

is written by dependency

receives telemetry from dynamic

sends data to dynamic

is constrained by control

is audited by assurance

Design guidance

Quantitative guidance

As stated by the sources; verify before use.

Classification

Patterns
Historical baseline comparisonPost-mortem trace archive
Technologies
PhoenixWeaveLangfuseDatadogJaegerZipkinGrafanaHoneycombLangSmith
Quality attributes
Maintainability (ISO/IEC 25010)Transparency and accountability (NIST AI RMF: accountable and transparent)
Risks mitigated
Unbounded trace storage costPrivacy exposure from full trace retention

Sources

  1. Ch1.1A: T. Nguyen, "Designing User Interfaces for Intuitive Human-Agent Interaction," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.1A. ISBN: 9798244538229.
  2. Ch1.2: T. Nguyen, "Core Agent Patterns," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.2. ISBN: 9798244538229.
  3. Ch1.3: T. Nguyen, "Multi-Agent Systems," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.3. ISBN: 9798244538229.
  4. Ch1.8: T. Nguyen, "Scalability and Production Deployment," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.8. ISBN: 9798244538229.
  5. Ch3.1A: T. Nguyen, "Implement Evaluation Pipelines and Task Benchmarks," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.1A. ISBN: 9798244538229.
  6. Ch3.1B: T. Nguyen, "Implement Evaluation Pipelines and Task Benchmarks - Guided Practice," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.1B. ISBN: 9798244538229.
  7. Ch3.2: T. Nguyen, "Compare Agent Performance Across Tasks and Datasets," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.2. ISBN: 9798244538229.
  8. Ch3.3: T. Nguyen, "Web Navigation and Interaction Benchmarks," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.3. ISBN: 9798244538229.
  9. Ch3.4: T. Nguyen, "Tuning Model Parameters for Production Performance," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.4. ISBN: 9798244538229.
  10. Ch3.5: T. Nguyen, "Prompt Optimization, Few-Shot Learning, Fine-Tuning," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.5. ISBN: 9798244538229.
  11. Ch3.6: T. Nguyen, "Trace Analysis and Execution Debugging," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.6. ISBN: 9798244538229.
  12. Ch3.7: T. Nguyen, "Tool Usage Auditing," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.7. ISBN: 9798244538229.
  13. Ch3.8: T. Nguyen, "Action Accuracy Metrics," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.8. ISBN: 9798244538229.
  14. Ch3.9: T. Nguyen, "Reasoning Quality," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.9. ISBN: 9798244538229.
  15. Ch3.10: T. Nguyen, "Efficiency Metrics," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.10. ISBN: 9798244538229.
  16. Ch4.2: T. Nguyen, "Deployment and Scaling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.2. ISBN: 9798244538229.
  17. Ch4.5: T. Nguyen, "NVIDIA NIM and Triton Inference Server," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.5. ISBN: 9798244538229.
  18. Ch6.5: T. Nguyen, "Production RAG Systems," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.5. ISBN: 9798244538229.
  19. Ch8.1: T. Nguyen, "Latency Metrics," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 8.1. ISBN: 9798244538229.
  20. Ch8.2A: T. Nguyen, "Error Rates and Reliability," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 8.2A. ISBN: 9798244538229.
  21. Ch9.1: T. Nguyen, "Output Filtering and Content Moderation," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.1. ISBN: 9798244538229.
  22. Ch10.1: T. Nguyen, "Conversational UI," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.1. ISBN: 9798244538229.
  23. Ch10.2: T. Nguyen, "Proactive Agents," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.2. ISBN: 9798244538229.
  24. Ch10.3: T. Nguyen, "RLHF Methodology," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.3. ISBN: 9798244538229.
  25. Ch10.4: T. Nguyen, "Human-in-the-Loop," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.4. ISBN: 9798244538229.
  26. Ch10.5: T. Nguyen, "Human-over-the-Loop," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.5. ISBN: 9798244538229.
  27. Ref1.01: NVIDIA, "NVIDIA NeMo Agent Toolkit overview," NVIDIA NeMo Agent Toolkit Documentation, v1.8. Accessed: Sep. 26, 2026. [Online]. Available: https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html
  28. Ref7.16: "Production Monitoring and Operations for Agentic AI," unpublished reference note (16-Production-Monitoring-Operations.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
  29. Ref8.01: LangChain, "LangSmith observability: AI agent observability platform," LangChain. Accessed: Sep. 27, 2026. [Online]. Available: https://www.langchain.com/langsmith/observability
  30. Ref8.06: "Error Troubleshooting and Incident Response for Agent Systems," unpublished reference note (06-Error-Troubleshooting-Incident-Response.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
  31. Ref10.05: "The Data Flywheel: Continuous Improvement Loop," unpublished reference note (05-Data-Flywheel-Continuous-Improvement.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note