Agentic AI Reference Architecture

Observability & Evaluation · Data store

User Feedback Store

Data storeObservability & EvaluationObservability & Evaluationarc:UserFeedbackStore

A store of explicit user feedback (ratings, comments, issue tags) and implicit behavioural events linked to agent interactions for analysis.

Responsibility. Persists user feedback signals for aggregation and analysis.

Also known as: Click-through behaviour log

is written byis read byis read byis read byis read byis read byis read byis read byis written byis read byis read byis read byis written byis read byis read byis read byis written byis read byFeedback Collector: is written byFeedback CollectorData Curator: is read byData CuratorOnline Evaluator: is read byOnline EvaluatorRegression Gate: is read byRegression GateExperiment Guardrail Monitor: is read byExperiment Guardrail Mon…Quality Drift Detector: is read byQuality Drift DetectorRLHF Policy Optimizer: is read byRLHF Policy OptimizerAlignment Drift Monitor: is read byAlignment Drift MonitorBehavioral Signal Tracker: is written byBehavioral Signal TrackerCompliance Evidence Collector: is read byCompliance Evidence Coll…Override Pattern Analyzer: is read byOverride Pattern AnalyzerTrajectory Harvester: is read byTrajectory HarvesterDecision Override Control: is written byDecision Override ControlFeedback Prioritizer: is read byFeedback PrioritizerRevealed Preference Learner: is read byRevealed Preference Lear…Satisfaction Score Calculator: is read bySatisfaction Score Calcu…User Complaint Intake: is written byUser Complaint IntakeAdoption Metrics Tracker: is read byAdoption Metrics Tracker+4 more (see relationships)
Direct neighbourhood (hover for relationship types)

Relationships

is read by dependency

is written by dependency

Sources

  1. Ch3.2: T. Nguyen, "Compare Agent Performance Across Tasks and Datasets," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.2. ISBN: 9798244538229.
  2. Ch4.1: T. Nguyen, "Introduction to AI Agent Deployment and Scaling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.1. ISBN: 9798244538229.
  3. Ch4.4: T. Nguyen, "Performance Profiling and Optimization," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.4. ISBN: 9798244538229.
  4. Ch5.7: T. Nguyen, "Episodic Memory," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 5.7. ISBN: 9798244538229.
  5. Ch5.8: T. Nguyen, "Semantic Memory," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 5.8. ISBN: 9798244538229.
  6. Ch5.10: T. Nguyen, "Utility-Based Decision Making Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 5.10. ISBN: 9798244538229.
  7. Ch6.4: T. Nguyen, "Data Quality Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.4. ISBN: 9798244538229.
  8. Ch6.5: T. Nguyen, "Production RAG Systems," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.5. ISBN: 9798244538229.
  9. Ch8.3: T. Nguyen, "Token Economics and Architecture," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 8.3. ISBN: 9798244538229.
  10. Ch8.4: T. Nguyen, "Success Metrics and Multi-Dimensional Measurement," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 8.4. ISBN: 9798244538229.
  11. Ch10.1: T. Nguyen, "Conversational UI," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.1. ISBN: 9798244538229.
  12. Ch10.2: T. Nguyen, "Proactive Agents," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.2. ISBN: 9798244538229.
  13. Ch10.3: T. Nguyen, "RLHF Methodology," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.3. ISBN: 9798244538229.
  14. Ch10.4: T. Nguyen, "Human-in-the-Loop," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.4. ISBN: 9798244538229.
  15. Ch10.5: T. Nguyen, "Human-over-the-Loop," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.5. ISBN: 9798244538229.
  16. Ref9.06: "Auditing and Compliance Monitoring for AI Systems," unpublished reference note (references/Chapter 9 - Safety, Ethics, and Compliance/06-Auditing-Compliance-Monitoring.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
  17. Ref10.03: "User Feedback and Iterative Improvement," unpublished reference note (03-User-Feedback-Iteration.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
  18. Ref10.04: "Change Management and Organizational Adoption," unpublished reference note (04-Change-Management-Adoption.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
  19. Ref10.05: "The Data Flywheel: Continuous Improvement Loop," unpublished reference note (05-Data-Flywheel-Continuous-Improvement.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note