Agentic AI Reference Architecture

Observability & Evaluation · Software component

Experiment Tracker

Software componentObservability & EvaluationObservability & Evaluationarc:ExperimentTracker

A tracking service that records each evaluation run's agent configuration, metrics and artifacts so runs are reproducible and comparable over time.

Responsibility. Logs evaluation runs and makes their history queryable for comparison.

Also known as: Experiment tracking, Experiment tracking platform, MLflow experiments and runs

receives data from; is invoked byis invoked bywritesreadsis invoked bywritesreceives data fromEvaluation Harness: receives data from; is invoked byEvaluation HarnessContinuous Integration Runner: is invoked byContinuous Integration R…Model and Agent Release Registry: writesModel and Agent Release …Configuration Repository: readsConfiguration RepositoryRelease Approver: is invoked byRelease ApproverEvaluation Result Store: writesEvaluation Result StoreEvaluation Score Aggregator: receives data fromEvaluation Score Aggrega…
Direct neighbourhood (hover for relationship types)

Relationships

is invoked by dependency

reads dependency

writes dependency

receives data from dynamic

Design guidance

Classification

Patterns
Consistent metric naming prefixes (e.g., custom/) and tags for filtering
Technologies
MLflowWeights & BiasesWeaveMLflow Tracking
Quality attributes
Reliability (ISO/IEC 25010 | NIST AI RMF: valid and reliable)Transparency and accountability (NIST AI RMF: accountable and transparent)
Risks mitigated
Ephemeral experiments whose conclusions cannot be validated

Sources

  1. Ch3.1A: T. Nguyen, "Implement Evaluation Pipelines and Task Benchmarks," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.1A. ISBN: 9798244538229.
  2. Ch3.1B: T. Nguyen, "Implement Evaluation Pipelines and Task Benchmarks - Guided Practice," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.1B. ISBN: 9798244538229.
  3. Ch3.7: T. Nguyen, "Tool Usage Auditing," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.7. ISBN: 9798244538229.
  4. Ch4.4: T. Nguyen, "Performance Profiling and Optimization," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.4. ISBN: 9798244538229.
  5. Ref3.07: NVIDIA, "NeMo-Agent-Toolkit," GitHub repository. Accessed: Sep. 27, 2026. [Online]. Available: https://github.com/NVIDIA/NeMo-Agent-Toolkit