Agentic AI Reference Architecture

Human Oversight · Human role

Domain Expert Annotator

Human roleHuman OversightExperience & Human Oversightarc:DomainExpertAnnotator

A human domain expert who demonstrates optimal task behaviour step by step, supplies seed examples and validates samples of machine-generated trajectories and preference annotations.

Responsibility. Provides expert-quality supervision for adaptation data.

Also known as: Manual annotation, Expert reviewer layer, Human reviewer, Expert demonstrator, DAgger expert labeller, Expert preference annotator, Demonstration author

Variant of Preference Annotator abstract

When to choose. Choose for specialized domains (healthcare, finance, autonomous vehicles, legal) where judging clinical accuracy, regulatory compliance or safety requires expertise; costs more than general annotators.

sends data to; evaluatesevaluatesspecializessends data toreceives data fromreceives data fromsends data toreceives data fromalternative tosends data toreceives escalation fromAgent Trajectory Dataset: sends data to; evaluatesAgent Trajectory DatasetPreference Dataset: evaluatesPreference DatasetPreference Annotator: specializesPreference AnnotatorSynthetic Data Generator: sends data toSynthetic Data GeneratorDAgger Trainer: receives data fromDAgger TrainerInverse Reward Learner: receives data fromInverse Reward LearnerExpert Demonstration Dataset: sends data toExpert Demonstration Dat…Neural Knowledge Extractor: receives data fromNeural Knowledge ExtractorGeneral Preference Annotator: alternative toGeneral Preference Annot…Instruction Demonstration Dataset: sends data toInstruction Demonstratio…Knowledge Gap Detector: receives escalation fromKnowledge Gap Detector
Direct neighbourhood (hover for relationship types)

Relationships

receives data from dynamic

receives escalation from dynamic

sends data to dynamic

evaluates assurance

alternative to variability

Design guidance

Quantitative guidance

As stated by the sources; verify before use.

Classification

Patterns
Expert demonstrationSample-based human validation
Quality attributes
Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)
Risks mitigated
Errors in LLM-generated training dataLow-quality annotations

Sources

  1. Ch3.5: T. Nguyen, "Prompt Optimization, Few-Shot Learning, Fine-Tuning," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.5. ISBN: 9798244538229.
  2. Ch5.10: T. Nguyen, "Utility-Based Decision Making Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 5.10. ISBN: 9798244538229.
  3. Ch5.12: T. Nguyen, "Learning-Based Decision Making Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 5.12. ISBN: 9798244538229.
  4. Ch5.13: T. Nguyen, "Hybrid Decision Systems Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 5.13. ISBN: 9798244538229.
  5. Ch10.3: T. Nguyen, "RLHF Methodology," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.3. ISBN: 9798244538229.