Agentic AI Reference Architecture

Human Oversight · Software component

Preference Annotation Console

Software componentHuman OversightExperience & Human Oversightarc:PreferenceAnnotationConsole

A structured comparison interface that presents annotators with randomized pairs of candidate responses and captures criterion-specific preference judgments.

Responsibility. Captures pairwise human preference judgments.

Also known as: Structured comparison interface, Annotation interface, Distributed annotation platform

writesis invoked byreceives data fromis configured byis configured byis routed to byis invoked bysends data toPreference Dataset: writesPreference DatasetPreference Annotator: is invoked byPreference AnnotatorCandidate Response Sampler: receives data fromCandidate Response SamplerAnnotation Task Format: is configured byAnnotation Task FormatAnnotation Guideline: is configured byAnnotation GuidelineAnnotation Task Router: is routed to byAnnotation Task RouterSenior Annotation Reviewer: is invoked bySenior Annotation ReviewerPreference Label Aggregator: sends data toPreference Label Aggrega…
Direct neighbourhood (hover for relationship types)

Relationships

is configured by structural

is invoked by dependency

writes dependency

is routed to by dynamic

receives data from dynamic

sends data to dynamic

Design guidance

Classification

Technologies
Crowd-sourcing platformsNVIDIA NeMo
Quality attributes
Interaction capability (ISO/IEC 25010)Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)
Risks mitigated
Anchor bias from fixed presentation order

Sources

  1. Ch3.5: T. Nguyen, "Prompt Optimization, Few-Shot Learning, Fine-Tuning," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.5. ISBN: 9798244538229.
  2. Ch9.5: T. Nguyen, "Constitutional AI," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.5. ISBN: 9798244538229.
  3. Ch10.3: T. Nguyen, "RLHF Methodology," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.3. ISBN: 9798244538229.