Human Oversight · Human role
Preference Annotator
Human roleHuman OversightExperience & Human OversightVariation point (abstract)arc:PreferenceAnnotator
A human evaluator who compares pairs of alternative agent responses and indicates which better satisfies specified criteria such as helpfulness, tone, accuracy or policy compliance.
Responsibility. Expresses human preferences through pairwise comparative judgments.
Also known as: Human evaluator, Human rater, Preference labeler, Fairness annotator, Cross-agent outcome annotator, Human annotator
Variants
| Variant | When to choose |
|---|---|
| Domain Expert Annotator | Choose for specialized domains (healthcare, finance, autonomous vehicles, legal) where judging clinical accuracy, regulatory compliance or safety requires expertise; costs more than general annotators. |
| General Preference Annotator | Choose for general-purpose content where volume and cost matter; unsuitable for specialized medical, legal, financial or safety-critical content. |
Relationships
is configured by structural
invokes dependency
sends data to dynamic
evaluates assurance
- Decision Engine abstract Ch9.6
is evaluated by assurance
is monitored by assurance
Design guidance
- SHOULD be drawn from diverse pools representing target user populations, with training on common bias patterns.
- SHOULD judge specific dimensions rather than generic overall quality.
- SHOULD be drawn from a diverse pool; training on one annotator pool embeds that pool's biases.
- SHOULD be drawn from a demographically and culturally diverse pool, since the reward model reflects the annotator population's values rather than universal values.
- SHOULD be rotated with adequate rest to prevent fatigue-induced inconsistency.
Quantitative guidance
As stated by the sources; verify before use.
- Inter-rater agreement can be as low as 60-70% on some tasks (Ch3.5).
- Production RLHF operations employ hundreds to thousands of annotators (Ch10.3).
- Scaling an annotation team from 50 to 500 annotators grew data ten-fold but degraded downstream model performance through fatigue, guideline drift, expertise gaps and reduced diversity (Ch10.3 case).
Classification
- Patterns
- Pairwise preference judgmentCriterion-specific annotation
- Quality attributes
- Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)
- Risks mitigated
- Inconsistent absolute ratings
Sources
- Ch3.5: T. Nguyen, "Prompt Optimization, Few-Shot Learning, Fine-Tuning," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.5. ISBN: 9798244538229.
- Ch9.5: T. Nguyen, "Constitutional AI," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.5. ISBN: 9798244538229.
- Ch9.6: T. Nguyen, "Value Alignment Frameworks," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.6. ISBN: 9798244538229.
- Ch10.3: T. Nguyen, "RLHF Methodology," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.3. ISBN: 9798244538229.
- Ch10.5: T. Nguyen, "Human-over-the-Loop," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.5. ISBN: 9798244538229.