Annotation Guideline

Data artifactHuman OversightExperience & Human Oversightarc:AnnotationGuideline

A version-controlled specification of the evaluation criteria, dimensions and instructions that tell annotators precisely what to judge when comparing responses.

Responsibility. Defines what preference annotators judge and how.

Also known as: Evaluation criteria, Annotation instructions

Direct neighbourhood (hover for relationship types)

Relationships

configures structural

Preference Annotation Console Ch3.5 Ch10.3
Preference Annotator abstract Ch3.5 Ch10.3

receives data from dynamic

Senior Annotation Reviewer Ch10.3

Design guidance

SHOULD be kept under version control and refined as understanding improves.
SHOULD be refined through regular calibration sessions and analysis of annotator disagreement patterns.

Classification

Quality attributes: Reliability (ISO/IEC 25010 | NIST AI RMF: valid and reliable)

Sources

Ch3.5: T. Nguyen, "Prompt Optimization, Few-Shot Learning, Fine-Tuning," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.5. ISBN: 9798244538229.
Ch10.3: T. Nguyen, "RLHF Methodology," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.3. ISBN: 9798244538229.