Preference Dataset

Data artifactModel AdaptationModelsarc:PreferenceDataset

A dataset of prompts with pairs of candidate responses labeled by which is preferred, per criterion, used to train reward models or directly optimize policies.

Responsibility. Records comparative human (or model) preference judgments for training.

Also known as: Preference comparisons, Pairwise preference data, AI-generated preference data, RLAIF preference labels, Preference comparison dataset

Direct neighbourhood (hover for relationship types)

Relationships

is read by dependency

is written by dependency

receives data from dynamic

Preference Annotator abstract Ch9.6
Red Team Tester Ch10.3

is evaluated by assurance

Domain Expert Annotator Ch3.5

Design guidance

SHOULD start with 500-1,000 comparison pilots before scaling to 5,000-50,000+ comparisons.
MUST be version-controlled as models and annotation guidelines evolve.
SHOULD retain disagreement metadata (e.g., 60% preferred A, 40% preferred B) instead of collapsing to artificial consensus.

Quantitative guidance

As stated by the sources; verify before use.

Pilots use 500-1,000 comparisons; production reward models need 5,000-50,000+ (Ch3.5).
InstructGPT collected approximately 33,000 preference comparisons from 40 trained annotators (Ch10.3).
Industry estimates put preference data for a large-scale RLHF run at hundreds of thousands to millions of dollars (Ch10.3).

Classification

Quality attributes: Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)

Sources

Ch3.5: T. Nguyen, "Prompt Optimization, Few-Shot Learning, Fine-Tuning," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.5. ISBN: 9798244538229.
Ch9.5: T. Nguyen, "Constitutional AI," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.5. ISBN: 9798244538229.
Ch10.3: T. Nguyen, "RLHF Methodology," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.3. ISBN: 9798244538229.
Ch10.5: T. Nguyen, "Human-over-the-Loop," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.5. ISBN: 9798244538229.