Agentic AI Reference Architecture

Model Adaptation · Software component

AI-Feedback Preference Labeler

Software componentModel AdaptationModelsarc:AIFeedbackPreferenceLabeler

A language-model judge that compares two candidate responses against a randomly selected constitutional principle and records which better adheres, producing AI-generated preference labels.

Responsibility. Labels response pairs by constitutional adherence.

Also known as: Constitutional judge, RLAIF labeler, AI feedback model

Variant of LLM Judge abstract

When to choose. Choose when preference labels must scale without human annotation time and be traceable to explicit principles; complement with human labels where contextual nuance matters.

reads; is configured byinvokesspecializesis orchestrated bywritesreceives data fromis configured byConstitution: reads; is configured byConstitutionLLM Inference Service: invokesLLM Inference ServiceLLM Judge: specializesLLM JudgeTraining Pipeline Orchestrator: is orchestrated byTraining Pipeline Orches…Preference Dataset: writesPreference DatasetCandidate Response Sampler: receives data fromCandidate Response SamplerPrinciple Priority Policy: is configured byPrinciple Priority Policy
Direct neighbourhood (hover for relationship types)

Relationships

is configured by structural

invokes dependency

reads dependency

writes dependency

receives data from dynamic

is orchestrated by control

Design guidance

Quantitative guidance

As stated by the sources; verify before use.

Classification

Patterns
Reinforcement Learning from AI Feedback (RLAIF)Principle-guided pairwise comparisonLLM-as-Judge
Quality attributes
Performance efficiency (ISO/IEC 25010)Reliability (ISO/IEC 25010 | NIST AI RMF: valid and reliable)Transparency and accountability (NIST AI RMF: accountable and transparent)
Risks mitigated
Inconsistent human annotator judgmentsAnnotator psychological burdenSlow preference collection

Sources

  1. Ch9.5: T. Nguyen, "Constitutional AI," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.5. ISBN: 9798244538229.
  2. Ch10.3: T. Nguyen, "RLHF Methodology," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.3. ISBN: 9798244538229.