Cognition · Data artifact
Critique Rubric
Data artifactCognitionCognition & Memoryarc:CritiqueRubric
A set of explicit evaluation criteria and reflection prompts that defines what a critic checks and how it judges output quality.
Responsibility. Defines evaluation criteria for reflection.
Also known as: Reflection prompt, Evaluation criteria, Manager quality standards, LLM judge scoring rubric, Explicit evaluation rubric, Structured evaluation prompt, Reasoning quality rubric, Domain-specific reasoning criteria, Hallucination evaluation rubric, 3-point grounding scale, Constitutional critique prompt, Revision prompt
Relationships
configures structural
Design guidance
- MUST state concrete evaluation criteria rather than vague prompts such as 'Improve this'.
- MUST operationalise vague quality concepts as specific answerable criteria with defined rating scales; avoid both overly prescriptive and overly vague criteria.
- SHOULD include domain-specific criteria (e.g., financial risk and compliance, medical likelihood ratios and differentials, code design trade-offs, legal precedent scope).
- SHOULD define anchored levels (0 hallucinated, 1 partially grounded, 2 fully grounded) focused on factual grounding, not writing quality.
- SHOULD be refined when evaluators disagree on more than 20% of cases.
- SHOULD ask the model to identify specific violations of the sampled principle and explain why they conflict with it.
Classification
- Patterns
- Rubric-operationalised criteria (coherence, comprehensive consideration, acknowledgment of limitations, goal alignment, efficiency)Step validity rating VALID/QUESTIONABLE/INVALID
- Quality attributes
- Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)
- Risks mitigated
- Superficial feedbackOptimising for the wrong objectiveInconsistent evaluation across evaluator instancesConfirmation bias in human assessment
Sources
- Ch1.2: T. Nguyen, "Core Agent Patterns," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.2. ISBN: 9798244538229.
- Ch2.4: T. Nguyen, "Multi-Agent Frameworks," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 2.4. ISBN: 9798244538229.
- Ch3.1B: T. Nguyen, "Implement Evaluation Pipelines and Task Benchmarks - Guided Practice," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.1B. ISBN: 9798244538229.
- Ch3.9: T. Nguyen, "Reasoning Quality," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.9. ISBN: 9798244538229.
- Ch3.10: T. Nguyen, "Efficiency Metrics," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.10. ISBN: 9798244538229.
- Ch9.5: T. Nguyen, "Constitutional AI," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.5. ISBN: 9798244538229.