Observability & Evaluation · Software component

Judge Adversarial Tester

Software componentObservability & EvaluationObservability & Evaluationarc:JudgeAdversarialTester

A meta-evaluation component that periodically injects known-incorrect agent outputs into the evaluation stream to verify that automated judges identify them as failures.

Responsibility. Verifies that automated evaluators are not deceived by confident but wrong outputs.

Also known as: Adversarial evaluation of evaluators

Direct neighbourhood (hover for relationship types)

Relationships

evaluates assurance

LLM Judge abstract Ch3.3

Design guidance

SHOULD periodically inject known-incorrect outputs to verify evaluators detect failures.

Classification

Patterns: Canary injection
Quality attributes: Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)
Risks mitigated: Agents gaming evaluator metricsPlausible but unfactual responses rated highly

Sources

Ch3.3: T. Nguyen, "Web Navigation and Interaction Benchmarks," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.3. ISBN: 9798244538229.