Observability & Evaluation · Software component
Failure Category Classifier
Software componentObservability & EvaluationObservability & Evaluationarc:FailureCategoryClassifier
A telemetry component that labels each unsuccessful request either as a safety violation (a guardrail block with its reason) or as an infrastructure failure (an execution exception passed through the guardrail layer), feeding separate metrics and span attributes.
Responsibility. Separates safety-violation outcomes from infrastructure failures in telemetry.
Also known as: Safety vs infrastructure error classification, Guardrail error classification
Relationships
emits telemetry to dynamic
receives data from dynamic
Design guidance
- MUST NOT count guardrail blocks against infrastructure reliability SLOs, because a block is correct system behaviour.
- SHOULD return a structured blocked response for guardrail blocks rather than raising an exception.
- SHOULD let infrastructure exceptions propagate unchanged through the guardrail layer so they are counted as execution failures.
- SHOULD keep the guardrail block reason at violation-type granularity (e.g., jailbreak, PII, factuality, brand safety) to target remediation.
- SHOULD tag spans with category attributes (e.g., guardrail.blocked, error.infrastructure) so traces can be queried per category.
Classification
- Patterns
- Separate error pipelines per failure categoryBlock-as-success semantics
- Technologies
- NVIDIA NeMo GuardrailsOpenTelemetryPrometheus
- Quality attributes
- Maintainability (ISO/IEC 25010)Reliability (ISO/IEC 25010 | NIST AI RMF: valid and reliable)Security (ISO/IEC 25010 | NIST AI RMF: secure and resilient)
- Risks mitigated
- Alert fatigue from conflated failure categoriesJailbreak attempts buried in infrastructure error noiseSafety blocks miscounted against reliability SLOs
Sources
- Ch8.2B: T. Nguyen, "NeMo Guardrails Integration," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 8.2B. ISBN: 9798244538229.