Observability & Evaluation · Data artifact
Evaluation Sampling Policy
Data artifactObservability & EvaluationObservability & EvaluationVariation point (abstract)arc:EvaluationSamplingPolicy
A configuration setting what fraction of production interactions are evaluated at each deployment stage and when sampling rates increase in response to anomalies.
Responsibility. Controls online evaluation coverage versus cost.
Also known as: Sampling strategy
Variants
| Variant | When to choose |
|---|---|
| Exhaustive Evaluation Sampling Policy | Choose when comprehensive per-trace ground truth justifies ~2-5% of inference cost and ~500 ms added latency per request. |
| Random Evaluation Sampling Policy | Choose when a representative baseline-quality estimate is needed with no production latency impact. |
| Strategic Evaluation Sampling Policy | Choose for production systems where evaluation should focus on problem areas; recommended default (5-10% with error oversampling). |
Relationships
configures structural
Design guidance
- SHOULD increase sampling rates when anomaly detection flags concerning patterns.
- SHOULD sample 5-10% of production traces with oversampling of errors (Ref8.03).
Quantitative guidance
As stated by the sources; verify before use.
- Monitor 100% early, then 10-20% as systems stabilize, then 1-5% for ongoing validation (Ch3.3).
- 10% of canary and stable conversations sampled for LLM-as-judge assessment (Ch4.2).
Classification
- Patterns
- Adaptive sampling
- Quality attributes
- Cost efficiency
- Risks mitigated
- Excessive evaluation cost at scale
Sources
- Ch3.3: T. Nguyen, "Web Navigation and Interaction Benchmarks," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.3. ISBN: 9798244538229.
- Ch4.2: T. Nguyen, "Deployment and Scaling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.2. ISBN: 9798244538229.
- Ref8.03: "Agent Evaluation Frameworks and Metrics," unpublished reference note (03-Agent-Evaluation-Frameworks.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note