Inference Metrics Endpoint

InterfaceObservability & EvaluationObservability & Evaluationarc:InferenceMetricsEndpoint

A scrapeable endpoint on a model-serving process publishing inference-specific metrics such as queue depth, latency percentiles, batch sizes, error rates and per-model GPU usage.

Responsibility. Exposes serving-level operational metrics for collection.

Also known as: Triton metrics port 8002, Prometheus metrics exporter, NIM /metrics endpoint, Inference server /metrics endpoint

Variant of Metrics Endpoint abstract

Direct neighbourhood (hover for relationship types)

Relationships

is exposed by structural

is invoked by dependency

Metrics Collector Ch4.5 Ch7.2 +3

Design guidance

SHOULD publish request duration, queue duration, batch size and per-model GPU memory so queue wait can be separated from compute time.

Quantitative guidance

As stated by the sources; verify before use.

Counters/gauges include nim_inference_requests_total, nim_inference_duration_seconds_bucket, nim_inference_errors_total, nim_gpu_utilization_percent, nim_inference_tokens_generated_total (Ch7.2).
Queue duration above 100 ms indicates insufficient GPU throughput for the offered load (Ch8.1).
800 ms average inference may be 600 ms queue wait plus 200 ms compute (Ch8.1).

Classification

Technologies: Prometheus exposition formatNVIDIA NIMNVIDIA Triton Inference Server metrics (port 8002)

Sources

Ch4.5: T. Nguyen, "NVIDIA NIM and Triton Inference Server," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.5. ISBN: 9798244538229.
Ch7.2: T. Nguyen, "Performance Optimization and Production Monitoring," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 7.2. ISBN: 9798244538229.
Ch8.1: T. Nguyen, "Latency Metrics," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 8.1. ISBN: 9798244538229.
Ref7.04: NVIDIA, "NVIDIA NIM," NVIDIA Docs. Accessed: Sep. 27, 2026. [Online]. Available: https://docs.nvidia.com/nim/
Ref7.08: NVIDIA, "NVIDIA Deep Learning Triton Inference Server Documentation," NVIDIA Docs. Accessed: Sep. 27, 2026. [Online]. Available: https://docs.nvidia.com/deeplearning/triton-inference-server/