Agentic AI Reference Architecture

Model Serving · Software component

Pre-compiled Engine Backend

Software componentModel ServingModelsarc:PrecompiledEngineBackend

An inference backend executing engines pre-compiled for a specific GPU architecture and memory size, with architecture-specific kernels, validated FP8/INT8 quantization and memory optimizations.

Responsibility. Delivers peak inference performance on supported GPU configurations.

Variant of Inference Backend abstract

When to choose. Choose when pre-compiled engines exist for the deployed GPU (e.g., A100 40/80GB, H100 80GB, L40S 48GB).

fails over to; is target of alternativeTohostshostshostsspecializesis routed to byis configured byPortable LLM Runtime Backend: fails over to; is target of alternativeToPortable LLM Runtime Bac…Optimized Inference Engine: hostsOptimized Inference EngineINT8 Quantized Engine: hostsINT8 Quantized EngineFP8 Quantized Engine: hostsFP8 Quantized EngineInference Backend: specializesInference BackendInference Engine Selector: is routed to byInference Engine SelectorModel-Specific Inference Image: is configured byModel-Specific Inference…
Direct neighbourhood (hover for relationship types)

Relationships

hosts structural

is configured by structural

is routed to by dynamic

fails over to control

alternative to variability

Quantitative guidance

As stated by the sources; verify before use.

Classification

Technologies
TensorRT-LLMTensorRT

Sources

  1. Ch7.1B: T. Nguyen, "Nvidia NIM and Colang," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 7.1B. ISBN: 9798244538229.