Agentic AI Reference Architecture

Model Serving · Software component

Portable LLM Runtime Backend

Software componentModel ServingModelsarc:PortableLLMRuntimeBackend

An inference backend that compiles dynamically at startup for any CUDA-capable GPU with sufficient memory, trading some peak performance for broad hardware compatibility.

Responsibility. Keeps inference running on hardware lacking pre-compiled engines.

Also known as: vLLM fallback

Variant of Inference Backend abstract

When to choose. Choose for uncommon or heterogeneous GPUs (RTX 4090, A10G, V100, T4), edge and mixed research clusters, or development prioritizing iteration speed.

alternative to; is failover forhostsspecializesis routed to byPre-compiled Engine Backend: alternative to; is failover forPre-compiled Engine Back…Foundation LLM: hostsFoundation LLMInference Backend: specializesInference BackendInference Engine Selector: is routed to byInference Engine Selector
Direct neighbourhood (hover for relationship types)

Relationships

hosts structural

is routed to by dynamic

is failover for control

alternative to variability

Quantitative guidance

As stated by the sources; verify before use.

Classification

Technologies
vLLM

Sources

  1. Ch7.1B: T. Nguyen, "Nvidia NIM and Colang," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 7.1B. ISBN: 9798244538229.