Agentic AI Reference Architecture

Model Serving · Software component

Edge Inference Runtime

Software componentModel ServingModelsarc:EdgeInferenceRuntime

An on-device inference runtime that executes compact, hardware-targeted model formats locally, operating autonomously when connectivity is absent.

Responsibility. Executes optimized models on edge hardware.

Also known as: On-device inference engine

is failover foris monitored byis monitored bydeployed onhostsLLM Inference Service: is failover forLLM Inference ServiceQuality Drift Detector: is monitored byQuality Drift DetectorEdge Device Agent: is monitored byEdge Device AgentEdge Device: deployed onEdge DeviceEdge-Optimized Model: hostsEdge-Optimized Model
Direct neighbourhood (hover for relationship types)

Relationships

deployed on structural

hosts structural

is failover for control

is monitored by assurance

Quantitative guidance

As stated by the sources; verify before use.

Classification

Patterns
Offline inferenceLocal fallback
Technologies
TensorFlow LiteONNX RuntimePyTorch MobileIntel OpenVINOApple Core MLTensorRT
Quality attributes
Performance efficiency (ISO/IEC 25010)Transparency and accountability (NIST AI RMF: accountable and transparent)
Risks mitigated
Loss of capability when offline

Sources

  1. Ch4.3: T. Nguyen, "Container Orchestration and Edge Deployment," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.3. ISBN: 9798244538229.