Model Routing Policy

Data artifactModel ServingModelsarc:ModelRoutingPolicy

A declarative policy mapping request characteristics such as token length or reasoning complexity to model tiers and providers, together with spending caps, applied centrally by the AI gateway.

Responsibility. Specifies which model and provider serve each class of request.

Also known as: AI gateway routing policy, Cost cap policy

Direct neighbourhood (hover for relationship types)

Relationships

configures structural

Model Router Ch4.2 Ch8.3 +2

Design guidance

SHOULD be configured once at the gateway layer rather than reimplemented across agent services.

Quantitative guidance

As stated by the sources; verify before use.

<500 tokens -> Claude 3.5 Haiku; >2,000 tokens multi-step reasoning -> Claude 3.5 Sonnet (Ch4.2 example).
Simple queries -> GPT-4o-mini, moderate and complex -> GPT-4o (Ch8.3).
Complexity score < 0.3 -> 4B model (~30x cheaper), < 0.6 -> 8B (~10x cheaper), else 70B (Ref8.05).
Tier cost/quality: 4B $0.0001/1K (75%), 8B $0.0005/1K (85%), 70B $0.003/1K (95%), 405B $0.01/1K (98%) (Ref8.05).

Classification

Patterns: Policy as configuration
Quality attributes: Cost efficiencyMaintainability (ISO/IEC 25010)
Risks mitigated: Runaway spendingInconsistent routing across services

Sources

Ch4.2: T. Nguyen, "Deployment and Scaling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.2. ISBN: 9798244538229.
Ch8.3: T. Nguyen, "Token Economics and Architecture," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 8.3. ISBN: 9798244538229.
Ref7.17: "Scaling Agentic AI Systems: Patterns and Strategies," unpublished reference note (17-Scalability-Patterns.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
Ref8.05: "Cost Optimization and Resource Monitoring for Agent Systems," unpublished reference note (05-Cost-Optimization-Resource-Monitoring.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note