Model Serving · Data artifact

Cache Policy

Data artifactModel ServingModelsarc:CachePolicy

A configuration specifying cache TTLs per layer, eviction policy, key normalization and coordination strategy.

Responsibility. Defines expiry, eviction and population rules for cache layers.

Also known as: TTL configuration, Eviction policy, Multi-tier cache configuration

Direct neighbourhood (hover for relationship types)

Relationships

SHOULD set TTLs according to data change frequency; too long risks staleness, too short reduces hit rate.
SHOULD prioritize retaining frequently accessed, expensive-to-compute entries.
SHOULD select TTLs per data type balancing freshness against hit rate (e.g., minutes for pricing, real-time for balances).
SHOULD consult cache tiers in order (reasoning-chain templates, then embeddings, then tool results) and populate all tiers on a full miss.
SHOULD combine TTL to bound staleness, event-based invalidation after updates, and LRU eviction to prevent memory exhaustion.

As stated by the sources; verify before use.

FAQ example TTL 3600 s (Ch1.8).
5-minute TTL acceptable for product pricing but inadequate for account balances (Ch3.4).
Tool-result TTL example 5 minutes combined with event invalidation (Ch4.7).
RAG worked example: average latency 500 ms to 446 ms (11%), 30% for frequent questions; total cache memory ~340 MB (Ch4.7).

Patterns: TTL expirationLRU evictionValue-aware evictionAsymmetric TTLMulti-tier cache hierarchy (L1 reasoning templates, L2 embeddings, L3 tool results)Layer-specific invalidationTime-based (TTL) invalidationEvent-based invalidationCapacity-based LRU eviction
Quality attributes: Maintainability (ISO/IEC 25010)
Risks mitigated: Stale responsesMemory exhaustion

Ch1.8: T. Nguyen, "Scalability and Production Deployment," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.8. ISBN: 9798244538229.
Ch3.4: T. Nguyen, "Tuning Model Parameters for Production Performance," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.4. ISBN: 9798244538229.
Ch4.7: T. Nguyen, "Scaling Strategies," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.7. ISBN: 9798244538229.
Ch6.5: T. Nguyen, "Production RAG Systems," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.5. ISBN: 9798244538229.