Distributed Response Cache

Data storeModel ServingModelsarc:DistributedResponseCache

A network-accessible key-value cache shared by all replicas so a response computed once benefits every instance.

Responsibility. Shares cached responses across all replicas over the network.

Also known as: Redis cache, Layer 2 cache, Exact-match cache, Shared cache, Redis query cache, Exact-match query cache

Variant of Response Cache abstract

When to choose. Choose for horizontally scaled systems with >20% query repetition or where cache must survive instance restarts.

Direct neighbourhood (hover for relationship types)

Relationships

SHOULD be the default cache for production agent systems.
SHOULD run in private subnets with authentication and encryption in transit for sensitive data.
SHOULD use replication with automatic failover so the cache layer is not a single point of failure.
MAY use snapshot persistence for reconstruction, or append-only logging when cached data is business-critical.
SHOULD set TTL by knowledge base volatility: 10-15 minutes for rapidly changing, ~1 hour typical, 24 hours for stable documentation.

As stated by the sources; verify before use.

Adds 1-3 ms network latency (Ch1.8).
Managed services cost $50-500/month; managed premium 20-40% over self-hosted (Ch1.8).
Example Layer 2 TTL 4 hours (Ch1.8).
Cache hits return in 5-15 ms vs 200-500 ms for retrieval plus generation (Ch6.5).

Patterns: Consistent hashing of keysConnection poolingPipeliningCluster shardingAutomatic failoverMD5-hashed cache keys
Technologies: RedisRedis ClusterRedis SentinelAWS ElastiCacheAzure Cache for RedisGoogle Cloud Memorystore
Quality attributes: Performance efficiency (ISO/IEC 25010)Cost efficiencyReliability (ISO/IEC 25010 | NIST AI RMF: valid and reliable)
Risks mitigated: Cache fragmentation across replicasCold start after restart

Ch1.8: T. Nguyen, "Scalability and Production Deployment," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.8. ISBN: 9798244538229.
Ch6.5: T. Nguyen, "Production RAG Systems," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.5. ISBN: 9798244538229.