Semantic Cache

Data storeModel ServingModelsarc:SemanticCache

A response cache that matches incoming queries to cached ones by embedding similarity, so paraphrased questions reuse the same answer.

Responsibility. Reuses cached responses for semantically similar queries.

Also known as: Vector-similarity cache, Semantic caching layer, Second-level semantic cache, Semantic caching

Variant of Response Cache abstract

When to choose. Choose when users ask the same questions in different wording and incremental savings exceed embedding costs.

Direct neighbourhood (hover for relationship types)

Relationships

SHOULD be adopted only if incremental inference savings exceed its embedding infrastructure cost.
SHOULD index cached queries in a secondary vector search and return a hit only above a high similarity threshold.

As stated by the sources; verify before use.

Raised hit rate from 70% to 85% for $300/month, saving $600/month (net $300) in the example (Ch1.8).
If ~30% of questions are variations on common themes, semantic lookup serves cached answers in <200 ms, dropping TTFT from 2-3 s (Ch2.9).
Cosine similarity threshold > 0.95; typical hit rate 40-60% (Ch6.5).
Semantic lookup over top_k = 5 similar cached queries with similarity threshold 0.95 (Ref8.05).

Ch1.8: T. Nguyen, "Scalability and Production Deployment," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.8. ISBN: 9798244538229.
Ch2.9: T. Nguyen, "Streaming and Real-Time Responses," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 2.9. ISBN: 9798244538229.
Ch6.5: T. Nguyen, "Production RAG Systems," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.5. ISBN: 9798244538229.
Ref8.05: "Cost Optimization and Resource Monitoring for Agent Systems," unpublished reference note (05-Cost-Optimization-Resource-Monitoring.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note