Multi-Layer Semantic Caching for Production LLM Systems
Introduction
In a previous article, I described building an agentic search framework in Go. While that architecture handled the functional requirements well, operating it at scale revealed significant cost and latency challenges. At millions of queries per month, LLM API costs, and P95 latency approached 5 seconds.
This article presents the semantic caching architecture we implemented to address these issues. The system reduced LLM costs by 45-50% and improved P95 latency to under 2 seconds, while maintaining response freshness guarantees.
The key insight: caching at multiple granularities within the agentic pipeline provides better results than end-to-end response caching alone. Specifically, caching the agent’s planning decisions—which are deterministic and independent of result freshness—achieved a 50% hit rate even with conservative similarity thresholds.
Problem Analysis
Cost Structure
A single query in an agentic search system involves multiple LLM calls:
-
Planning/Tool Selection (~8,500 input tokens): Agent reads tool definitions and decides which tools to invoke
-
Tool Execution (minimal cost): External API calls
-
Summarization (~24,000 tokens): LLM formats tool outputs into natural language
-
And couple of smaller models for rewriting the query, and selecting the tools based on the query, and tool responses.
Why Traditional Caching Fails
String-based caching provides minimal hit rates for natural language queries:
"weather in san francisco" → cache key: hash_1
"what's the weather in sf" → cache key: hash_2 (miss)
"san francisco weather" → cache key: hash_3 (miss)
"tell me about san francisco weather"→ cache key: hash_4 (miss)
These queries are semantically identical but produce different cache keys. Our initial implementation with exact string matching achieved only ~15% hit rate.
The Freshness Challenge
Semantic similarity matching introduces a new problem: distinguishing between queries that should produce identical responses versus those requiring fresh data.
Consider:
- “who invented the telephone” (answer never changes)
- “what’s the weather today” (answer changes daily)
Both queries might have similar embeddings to cached entries, but only the first should retrieve cached results. This requires query classification and validation beyond pure similarity matching.
Architecture Design
Multi-Layer Caching Strategy
Rather than caching only the final output, we implemented caching at three distinct layers in the processing pipeline:
Query Input
↓
┌─────────────────────────────┐
│ Agent Planning & Tool │ ← Layer 1: Planner Cache (50% hit rate)
│ Selection │ Cache tool calls, not tool results
└─────────────────────────────┘
↓
┌─────────────────────────────┐
│ Tool Execution │ No caching (requires fresh data)
└─────────────────────────────┘
↓
┌─────────────────────────────┐
│ Response Generation/ │ ← Layer 2: Summarization Cache (35% hit rate)
│ Summarization │ Cache only when tools used static data
└─────────────────────────────┘
↓
Final Response ← Layer 3: End-to-End Cache (3% hit rate)
Each layer targets different characteristics:
- Layer 1 (Planning): High hit rate, deterministic outputs, no freshness concerns
- Layer 2 (Summarization): Medium hit rate, requires freshness validation
- Layer 3 (End-to-End): Low hit rate but zero computation when hits occur
Core Components
The semantic cache system consists of:
- Embedding Generator: Converts queries to dense vector representations (768-dimensional)
- Vector Database (ANN): Stores embeddings with approximate nearest neighbor search
- Key-Value Store: Stores response payloads separately from embeddings
- Query Classifier: Categorizes queries as evergreen vs time-sensitive
Implementation
Layer 1: Agent Planning Cache
The agent planning step is particularly well-suited for caching because:
- Tool selection depends only on query intent, not result freshness
- Planning outputs are deterministic for semantically similar queries
- This is the most expensive operation (6,500+ tokens per call)
type PlannerCache struct {
embeddingClient EmbeddingClient
vectorDB VectorDB
kvStore KVStore
}
type CacheContext struct {
ModelVersion string
ToolVersions []string // sorted
Locale string
EmbeddingVersion string
Temperature float32
}
func (pc *PlannerCache) Get(query string, ctx CacheContext) (*ToolCalls, bool) {
// Generate embedding
embedding := pc.embeddingClient.Embed(query)
// Create cache key from context
contextHash := hashContext(ctx)
// Search with conservative threshold
candidates := pc.vectorDB.Search(VectorSearchRequest{
Embedding: embedding,
Tags: contextHash,
Threshold: 0.98, // Near-exact matching
Limit: 10,
})
// Verify context match and retrieve from KV store
for _, candidate := range candidates {
if candidate.ContextHash == contextHash {
if toolCalls := pc.kvStore.Get(candidate.Key); toolCalls != nil {
return toolCalls, true
}
}
}
return nil, false
}
func (pc *PlannerCache) Set(query string, ctx CacheContext, toolCalls *ToolCalls, ttl time.Duration) {
embedding := pc.embeddingClient.Embed(query)
contextHash := hashContext(ctx)
key := generateKey(embedding, contextHash)
// Store in both vector DB (for similarity search) and KV store (for retrieval)
pc.vectorDB.Insert(embedding, key, contextHash)
pc.kvStore.Set(key, toolCalls, ttl)
}
Cache Context Importance: Including model version, tool versions, and other configuration parameters in the cache key prevents serving stale plans after system updates. When tool definitions change, the context hash changes, effectively invalidating old cache entries.
Layer 2: Summarization Cache with Freshness Validation
The summarization cache requires additional logic to prevent serving stale responses:
func shouldCacheSummarization(query Query, toolResults []ToolResult) bool {
// Multi-turn queries have context dependencies
if query.IsFollowUp {
return false
}
// Check tool result freshness
for _, result := range toolResults {
switch result.Source {
case "web_fresh", "web_daily":
// Results from frequently-updated sources
return false
case "static_index":
// Results from weekly-updated index are cacheable
continue
}
}
// Additional constraints
return query.IsSimpleQuery &&
query.IsSingleStep &&
len(toolResults) == 1
}
This conservative approach accepts a lower hit rate (35%) to ensure response freshness. Only queries that exclusively use static data sources are cached.
Layer 3: End-to-End Cache
The end-to-end cache serves as a catch-all for repeated identical queries:
func (sc *SemanticCache) GetEndToEnd(query string, ctx CacheContext) (*Response, bool) {
// First try exact match
exactKey := md5Hash(normalize(query))
if resp := sc.kvStore.Get(exactKey); resp != nil {
return resp, true
}
// Fall back to semantic search with high threshold
embedding := sc.embeddingClient.Embed(query)
candidates := sc.vectorDB.Search(VectorSearchRequest{
Embedding: embedding,
Tags: hashContext(ctx),
Threshold: 0.98,
Limit: 5,
})
for _, candidate := range candidates {
if resp := sc.kvStore.Get(candidate.Key); resp != nil {
return resp, true
}
}
return nil, false
}
Freshness Control: Two-Stage Gating
To prevent serving stale responses, we implement a two-stage freshness check:
Stage 1: Query Classification
A BERT-based classifier categorizes queries as evergreen (static answers) or time-sensitive (dynamic answers):
type QueryClassifier struct {
model BERTClassifier
}
func (qc *QueryClassifier) Predict(query string) (isEvergreen bool, confidence float64) {
features := qc.extractFeatures(query)
logits := qc.model.Forward(features)
isEvergreen = logits[0] > logits[1] // [evergreen_logit, time_sensitive_logit]
confidence = softmax(logits)[0] if isEvergreen else softmax(logits)[1]
return isEvergreen, confidence
}
The classifier is trained with 95%+ precision at the cost of recall. This conservative tuning ensures we rarely cache time-sensitive queries incorrectly.
Stage 2: Post-Execution Validation
Even if the classifier isn’t confident, we can still cache if we verify that only static data sources were used:
func (sc *SemanticCache) GetOrCompute(
query string,
ctx CacheContext,
compute func() (Response, error),
) (Response, error) {
// Stage 1: Query classification
isEvergreen, confidence := sc.classifier.Predict(query)
if isEvergreen && confidence > 0.95 {
// High confidence evergreen - try cache
if cached, found := sc.Get(query, ctx); found {
return cached, nil
}
}
// Execute computation
response, err := compute()
if err != nil {
return nil, err
}
// Stage 2: Validate based on execution results
shouldCache := false
if isEvergreen && confidence > 0.95 {
shouldCache = true
} else if response.OnlyUsedStaticSources() {
shouldCache = true
}
if shouldCache && response.IsCacheable() {
sc.Set(query, ctx, response, 7*24*time.Hour)
}
return response, nil
}
Embedding Generation
We use a 768-dimensional embedding model (similar to BERT base) for semantic similarity:
type EmbeddingClient struct {
endpoint string
model string
}
func (ec *EmbeddingClient) Embed(text string) []float32 {
// Normalize text
normalized := strings.ToLower(strings.TrimSpace(text))
// Call embedding service
resp := ec.callEmbeddingAPI(EmbeddingRequest{
Text: normalized,
Model: ec.model,
})
// L2 normalize for cosine similarity
return l2Normalize(resp.Embedding)
}
func l2Normalize(vec []float32) []float32 {
var norm float32
for _, v := range vec {
norm += v * v
}
norm = sqrt(norm)
normalized := make([]float32, len(vec))
for i, v := range vec {
normalized[i] = v / norm
}
return normalized
}
Latency: ~15-20ms per embedding generation call.
Vector Search with ANN
For efficient similarity search at scale, we use an approximate nearest neighbor (ANN) index:
type VectorDB struct {
index HNSWIndex // Hierarchical Navigable Small World graph
metadata map[string]Metadata
}
func NewVectorDB(dimension int) *VectorDB {
return &VectorDB{
index: NewHNSWIndex(HNSWConfig{
Dimension: dimension,
M: 16, // connections per node
EfConstruction: 200, // search quality during construction
Metric: "cosine",
}),
metadata: make(map[string]Metadata),
}
}
func (vdb *VectorDB) Search(req VectorSearchRequest) []Candidate {
// ANN search returns approximate nearest neighbors
neighbors := vdb.index.Search(req.Embedding, req.Limit*2)
var candidates []Candidate
for _, neighbor := range neighbors {
// Filter by tags and threshold
meta := vdb.metadata[neighbor.ID]
if meta.Tags == req.Tags && neighbor.Similarity >= req.Threshold {
candidates = append(candidates, Candidate{
Key: neighbor.ID,
Similarity: neighbor.Similarity,
ContextHash: meta.Tags,
})
}
if len(candidates) >= req.Limit {
break
}
}
return candidates
}
Latency: ~10-15ms for search across millions of vectors.
Production Results
After deploying the multi-layer semantic cache to production (serving 10M+ queries/month), we observed:
Hit Rates by Layer
|
Layer |
Hit Rate |
Avg Latency Saved |
|---|---|---|
|
Planner Cache |
44% |
380ms |
|
Summarization Cache |
35% |
950ms |
|
End-to-End Cache |
18% |
1,850ms |
Aggregate Impact
- Cost Reduction: 48% reduction
- Latency Improvement: P95 latency from 3.2s to 1.9s (41% reduction)
- Freshness: Zero incidents of stale responses since implementing two-stage gating
Layer Contribution to Savings
The planner cache (Layer 1) contributes disproportionately to total savings:
- Planner cache: ~56% of cost savings
- Summarization cache: ~30% of cost savings
- End-to-end cache: ~10% of cost savings
This validates the strategy of caching deterministic intermediate computations rather than focusing solely on final outputs.
Design Considerations and Trade-offs
Similarity Threshold Selection
We experimented with thresholds from 0.85 to 0.99:
- Threshold < 0.90: Unacceptable false positive rate. Example: “weather in Seattle” matched “weather in San Francisco”
- Threshold 0.90-0.95: Better hit rate but occasional semantic mismatches
- Threshold ≥ 0.98: Near-exact matching, very low false positive rate
We chose 0.98 as the default. This conservative approach sacrifices some hit rate for quality guarantees.
TTL Strategy
Cache TTLs vary by layer:
- Planner cache: 7 days (tool definitions change infrequently)
- Summarization cache: 2 days (more conservative due to potential freshness issues)
- End-to-end cache: 1 days
Shorter TTLs provide additional freshness guarantees at the cost of reduced hit rates.
Cache Context Granularity
Including too many parameters in cache context reduces hit rate. Including too few causes incorrect cache hits after configuration changes.
Our context includes:
- Model name and version
- Tool definitions and versions (sorted for consistency)
- User locale
- Embedding model version
- Temperature and top-p parameters
We exclude:
- Request timestamp
- User ID
- Session ID
Embedding Model Selection
We evaluated several embedding models:
- BERT-base (768-dim): Baseline performance
- Sentence-BERT (768-dim): Similar performance to BERT-base
- MiniLM (384-dim): Faster but slightly lower quality
- Larger models (1024-dim+): Marginal improvement, significant latency cost
We selected a 768-dimensional model as the best performance/latency trade-off.
Operational Considerations
Monitoring
Critical metrics tracked in production:
type CacheMetrics struct {
HitRate float64 // by layer
MissRate float64 // by layer
LatencySaved time.Duration
CostSaved float64
FalsePositiveRate float64 // classifier accuracy
CacheSize int64
EvictionRate float64
}
We alert on:
- Hit rate drops >10% week-over-week
- False positive rate >5% for evergreen classifier
- P99 cache lookup latency >100ms
Cache Invalidation
Beyond TTL-based expiry, we implement targeted invalidation:
func (sc *SemanticCache) InvalidateByContext(ctx CacheContext) {
contextHash := hashContext(ctx)
// Find all cache entries with this context
keys := sc.vectorDB.GetKeysByTag(contextHash)
// Delete from both vector DB and KV store
for _, key := range keys {
sc.vectorDB.Delete(key)
sc.kvStore.Delete(key)
}
}
This allows immediate invalidation when tool definitions or models are updated, rather than waiting for TTL expiry.
Storage Requirements
Approximate storage per 1M cached queries:
- Vector DB: ~3GB (768-dim floats + metadata)
- KV Store: ~5GB (compressed response payloads)
- Total: ~8GB per 1M cache entries
With 7-day TTLs and 10M queries/month, steady-state storage is approximately 50-60GB.
Conclusion
Multi-layer semantic caching is a critical component for production LLM systems operating at scale. By caching at multiple granularities—particularly at the agent planning layer—we achieved significant cost reduction (48%) and latency improvement (41%) while maintaining response quality and freshness.
The key architectural insight is that deterministic intermediate computations (agent planning, tool selection) are more cacheable than final outputs, which depend on fresh data. This inverts the typical caching strategy and provides better hit rates where they matter most.
For teams building agentic systems, the planner cache should be the first caching layer implemented. It provides the highest return on investment and requires no freshness validation.
References
- Previous article: Building a Production-Ready Agentic Search Framework
