Prompt Caching: Save 90 Percent on Repeated LLM Calls

Prompt Caching: Save 90 Percent on Repeated LLM Calls

We’ve all been there. Building an agentic system that feels magical… until the cloud bill arrives. In 2026, with models like Gemini 1.5 Pro, Claude 3 Opus, and even increasingly capable open-source options like Mixtral 8x22B dominating the landscape, the cost of Large Language Model (LLM) inference has become a critical bottleneck. It’s not just about token usage anymore; it's the cumulative cost of repeated prompts, especially in applications with user interaction or iterative workflows. I’ve seen projects easily spend tens of thousands of dollars per month on LLM calls that, in retrospect, were largely redundant. The good news? Prompt caching, implemented strategically, can slash those costs – often by 90% or more – without significantly impacting performance. This isn't a new concept, but the scale of LLM cost makes it a necessity, and the sophistication of caching strategies has evolved rapidly.

Why is Prompt Repetition So Common?

Before diving into solutions, let's understand why we see so much prompt repetition. It's rarely malicious, more often a byproduct of how we architect agentic systems. Here are a few common culprits:

Implementing a Basic Prompt Cache with Redis

The simplest form of prompt caching involves storing the prompt and the LLM's response in a key-value store. Redis is a popular choice due to its speed and in-memory nature. Here's a Python example using the redis and openai libraries (although the principle applies to any LLM provider):

import redis
import openai
import hashlib
Configure Redis connection
redis_client = redis.Redis(host='localhost', port=6379, db=0)
OpenAI API key (replace with your actual key)
openai.api_key = "YOUR_OPENAI_API_KEY"
def get_llm_response(prompt):     """     Retrieves an LLM response, checking the cache first.     """     # Generate a cache key based on the prompt     prompt_hash = hashlib.sha256(prompt.encode('utf-8')).hexdigest()     cache_key = f"llm_response:{prompt_hash}"
# Check if the response is in the cache     cached_response = redis_client.get(cache_key)
if cached_response:         print("Cache hit!")         return cached_response.decode('utf-8')  # Decode from bytes
print("Cache miss. Calling LLM...")     # Call the LLM (replace with your preferred model)     response = openai.Completion.create(         engine="gpt-3.5-turbo-instruct", #Or your preferred model         prompt=prompt,         max_tokens=150     ).choices[0].text.strip()
# Store the response in the cache with a TTL (Time To Live)     redis_client.set(cache_key, response, ex=3600)  # Cache for 1 hour
return response
Example usage
prompt1 = "What is the capital of France?"
response1 = get_llm_response(prompt1)
print(f"Response 1: {response1}")
prompt2 = "What is the capital of France?" # Same prompt response2 = get_llm_response(prompt2) print(f"Response 2: {response2}")
prompt3 = "What is the capital of Germany?" #Different prompt response3 = get_llm_response(prompt3) print(f"Response 3: {response3}") 

This code snippet hashes the prompt using SHA256 to create a unique cache key. This is crucial; you don't want slight variations in whitespace or punctuation to result in cache misses. The response is then stored in Redis with a Time To Live (TTL) of one hour. Adjust the TTL based on how frequently the underlying information changes.

Tradeoffs:

Beyond Basic Caching: Semantic Hashing and Fuzzy Matching

A simple hash-based cache is effective for exact prompt matches. But what about the user who asks "Help me with my password" instead of "How do I reset my password?" That’s where more sophisticated techniques come into play.

Semantic Hashing: Instead of hashing the raw prompt string, you can embed it using a sentence transformer (like those from Sentence Transformers library). This creates a vector representation of the prompt's meaning*. You can then store these vectors in a vector database alongside the corresponding LLM responses. When a new prompt arrives, you embed it and perform a similarity search in the vector database. If a similar prompt is found (above a defined threshold), you return the cached response. This handles paraphrasing and semantic variations.
• Fuzzy Matching: Libraries like fuzzywuzzy allow you to calculate the similarity between two strings. You can use this to check if a new prompt is "close enough" to a cached prompt. This is a simpler approach than semantic hashing, but it can be surprisingly effective for common variations.

Considerations:

Advanced Caching Strategies for Agentic Systems

For complex agentic systems, you need to go beyond simple prompt/response caching.

Contextual Caching: Cache prompts along with* the agent's state. For example, if the agent is helping a user book a flight, cache the prompt and response based on the user's current location, desired destination, and travel dates. This avoids re-prompting for the same information when the user modifies a single parameter.
• Response Summarization: In iterative loops, instead of re-prompting with the full previous conversation, summarize the key information from the previous responses and include that in the prompt. This reduces the token count and the likelihood of cache misses.
• Cache Partitioning: Divide your cache into partitions based on user ID, session ID, or other relevant criteria. This improves scalability and reduces the risk of cache collisions.
• Hybrid Approach: Combine different caching strategies. For instance, use a hash-based cache for exact matches, semantic hashing for paraphrased prompts, and response summarization for iterative loops.
• Monitoring and Analytics: Crucially, track your cache hit rate, cache size, and the cost savings achieved. Use this data to fine-tune your caching strategy and identify areas for improvement. Tools like Prometheus and Grafana are valuable here.

Key Takeaways