Local LLMs in Production: Ollama, vLLM and llama.cpp Compared

Local LLMs in Production: Ollama, vLLM and llama.cpp Compared

For the last two years, we’ve been building autonomous agents. Initially, like most, we leaned heavily on OpenAI and Anthropic APIs. It was fast to prototype, but the costs quickly became unsustainable, and the limitations around data privacy and customisation started to bite. In 2026, the pendulum has swung. While API access remains valuable, running Large Language Models (LLMs) locally – truly on your infrastructure – is no longer a niche pursuit, but a critical path for production-grade autonomous systems. But choosing the right local LLM serving solution isn’t straightforward. This article dives deep into three popular options – Ollama, vLLM, and llama.cpp – comparing their strengths, weaknesses, and suitability for different production scenarios. We’ll go beyond the marketing hype and focus on the practical realities of performance, cost, and complexity.

Ollama: The Developer-Friendly Gateway

Ollama has become the de-facto standard for getting started with local LLMs. It’s ridiculously easy to install (a single command on macOS and Linux, Windows support maturing rapidly), and even simpler to run models. Under the hood, Ollama leverages llama.cpp for inference, but crucially, it abstracts away all the complexities of model downloading, format conversion, and basic serving.

The real power of Ollama lies in its model ecosystem and the ollama run command. You can pull models directly from the Ollama Hub, a growing repository of quantized LLMs. For example, to run the Mistral 7B Instruct model:

ollama run mistral

Ollama handles the download, unpackaging, and setup. It then presents a chat interface, but more importantly, exposes a REST API on localhost:11434. This API allows you to integrate the model into your applications with minimal code. Here’s a Python example using the ollama library:

import ollama
def generate_text(prompt):   response = ollama.chat(model='mistral', messages=[       {           'role': 'user',           'content': prompt,       },   ])   return response['message']['content']
prompt = "Write a short story about a robot who learns to paint." story = generate_text(prompt) print(story) 

The Good: Incredibly easy setup and use. Excellent for prototyping and smaller applications. The Ollama Hub simplifies model discovery and management. Good community support. Automatic handling of quantization and model formats.

The Not-So-Good: Ollama's simplicity comes at a cost. It's primarily designed for single-model serving. Scaling to handle multiple concurrent requests or deploying across a cluster is significantly harder and requires you to build your own orchestration layer on top. Performance, while improved with recent versions, isn't optimized for high-throughput scenarios like vLLM. The underlying llama.cpp engine can become a bottleneck. Limited control over advanced inference parameters. Its reliance on a central Hub, while convenient, introduces a potential single point of failure and dependency.

vLLM: Optimised for Throughput and Scalability

vLLM (Very Large Language Model) takes a fundamentally different approach. While Ollama aims for ease of use, vLLM prioritizes performance and scalability. It's built on a PagedAttention mechanism, which dramatically reduces memory fragmentation during inference, allowing you to serve models with higher throughput, especially with longer sequences.

vLLM is more complex to set up than Ollama. It requires a compatible GPU (Nvidia is best supported, AMD support is improving) and a deeper understanding of LLM serving concepts. It’s designed to be a serving engine and doesn't include a built-in model hub or chat interface like Ollama. You’re responsible for downloading and preparing the models yourself, typically using Hugging Face Transformers.

The key to vLLM's performance is its architecture. It efficiently manages attention keys and values in GPU memory, avoiding the costly reallocations that plague traditional serving methods. It also supports dynamic batching, where it groups incoming requests to maximize GPU utilization.

Here’s a simplified example of how you might deploy a model with vLLM using its API:

from vllm import LLM, SamplingParams
Load the model
llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.2")
Define sampling parameters (adjust for desired output)
sampling_params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=256)
Generate text
prompts = ["Write a short story about a robot who learns to paint."]
outputs = llm.generate(prompts, sampling_params)
Print the generated text
for output in outputs:
    print(output.outputs[0].text)

To truly unlock vLLM’s capabilities, you’ll want to use its distributed inference features, allowing you to spread the model across multiple GPUs and nodes. This requires using a cluster management system like Kubernetes or Ray.

The Good: Exceptional throughput and scalability, especially for long sequences. PagedAttention significantly reduces memory overhead. Dynamic batching improves GPU utilization. Supports distributed inference. Good control over inference parameters. Integrates well with Hugging Face ecosystem.

The Not-So-Good: Significantly more complex to set up and manage than Ollama. Requires a powerful GPU. Less developer-friendly; steeper learning curve. No built-in model hub. Debugging can be challenging. The initial overhead of setting up the serving infrastructure is substantial.

llama.cpp: The Bare-Metal Powerhouse

llama.cpp is the granddaddy of local LLM inference. Originally designed to run LLMs on CPUs, it has evolved to leverage GPUs (with varying degrees of efficiency) and is the foundation for many other projects, including Ollama. It's written in C++, making it extremely performant, but also demanding of technical expertise.

llama.cpp shines when you need maximum control over the inference process and are willing to trade off ease of use for performance. It’s ideal for resource-constrained environments or when you need to squeeze every last bit of performance out of your hardware.

Unlike Ollama and vLLM, llama.cpp doesn't offer a high-level API or automatic model management. You need to download the model (typically in GGML or GGUF format), convert it if necessary, and then use the command-line interface or build your own application on top of its C++ API.

Here’s a very basic example of running a model from the command line:

./main -m /path/to/your/model.gguf -p "Write a short story about a robot who learns to paint."

You can also use it as a library within your C++ projects to build custom inference pipelines.

The Good: Maximum performance and control. Excellent for resource-constrained environments. Mature and well-optimized. Large community and extensive documentation. Supports a wide range of model formats and hardware. Can be used as a library in C++ applications.

The Not-So-Good: Steepest learning curve of the three. Requires C++ knowledge. No built-in API or model management. Manual model downloading and conversion. Complex configuration. Not ideal for rapid prototyping. Maintaining compatibility with evolving model formats can be challenging.

A Quick Comparison Table

| Feature | Ollama | vLLM | llama.cpp | |-------------------|-------------------|-------------------|-------------------| | Ease of Use | Very High | Medium | Low | | Performance | Good | Excellent | Excellent | | Scalability | Limited | High | Medium (requires custom orchestration) | | Model Hub | Built-in | None | None | | API | REST | REST | C++ Library/CLI | | Hardware | CPU/GPU | GPU (Nvidia preferred)| CPU/GPU | | Complexity | Low | High | Very High | | Use Cases | Prototyping, Small Apps | High-Throughput Serving | Resource-Constrained Environments, Custom Pipelines |

Key Takeaways

Choosing the right tool depends heavily on your specific needs and resources. We've found that a hybrid approach often works best – using Ollama for initial development and prototyping, then migrating to vLLM for production deployment. And always remember to benchmark performance with your models and your data to make an informed decision. The local LLM landscape is evolving rapidly, and staying current is crucial for building robust and cost-effective autonomous systems.