Running LLMs Locally with Ollama: A Getting-Started Guide

Running a large language model on your own laptop used to sound like something only people with a rack of GPUs in their garage could pull off. That's not true anymore. With a tool called Ollama, you can go from nothing installed to chatting with a real model in about three commands. Let me walk you through why you'd want to do this, and exactly how it works.

So — why run a model locally at all, when the cloud APIs are right there? Three reasons. First, privacy. Whatever you type never leaves your machine, which matters a lot when you're feeding it sensitive data. Second, learning. There's no API wrapper sitting between you and the model — you're poking at its raw behavior, which is a fantastic way to actually understand how these things tick. And third, cost. You pay for the hardware once, up front, and after that every single token you generate is free. No metered billing, no surprise invoice at the end of the month.

Here's the whole thing in a nutshell. You install Ollama with one command. You pull down a model with one command. You start chatting in your terminal with one more. And if you want to build something on top of it, there's a REST API waiting for your own code.

Let's talk hardware for a second, because that's the first question everyone asks. The rule of thumb is about your RAM. With 8 gigabytes, you can comfortably run the small 3-billion-parameter models — think Llama 3.2 in its 3B flavor, or Gemma 2 at 2B — and they run smoothly. Bump up to 16 gigs and you're into the 7 and 8 billion range, Mistral 7B, Llama 3.2 8B, at acceptable speeds. With 32 gigs you can smoothly run a 14-billion model like Qwen 2.5. And if you've got 64 gigs or more, you're playing in the 30 to 70 billion territory, though how well that goes depends on the specific model.

Now, do you need a fancy graphics card? No. But it helps enormously. Ollama automatically detects your GPU — NVIDIA CUDA, Apple's Metal on Mac, or AMD's ROCm — and when it finds one, you're looking at anywhere from ten to fifty times faster generation. Without a GPU, it falls back to your CPU. It still works. It's just slower. And on the operating system front, you're covered everywhere: macOS, Linux, and Windows, either natively or through WSL2.

Installing is genuinely a one-liner. On Mac or Linux, you pipe a script from ollama.com straight into your shell. Mac users who live in Homebrew can just brew install it instead. Windows folks download an installer from the site. Then you check that it landed by asking it for its version number.

Once it's installed, you pull a model. If you just want to kick the tires, grab Llama 3.2 — that's the 3B model, about 2 gigabytes, perfect for testing. Want noticeably better answers? Pull the 8B version instead, around 5 gigs, and the jump in quality is real. And if you're working across languages, Qwen 2.5 at 7B has much stronger multilingual support.

To actually talk to it, you run the model by name, and you get a little prompt in your terminal. You type a question — say, "what is zero-copy in Kafka?" — and it just... answers, right there, offline. When you're done, you type slash-bye, or hit Control-D, and you're out.

Here's where it gets interesting for anyone who writes code. Ollama runs a web server on your machine, on port 11434, and it speaks the same language as the OpenAI API. That's the killer feature. It means if you've already got code written against OpenAI, you can point it at your local Ollama by changing exactly one thing — the base URL. You swap in localhost, you hand it any random string as an API key because Ollama doesn't even check it, and everything else stays the same. Your existing OpenAI code now runs entirely on your laptop, for free. You can also hit it with a plain curl request if you just want to fire a prompt and get JSON back.

A quick word on picking a model, because the menu can be overwhelming. If you want the short version: the 3B Llama is your quick-test, low-memory option. The 8B Llama is the best all-around value for general use. Qwen 2.5 shines on Chinese and other languages. Mistral 7B is strong at English reasoning and code. Code Llama is your specialist for generating code. And there's a tiny one called nomic-embed-text, only about 300 megabytes, that doesn't chat at all — its whole job is turning text into embeddings, those numerical fingerprints you need for search and retrieval.

Which brings us to the payoff: building a complete retrieval-augmented generation pipeline that runs fully offline. Here's the shape of it, in plain terms. You start with a small knowledge base — just a handful of facts, say a few sentences about how Kafka gets its speed and what Ollama does. You take each of those sentences and run them through that embedding model to turn them into vectors. Then a question comes in — "how does Kafka achieve performance?" — and you embed that question the same way. Now you compare the question's vector against every document's vector using cosine similarity, which is really just a measure of how close two vectors point in the same direction. The document that scores highest is your most relevant match. Finally, you take that winning document, staple it onto the question as context, and hand the whole package to the chat model to write the answer. Retrieve the right fact, then generate a grounded response — and the entire loop, embedding and generation both, never touches the internet.

Before I wrap up, a few troubleshooting notes worth keeping in your back pocket. If generation feels sluggish, check whether it's actually using your GPU — there's a status command that shows you a GPU column. On Apple Silicon, Metal acceleration is on by default, so you're usually fine. On NVIDIA, make sure your CUDA drivers are actually installed. If the output looks garbled or just low quality, the culprit is often heavy quantization — those compressed q4 versions trade precision for size, so try a less-compressed variant. And if you're running Ollama as a shared server for multiple people, know that it handles concurrent requests fine, but GPU memory is shared — try to run several big models at once and you'll run out of memory.

So let me leave you with the three things that matter. One: local LLMs have crossed the line from "hobbyist project" to genuinely practical, and Ollama is the gentlest on-ramp there is — install, pull, run, done. Two: because it mimics the OpenAI API, migrating existing code is a one-line change, which makes local a low-risk thing to experiment with. And three: with a chat model and a tiny embedding model working together, you can stand up a real, private, fully offline RAG system on hardware you already own. The barrier to running your own AI has basically disappeared — the only thing left is to try it.