Running Your Own LLM at Home: Ollama on a Homelab Box
Running a real language model used to mean either a cloud subscription or a GPU rig that sounds like a leaf blower. Ollama changes that math: it packages open-weight models into something you can run with one command, on hardware you probably already own. It won’t beat the biggest hosted models, but for a huge chunk of homelab use cases, it doesn’t need to.
What you need (be honest with yourself here)
- 8GB+ RAM minimum for small models (7B-parameter range), 16GB+ if you want anything more capable
- A GPU helps a lot but is not required, CPU-only inference works, it’s just slower
- Decent disk space, models range from a few GB to tens of GB each
A Raspberry Pi will technically run Ollama’s smallest models, technically being the operative word. If you want something usable day to day, point this at whatever’s the beefiest box in your homelab, not the Pi running Pi-hole.
Install it
curl -fsSL https://ollama.com/install.sh | shOr, if you’d rather keep it contained like everything else in this blog’s homelab:
docker run -d --name ollama \
-v ollama_data:/root/.ollama \
-p 11434:11434 \
--restart unless-stopped \
ollama/ollamaPull and run a model
ollama run llama3.2First run downloads the model, which takes a while depending on size and your connection. After that it’s cached locally, and you get a prompt straight in your terminal. Swap llama3.2 for mistral, phi3, or whatever else is in Ollama’s library depending on what you’re after.
Smaller models (7B and under) are genuinely fine for summarizing text, drafting emails, or basic Q&A. Don’t expect them to write flawless code or reason through anything complex, that’s still where hosted, larger models win.
Give it an API instead of a terminal
Ollama exposes a local API on port 11434, so anything on your network that speaks HTTP can use it, including your own scripts:
curl http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "Summarize this log file in one sentence"
}'This is the bit that turns it from “a chatbot in my terminal” into “something my homelab scripts can actually call.”
Is it worth it over just using a hosted model
If privacy matters (you don’t want your data leaving your network), if you want zero ongoing cost, or if you just want to understand how this stuff actually works under the hood, yes. If you need the absolute best output quality for complex tasks, a hosted frontier model still wins. Most homelabs land somewhere in the middle, and that’s exactly where Ollama is strongest.
Related reading
- Running this in a container? the Docker Compose cheat sheet covers exactly this pattern
- Want to know if your Ollama box is still up at 3am? Uptime Kuma will tell you



