Running open models privately: a practical starter with Ollama
When information cannot leave your machines, open models can run locally. Here is how to start without a big infrastructure project.
By the Halden editorial team
Why local
Privacy and control.
Running a model on your own hardware keeps prompts and files on your side, avoids per-token costs and works offline. It suits sensitive drafts, internal documents and experimentation.
Getting started
Small steps.
Install a runner such as Ollama, download a model sized for your hardware and test it on a real task. Smaller models run on ordinary laptops; larger ones need a strong GPU or Apple silicon.
- Pick a model size your machine can run.
- Test on non-sensitive material first.
- Compare with a hosted model on the same task.
Add your documents
Retrieval helps.
Pair a local model with a search index of your own documents so answers are grounded in your material. Keep the index private and refresh it.
Own it
You are the operator.
You are responsible for updates, access control and monitoring. Check each model's licence and keep a record of which versions are in use.
A next step
Turn this into a decision for your organization.
Explore how AI Implementation works with leaders and teams, or start with a short reflection on where you are today.