NAME
private-llm-server — A private AI assistant on my own hardware
SYNOPSIS
Open-weight language models on a desktop graphics card, behind a sign-in, so documents never leave the building.
SITUATION
My own lab in the Fraser Valley. I wanted to try AI assistants on real documents without sending them to a cloud provider, on hardware a small office could afford.
WHAT I CHANGED
- Ollama runs 7B to 14B open-weight models on a desktop graphics card with 6 GB of memory.
- A web chat sits behind a sign-in; the model server itself listens only on the machine.
- The search (embedding) model runs on the processor, so it never pushes the chat model out of the graphics card.
- The chat model loads on start-up, so the first question of the day does not wait for it.
HOW IT WAS CHECKED
| Measure | Before | After |
|---|---|---|
| Chat model unloaded by every document search | yes | no |
| Models kept loaded at once | 1 | 2 |
- Slow first answers came from the chat model being unloaded every time the search model ran. After the change, the chat model stays loaded through searches.
- On a modest graphics card, one request at a time beats parallel requests, and good document retrieval matters more than a bigger model.
STACK
- Ollama
- Open WebUI
- NVIDIA GPU (6 GB)
- nginx sign-in