RIVET
Home / Guides / Run an AI assistant locally
How to run an AI assistant locally on Windows
You need a runtime that runs open-weight language models on your own hardware, such as llama.cpp, Ollama or Microsoft's Foundry Local, plus a model file small enough for your PC. A 4-bit quantised model of 1 to 4 billion parameters runs on an ordinary CPU with a few gigabytes of free memory, and once it is downloaded it works offline.
Running a model locally means your questions never leave your PC, there is no account or API key, and it keeps working with the network off. The trade-off is that small local models are less capable than the largest cloud models, and they use your PC's memory and CPU while they answer.
What you need
- A runtime: llama.cpp, Ollama or Foundry Local, among others.
- A model: usually a GGUF file. Smaller models (around 0.5 to 2 billion parameters) are quick on most PCs; larger ones (4 to 8 billion) give better answers but need more memory.
- Memory: roughly the size of the model file, plus some headroom.
- Disk space for the model, from under 1 GB to several GB.
The general steps
- Install a runtime from its official site.
- Download a model sized for your PC, from a source you trust.
- Start the runtime with the model and ask your question, in a terminal or a chat window.
- Unplug the network to check it really runs locally.
How RIVET does it
Ask RIVET (Pro) runs a Qwen3 model through llama.cpp, on your CPU, inside the app. RIVET recommends a size for your PC from four (0.6B, 1.7B, 4B and 8B, 4-bit), a download of 0.48 GB to 5.03 GB that happens only when you choose it, or it uses a GGUF file you already have. While running it uses about 2 GB of memory. If Foundry Local or Ollama is already running, RIVET uses that instead and downloads nothing. Answers are built from readings RIVET takes of your PC, and nothing you ask leaves the machine.
Questions
Do I need a graphics card to run AI locally?
No. Small quantised models run on the CPU. A graphics card speeds things up but isn't required; Ask RIVET runs on the CPU.
Does a local AI model need the internet?
Only to download the model. After that it runs entirely offline.
Is a local AI assistant private?
Yes, if the runtime and the app around it don't send data anywhere. Ask RIVET makes no network connection after the model download, and RIVET has no telemetry.