How to run a local LLM with Ollama on macOS, Windows or Linux

Key takeaways

• Ollama is a free, MIT-licensed tool that lets you run a local LLM on macOS, Windows or Linux and talk to it from the terminal or over a REST API on http://localhost:11434.
• Small models are light: Meta's Llama 3.2 in its 3B version is a 2.0 GB download, and the 1B version is 1.3 GB, according to the Ollama model library.
• Memory is the main limit. Earlier versions of Ollama's own README advised at least 8 GB of RAM for 7B models, 16 GB for 13B models and 32 GB for 33B models.
• A short text file called a Modelfile lets you save a custom system prompt and settings as your own named model.
• When you run models locally, Ollama says it does not see your prompts or data. Cloud-tagged models are different: they are processed on Ollama's servers.

To run a local LLM with Ollama, you install one free program, download a model and chat with it in a terminal or through a REST API on localhost. Your prompts stay on your machine, and once downloaded the model works without an internet connection. Ollama also supports Open Responses, the API spec OpenAI donated to the Agentic AI Foundation, as we report in OpenAI donates the Open Responses spec.

What you need to run a local LLM with Ollama

Ollama's documentation lists these minimum platform requirements:

• macOS: macOS Sonoma (version 14) or newer. Apple M-series chips get both CPU and GPU support; Intel (x86) Macs run on the CPU only.
• Windows: Windows 10 22H2 or newer, Home or Pro. NVIDIA cards need driver 551.61 or newer. AMD Radeon cards need a ROCm v7 driver stack or a Vulkan-capable driver.
• Linux: a 64-bit distribution with systemd is the simplest case, because the installer sets Ollama up as a service.
• GPU (optional): NVIDIA GPUs need compute capability 5.0 or higher and driver 550 or newer, per the GPU page. Apple chips are accelerated through Metal, and Vulkan adds extra GPU support on Windows and Linux.
• Disk: the Windows install alone needs at least 4 GB. Ollama's docs warn that models can be "tens to hundreds of GB" in size, so leave plenty of room.

How much RAM or VRAM does each model size need?

The weights must fit in memory. Ideally that is your GPU's video memory (VRAM), and system RAM is the fallback. Version 0.3.0 of Ollama's README gave a simple rule of thumb, and it is still a sensible starting point. Actual use rises with longer context windows, and it falls with heavier quantization (explained below). For a newer free coding model built to run on a laptop, see our report on Recursive Flint.

Sources: RAM guidance from the Ollama README (v0.3.0); download sizes from the Ollama library pages for llama3.2 and gemma4, as listed on October 22, 2026.

Step 1: Install Ollama

On macOS, download Ollama.dmg from ollama.com/download and drag the app into Applications. On first launch, it offers to link the ollama command into /usr/local/bin so that it works in Terminal. The README also lists a one-line install:

curl -fsSL https://ollama.com/install.sh | sh

On Windows, run OllamaSetup.exe. It installs into your home folder and does not need administrator rights. Alternatively, run this in PowerShell:

irm https://ollama.com/install.ps1 | iex

Ollama then runs in the background, and the ollama command works in cmd or PowerShell.

On Linux, the official script installs the binary and creates a systemd service called ollama:

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl status ollama

As with any script piped into a shell, you can download it and read it first. To update on Linux later, run the same command again. The macOS and Windows apps download updates themselves and ask you to restart.

Step 2: Pull a model and chat in the terminal

Browse ollama.com/library to see what is available. For a first test, pick something small. The commands below come from Ollama's CLI reference:

ollama pull llama3.2      # download the model (about 2.0 GB)
ollama run llama3.2       # start an interactive chat
ollama ls                 # list downloaded models
ollama ps                 # show models loaded in memory
ollama stop llama3.2      # unload a running model
ollama rm llama3.2        # delete a model from disk

ollama run also pulls the model if it is missing, so you can skip the first line. Inside the chat, type your question and press Enter. Type /bye to leave. You can also pass a one-off prompt, which is handy in scripts:

ollama run llama3.2 "Explain what a context window is in two sentences."

Choosing a quantized model

Most library models come in several quantizations: copies of the same weights stored at lower numeric precision. Lower precision means a smaller file and less memory, with some loss of quality. The llama3.2 tags page shows the trade-off clearly. The 3B instruct model is 6.4 GB at full 16-bit precision (fp16), 3.4 GB at 8-bit (q8_0), 2.0 GB at the common 4-bit q4_K_M level, and 1.4 GB at 2-bit (q2_K). The default llama3.2 tag points to the q4_K_M build.

Source: Ollama library, llama3.2 tags page, as listed on October 22, 2026.

A practical rule: start with the default 4-bit tag. Move up to q8_0 only if you have spare memory and notice quality problems. A larger model at 4-bit usually beats a smaller model at 8-bit for the same memory, but test it on your own tasks. Rankings like the ones in our model benchmark comparison are a guide, not a guarantee.

Step 3: Save a custom system prompt with a Modelfile

A Modelfile is a plain-text recipe for a new model. It must start with FROM and can add PARAMETER lines and a SYSTEM message. Create a file named Modelfile:

FROM llama3.2
PARAMETER temperature 0.3
PARAMETER num_ctx 8192
SYSTEM """You are a concise technical assistant. Answer in plain English, use short lists, and say clearly when you are not sure."""

Then build and run it:

ollama create tech-helper -f ./Modelfile
ollama run tech-helper

Here, temperature controls randomness: the documented default is 0.8, and lower values give steadier answers. num_ctx sets the context window in tokens. Ollama's FAQ says the default is 4,096 tokens, and you can raise it, at the cost of more memory. To see how an existing model was built, run ollama show --modelfile llama3.2.

Step 4: Call the local REST API with curl

While Ollama is running, it serves an HTTP API that by default listens only on 127.0.0.1, port 11434. The chat endpoint takes a model name and a list of messages, each with a role (system, user, assistant or tool) and content. Responses stream by default. Set "stream": false to get one JSON object back:

curl http://localhost:11434/api/chat -d '{
  "model": "tech-helper",
  "messages": [
    {"role": "user", "content": "What is quantization in one paragraph?"}
  ],
  "stream": false
}'

The reply includes message.content, plus timing statistics such as eval_count (tokens generated) and eval_duration (in nanoseconds). The API documentation notes that tokens per second equals eval_count / eval_duration × 10^9, a quick way to benchmark your hardware. GET /api/tags lists your installed models.

Step 5: Call Ollama from Python

Option A: plain HTTP with requests

This version has no Ollama-specific dependency, so the same code works against any machine that runs Ollama:

import json
import requests

URL = "http://localhost:11434/api/chat"

def ask(prompt, model="llama3.2"):
    payload = {
        "model": model,
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": prompt},
        ],
        "stream": True,
        "options": {"temperature": 0.3},
    }
    with requests.post(URL, json=payload, stream=True, timeout=300) as r:
        r.raise_for_status()
        for line in r.iter_lines():
            if not line:
                continue
            chunk = json.loads(line)
            print(chunk["message"]["content"], end="", flush=True)
            if chunk.get("done"):
                print()
                break

ask("Give me three tips for writing clear commit messages.")

Option B: the official ollama library

The MIT-licensed ollama Python package, which supports Python 3.8 and later, wraps the same API:

pip install ollama

from ollama import Client

client = Client(host="http://localhost:11434")
response = client.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
print(response.message.content)

Pass stream=True to chat() to receive chunks as they are generated. Change host to point at another computer on your network.

Next projects with your local model

The same local endpoint is the foundation for more ambitious builds. To answer questions over your own files, follow our tutorial on building a RAG app in Python with Ollama and Chroma. To expose your own tools and data to an assistant, see how to build an MCP server in Python. If a base model still is not good enough for a narrow task, our guide to fine-tuning an LLM with LoRA and QLoRA on a single GPU covers the next step, and the nine prompt engineering techniques in our guide are worth trying before you train anything.

Privacy trade-offs: what stays on your machine

The main reason to run a local LLM is control. Ollama's FAQ says that when you run locally, "we don't see your prompts or data." There are still three caveats:

• Cloud models are not local. The library also lists tags ending in cloud (for example gemma4:cloud), which run on Ollama's servers. Ollama says it does not store or log those prompts or use them for training. Even so, check the tag before you paste sensitive material.
• Opening the port widens exposure. The default bind address is 127.0.0.1. If you set OLLAMA_HOST to listen on your whole network, anyone who can reach port 11434 can use your models, because the documented API examples use no authentication. Put a firewall or an authenticating reverse proxy in front of it.
• Local still means stored. Models live in ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux and C:\Users\%username%\.ollama\models on Windows, and logs are kept on disk. Treat those files like any other data on the machine.

Organisations in Europe that deploy models internally should also check their obligations under the AI rules described in our EU AI Act explainer.

Troubleshooting common problems

• "Connection refused" on port 11434: the server is not running. Open the app on macOS or Windows, run sudo systemctl start ollama on Linux, or start it by hand with ollama serve.
• Very slow answers: run ollama ps and check the Processor column. "100% GPU" means the model fits in video memory. "100% CPU" or a split such as "48%/52% CPU/GPU" means it does not. Try a smaller model or a lower quantization.
• GPU not detected: update your drivers to at least the versions listed above. Background on the vendor software stacks is in our explainer on CUDA and its rivals.
• Running out of disk: set the OLLAMA_MODELS environment variable to a bigger drive and restart Ollama. On Linux, the ollama user needs read and write access to that folder.
• The model forgets earlier text: raise num_ctx in the Modelfile or in the API options.
• Checking logs: on Linux run journalctl -e -u ollama. On macOS, look in ~/.ollama/logs/server.log. On Windows, look in %LOCALAPPDATA%\Ollama.
• Model stays in memory, or unloads too soon: by default, a model stays loaded for five minutes after use. Change this with the keep_alive API field or the OLLAMA_KEEP_ALIVE variable.

What it means

Running a local LLM with Ollama has become a routine developer task rather than a research project. One installer, one pull command and a documented REST API give you a private model you can script against, with no per-token cost and no data leaving the machine. The trade-off is capability: a model that fits in 8 or 16 GB will not match the largest hosted systems on hard reasoning tasks. A sensible pattern is to use local models for drafts, private documents, offline work and prototyping, and to keep hosted APIs for the jobs that truly need them. Because the code above talks to a plain HTTP endpoint, switching between the two later is a small change, not a rewrite.

Sources

• Ollama: Download
• Ollama on GitHub (README, MIT license)
• Ollama README v0.3.0 (RAM guidance)
• Ollama API reference
• Ollama documentation: Modelfile reference
• Ollama documentation: CLI reference
• Ollama documentation: FAQ
• Ollama documentation: Hardware support
• Ollama documentation: macOS
• Ollama documentation: Windows
• Ollama documentation: Linux
• Ollama library: llama3.2 tags
• Ollama library: gemma4
• ollama-python library on GitHub