How to Install Ollama on Ubuntu to Run AI Models Locally

If you are a developer, you're probably familiar with ChatGPT, Claude, or Gemini. AI assistants like these are incredibly helpful, from writing code and summarizing documents to answering technical questions. But all of them run on someone else's servers.

That means every prompt you send, including snippets of work code or personal data, travels over the internet. On top of that, API costs keep growing with usage, and rate limits tend to show up at the worst possible moment.

So what if the AI model ran directly on your own computer or server? That's exactly what Ollama offers: an open source application that makes running Large Language Models (LLMs) locally as easy as typing a single command in the terminal.

In this article, we'll cover everything about installing Ollama on Ubuntu, from understanding what Ollama is, choosing a model that fits your RAM, chatting with AI in the terminal, and using its API from your apps, to setting up a ChatGPT-like web interface with Open WebUI. Let's get started!

What is Ollama?

Ollama is an application for downloading, managing, and running large language models on your own machine. It's available for Linux, macOS, and Windows, and its source code is open on GitHub.

If you've used Docker before, Ollama will feel very familiar. Just like docker pull nginx downloads an image, ollama pull llama3.2 downloads a model from the Ollama library. And just like docker run, ollama run starts the model right away and opens a chat session.

Under the hood, Ollama runs as a server on port 11434. This server loads models into RAM or GPU VRAM and handles requests from the terminal, applications, and web interfaces. There's a wide choice of models too, from Meta's Llama, Google's Gemma, and Alibaba's Qwen, to gpt-oss, OpenAI's open-weight model.

Why Run AI Locally?

Running AI on your own machine comes with some compelling benefits:

• Privacy: your prompts and data never leave your computer or server. Great for internal code, work documents, or sensitive data.
• Free with no quotas: no per-token costs and no rate limits. The only limit is your hardware.
• Works offline: once a model is downloaded, you don't need an internet connection anymore.
• Easy to integrate: Ollama provides a REST API, including endpoints compatible with the OpenAI API, so many apps and libraries can use it right away.
• Freedom to experiment: you can try dozens of different models and compare them yourself.

Drawbacks to Keep in Mind

To be fair, there are also a few things you should know before you begin:

• Not as smart as the biggest commercial models: models that fit on a typical laptop or VPS are generally still behind the flagship models of ChatGPT or Claude, especially for complex tasks.
• Needs a decent amount of RAM: even small models need a few GB of RAM, and smarter models need much more.
• Slow without a GPU: on CPU only, you'll still get answers, but medium-sized and larger models can feel slow.

Preparation

Before getting started, make sure you have:

• Ubuntu 24.04 LTS or 26.04 LTS, on a PC, laptop, or VPS. The commands in this article are intended for both versions, and the process is similar on other Linux distros.
• At least 8 GB of RAM. With 4 GB you can still try very small models, but your options are limited.
• At least 10–20 GB of free disk space, since a single model can be several GB.
• A GPU (optional): an NVIDIA or AMD graphics card makes Ollama much faster. Without a GPU, Ollama automatically runs on the CPU.
• A user account with sudo access.

Using a VPS? Make sure your VPS is secured first. Follow the guide on how to secure a new Linux VPS. If your VPS is tight on RAM, adding swap memory can prevent the Ollama process from being killed when memory runs out, although swap is much slower than RAM.

Choosing a Model for Your RAM

This is the part that confuses beginners the most. In general, the model has to fit in RAM (or in GPU VRAM to be fast), plus some headroom for the system and the conversation context.

Here are some popular models from the Ollama library and their download sizes at the time of writing:

The Suggested RAM column above is a rough estimate to keep the system comfortable, not an official figure. Model sizes can also change when models are updated. Always check the model page on ollama.com/library before downloading.

The number after a model name, like 3b or 7b, is the number of parameters in billions. The bigger the number, the smarter the model usually is, but also the heavier. For your first try, llama3.2 is a safe choice because it's light and responsive.

How to Install Ollama on Ubuntu

Step #1: Update the System and Install Dependencies

First, update the package list and install curl and zstd. The zstd package is needed by the Ollama install script to extract its release files, and some minimal Ubuntu images (especially on VPSes) don't ship with it.

sudo apt update
sudo apt install -y curl zstd

Step #2: Run the Official Install Script

Ollama provides an official install script that automatically downloads the binary, creates an ollama system user, registers a systemd service, and detects your GPU. Run the following command:

curl -fsSL https://ollama.com/install.sh | sh

Security tip: piping a script from the internet straight into sh should only be done for sources you trust. If you'd like to inspect it first, download it with curl -fsSL https://ollama.com/install.sh -o install.sh, read it, then run it with sh install.sh.

This takes a few minutes depending on your internet speed. At the end of the installation, the output looks roughly like this:

>>> The Ollama API is now available at 127.0.0.1:11434.
>>> Install complete. Run "ollama" from the command line.
WARNING: No NVIDIA/AMD GPU detected. Ollama will run in CPU-only mode.

The last line only appears if no NVIDIA or AMD GPU was detected on your machine. It's not an error; Ollama still works normally on the CPU.

Step #3: Make Sure Ollama Is Running

Check the installed Ollama version with this command:

ollama -v

Then make sure the systemd service is active:

systemctl status ollama

If you see active (running), the Ollama server is up and will start automatically every time the system boots. Press q to exit the status view.

You can also verify that the API responds with the following command:

curl http://localhost:11434

The output is the text Ollama is running. Simple, right?

Step #4: Download and Run Your First Model

Now for the fun part. Run the llama3.2 model with this command:

ollama run llama3.2

If the model isn't on your machine yet, Ollama downloads it first (about 2 GB), then opens a chat session with a >>> prompt. Go ahead and ask anything:

>>> Explain what Docker is in two sentences.

The model answers right in the terminal, word by word. Pretty cool, right? All of this runs 100% on your machine, without sending anything to the internet.

Some useful commands inside the chat session:

• Type /? to see the list of available commands.
• Use """ to type a multi-line message, then close it again with """.
• Type /bye or press Ctrl+D to exit.

Curious how fast the model runs on your machine? Add the --verbose option:

ollama run llama3.2 --verbose

After each answer, Ollama shows statistics such as eval rate in tokens per second. As a rough guide, anything above 10 tokens/second usually feels comfortable to read.

How to Use Basic Ollama Commands

Once your first model is running, here are the commands you'll use most often.

To download a model without running it right away, use pull:

ollama pull qwen2.5-coder:7b

To list all downloaded models along with their sizes:

ollama list

To see which models are currently loaded in memory:

ollama ps

The output looks roughly like this:

NAME               ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:latest    a80c4f17acd5    3.4 GB    100% CPU     4096       4 minutes from now

The PROCESSOR column is very handy for checking whether a model runs on the GPU. 100% GPU means the model is fully loaded on the GPU. A mix like 48%/52% CPU/GPU means your GPU doesn't have enough VRAM and part of the model runs on the CPU, which is slower.

The UNTIL column shows when the model will be unloaded from memory. By default, a model stays loaded for 5 minutes after it was last used. To unload it right now, use this command:

ollama stop llama3.2

Finally, to delete a model you no longer use and free up disk space:

ollama rm qwen2.5-coder:7b

On Linux, all models downloaded through the systemd service are stored in /usr/share/ollama/.ollama/models. If your main disk is small, you can move this location with the OLLAMA_MODELS variable (set the same way as in the next section), as long as the ollama user has read and write access to the new folder.

Accessing Ollama Through the API

This is the feature that makes Ollama so appealing to developers. Besides the terminal, models can be called from any application through the REST API at http://localhost:11434.

As the simplest example, send a message to a model through the /api/chat endpoint with this command:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{ "role": "user", "content": "Why is the sky blue?" }],
  "stream": false
}'

The "stream": false option makes the response arrive all at once as a single JSON object. The model's answer is in message.content.

Using the OpenAI-Compatible Endpoint

Ollama also provides OpenAI API-compatible endpoints under /v1. That means apps or libraries originally built for OpenAI can be pointed at Ollama just by changing the base URL.

Here's an example calling the /v1/chat/completions endpoint with curl:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [{ "role": "user", "content": "Write a JavaScript function that reverses a string." }]
  }'

If you use the official OpenAI library for Python, just change base_url and set api_key to any text (it's required, but ignored by Ollama):

from openai import OpenAI
 
client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",
)
 
response = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "What is Ubuntu?"}],
)
 
print(response.choices[0].message.content)

This way, you can build your own chatbot, document summarizer, or coding assistant that runs entirely locally.

Setting Up a ChatGPT-Like Web Interface with Open WebUI

Chatting in the terminal is handy, but for daily use, a web interface feels much more comfortable. Open WebUI is an open source web interface that looks a lot like ChatGPT and connects directly to Ollama.

The easiest way to install it is with Docker. If Docker isn't installed yet, first read getting to know Docker containers and install Docker Engine from its official documentation.

By default, Ollama only listens on 127.0.0.1, so it can't be reached from Docker's regular internal network. The simplest solution is to run Open WebUI with host networking, as recommended by the Open WebUI documentation:

docker run -d --network=host \
  -v open-webui:/app/backend/data \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
  --name open-webui --restart always \
  ghcr.io/open-webui/open-webui:main

Wait about a minute for the container to finish starting, then open http://localhost:8080 in your browser. Create your first account, which automatically becomes the administrator. After that, pick a model at the top of the page and start chatting.

Running on a VPS? Don't just open port 8080 to the internet. Access Open WebUI through an SSH tunnel, for example ssh -L 8080:localhost:8080 [email protected], then open http://localhost:8080 on your computer. If you want to use it permanently with a domain, set up a reverse proxy with HTTPS. To manage several containers at once, you can also define them in a Compose file as shown in learning Docker Compose.

Advanced Ollama Configuration

On Linux, Ollama is configured through environment variables on the systemd service. The recommended way is to create an override file with this command:

sudo systemctl edit ollama

An editor will open. Add the settings you need under the [Service] section, for example:

[Service]
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_CONTEXT_LENGTH=8192"

Save the file, then reload the systemd configuration and restart Ollama:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Some of the most commonly used variables:

• OLLAMA_KEEP_ALIVE: how long a model stays loaded in memory after it was last used (5 minutes by default). A longer value makes the first response faster, but keeps memory in use.
• OLLAMA_CONTEXT_LENGTH: the context length (how many tokens the model can "remember" in one conversation). A larger value helps with long documents, but increases memory usage.
• OLLAMA_MODELS: where models are stored.
• OLLAMA_HOST: the address and port the server listens on, for example 0.0.0.0:11434 to make it reachable from other machines on the network.

Warning: the Ollama API has no authentication. If you set OLLAMA_HOST to 0.0.0.0 on a VPS with a public IP, anyone on the internet can use your models (and your server's resources). Only do this on a trusted local network, restrict it with a firewall (for example sudo ufw allow from 192.168.1.0/24 to any port 11434), or use an SSH tunnel.

How to Update and Uninstall Ollama

To update Ollama to the latest version, simply re-run the install script. Your downloaded models won't be deleted.

curl -fsSL https://ollama.com/install.sh | sh

If you ever want to remove Ollama completely, stop and remove the service first:

sudo systemctl stop ollama
sudo systemctl disable ollama
sudo rm /etc/systemd/system/ollama.service

Then remove the libraries, binary, system user, and models folder:

sudo rm -r $(which ollama | tr 'bin' 'lib')
sudo rm $(which ollama)
sudo userdel ollama
sudo groupdel ollama
sudo rm -r /usr/share/ollama

The last command deletes all the models you've downloaded. Make sure you really don't need them anymore.

Troubleshooting Common Problems

Here are some fairly common problems and how to solve them:

To see the full Ollama logs, use the following command:

journalctl -e -u ollama

Conclusion

From this article, we can conclude that running AI locally is now very easy thanks to Ollama. With one install script and a single ollama run command, you can chat with models like Llama, Gemma, or Qwen right on your own machine, with no per-token costs and without sending your data to anyone else.

The key is choosing a model that fits your RAM. Start with a small model like llama3.2, measure its speed with --verbose, then gradually move up in size if your hardware can handle it.

Who Should Use Ollama?

• Developers who want to build AI apps without API costs, thanks to the OpenAI-compatible API.
• Privacy-conscious users who want to process sensitive documents or code locally.
• Self-hosting enthusiasts who want their own "private ChatGPT" on a home server or VPS with Open WebUI.

Next Steps

Once Ollama is running, try different models for different needs, for example qwen2.5-coder to help you write code. If you'd like to compare it with a cloud-based AI coding assistant, check out our review of Grok Code Fast 1.

Resources and Useful Links

• Official Ollama website
• Ollama Linux installation docs
• Ollama FAQ
• Ollama model library
• Open WebUI documentation

If you have any questions or run into problems while following this tutorial, feel free to ask. Stay tuned to Teknotut!

I hope this article helps.