TeknoTUT

How to Use Ollama with Python: Build a Local AI Chatbot

A practical guide to using Ollama with Python: install the official library, stream responses, build a chatbot with memory, and get structured JSON.

Written by 14 min readBahasa Indonesia
Illustration of a Python code window connected to an AI chip and chat bubbles
On This Page

If you have already run an AI model on your own computer with Ollama, you know how fun it is to chat with an AI straight from the terminal. But chatting alone isn't quite enough, right? As developers, we usually want to go one step further: calling that model from our own code.

Imagine building a personal chatbot, a document summarizer, or a script that automatically classifies customer reviews, all running on your own machine. No per-token API bills, and no data sent to someone else's server.

The good news is that Ollama ships an official Python library that makes all of this possible with just a few lines of code. You don't have to craft HTTP requests by hand.

In this article, we'll cover how to use Ollama with Python from start to finish: installing the library, sending your first message, streaming the response, building a chatbot that remembers the conversation, and asking for answers in structured JSON. Let's get started!

What is the Ollama Python Library?

Ollama is an application for running Large Language Models (LLMs) locally. While it runs, Ollama exposes a REST API at http://localhost:11434 that any application can call.

The ollama library for Python is the official package from the Ollama team that wraps that REST API in clean Python functions. Instead of writing HTTP requests and parsing JSON yourself, you simply call a function like chat() and get a response object back.

Here's an analogy: the REST API is the restaurant kitchen, and this library is the waiter. You just say what you want, and the waiter handles the communication with the kitchen.

Note: the library is only a connector. The AI model is still run by the Ollama application, so Ollama must already be installed and running on your computer (or on a server you can reach).

Why Use the Official Ollama Library?

There are actually several ways to call Ollama from Python, such as the requests library or the OpenAI library pointed at the /v1 endpoint. So why pick the official one?

  • Less code: a single chat() call is enough to send a message and receive the answer.
  • Supports every Ollama feature: streaming, structured outputs, tool calling, embeddings, and model management (pull, list, delete) are all available directly.
  • Typed response objects: you can write response.message.content, and editors like VS Code can offer autocomplete.
  • Sync and async versions: there's Client for regular scripts and AsyncClient for asyncio-based applications such as FastAPI.

Drawbacks You Should Know

To keep things balanced, here are a few points to consider:

  • Tied to Ollama: if you ever want to switch to another AI provider, your code will need changes. If portability matters to you, the OpenAI-compatible endpoint may be the more flexible choice.
  • Speed depends on your hardware: the library doesn't make the model any faster. Without a GPU, answers from large models will still feel slow.

Preparation

Before we begin, make sure you have:

  • Ollama installed and running. If you don't, follow the guide on how to install Ollama on Ubuntu first.
  • At least one downloaded model. The examples in this article use the lightweight llama3.2 (about a 2 GB download). Feel free to swap in another model.
  • Python 3.8 or newer. Ubuntu 24.04 LTS and 26.04 LTS already ship a much newer Python 3 than that.
  • A code editor, such as VS Code or simply nano.

The commands in this article are intended for Ubuntu 24.04 LTS and 26.04 LTS. On macOS and Windows the Python code is exactly the same; only the way you create the virtual environment differs slightly.

If the model isn't on your computer yet, download it with the following command:

ollama pull llama3.2

Then make sure the Ollama server responds:

curl http://localhost:11434

If the output is the text Ollama is running, you're ready to continue.

How to Use Ollama with Python

Step #1: Create a Project Folder and Virtual Environment

On recent Ubuntu releases, running pip install directly on the system is rejected with the message error: externally-managed-environment. This isn't a bug; it's a safeguard that keeps the system's own Python packages from breaking. The solution is a virtual environment (venv), a separate Python environment dedicated to your project.

First, install the venv support packages with the following command:

sudo apt update
sudo apt install -y python3-venv python3-pip

Next, create a project folder and move into it:

mkdir ~/chatbot-ollama
cd ~/chatbot-ollama

Create a virtual environment in the .venv folder, then activate it:

python3 -m venv .venv
source .venv/bin/activate

If it worked, (.venv) appears at the beginning of your terminal prompt. From now on, every package you install applies to this project only.

Each time you open a new terminal, reactivate the venv with source .venv/bin/activate from inside the project folder. To leave the venv, type deactivate.

Step #2: Install the Ollama Library

With the venv active, install the ollama library with the following command:

pip install ollama

This command also installs its two dependencies automatically: httpx (for HTTP connections) and pydantic (for data validation). We'll use pydantic later in the structured JSON section.

To confirm the library is installed, check it with the following command:

pip show ollama

If the output shows the package name, version, and location, the installation succeeded.

Step #3: Send Your First Message to the Model

Now for the part you've been waiting for. Create a new file named halo.py:

nano halo.py

Add the following code:

from ollama import chat
 
response = chat(
    model="llama3.2",
    messages=[
        {"role": "user", "content": "Explain what Python is in two sentences."},
    ],
)
 
print(response.message.content)

Save the file by pressing Ctrl+X, then Y, and Enter. Run it with the following command:

python halo.py

After a few seconds, the model's answer appears in your terminal. Done! You've just called a local AI model from Python.

Let's break the code down:

  • model: the name of the model to use; it must match a name in the output of ollama list.
  • messages: the list of messages in the conversation. Each message has a role and a content.
  • response.message.content: the text of the model's answer. You can also write it dictionary-style: response["message"]["content"].

Ollama recognizes four role values:

Role Purpose
system Base instructions that shape the model's behavior
user A message from the user
assistant An answer from the model
tool The result of a tool call (used in tool calling)

The first call is usually slower because Ollama has to load the model into memory first. Subsequent calls are faster as long as the model stays loaded (by default, 5 minutes after it was last used).

Step #4: Stream the Response

In the previous example, we had to wait until the entire answer was generated before any text appeared. For long answers, that feels slow. The solution is streaming: the answer is displayed word by word as the model writes it, just like in ChatGPT.

One thing to know: in Ollama's REST API streaming is enabled by default, but in the Python library it is disabled by default. So we need to add stream=True.

Create a file named streaming.py with the following content:

from ollama import chat
 
stream = chat(
    model="llama3.2",
    messages=[
        {"role": "user", "content": "Write three tips for beginners learning to code."},
    ],
    stream=True,
)
 
for chunk in stream:
    print(chunk.message.content or "", end="", flush=True)
 
print()

Run it with python streaming.py, and you'll see the answer flow in bit by bit.

With stream=True, the chat() function no longer returns one complete answer; it returns a series of pieces (chunks). The end="" option stops print from starting a new line for every chunk, and flush=True makes sure the text shows up immediately instead of sitting in a buffer.

Step #5: Build a Chatbot That Remembers the Conversation

Here's something important that often surprises beginners: the model remembers nothing between two calls. Every chat() call stands on its own. If you ask "What's my name?" in the second call, the model has no idea you introduced yourself in the first.

So how does a chatbot "remember"? The trick is simple: we keep the whole conversation history in a list and send all of it every time we call the model.

Create a file named chatbot.py with the following content:

from ollama import chat
 
MODEL = "llama3.2"
 
messages = [
    {
        "role": "system",
        "content": "You are a friendly assistant. Answer briefly and clearly.",
    },
]
 
print("Chatbot ready. Type 'exit' to quit.\n")
 
while True:
    try:
        user_input = input("You: ").strip()
    except (EOFError, KeyboardInterrupt):
        print()
        break
 
    if not user_input:
        continue
    if user_input.lower() in ("exit", "quit"):
        break
 
    messages.append({"role": "user", "content": user_input})
 
    print("AI: ", end="", flush=True)
    reply = ""
    stream = chat(model=MODEL, messages=messages, stream=True)
    for chunk in stream:
        text = chunk.message.content or ""
        print(text, end="", flush=True)
        reply += text
    print("\n")
 
    messages.append({"role": "assistant", "content": reply})

Run it with the following command:

python chatbot.py

Try introducing yourself, then ask for your name again in the next message. The conversation looks roughly like this (the model's answers will of course vary):

Chatbot ready. Type 'exit' to quit.
 
You: Hi, my name is Budi and I'm learning Python.
AI: Hi Budi! Nice to meet you. How can I help you with Python?
 
You: What's my name?
AI: Your name is Budi.

Pretty neat, right? Here's how the script works:

  1. The messages list starts with a system message that sets the chatbot's personality. Change it however you like.
  2. Every user input is appended to messages with the user role.
  3. The entire messages list is sent to the model, and the answer is streamed to the screen while being collected in the reply variable.
  4. The full answer is appended back to messages with the assistant role, so it gets sent along on the next turn.

Watch the history length. Every model has a context window, the maximum number of tokens it can read at once. Because the whole history is resent on every turn, a very long conversation will eventually exceed that limit, and the earliest messages start to be "forgotten". The easiest fix is to cap the history, for example by adding the line messages = messages[:1] + messages[-20:] before calling chat(). That line keeps the system message and only the last 20 messages.

Step #6: Tune Model Behavior with Options

The chat() function accepts an options parameter that controls how the model generates its answer. Here's an example:

from ollama import chat
 
response = chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Write one slogan for a coffee shop."}],
    options={
        "temperature": 0.8,
        "num_ctx": 8192,
        "num_predict": 200,
    },
    keep_alive="10m",
)
 
print(response.message.content)

The most commonly used options:

Option Purpose
temperature The level of "creativity". Low values (e.g. 0) make answers consistent, high values make them more varied
num_ctx The context window size in tokens. Bigger fits more history, but memory usage goes up too
num_predict The maximum number of tokens generated in one answer
seed A reference number that makes results repeatable for the same input
stop A list of strings that make the model stop writing

The keep_alive parameter (outside options) controls how long the model stays loaded in memory after use. A value of "10m" means 10 minutes, while 0 unloads the model right after it answers.

Step #7: Ask for Structured JSON Output

Free-form text is pleasant for humans to read, but hard for a program to process. Say you want to sort customer reviews into positive or negative: you need an answer in a guaranteed format, not a long paragraph.

This is where structured outputs come in. We define the shape of the data we want with pydantic, then pass its schema through the format parameter. Ollama forces the model to answer according to that schema.

Create a file named analisis_ulasan.py with the following content:

from typing import Literal
 
from ollama import chat
from pydantic import BaseModel
 
 
class Review(BaseModel):
    product: str
    sentiment: Literal["positive", "negative", "neutral"]
    score: int
    summary: str
 
 
text = (
    "I bought this mechanical keyboard last week. The keys sound great "
    "and shipping was fast, but the RGB lighting died after three days."
)
 
response = chat(
    model="llama3.2",
    messages=[
        {
            "role": "user",
            "content": f"Analyze the following review. Give a score from 1 to 5.\n\n{text}",
        },
    ],
    format=Review.model_json_schema(),
    options={"temperature": 0},
)
 
review = Review.model_validate_json(response.message.content)
 
print("Product  :", review.product)
print("Sentiment:", review.sentiment)
print("Score    :", review.score)
print("Summary  :", review.summary)

Run it with python analisis_ulasan.py. The output looks roughly like this:

Product  : mechanical keyboard
Sentiment: neutral
Score    : 3
Summary  : The keys sound great and shipping was fast, but the RGB lighting died after three days.

A few things worth noting in the code above:

  • Review.model_json_schema() converts the pydantic class into a JSON Schema that Ollama understands.
  • Literal[...] restricts sentiment to just those three choices.
  • Review.model_validate_json(...) turns the model's JSON text into a Python object and validates its types at the same time. The result is ready to use, for example to store in a database.
  • temperature: 0 is a tip from the Ollama documentation for more consistent results.

A schema guarantees the shape, not the correctness of the content. The model can still misjudge the sentiment or give a score outside the 1–5 range, especially a small model. For important tasks, double-check the results or use a larger model.

Step #8: Handle Errors Gracefully

A good script doesn't just die with a long traceback when something goes wrong. Two kinds of errors show up most often:

  • ConnectionError: the library can't connect to the Ollama server, usually because Ollama isn't running.
  • ollama.ResponseError: the Ollama server replied with an error, for example because the model hasn't been downloaded (status 404).

Here's how to handle them, in a file named aman.py:

import ollama
 
MODEL = "llama3.2"
 
try:
    response = ollama.chat(
        model=MODEL,
        messages=[{"role": "user", "content": "Hello!"}],
    )
    print(response.message.content)
except ollama.ResponseError as e:
    print("Error from Ollama:", e.error)
    if e.status_code == 404:
        print(f"Model not found. Download it with: ollama pull {MODEL}")
except ConnectionError:
    print("Cannot connect to Ollama. Check it with: systemctl status ollama")

With this pattern, the people using your script get a clear hint about what to do next.

Tips for Getting Better Results

Tip #1: Measure Model Speed from Python

Every (non-streaming) response includes processing statistics. You can calculate the model's speed in tokens per second by adding the following code after the chat() call:

tokens = response.eval_count
seconds = response.eval_duration / 1_000_000_000
print(f"{tokens} tokens in {seconds:.1f} seconds ({tokens / seconds:.1f} tokens/second)")

The eval_duration value is in nanoseconds, which is why we divide by one billion. This number is handy for comparing several models on your machine.

Tip #2: Connect to Ollama on Another Computer

By default, the library connects to http://localhost:11434. If Ollama runs on a different machine, such as a GPU-equipped PC on your home network, create a Client object with that server's address:

from ollama import Client
 
client = Client(host="http://192.168.1.50:11434")
 
response = client.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Hello from another computer!"}],
)
print(response.message.content)

Alternatively, set the OLLAMA_HOST environment variable before running your script, and the library picks it up automatically.

Warning: the Ollama API has no authentication. To be reachable from other machines, the Ollama server has to listen on 0.0.0.0, and that is only safe on a trusted local network. Do not open port 11434 to the internet. For a server on a VPS, use an SSH tunnel and make sure you've followed the guide on how to secure a Linux VPS.

Tip #3: Use AsyncClient for Web Applications

If you're building a web app or a bot that serves many users at once, use AsyncClient so your application doesn't "freeze" while waiting for the model:

import asyncio
 
from ollama import AsyncClient
 
 
async def main():
    response = await AsyncClient().chat(
        model="llama3.2",
        messages=[{"role": "user", "content": "What is asyncio?"}],
    )
    print(response.message.content)
 
 
asyncio.run(main())

It works the same way as the sync version, just with await added.

Troubleshooting Common Problems

Problem Cause Solution
error: externally-managed-environment pip install was run outside the venv Activate the venv with source .venv/bin/activate
ModuleNotFoundError: No module named 'ollama' The script was run outside the venv, or the library isn't installed Activate the venv, then run pip install ollama
ConnectionError: Failed to connect to Ollama The Ollama server isn't running Run sudo systemctl start ollama
ResponseError with status 404 The model isn't downloaded or its name is misspelled Check ollama list, then ollama pull <model>
The first answer is very slow The model is being loaded into memory Wait, or extend keep_alive
The chatbot forgets the start of the conversation The history exceeds the context window Cap the history or increase num_ctx
A file named ollama.py causes an import error The file name clashes with the library name Rename your file; don't call it ollama.py

Conclusion

From this article, we can conclude that using Ollama with Python is remarkably easy. With the official ollama library, a few lines of code are enough to send messages to a model, stream the answer, build a chatbot that remembers the conversation, and get structured JSON that's ready for your program to process.

Things to Remember

  • The model has no memory. You have to store the conversation history and resend it on every call.
  • Streaming in the Python library is disabled by default, so add stream=True.
  • Structured outputs guarantee the shape of the data, not the correctness of its content.
  • Always work inside a virtual environment to keep the system Python clean.

Next Steps

Once you've mastered the basics, you can take this chatbot further. For example, learn tool calling so the model can call your Python functions, or embeddings to build document search. If you want your script to run continuously on a server, say as a bot, check out the guide on how to create a systemd service on Linux.


Have fun experimenting with different models and system prompts, and stay tuned to Teknotut!

I hope this article helps.

Continue Reading

More articles you might enjoy.