Using IPWorks LLM with Ollama

Requirements:
IPWorks LLM, Ollama

Introduction

The LLMClient component in IPWorks LLM can generate and stream replies from an Ollama model running locally, and list models available to the Ollama server. Ollama exposes an OpenAI-compatible Chat Completions endpoint; select spOllama to use its default local address.

This guide shows how to prepare a local model, generate a reply, stream it, and list available models.

Tip: Download the latest version of IPWorks LLM to follow along. Install Ollama before starting the local model.

Step 1: Install a Model

Install and start Ollama, then download the example model. The examples use qwen2.5:0.5b because it is small enough for a quick local test; another installed chat model may be used instead:

ollama pull qwen2.5:0.5b

List the installed models to confirm the model name:

ollama list

The Ollama desktop application normally starts the local service automatically. On systems where the service is not already running, start it from a terminal:

ollama serve

By default, Ollama's OpenAI-compatible base URL is http://localhost:11434/v1. This is also the default endpoint of spOllama.

Step 2: Generate a Response

Create an LLMClient, select spOllama, and set the model:

using nsoftware.IPWorksLLM;
LLMClient llmclient = new LLMClient();
llmclient.ServiceProvider = LLMClientServiceProviders.spOllama;
llmclient.Model = "qwen2.5:0.5b";
llmclient.AddMessage("user", "Reply with one sentence about local language models.");
llmclient.Generate();
Console.WriteLine(llmclient.ResponseText);

With ServerURL left empty, the component sends this request to http://localhost:11434/v1/chat/completions. If Ollama runs on another host or port, set ServerURL to that server's base URL, including /v1. The model name must match one available to that server. The default local endpoint does not require an API key.

Use spOllama rather than spOpenAI with a custom ServerURL. Ollama reads the output limit only from max_tokens, and spOllama sends MaxOutputTokens in that field; spOpenAI sends max_completion_tokens, which Ollama ignores.

Step 3: Stream a Response

Ollama's compatible endpoint supports streaming. Register ContentDelta and enable Stream before Generate. The event can also carry non-text parts, so display only ptText fragments:

llmclient.OnContentDelta += (s, e) =>
{
  if (e.PartType == (int)PartTypes.ptText)
    Console.Write(e.Delta);
};
llmclient.Stream = true;
llmclient.AddMessage("user", "Write a short poem about a quiet computer.");
llmclient.Generate();
Console.WriteLine();

After the stream finishes, the assembled reply is also available through ResponseText and the conversation collections.

Step 4: List Available Models

Use ListModels to obtain the model identifiers exposed by the local server:

llmclient.OnModelList += (s, e) => Console.WriteLine(e.Id);
llmclient.ListModels();

Use one of the returned identifiers for Model. This avoids assuming that a model downloaded on one machine is also present on another.

Troubleshooting

• Connection refused: Confirm that Ollama is running. If it does not listen on localhost:11434, confirm that ServerURL names its host and port and includes /v1.
• Model not found: Run ollama list, then set Model to an installed identifier or download it with ollama pull.
• Slow first response: Loading a model into memory can make the first request slower than later requests. Model size and available CPU, GPU, and memory also affect response time.
• Error 804: With spOllama, the ToolChoice configuration setting must be empty or auto. CountTokens is not available because the Chat Completions API has no token-counting endpoint.
• Unsupported option: Ollama implements a subset of the OpenAI API. A field Ollama does not recognize may be ignored rather than rejected, so verify options against the Ollama compatibility documentation.

To try a local model without writing code, run the LLMClient Chat demo included with IPWorks LLM with --provider ollama --model qwen2.5:0.5b. It holds a multi-turn conversation, streams replies with --stream, and can switch the same conversation to another provider.

Conclusion

The spOllama setting uses Ollama's local API endpoint. With a model installed, the same LLMClient instance can continue the conversation or stream later replies. For provider switching, see Getting Started with IPWorks LLM. For current compatibility details, see the Ollama OpenAI compatibility documentation.