[Suggestion] Ability to send num_ctx parameter to Ollama to change Context Window size

[krissrex]

By default, ollama will use the num_ctx set in a modelfile parameters, or fall back to a low value between 1k and 8k. I think the default depends on how ollama is used (cli vs api).

In a chat, I can change the context window with /set parameter num_ctx 131072 for much higher memory usage and full context of llama3.2.
In the API, the options object can take a num_ctx https://ollama.readthedocs.io/en/api/#request_7 .

For some tasks, we want a much higher context window.

Workaround

The currently available option is to create a new model with num_ctx, like installing a new Modelfile or running /set parameter num_ctx 20000 followed by /save llama3.2-20k_ctx .
Or set the global default when starting ollama, with environment variable OLLAMA_CONTEXT_LENGTH=20000.

ollama logs:

llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 131072
llama_context: n_ctx_per_seq = 131072
llama_context: n_batch       = 512
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 1
llama_context: freq_base     = 500000.0
llama_context: freq_scale    = 1

---

Details and notes

Enable debug logging

OLLAMA_DEBUG=1 ollama serve

Ollama will log during model loading, pay attention to runner.num_ctx=8192:

time=2025-06-17T15:46:27.259+02:00 level=DEBUG source=sched.go:495 msg="finished setting up" runner.name=registry.ollama.ai/library/llama3.2:latest runner.inference=metal runner.devices=1 runner.size="3.3 GiB" runner.vram="3.3 GiB" runner.parallel=2 runner.pid=36317 runner.model=/Users/kristian/.ollama/models/blobs/sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff runner.num_ctx=8192

llama_context: constructing llama_context
llama_context: n_seq_max     = 2
llama_context: n_ctx         = 8192
llama_context: n_ctx_per_seq = 4096
llama_context: n_batch       = 1024
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 1
llama_context: freq_base     = 500000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized

If I do /set parameter num_ctx 5 in a ollama run llama3.2:latest chat, I get a stupid assistant and this log:

time=2025-06-17T15:50:28.050+02:00 level=DEBUG source=sched.go:495 msg="finished setting up" runner.name=registry.ollama.ai/library/llama3.2:latest runner.inference=metal runner.devices=1 runner.size="2.8 GiB" runner.vram="2.8 GiB" runner.parallel=2 runner.pid=37551 runner.model=/Users/kristian/.ollama/models/blobs/sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff runner.num_ctx=10

llama_context: constructing llama_context
llama_context: n_batch is less than GGML_KQ_MASK_PAD - increasing to 64
llama_context: n_seq_max     = 2
llama_context: n_ctx         = 10
llama_context: n_ctx_per_seq = 5
llama_context: n_batch       = 64
llama_context: n_ubatch      = 64
llama_context: causal_attn   = 1
llama_context: flash_attn    = 1
llama_context: freq_base     = 500000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (5) < n_ctx_train (131072) -- the full capacity of the model will not be utilized

Also lots of warnings:

time=2025-06-17T15:50:29.721+02:00 level=DEBUG source=cache.go:240 msg="context limit hit - shifting" id=0 limit=5 input=5 keep=4 discard=1

See also ollama/ollama#2714

[ptitjes]

@krissrex I am quite confused by your report. Why doesn't Ollama use a context window appropriate to the given prompt? (capped by the actual model's max context length) Isn't that a bug in Ollama?

[krissrex]

I am quite confused by your report.

tl;dr llama3.2 runs at 4k context despite the model handling 128k. short context = forgetful AI

Why doesn't Ollama use a context window appropriate to the given prompt? (capped by the actual model's max context length) Isn't that a bug in Ollama?

I don't know. Changing the context size requires a model reload. My guess is an optimization by ollama to make models easier to run at the expense of quality.
(Note, they are adding new optimizations as well, like OLLAMA_KV_CACHE_TYPE="q4_0". See https://mitjamartini.com/blog/kv-cache-quantization-in-ollama/)

https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-specify-the-context-window-size

The llama3.2 modelfiles do not set the num_ctx parameter, so they use the default 4k instead of 128k.
This means the AI will forget previous input when the prompt passes approx 3500 words. If you search documents without using RAG, and verbose tools, this is an issue. You will also see warnings like

time=2025-06-17T23:33:58.093+02:00 level=WARN source=runner.go:128 msg="truncating input prompt" limit=4096 prompt=5531 keep=5 new=4096

People have been tripped by this before
https://news.ycombinator.com/item?id=42833427

[aozherelyeva]

Hi @krissrex, thanks for opening the issue!
@Rizzen, could you please take a look at the suggestion? Thanks!

[ptitjes]

@krissrex I took some time looking at the (many) different issues around in the Ollama repository (and related repositories) and I think you are right that we need to handle this. I don't think there will be an option anytime soon on Ollama's side to compute an optimal num_ctx based on the provided prompt (though they have the tokenizer at hand).

However, the only designs I see would be:

Extending LLMParams, not really possible

We would add an additional parameter in LLMParams, though this would need a consistent treatment amongst all the LLM client, which will be very difficult to achieve.

For instance, OpenAI API has a max_tokens parameter, but it is deprecated in favor of max_completion_tokens which only dictates the number of tokens of the output.

Making it an implementation detail of OllamaClient

We would add a functional parameter to OllamaClient, that would compute a num_ctx for a particular prompt.

class OllamaClient(
  // ...
  allocatedContextSize: (Prompt) -> Int? = { null }, // or in a future `OllamaClientSettings`
)

If allocatedContextSize returns something other than null, then num_ctx would be set in the Ollama request.

In the simplest cases, the user would for example do:

val client = OllamaClient(
  // ...
  allocateContextSize = { 8196 },
)

@Rizzen WDYT?

[ptitjes]

Or maybe, to make it more future-proof:

interface OllamaRequestBuilder {
  var numCtx: Int? = null
}

class OllamaClient(
  // ...
  requestBuilder: OllamaRequestBuilder.(Prompt) -> Unit = {},
)

and

val client = OllamaClient(
  // ...
  requestBuilder = { numCtx = 8196 }
)

which allows us to add more request customisation options in the future.

[krissrex]

There are many other options in the API: https://ollama.readthedocs.io/en/api/#request_7 . I don't know how many would be relevant for users to tweak.

An important thing to note, is that if Koog ever uses the OpenAI Compatible API of Ollama, num_ctx is not possible to change: https://ollama.readthedocs.io/en/openai/#setting-the-context-size .

---

Regarding dynamic context size:

  allocatedContextSize: (Prompt) -> Int? = { null }, // or in a future `OllamaClientSettings`

I am not sure if this is clever or not. If the model needs to reload when changing size, this could slow down prompt completion speeds due to reloads. I don't know the actual impact of changing this parameter on the fly.

A third option is to hope a user has enough memory to use the max context window, and set the value according to the Ollama modelfile. https://ollama.readthedocs.io/en/api/#response_22 $.model_info.llama.context_length. (If ollama is started with OLLAMA_KV_CACHE_TYPE="q4_0" then using the full context window is less straining on the hardware).

[ptitjes]

I don't know how many would be relevant for users to tweak.

For now, I would add seed (would be practical for testing) and maybe numPredict.

I am not sure if this is clever or not. If the model needs to reload when changing size, this could slow down prompt completion speeds due to reloads. I don't know the actual impact of changing this parameter on the fly.

For sure, but I would rather have the option than not. We can always document that changing numCtx might force the model to be reloaded, and recommend using a "stable" value. One could for example use it like this:

  requestBuilder = { prompt ->
    val tokenCount = prompt.lastTokenUsage
    numCtx = maxOf(2048, ((tokenCount + 500) / 1024 + 1) * 1024)
  }

@Rizzen @Ololoshechkin would you be against introducing such a requestBuilder parameter to OllamaClient?

[ptitjes]

Can I take this?

[Ololoshechkin]

@ptitjes sure, please feel free to take this one!
From my side, I'd rather have this as a normal constructor parameter of OllamaClient.

Please also note that after being wrapped into a PromptExecutor (which may be a multi-llm one, with Ollama+OpenAI+Anthropic, for instance) you'll be unable to use the Ollama specific settings. So everything should be configured at construction.

@Rizzen WDYT?

[ptitjes]

From my side, I'd rather have this as a normal constructor parameter of OllamaClient.

Please also note that after being wrapped into a PromptExecutor (which may be a multi-llm one, with Ollama+OpenAI+Anthropic, for instance) you'll be unable to use the Ollama specific settings. So everything should be configured at construction.

Yeah, that's what I figured out. Hence, I proposed the requestBuilder parameter to OllamaClient.
I will do the PR using this approach.

[krissrex]

@ptitjes When you build this, it is interesting for the builder to know which model is being executed. Because the different ollama models support different maximum context windows. Some are 32k, some are 128k etc.