Pin Your Context Window or Pay the Reload Tax
Change num_ctx in Ollama and it reloads the whole model. On my RTX 5090 that meant 140 seconds of dead air per context swap. Here is how to stop paying it.
Jul 12, 20265 min read0 reactions0 comments
Tag archive
Change num_ctx in Ollama and it reloads the whole model. On my RTX 5090 that meant 140 seconds of dead air per context swap. Here is how to stop paying it.
Preview the files your local LLM will receive before inference. This simple gate catches bad retrieval while the fix is still cheap and clear.