Local Inference: Running Models on Your Own Machine
Ollama, llama.cpp, and what hardware actually gets you usable speed.
Tag archive
Ollama, llama.cpp, and what hardware actually gets you usable speed.

llama.cpp and ggml hit 1.0.0 on 13 Nov 2026 with semver guarantees and FP8 KV caches. On a Windows call the audio engine still sets the floor for any local STT or translation overlay.
Our small open model answered a stock question in 4.25 s on a 4 GB Galaxy A05s. In that same run, its...
POCKET GGUF의 헤더와 텐서 목록을 직접 읽어 가중치당 비트 수, 문맥 길이별 KV 캐시, 토큰당 읽는 가중치를 계산하고 램 예산표를 만들었습니다.

On a Surface Laptop Ultra (RTX Spark, unified memory), I measured local LLM speed and Windows' free memory with the GPU's dedicated memory set to 4, 64, 82 and 96GB. Speed barely moved; memory left for Windows did.
I made my AI agent's notes vault retrievable: 35 MB embedder, llama.cpp reranker,...

Jan open source ChatGPT alternative: the janhq/jan repository calls itself an « Open-source...

Same llama.cpp build, same commands, one Surface Laptop Ultra (Windows on Arm, RTX Spark): generation speed lined up by active parameters, not total size, and the fastest model couldn't fix a one-character config error.
Off Grid AI Desktop on Linux keeps llama.cpp, stable-diffusion.cpp and whisper.cpp current in the background, Vulkan and CUDA builds included.
Chat on your Windows laptop with models running on another PC's Ollama or llama.cpp server, with thinking, context and server status shown.
Off Grid AI Desktop on Windows keeps llama.cpp, stable-diffusion.cpp and whisper.cpp current in the background, CUDA builds included, with no reinstall.
A Cline user running a fully local stack — Cline 3.0.68, Windows 11, llama.cpp serving...