Tag archive
Off Grid AI Desktop on Windows keeps llama.cpp, stable-diffusion.cpp and whisper.cpp current in the background, CUDA builds included, with no reinstall.
Chat on your Mac with models running on another computer's Ollama or llama.cpp server, with thinking, context and status shown right.
Out of nowhere, on August 26, 2026, the Qwen team posted an announcement on ModelScope and Hugging...
An 8 GB RTX 3050 serves Gemma 4 E2B at 109 tokens per second, and that speed makes it a training card too. One card samples its own training data, fine-tunes the model to think less and grades it, in 6 h 44 min.
Why Supporting Specialized Large Models Matters The rapid evolution of large language...
Tuần này trên Hacker News có một project khá thú vị tên Janas: chạy LLM lớn hơn RAM bằng cách stream...
llama.cpp's thinking cap trades accuracy for tokens. Fine-tuning Gemma 4 E2B on its own answers sampled under a 64-token cap halves tokens per correct answer on all 1,319 GSM8K test problems, with no cap at inference, for 12 to 33 fewer right answers across four training runs, and matches the accuracy of the best cap with 9.5% to 16.6% fewer tokens.
A self-hosted AI agent needs more than a long context window. It needs a way to retain useful facts,...
Gemma 4 31B, a dense 17.7 GB model, serves from an 8 GB RTX 3050 with 40 of its 60 layers in system RAM: the highest GSM8K and BFCL scores of the family, at about 4 tok/s and 11x the 12B's GSM8K time.
TL;DR: llama-server's RAM prompt cache (--cache-ram, on by default) saves a slot's KV when another...
By Georgii Kharlampiiev, Mindscend Our company website is a single chat page backed by a model we...
Yes. An exact Q4_0 rebuild of Google's Gemma 4 12B QAT GGUF runs with every layer on an 8 GB RTX 3050 at up to 40,960 tokens of context, and Google's 12B drafter, rebuilt the same way, doubles its generation speed to 56 tok/s with llama.cpp. A 4-bit context cache stretches the drafter setting to 81,920 tokens at the same answers.