
Why Do AI Models Stream Their Responses?
AI models stream their responses because they generate text one small piece at a time, and waiting...
Tag archive

AI models stream their responses because they generate text one small piece at a time, and waiting...

Spending compute at inference instead of packing it into weights looks like a free win until the per-token latency and the serving fleet size are added back.
vLLM is an open-source LLM serving engine. PagedAttention and continuous batching let it fit far more requests per GPU than a plain generate() loop.
MLPerf Inference has no rows for Google TPU or AWS Trainium, so the industry benchmark cannot compare the chips you actually want to compare. A five-axis framework that fixes that.
DSpine 카드의 에이전트 수치 10칸을 계산하면 수락 길이는 평균 22.8%, 처리량은 14.0% 늘었습니다. 초안 모델의 파라미터와 KV 메모리도 공개 파일로 확인했습니다.

Reusing a shared prompt prefix looks like free compute until cache memory, eviction and tenant isolation are added to the per-token bill.
clyde came back twice in the comments on my last piece, both times arguing from a consumer card...

Skipping layers per token looks like free throughput until the router, the gather and the batch imbalance are added to the bill.

Most inference benchmarks look like a chatbot. One question, one answer, next. Agents don't work...

Hugging Face shipped a free 70k-word MoE training handbook grounded in ~800k GPU hours. The numbers on all-to-all and router precision explain why sparse models keep fitting inside desktop latency budgets.
A reader asked me to run the cache-versus-weights crossover on Qwen3-32B. On 80 GiB it has no answer, and the reason is the card, not the model.

Most questions don't need your most expensive model. Routing, cascades, distillation and caching: the four ways to send each request to the cheapest model that can handle it, and how to know it's still good enough.