Tag archive
If you run vllm serve --api-key ... and think the server is now private, read vLLM's own security...
Slash enterprise inference bills and latency by compressing frontier model capabilities into 8B architectures using modern KD pipelines and AWQ quanti
If you serve models with vLLM, pip-audit or Dependabot probably told you nothing new this week. That...
vLLM officially supports Linux only. On Windows the usual answer is WSL or Docker. I wanted it...
Fourteen Apache-2.0 and MIT open-weight models that fit one TPU v5e chip, served one after another through vLLM on the same pinned image and scored on GSM8K and BFCL against Gemma 4 E2B. Qwen3-4B matches Gemma's accuracy at 0.57x its output, SmolLM3-3B and Granite 3.3 2B match it on tool calls, and Gemma keeps the lead on long prompts.
FP8 and INT8 KV cache quantization can double your context budget, but a silent config.json setting can change your accuracy without warning. Here is
A practical cost model for self-hosted LLM serving: how continuous batching, token mix, and GPU utilization decide your real price per million tokens.
Prefix cache hits, SSE chunk counting, silent FP8 KV cache, multi-knob tuning and lazy gates: real numbers from vLLM benchmarks that went wrong.
TL;DR: vLLM 0.30.0 can keep a model's weights in a long-running daemon so that engines restart...
TL;DR: A PEFT LoRA adapter can give individual modules their own rank and alpha through rank_pattern...
A serving setup for Qwen3.8-27B on a single RTX 3090 with vLLM whose README measures when `DFLASH_TOKENS=15` pays and what it costs in reque
An engineering evaluation of speculative decoding methods in vLLM on AMD Instinct MI300X hardware, analyzing throughput, acceptance rates, and tradeoffs.