Serving 753B MoE Across Nodes: vLLM & RDMA Limits
Evaluating multi-node tensor parallelism for GLM-5.3 NVFP4 across DGX Stations reveals severe throughput cliffs past concurrency 16.
Tag archive
Evaluating multi-node tensor parallelism for GLM-5.3 NVFP4 across DGX Stations reveals severe throughput cliffs past concurrency 16.
Loading a model with a native Hugging Face pipeline (AutoModelForCausalLM) inside a web server is...
A user pastes an 8,000-token document into a chat box and hits send. On the server, that prompt takes...
vLLM is an open-source LLM serving engine. PagedAttention and continuous batching let it fit far more requests per GPU than a plain generate() loop.
vLLM 0.33.0 shipped on November 2, 2026 — 927 commits from 389 contributors — and the headline is not...
We examine llm-compressor's quantization-to-vLLM workflow, with a reproducible FP8 example, explicit calibration controls, and deployment gates for VRAM, accuracy, and serving cost.
I read 36 Dynamo release notes. Stable pins trail vLLM by a median 30 days, and 16 of 26 model snapshots sit on patched or custom engine bases.
ERNIE-5.5-Flash-Base: 288 GB of FP8 weights, about 5 GB of KV cache at 1M tokens (my estimate), and a license that excludes the EU, UK and Korea.
Transcription queues, synthesis hangs, OOM crashes—adding local text generation to VoiceStudio without resource isolation destroys audio responsiveness.
We tried to reproduce verl's GRPO plus vLLM rollout tuning on a fixed GPU budget. The GPU leg never ran, so here is the phase-budget model, the version traps, and the exact config that decides whether your actor and rollout engine can share a card.
vLLM 0.33 ships FLy, an opt-in lossy speculative decoding verifier. Speedups, why the accuracy tables prove little, and a paired gate you can run.
vLLM's Kimi K3 RFC balances CuTe DSL kernel fusion on Blackwell GPUs with strict isolation and automated gates to manage agentic code generation.