#gpu

Liquid AI's 24B MoE reads 2.45B parameters per token. What that buys on one card, which vLLM flags matter, and where the per-token saving ends.

Renting a cloud GPU looks simple until the invoice arrives. Before you pick between an H100, an A100...

Are you a Rust developer who wants to learn CUDA programming? Try a GPU programming...

SGLang 0.5.20 cut decode CPU dispatch with the Gumbel-max trick, but only for requests with no top_k, top_p or min_p. How to tell which path your traffic takes, and what a free T4 measured.

Run an LLM locally when it fits your memory and data must stay put; use the cloud for 70B+ models, fine-tuning and batch jobs. Break-even math inside.

Six single-GPU machines launched through an API, timed from create call to first SSH and first CUDA matmul. 3 to 12 minutes, and why.

Plain DDP on a rented 4x RTX A6000 gave 2x. Measured all-reduce bandwidth, then bf16 gradients and accumulation took it to 3.6x. Commands inside.

Destroying a rented GPU deletes its disk. Copy checkpoints, LoRA adapters and outputs off the machine with rsync, scp, Hugging Face or rclone.

gpu-unlock, which brings RTX 3000-series FP16/BF16 performance in line with workstation cards, has been released. Based on reports in the README and issues, we’ll cover which workloads get faster, which don’t, and what to consider before installing it.

This week's theme: passing tests is wonderful. Knowing what they didn't catch is even better. ...

Same model, same vLLM benchmark, three rented GPUs. Cost per 1M output tokens at concurrency 1 to 64, with the commands to rerun it.

How to inject a custom rendering pass into Unreal's frame without modifying the engine