Scaling GPUs past one box
Past one node the network becomes the machine. InfiniBand vs RoCE, gang scheduling, FSDP and Megatron, and why a 16k-GPU cluster fails every three hours.
Jul 2, 202612 min read0 reactions0 comments
Back to articles Jul 2, 2026
Tag archive
#distributedtraining
R
Apr 4, 2026Ring Attention: Train 1M Tokens on 8GB GPUs in 2026
Training Million-Token Context Without Selling Your Kidney You can fit a million-token...
Apr 4, 20261 min read0 reactions0 comments