Warmup Did Nothing for My Learning Rate — Then It Fully Rescued the Wrong One
Learning-rate warmup is the most cargo-culted schedule fragment in deep learning: transformers need it, so everyone ships it. I measured
Nov 4, 20264 min read0 reactions0 comments