#optimizers

A single-author preprint shows that a mixture-of-experts model can drop momentum entirely for its expert layers, shrinking persistent optimizer state from 50.55 gigabytes to 1.29 with almost no effect on final quality.

OmniOpt ran a controlled bake-off of more than two dozen modern training optimizers across model sizes from 60M to 1B parameters, and its main lesson is deflating: no challenger cleanly dethrones AdamW, because an optimizer's advantage depends heavil

Ok, it's midnight during Halloween, and I'm talking about Optimizers. Such a thrill lol. And the goal...