New training speed record for @karpathy’s 124M-parameter NanoGPT setup: 3.28 Fineweb validation loss in 3.7B training tokens Previous record: 5B tokens Changelog: new optimizer 1/8The new optimizer is defined as follows. It is based on orthogonalizing the update given by SGD-Nesterov-momentum in an efficient wayIt uses half the memory of AdamW and takes 3% extra wallclock time per step for this setup Here's code to reproduce the result:A variant of this optimizer also yielded a training speed record for CIFAR-10. Scaling it up to NanoGPT was surprisingly simple 4/8The optimizer's core design decisions are: - For each parameter, orthogonalize the update *after* applying Nesterov momentum - Use a fast iterative method for the orthogonalization. Quintic Newton-Schulz works well - Run the iteration in bfloat16 5/8And - To minimize the number of Newton-Schulz steps, select the quintic coefficients aggressively to maximize slope at zero, even at the cost of some variance / non-convergence. There's an interesting tradeoff space here 6/8I would like to thank & acknowledge @jxbz for sending me his recent paper, which is where I learned about the crucial Newton-Schulz iteration method. He also had the insight that my initial quintic coefficients could be improved. 7/8I'm still thinking about names for the optimizer, & there may be a full arXiv writeup at some point. I hope the community finds this intermediate result interesting; critiques are welcomed 8/8

New training speed record for @karpathy’s 124M-parameter NanoGPT setup: 3.28 Fineweb validation loss in 3.7B training tokens Previous record: 5B tokens Changelog: new optimizer 1/8 ... The new optimizer is defined as follows. It is based on orthogonalizing the update given by SGD-Nesterov-momentum in an efficient way ... It uses half the memory of AdamW and takes 3% extra wallclock time per step for this setup Here's code to reproduce the result: ... A variant of this optimizer also yielded a training speed record for CIFAR-10. Scaling it up to NanoGPT was surprisingly simple 4/8 ... The optimizer's core design decisions are: - For each parameter, orthogonalize the update *after* applying Nesterov momentum - Use a fast iterative method for the orthogonalization. Quintic Newton-Schulz works well - Run the iteration in bfloat16 5/8 ... And - To minimize the number of Newton-Schulz steps, select the quintic coefficients aggressively to maximize slope at zero, even at the cost of some variance / non-convergence. There's an interesting tradeoff space here 6/8 ... I would like to thank & acknowledge @jxbz for sending me his recent paper, which is where I learned about the crucial Newton-Schulz iteration method. He also had the insight that my initial quintic coefficients could be improved. 7/8 ... I'm still thinking about names for the optimizer, & there may be a full arXiv writeup at some point. I hope the community finds this intermediate result interesting; critiques are welcomed 8/8 ...