It turns out multi step backpropaganda is better.
paper has a beautiful way of improving backpropagation. One iteration cleanly gets us backprop, multiple iterations get us a preconditioned update.
Jensen is reason why many unreasonable audacious bets by young companies thrived and succeeded, the counterfactual would be a sad world without the age of wonder that we can all see glimpses of. Crazy world!
Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
thinkingmachines.ai/blog/end-to-en…
Just a random fact of trivia: When I left microsoft frustrated not being able to work on machine learning in 2013 to google, I yolo wrote in the exit survey that I would stay if I get to work with John, and now destiny and world has manifested this :)
This is a huge get, congrats to my ppl at @coreauto.
@JohnCLangford is a galaxy-class researcher and true intellectual of a style that's impossibly hard to come by.
Excited to see what they accomplish together.