Kaiyue Wen

@wen_kaiyue

A continuous learner

Mountain View, CA

wenkaiyue.com

2023年6月からXを利用しています

1,288 フォロー中

3,222 フォロワー

固定
Kaiyue Wen
@wen_kaiyue
1月21日
(1/n) Introducing Hyperball — an optimizer wrapper that keeps weight & update norm constant and lets you control the effective (angular) step size directly. Result: sustained speedups across scales + strong hyperparameter transfer.
30
Kaiyue Wen
@wen_kaiyue
11月12日
Optimize to generalize
Xingyu Dang
@xingyudang
11月12日
Compute grows fast but pretraining corpora don’t, so data-constrained optimization matters! We propose Training-Time Testing (T³): each step takes a trial step on half the batch, tests it on the other half, and updates with the meta-gradient. It achieves SOTA on SlowRun!
1
Kaiyue Wen
@wen_kaiyue
11月12日
Interestingly the norm corresponding to each of the neurons are almost exactly 1 🤪 Weight normalization for the win!
Songlin Yang
@SonglinYang4
11月12日
Meet Inkling 2 Small! We’ve tweaked the architecture a bit since the last generation, adding attention residuals (with a q/k/v split), gated attention, dynamic attention sink, dynamic shortconv, value norm and more. Don’t ask me “linear attention wen"
4
Kaiyue Wen
@wen_kaiyue
11月12日
Amin's new work on using activation preconditioning to double critical batch size! A long overdue comeback of K-FAC 🤪
Amin Mohamadi
@QuelMohamadAmin
11月12日
Check out Toner! A general rule that preconditions the optimizer's update with each layer's input activations. Applied to Muon and AdamW as MuonT and AdamT, it speeds up LM pretraining, improving the critical batch size by ~2× over Muon.
1
Kaiyue Wen
@wen_kaiyue
11月9日
Super cool work by @bremen79 and clean blog! For scale-invariant L, the weight norm is a free accumulator. If L is geodesically convex on a hemisphere centered at a minimizer, w ← w − ∇L(w/‖w‖) gives schedule-free GD with AdaGrad-style optimal regret. chatgpt.com/s/t_6af2082e74… See also Lyu et al., App. B arxiv.org/abs/2206.07085
jianlin.su
@Jianlin_S
11月9日
Revisiting Convergence Results in Convex Optimization (XI) kexue.fm/archives/12003 New ideas for scheduled-free learning from arxiv.org/abs/2610.39531
1

Kaiyue Wen

Kaiyue Wenさんのポスト