elie

@eliebakouch

research @PrimeIntellect (prev: @huggingface) anon feedback: admonymous.co/eliebakouch

2024年1月からXを利用しています

4,767 フォロー中

2.6万フォロワー

返信リポストメディア

固定
elie
@eliebakouch
8月15日
to my knowledge this is the largest open experiment on autonomous agents iterating on a research environment we scaled runtime, compute, diversity of models and harnesses. as a comparison, similar tasks on oai/anthropic system cards are anthropic "optimizing an llm training on
Prime Intellect
@PrimeIntellect
8月15日
We ran the largest open experiment on how frontier models do AI research. 100+ autonomous runs across 10+ models, sandboxed on 8xH200s for up to 8 days, iterating on the nanoGPT optimizer track. Best runs closed 82% of the gap to a record built by dozens of humans over months.
54
elie
@eliebakouch
11月12日
i was going to do a deep dive about the arch but @SonglinYang4 already cover most of it! i'd add: > experts NVFP4 in layers 0-39 and MXFP8 in last 8 (very cool!) > softcaps in the lm head and swiglu > SWA size 512-> 1k very cool model, congrats thinky folks!
Songlin Yang
@SonglinYang4
11月12日
Meet Inkling 2 Small! We’ve tweaked the architecture a bit since the last generation, adding attention residuals (with a q/k/v split), gated attention, dynamic attention sink, dynamic shortconv, value norm and more. Don’t ask me “linear attention wen"
5
elie
@eliebakouch
11月11日
i added 5 open models to @dwarkesh_sp's blood on the clocktower benchmark to see how they perform, and look at their chains of thought while they lie, bluff and try to catch each other 👁️ kimi k3.1 pro is the best open model and lands in the middle of the closed ones, just
Dwarkesh Patel
@dwarkesh_sp
11月5日
18:13
I spent 20,000 dollars in API credits having different AI models play each other at Blood on the Clocktower (a Mafia or Werewolf type social deception game). The results were fascinating. Models have gotten much better over the course of this year at creative on-the-spot
25
elie
@eliebakouch
11月10日
no single dataset is enough to do pre-training ablations there is some changes where it matter more than other, but generally epoch aware scaling ladder (like MAI and more recently beam as well) is the way. also you have to be careful on how you evaluate it, and which metrics you look at depending on what you are ablating
Benhao Huang
@huskydogewoof
11月10日
+1
Some unpopular-but-true notes on pretraining research: → 𝗗𝗮𝘁𝗮 𝗿𝗲𝗽𝗲𝘁𝗶𝘁𝗶𝗼𝗻 is a very common practice. → 𝗢𝘃𝗲𝗿-𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 (~1000 TPP) is a very common practice. → In the long run, 𝗱𝗮𝘁𝗮-𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 architectures and algorithms will matter more
5
elie
@eliebakouch
11月9日
while this personally makes me really sad as i love playing with llms on this kind of topic, i appreciate the transparency from anthropic here and also not nerfing domains like interpretability
elie
@eliebakouch
11月9日
with this, i think it's quite likely that anthropic will never publicly release (or at least not in the coming months) a model more capable than opus 5.5 at llm training
8

elie

elieさんのポスト