FineWeb-Edu may not capture all architectural gains, but I think it’s still useful for small-scale controlled analysis. More realistic data mixtures would ofc help, but “realistic” is hard to define, and small models may struggle to learn advanced tasks.
If we want to eval math
Some unpopular-but-true notes on pretraining research:
→ 𝗗𝗮𝘁𝗮 𝗿𝗲𝗽𝗲𝘁𝗶𝘁𝗶𝗼𝗻 is a very common practice.
→ 𝗢𝘃𝗲𝗿-𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 (~1000 TPP) is a very common practice.
→ In the long run, 𝗱𝗮𝘁𝗮-𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 architectures and algorithms will matter more