Ryan Yixiang Wang

@RyanYixiang

Phding at @berkeleynlp with @sewon__min | prev @nlp_usc

Berkeley, CA

ryanyxw.github.io

2017年6月からXを利用しています

588 フォロー中

437 フォロワー

固定
Ryan Yixiang Wang
@RyanYixiang
5月8日
MoEs are everywhere in frontier models, and they are deployed as a monolith system. But many applications only need a narrow slice of capabilities, e.g., math, code, biomedical, etc. So what if "modularity" is actually the missing opportunity for MoEs? Today, we're releasing EMO: an end-to-end pretrained MoE where modularity emerges naturally, enabling selective use of experts!
Ai2
@allen_ai
5月8日
Today we’re releasing EMO, a new mixture-of-experts (MoE) model trained so modular structure emerges directly from data without human-defined priors. EMO can use a small subset of its experts for a given task while keeping near full-model performance. 🧵
8
Ryan Yixiang Wang
@RyanYixiang
11月5日
Almost all model behavior can be traced back to its training data. Unexpected LLM behaviors are often just hard to notice patterns in the data! Human inspection just can’t catch this sort of stuff 🥸
Yaowen Ye
@yaowenye123
11月5日
To build a system that can predict arbitrary side effects of post-training, we first need a proxy evaluation set of examples. This is a first step toward that. Really enjoyed working on it with the amazing folks at @TransluceAI!
引用
Transluce
@TransluceAI
11月5日
We built a pipeline that discovers thousands of datasets with surprising side effects (DSSEs): datasets that read like ordinary post-training data yet induce unexpected model behaviors (e.g. recommends self-harm) after finetuning. 🧵
1
Ryan Yixiang Wang
@RyanYixiang
9月28日
When I use coding agents these days, I often get the feeling that some questions are inherently harder than others. I'd be comfortable asking an agent, “Is this README still up to date?”, but much less comfortable asking something broad like, “Are there any remaining bugs in this
Prasann Singhal
@prasann_singhal
9月28日
Could long-context architectures “find all contradictions” in a science literature? Not yet! 🧵 We study a new class of "high-complexity” tasks whose difficulty scales quadratically with corpus size (as opposed to linearly), reversing common LCLM decisions! (block-sparse
3
Ryan Yixiang Wang
@RyanYixiang
2024年8月11日
Presenting our work at ACL on using data watermarks for detecting if an LM used your data during pretraining (with statistical guarantees)! Come find me and @johntzwei Monday 5:45 pm! Also happy to chat about how hot Bangkok is, or anything LM pre-training/memorization, etc
Ryan Yixiang Wang
@RyanYixiang
2024年2月19日
We could detect SHA hashes that only occurred in BLOOM-176B pre-training data 90 times! For reference BLOOM has a training corpus of 341B tokens 🫣🥸
Ryan Yixiang Wang
@RyanYixiang
2024年2月19日
We could detect SHA hashes that only occurred in BLOOM-176B pre-training data 90 times! For reference BLOOM has a training corpus of 341B tokens 🫣🥸
Johnny Tian-Zheng Wei
@johntzwei
2024年2月19日
To detect if your data was used for LLM pretraining, consider using data watermarks: arxiv.org/pdf/2402.10892… Detection can be framed as hypothesis testing (statistical guarantees!), if you contributed multiple training documents and watermarked them before public release. 🧵

Ryan Yixiang Wang

Ryan Yixiang Wangさんのポスト