Joël Niklaus

@joelniklaus

Harness Optimization @HuggingFace

Berne, Switzerland

固定
Joël Niklaus
@joelniklaus
11月5日
I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard. Which harness should you run your open model in? I ran 28,849 trials to find out. The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness
276
Joël Niklaus
@joelniklaus
22時間
High reasoning effort makes DeepSeek-V4.1-Flash 28% more expensive and not one point better. Reasoning effort sets how much the model thinks before it acts. On the @huggingface Harness Leaderboard, I compared low and high effort with five coding agents: Claude Code, Codex,
58
Joël Niklaus
@joelniklaus
11月12日
One default cost @openclaw 16 to 21 points on DeepSeek-V4.1-Flash: 8,192 tokens per reply. OpenClaw asks for at most 8,192 tokens per reply when a model's config sets no output limit. Mine set none. DeepInfra, the provider serving the model, cut its thinking at 4,916 tokens,
19
Joël Niklaus
@joelniklaus
11月11日
You asked for dsh, I added it DeepSeek Harness is the open-source coding agent by @deepseek_ai. It's now on the @huggingface Harness Leaderboard, which compares ten coding agents on the same open models. DSH is statistically tied for first in all five evaluations:
78
Joël Niklaus
@joelniklaus
11月10日
Wow, the @huggingface Harness Leaderboard is the #2 trending Space this week 🤯 So glad to see it being useful to so many people, and right behind my colleagues' Ultimate MoE Training Handbook! For newcomers: the leaderboard shows which coding-agent harness gets the most out of
4

Joël Niklaus

Joël Niklausさんのポスト