We’re spending up to 100B tokens across 26,040 agent runs to answer one question: when you switch models, should you switch harnesses too?
retained text
FrontierHarness v1.0 compared 9 harnesses on a single model. Its obvious limitation was that we couldn’t say whether a harness that wins on one model wins on another. V1.1 is the full matrix: 21 harnesses × 5 models × 62 terminal tasks × 4 attempts, 26,040 runs, with a budget of up to 100B tokens. Every run gets the same environment and the same limits, and lands on the board with its full outcome, token count and outcome as soon as it finishes. All five models are open-weight: GLM-5.3, GLM-5.3-Flash, DeepSeek-V4.1-Flash-1017, Muse Spark Open and Kimi-K3. Which harness + open model combo do you use day to day?
document fetched 2026-11-11 00:18 · canonical https://reddit.com/r/AI_Agents/comments/1xsv7xm/were_spending_up_to_100b_tokens_across_26040