← sources

Context beat model choice by a wide margin in a new data agent benchmark (91% vs 58%)

source: reddit (reddit:AI_Agents)published 2026-11-14 11:56ingested 2026-11-15 02:34

retained text

Databricks just published a benchmark they built with Turing that compares Genie (their data agent, using Genie Ontology as a context layer) against two frontier coding/knowledge agents. Those are anonymized as Agent A and Agent B. All three got the same questions against the same live Databricks workspace.

What I found interesting is how the benchmark is set up. Most data agent benchmarks only give the agent a set of tables, and on those the agents score about the same. This one builds a fake UK energy supplier with hundreds of tables (the biggest has around 120M rows of half-hourly meter data). On top of the tables sit the usual notebooks, dashboards, saved SQL queries, docs and scheduled jobs. It also has the realistic mess: copied notebooks, abandoned analyses, dev copies, stale dashboards nobody has opened in months. To get the right answer, the agent often has to find a business rule that lives in one of those assets and not in the schema.

Each agent ran every task 5 times and was graded against a rubric fixed before the runs.

Results:

Genie: 91% pass rate, \~100s per run, \~432K tokens

Agent A: 45%, \~435s, \~889K tokens

Agent B: 58%, \~307s, \~1.05M tokens

So about 1.6x the accuracy of the best frontier agent, with less than half the tokens, and 3 to 4x faster.

The example task explains the gap well. The question asks for the FY24/25 planning rate. The obvious table has a column with 0.31 p/kWh on every FY24/25 row, so a lookup looks easy. But the notebook that builds that table says the mart applies FY25/26 assumptions to older periods, and it keeps the real historical rate (0.52) in a separate section. Genie's ontology surfaced that snippet and linked it to the source notebook, and Genie answered 0.52 on all 5 runs. The other two agents never got it right. One said 0.31 every time, and the other either said 0.31 or wondered whether a year-specific rate was documented at all.

None of the agents had trouble writing SQL. The difference was that the frontier agents had to rediscover the business context from scratch on every run, which burns tokens and still misses things. Genie started from context the ontology had already pulled out of the workspace.

Caveats: it's a vendor benchmark, the other agents connected through a Databricks toolbelt (with the Genie MCP excluded on purpose), and Databricks itself says the gap will be smaller in new workspaces without much accumulated knowledge, or when the answer is right there in the tables. They say the tasks and assets are shareable, so you can run it yourself through your account team.

To me it confirms what a lot of us see in practice: once models are good enough, harness and context matter more than which model you use. Curious how others handle this. Are you building your own context layers, stuffing definitions into AGENTS.md style files, or just letting the agent explore every time?

document fetched 2026-11-15 00:18 · canonical https://reddit.com/r/AI_Agents/comments/1xw0nyc/context_beat_model_choice_by_a_wide_margin_in_a