Executive summary
AI leaderboard rankings measure a model working alone, on tasks that look little like your warehouse. On the 410 business questions in the BIRD-Interact benchmark, what sat under the model made a bigger difference than which model it was. Episode one of The AI Accuracy Files digs into why, and this post turns it into four questions to ask before you sign off on a model.
- Leaderboard ranks are a narrow claim. They can be tuned for, and strong scores on easy tests often don’t carry over to enterprise data.
- Model choice doesn’t determine accuracy for business questions. Working straight against the database, six of the seven models we tested finished within about five points of each other on the 410 business questions (one no-layer run per model, preliminary).
- What the model queries mattered more. With AtScale’s semantic layer underneath, every model improved, and the layer made a bigger difference than switching models (best of three runs vs. one no-layer run).
N.B. BIRD-Interact benchmark results are preliminary.
Do AI leaderboards matter for AI accuracy?
If someone on your team is building the business case for an enterprise LLM, you’ve probably seen the slide. A public leaderboard appears with the favored model circled near the top, along with a recommendation. But is the leaderboard really what matters when enterprises need to solve for AI accuracy?
We ran seven models through BIRD-Interact, one of the hardest public tests of AI on business data, and came away with a different view. For enterprise analytics, a leaderboard rank tells you less than you’d think. Before a ranking decides which model you pick, ask what the model will query: raw tables or defined business metrics.
The same idea runs through the first episode of The AI Accuracy Files, our new podcast on AI accuracy and semantic layers. Ryan Siegler, Head of Developer Relations at AtScale, sat down with CTO and Co-Founder Dave Mariani and Chief Strategy and Marketing Officer Mark Palmer to talk through what we found in the BIRD-Interact benchmark. This post picks up their argument for anyone building or reviewing that case.
Are AI leaderboard rankings reliable for choosing an enterprise model?
Not on their own. A leaderboard measures a model working alone, usually on tasks that don’t look anything like your warehouse.
Researchers from Cohere Labs, Stanford, MIT and others found that private pre-release testing distorts public arena rankings, with one provider testing 27 private variants before a single launch (The Leaderboard Illusion). In another case, a model’s chat-optimized test version placed near the top of a popular leaderboard, while the version that actually shipped ranked 32nd (TechCrunch).
An honest score may still not carry over to your data. On Spider 1.0, a well-known text-to-SQL test, one model scored 91.2%. On Spider 2.0, built to look like real enterprise databases, the same model scored 17.0% (Spider 2.0). The 91.2% was accurate, but it described clean, simple databases, not the messy ones enterprises run on.
BIRD-Interact is built to test the messy kind.
Its 600 tasks run against 22 messy PostgreSQL databases. The prompts are as vague as the ones real people type, and a simulated user pushes back the way a colleague would. “It’s a dirty benchmark and the world is really dirty,” Mark said. Dave explained why that matters: “It’s a conversation and there are refinements and further follow up… in real life you’re not going to ask one question and just get an answer.”
How much does the model matter for business analytics accuracy?
Less than AI leaderboards imply. Of the benchmark’s 600 tasks, 410 are business questions. Working straight against the database, six of the seven models we tested finished within about five points of each other on those questions, and one, Gemini 3.1 Pro, stood clear of the pack. Even that gap was smaller than what the semantic layer added to any of them.
Stanford’s 2025 AI Index found the gap between the first- and tenth-ranked models fell from 11.9% to 5.4% in a single year (Stanford AI Index 2025), so the bunching isn’t unique to our test.
Dave pointed out that every frontier and open-weight model already knows how to write SQL, so the edge has to come from somewhere else:
“It’s not about the number of parameters the model is trained on… You don’t need a billion parameter model to do better at generating Postgres SQL.”
Where the model does make a difference is in how it handles a vague request. Some models stop and ask a clarifying question, and others take a guess. You can test for that in a pilot, but a leaderboard rank won’t tell you which kind you’re buying.
What’s under the model, and why does it matter more?
The data the model queries: raw warehouse tables, or business metrics defined once in a semantic layer. In our runs, that choice is what moved the score.
Picture the same agent working in two lanes. In the top lane, it writes its own SQL against raw warehouse tables, which is how the public leaderboard runs. It gets the knowledge base, the schema, column descriptions and sample data, and it still has to work out the joins and the metric math on every question. As Dave put it, the model gets all of that information “and it still guesses.”
In the bottom lane, the same agent works through AtScale’s semantic layer. Metrics, relationships and aggregation rules are defined once in a semantic model. The agent picks a metric and a dimension, and AtScale’s engine compiles the SQL the warehouse runs, so the business logic isn’t rebuilt from scratch on every question.
“The LLMs know how to write SQL. We just allow it to write really simple SQL.”
– Dave Mariani
In the episode, Dave walked through one benchmark question where the model’s own SQL ran without an error and returned the wrong answer. Through the semantic layer, the same question became a short query against a single virtual table, and it came back right.
Across the 410 business questions, every one of the seven models improved with AtScale underneath (best of three runs vs. one no-layer run, preliminary). Every model also improved on its first, cold run, before anything carried over from earlier runs. Turning the layer on made a bigger difference than switching models did. In each model’s best run with AtScale in place, the gap between the strongest and weakest model was smaller than the gain any one of them got from the layer.
To keep the comparison fair, an agent generated AtScale’s semantic models from the same documentation the raw lane received, and engineers then turned them on the benchmark. A September 23 audit removed anything traceable to the answer key, and every reported run uses the cleaned models.
In AtScale’s product, the same foundation goes further. AtScale Ava asks which metric you meant when a question could mean more than one, and ACE, the AI Computation Engine, computes the answer the same way every time.
Anthropic’s own data team saw the same pattern outside the benchmark. After it wrote down what its metrics meant, accuracy jumped from about 21% to above 95% without changing the model (AtScale’s analysis).
Once the business logic lives in the layer, picking a model comes down to price, latency and where your data is allowed to go. You can swap it out as better options ship without starting over.
What should I ask an AI analytics vendor instead of its leaderboard rank?
Ask these four questions of anyone proposing an AI analytics setup, whether it’s a vendor or your own team. Each one tells you more about the numbers you’ll get than a rank does.
- Does the model write SQL against raw tables, or call defined metrics? If it’s raw tables, the model is working out your joins and business rules on every question.
- Who computes the metric? Models are probabilistic. “If I ask a question on Monday, it’s going to come up with a different answer on Tuesday,” Dave warned. A metric computed by an engine from one definition comes back the same every time.
- What happens to accuracy if you swap the model? If the answer is “we’d have to start over,” the logic lives in the model, and you’re locked into it.
- Was it scored on a public test built like your warehouse? Ask for results on ambiguous, multi-turn business questions, and ask how the run was set up: how many runs, which run is reported, and what the comparison ran against.
Buy the foundation, rent the model
Even our best result on BIRD-Interact isn’t good enough yet.
Dave described what’s at stake when a business user gets one bad number from an AI tool: “If you get it wrong, even if you just get it wrong once, they’ll never come back.”
The rest of this series covers the path toward 100% accuracy. It starts with the foundation: choose what the model queries first. Then pick the model that fits your budget and data rules, knowing you can change it later.
Keep going:
- See the full method and preliminary results at atscale.com/benchmarks/bird-interact-results/
- Download the ebook, AtScale’s BIRD-Interact Benchmark Results, Methodology and Analysis [link TK]
- Watch episode 1.1 of The AI Accuracy Files