Book a demo
Published October 28, 2026 / Posted by: Ryan Siegler / Reviewed by: Dave Mariani

A better LLM isn’t the biggest upgrade for Text-to-SQL

Across Seven LLMs, a Semantic Layer Improved Accuracy More Than Switching Models
Estimated reading time: 11 minutes
A better LLM isn’t the biggest upgrade for Text-to-SQL

Executive Summary

Seven LLMs answered the same 410 business questions from the BIRD-Interact benchmark, once against PostgreSQL with full documentation and once through an AtScale semantic layer. Their tool-call logs show seven working styles. Grok 4.3 submits SQL more often than it tests. Kimi K2.5 tests six times a task. None of those habits predicted the score. Switching LLMs moved accuracy by up to 17.6 points. The semantic layer added 20 to 31 points to every one of them.

Key takeaways:

  • Every LLM worked the problem its own way. Grok 4.3 submits SQL 4.5 times a task and runs a test 0.7 times. Kimi K2.5 tests six times a task. Gemini 3.1 Pro asks the user most, at 2.2 clarifying questions per task.
  • Those habits did not predict the score. Kimi ran out of budget on 81% of tasks and Gemini on 75%, and they finished 12.9 points apart.
  • The semantic layer mattered more than the LLM. Switching LLMs moved accuracy by up to 17.6 points. The semantic layer added 20 to 31 points to every one of them, including the LLM already in the lead.
  • An open-weight LLM kept pace with a frontier LLM. Kimi K2.5 scored 49.3% against GPT-5’s 50.5%, at a penny less per correct answer.
  • Full documentation was not enough. The no-layer arm received the schema, a plain-English meaning for every column, and a knowledge base with every formula the questions rely on. It still scored about half as well.

Grok 4.3 does something I didn’t expect. On an average task, it sends SQL to the grader 4.5 times and runs a query itself 0.7 times. It would rather submit an answer and find out whether it was right than run the query and look at the rows.

That’s a strategy. It’s a bad one, and Grok finished last. But six other LLMs had strategies just as distinct.

The logs come from a BIRD-Interact Text-to-SQL benchmark sweep: seven models, the same 410 business questions asked straight against PostgreSQL with documentation, and then through an AtScale semantic layer. The accuracy numbers were what we were after. The tool-call counts showed how differently each model attacks the same question.

Below I’ll go through how each LLM worked the problem, what their habits predicted about the final score, the difference in generated SQL, what a correct answer cost from each, and whether any of it argues for buying a better model. This is the third piece in my hundred days of learning AtScale. You can find the first post, The Missing Step in Text-to-SQL and second, Will the real semantic layer please stand up? on the AtScale blog.

New to the benchmark? Start with our BIRD-Interact explainer. The short version: 600 tasks over 22 messy PostgreSQL databases, questions left vague on purpose, and a simulated user holding the definitions you need. For full methodology and per-run detail, read the eBook. Results produced and submitted by AtScale. Not yet verified by the BIRD team.

How each LLM works the benchmark

ModelClassNo layerWith AtScaleGain
Gemini 3.1 Profrontier36.1%62.2%+26.1
Claude Sonnet 5frontier25.9%56.6%+30.7
Claude Opus 4.6frontier22.7%53.7%+31.0
GPT-5frontier21.5%50.5%+29.0
Kimi K2.5open weight23.4%49.3%+25.9
Nemotron 3 Ultraopen weight23.2%45.9%+22.7
Grok 4.3frontier24.9%44.6%+19.7

Phase-1 success rate on the 410 business questions. AtScale figures are the best of three runs, the benchmark’s own convention; the no-layer control is a single run.

All the figures below come from the AtScale runs: tool calls per task on the 410 query tasks, budget exhaustion across all 600.

Gemini 3.1 Pro asks. At 2.2 clarifying questions per task, it talks to the simulated user nearly twice as often as anyone else, and it arrived with the strongest no-layer score in the set.

Sonnet 5 does the reading. 10.4 tool calls a task, tests before it submits, asks about once a task.

Opus 4.6 works alone. It asks the user 0.43 times a task, less than anyone in the set, and submits more often than it tests. Form a theory, send it, revise.

GPT-5 is the disciplined one. Fewest exploration calls at 1.9 a task, and it finishes inside its budget on 77% of tasks, while the rest range from 19% to 60%.

Kimi K2.5 checks everything. Six test queries a task, the most in the set, ahead of Nemotron’s 5.2, and 13.0 calls in total. It also runs out of budget more than any model in the set.

Nemotron 3 Ultra burns its budget almost as hard on 79% of tasks and lost 28 tasks to context overflow across its three runs.

Grok gambles, as above.

Every LLM got the same layer, the same questions, and the same budget, and each still found its own route through.

Do the LLMs’ habits make a difference?

Tool use and budget discipline look like clues to model performance. But neither explained the results.

Line the seven models up by tool calls and the ranking scatters. Kimi makes the most at 13.0 and finishes fifth. Nemotron is second at 11.9 and finishes sixth. Gemini is third at 11.6 and finishes first. The middle of the pack, Sonnet at 10.4, Opus at 8.7, and GPT-5 at 8.0, finish in that order, and then Grok makes the fewest calls at 7.8 and finishes last.

Budget use was an even weaker indicator. Kimi exhausted its budget on 81% of tasks and Gemini on 75%, yet they finished 12.9 percentage points apart, at opposite ends of the field. GPT-5, the most disciplined model in the set, finished fourth.

The clearest finding was the impact of the semantic layer. Without it, six of the seven models scored between 21.5% and 25.9%. Gemini stood apart at 36.1%. Adding the layer improved every model’s accuracy by 20 to 31 percentage points.

By comparison, switching models changed accuracy by at most 17.6 points. Gemini defined one end of the range in both conditions. Without the layer, the other six models were separated by just 4.4 points.

The layer mattered more than the choice of model. And neither tool use nor budget discipline reliably predicted which model would perform best.

21 lines of SQL, or 3

One task from the insider-trading database asks for the average order modification intensity among volatile event speculators. Kimi, going to PostgreSQL with the schema and documentation available to it, wrote 21 lines: four chained subqueries, five tables, six joins, JSON extraction across three nested columns, two COALESCE guards, and a CASE to handle a divide-by-zero in the denominator.

Through the semantic layer, the same answer is three lines:

21 lines of SQL vs 3 with a semantic layer

Each BIRD task has two questions: the first one, and a follow-up asked only if the first one passes. Both versions got the first question right. Only the AtScale run got as far as the follow-up, because the other had spent its budget by then. 

The subqueries, the joins, and the divide-by-zero guard still exist. They live in the semantic model instead of being rebuilt from English on every request. 

On that one task, Kimi went from 19 tool calls to 7, from 16 BIRD-coins to 10, and from 1,669 characters of SQL to 155.

That isn’t one lucky task. Across the sweep, Sonnet averaged 2.9 queries per task through the layer against 5.2 going at the database directly, and 21.3 agent turns against 27.9.

How much does a correct Text-to-SQL answer cost?

Blended over all 600 tasks, with the layer: Grok $0.14, Nemotron $0.30, Kimi $0.49, GPT-5 $0.50, Sonnet $0.52, Gemini $0.63, Opus $0.73.

Grok is the cheapest per correct answer and the least accurate model in the set, which tells you what the metric rewards. A model that submits fast and tests rarely burns few tokens per task, and cheap answers are not the same as answers you want.

Nemotron is the more useful cheap option. At $0.30, it costs double Grok and finishes 1.3 points ahead of it, which is the trade most teams would take.

The comparison worth having is Kimi against GPT-5. Kimi is open weight and scored 49.3% against GPT-5’s 50.5%, a gap of 1.2 points, at a penny less per correct answer.

The layer also made every model cheaper per correct answer, and more accurate. GPT-5 went from $0.79 without it to $0.50 with it, Nemotron from $0.54 to $0.30, and Opus from $0.87 to $0.73.

Why not just upgrade the LLM?

Because switching models never beat adding the semantic layer. Not for one of the seven. 

Say you’re on GPT-5 today, at 21.5%. Switching to Gemini, the highest scorer here, gets you to 36.1%. Keeping GPT-5 and adding the layer gets you to 50.5%. The upgrade is worth 14.6 points. The layer is worth 29. Even Gemini, already the strongest model before the layer, gained 26.1.

And the winner changes. Every vendor on this list ships often, and the table above will be out of date before long.

The business definitions in a semantic layer are files in your own Git repository, written in SML, not configuration inside a model vendor’s product. Point a different model at them and it reads the same metrics. Run two models, and they compute the same numbers. Your definitions stay yours when the model changes.

This benchmark is the evidence for that. Seven models from six vendors, two of them open weight, and the semantic layer helped every one.

Give the next LLM less to figure out

The query that went from 21 lines to three is a good microcosm of this topic. Both versions returned the right number. In one, the LLM had to rebuild the joins, dig the fields out of the JSON, and guard the denominator. In the other, that work was already written down and ready to use.

That is why the semantic layer helped each LLM even with such different habits. Each one still worked the question its own way. None of them had to rediscover the business logic to get there.

You should still test LLMs and pick carefully. But the smallest gain from the semantic layer beat the gap between the best and worst LLMs. Get that in place first.

The definition is work you do once, and every model after that inherits it.

Resources

Full methodology and per-run detail: BIRD-Interact Benchmark Results: Methodology and Analysis.

Connect with me on LinkedIn!

Frequently asked questions

What is a semantic layer?

A semantic layer sits between an AI system and the database and stores the business definitions a question depends on: how a metric is calculated, which tables join to which, and the edge cases the calculation must handle. The LLM asks for a named measure instead of writing the SQL to derive it.

Does a semantic layer improve text-to-SQL accuracy?

Yes. In a BIRD-Interact sweep of seven LLMs on 410 business questions, accuracy without a semantic layer ranged from 21.5% to 36.1%. With an AtScale semantic layer, it ranged from 44.6% to 62.2%. The layer added 20 to 31 points to every model tested.

Why do AI agents give wrong answers on business data?

The definitions a business question depends on sit outside the database. Joins, metric logic and edge cases are not in the schema, so a model guesses them from English on every request. In this benchmark, the no-layer arm received the schema, a plain-English meaning for every column, and a knowledge base with every formula the questions rely on, and still scored about half as well.

Which LLM is best for text-to-SQL?

Gemini 3.1 Pro scored highest in this sweep at 62.2% on the 410 business questions, ahead of Claude Sonnet 5 at 56.6%, Claude Opus 4.6 at 53.7%, GPT-5 at 50.5%, Kimi K2.5 at 49.3%, Nemotron 3 Ultra at 45.9%, and Grok 4.3 at 44.6%. Switching models moved accuracy by up to 17.6 points, less than the layer added to any one of them.

How does a semantic layer change the SQL a model writes?

It moves the hard parts out of the model’s query. One insider-trading task took 21 lines of model-written SQL against PostgreSQL: four subqueries, six joins, nested JSON extraction and a divide-by-zero guard. The same question took three lines through the layer. Both got the first question right, and only the semantic-layer run had budget left for the follow-up.

What does a correct answer cost?

Blended across all 600 tasks with the layer: Grok 4.3 $0.14, Nemotron 3 Ultra $0.30, Kimi K2.5 $0.49, GPT-5 $0.50, Claude Sonnet 5 $0.52, Gemini 3.1 Pro $0.63, Claude Opus 4.6 $0.73 per correct answer. The layer lowered cost per correct answer by 7% to 44% across models, with the biggest drop on Nemotron 3 Ultra ($0.54 to $0.30).

What is the BIRD-Interact benchmark?

BIRD-Interact is an independent benchmark built by the BIRD Team and Google Cloud, accepted at ICLR 2026 as an oral. It holds 600 tasks across 22 messy PostgreSQL databases. The questions are intentionally vague, and a simulated user holds the definitions the model needs. This sweep ran the 410 business questions.