MEET ATSCALE / HOW AI SCORES ON BIRD-INTERACT
What BIRD-Interact says about AI and your numbers.
On BIRD-Interact, adding an AtScale semantic layer made an LLM twice as accurate. The same benchmark shows how much work is left before a business can hand its metrics to AI.
Read the full resultsWHY BIRD-INTERACT MATTERS
Enterprises use a fraction of what AI can do.
“Only 5% to 10% of the current value of the technology” is actually being used.
–Dario Amodei, CEO of Anthropic, on stage with Marc Benioff, September 15, 2026
AI gives persuasive answers that are often wrong, so businesses don’t trust it with the numbers they run on. Benchmarks with clean data and obvious questions don’t catch that. BIRD-Interact uses vague questions, messy data and strict scoring, the way a real warehouse behaves, so it tests the gap Dario describes.
BIRD-INTERACT 101
A simulated user, a real database and a budget.
BIRD-Interact gives an AI model a business question and a PostgreSQL database to answer it from, without the documentation that says which columns to use or how to calculate the answer. A simulated user plays the person asking. The model can ask it what it meant, and the user says when an answer is wrong. Each query costs BIRD COINs from a budget of 17.86 per task. A task passes only if the model returns the right result before the budget runs out.
FIG. 1 · SAME QUESTION, TWO RUNS
BIRD-Interact, released October 2025: 600 tasks, 22 PostgreSQL databases, 2,010 columns adapted from real schemas. AtScale ran on the public test harness in stress mode with Claude Sonnet 5, September 11 to 20, 2026. Results are preliminary.
ALL 600 TASKS, SIX MODELS
Every model scored higher with AtScale, by up to 78%.
We ran BIRD-Interact twice on each of seven LLMs: once with no semantic layer, and once with an automatically generated AtScale layer. Every figure below is scored across all 600 tasks, the same total the public leaderboard uses. Two of the six models are open weight, and four are frontier models.
FIG. 2 · PASS RATE ON ALL 600 TASKS, PRELIMINARY
| Model | Without AtScale | With AtScale | Improvement |
|---|---|---|---|
| Claude Sonnet 5 · Anthropic · frontier | 32.3% | 53.2% | +64% |
| Claude Opus 4.6 · Anthropic · frontier | 28.3% | 50.3% | +78% |
| Kimi 2.5 · Moonshot AI · open weight | 28.4% | 49.0% | +72% |
| GPT-5 · OpenAI · frontier | 27.7% | 48.2% | +74% |
| Grok 4.3 · xAI · frontier | 32.5% | 46.0% | +42% |
| Nemotron 3 Ultra · NVIDIA · open weight | 29.2% | 44.8% | +54% |
Preliminary. Combined phase-1 pass rate across all 600 tasks, BIRD-Interact stress mode. With AtScale, best of three runs. Without AtScale, a single run on the same test harness.
No model can pass all 600 tasks.
The top score on the leaderboard is about 53%, and that ceiling is one reason we chose this benchmark. In a real business, some questions don’t have a single right answer, and some data can’t answer them at all.
- 190 tasks are database-management operations (create, read, update, delete), so they aren’t questions. Both runs send them straight to Postgres, so a semantic layer can’t change their score.
- 93 tasks have a broken answer key. The graded answer contradicts the benchmark’s own business-terms document, reads a field the database doesn’t contain, or doesn’t specify a row order, so the same result can pass one run and fail the next.
THE PATH TO 100%
A few wrong answers are still too many.
A business can’t depend on AI that’s sometimes wrong, and an agent that feeds one answer into the next step makes each miss worse. On BIRD-Interact, the misses came from four problems: vague prompts, undefined terms, messy data and calculations with several steps. The AtScale AI Accuracy Flywheel fixes each miss. When Canary arrives this fall, it’ll test each fix on every run so the fix holds.
FIG. 3 · THE ATSCALE AI ACCURACY FLYWHEEL
For the bigger picture:
- The path to 100% AI accuracy: five real misses from this benchmark and how each one was fixed.
- AI accuracy in your industry: the metrics people and agents get wrong most often in each industry.
- How AtScale powers AI agents: how agents get the same answers your team does.
THE EVIDENCE
Check the numbers yourself.
Every AtScale figure on this page comes from runs on the public BIRD-Interact test harness in stress mode.
- AtScale’s full findings: per-run detail, control runs, generated models and all 93 excluded questions.
- The benchmark: the test harness, 600 tasks, 22 databases and the scoring code.
- The live leaderboard, updated as new entries are scored.
- The paper: how the BIRD team built and scored the tasks.
To see how AI scores on your own data, bring your own questions to a workshop.
Book a workshop.
- Bring us your most controversial metric and the business questions your team and your agents ask about it every week.
- We benchmark your AI on them, with AtScale and without it.
- You keep the semantic model we built to run the test.