Executive summary
AI tools, like AtScale’s CoModel, can now draft most of a semantic model, and that will change the data modeler’s job. Claude Sonnet 5, working through a semantic layer that AI generated and engineers then tuned, answered 296 of the 317 answerable questions correctly (93%, best of three runs). The misses we examined ran without an error, which is what makes them hard to catch. This post covers what that means for data modelers and why reviewing AI-drafted definitions should be a vital component of their role.
- AI can now draft the model. Data modelers can spend less time writing definitions and more time reviewing what AI drafts.
- AI’s wrong definitions look correct. Several incorrect AI-generated definitions ran without flagging an error in AtScale’s preliminary BIRD-Interact benchmark runs.
- AI semantic layer drafts can fail without human review. Anthropic found LLM-generated definitions “plausible-looking” and net-negative on its evals, so a person has to own each one.
- Ownership is the gap. More than 50% of data practitioners name “lack of clear ownership” as a modeling pain point.
- What gets valued is starting to change. As AI takes over more drafting, typing SQL and model files matters less; knowing what the business needs and catching a metric that double counts or averages a ratio becomes far more important as a skill.
The hardest question in analytics is still “How do you define this metric?” TELUS runs a wireless network of more than 200,000 cells from four hardware vendors. Cornell Lee, who leads its data platform and engineering team, admits: “Writing SQL for this is almost impossible. Even just knowing what table and what column to query is already a challenge.”
For years, the fix was to write the semantic model by hand. AI can now draft most of it, which requires the data modeler’s job to expand from writing definitions to reviewing AI-generated models. Without human oversight, AI’s wrong definitions can go undetected and reach production.
Will AI replace data modelers?
AI can now do most of the model writing. When that shift is fully realized, the job evolves to guiding the AI and identifying where it’s wrong. In AtScale’s preliminary runs on the 317 answerable questions, Claude Sonnet 5 writing raw SQL on its own got 46% right on its best of three runs (at a 45.8% mean). Working through an AI-generated semantic layer that engineers then tuned, the same model scored 93% on its best of three runs (296 of 317), with a mean of 90.6%. These rates leave out 93 of the 410 business questions whose answer keys are flawed, so they can’t be compared with the public leaderboard.
If a machine can draft a model scoring over 90% of these questions right, the role built around writing the layer is the next thing to change.
AtScale’s BIRD-Interact benchmark results ebook sums them up as “a definition someone has to write down, a question someone has to clarify, a test someone has to keep.”
In late 2025, only about 13% of data professionals said data modeling was where AI helped them most, far behind writing code (Practical Data Community’s 2026 State of Data Engineering Survey). In March 2026, 49% of AI-using practitioners in the community’s AI Tooling Pulse named data modeling and semantic layers as what will matter most in 2027. Together, the surveys confirm practitioners expect AI to be a part of modeling going forward.
Should you still write your semantic model by hand?
No. Stop writing it and start reviewing it. Hand-writing every definition doesn’t scale to the number of models AI work needs. Dave Mariani, AtScale’s CTO and founder, said as much about building the benchmark models: “If we had to build 22 semantic models by hand, we wouldn’t be recording this session right now.”
Anthropic’s data team tested the other extreme. It tried “bootstrapping the semantic layer by having an LLM auto-generate metric definitions from raw tables and query logs.” The approach “produced plausible-looking definitions that encoded the very ambiguities we were trying to eliminate,” and it “was net-negative on our evals versus a smaller, human-curated layer.” Anthropic’s advice is “generating the documentation with Claude, but having a human own the definition.”
Atscale agrees that a human has to own every definition. Owning a definition doesn’t mean typing it. Let AI draft the model, then put a person in charge of what ships. Anthropic’s phrase “plausible-looking definitions” is the reason that person’s role matters. We think of that role as the semantic model engineer.
What does a semantic model engineer do?
A semantic model engineer reviews what AI drafts, approves each change the way a team approves code, and helps decide what a metric means when departments disagree. In practice, AI proposes metrics and dimensions, and the engineer approves, edits, or rejects each one before an agent can use it. Many teams haven’t named anyone for that specific workflow yet. In the Practical Data Community survey, 50.7% of practitioners named “lack of clear ownership” as a data modeling pain point.
TELUS shows the impact of placing someone in this role. Its VoLTE Drop Call Rate is “defined simply as total drops divided by total calls, vendor-agnostic terms that stay consistent regardless of which equipment vendor a subscriber is on.” This definition is maintained even though “each vendor’s underlying counters may differ significantly.” Someone created that definition once, and every tool now uses it. This is the desired outcome from ensuring that someone in the business is in this role.
Why is reviewing a semantic model harder than writing one?
A draft only has to look like it could be right. A review has to prove that it is. Three of our misses show what a reviewer has to catch:
- An undefined business term. One question asked for a ranking of accounts by VPN abuse. No column defines a VPN abuser, so the model invented a threshold and computed its guess faithfully.
- A metric that lives in a document. Composite Reliability Index is a formula in the benchmark’s knowledge base, and no table holds it. The fix: “A data engineer carefully specifies how to compute Composite Reliability Index, using the RELIAB_IDX and BSCSI columns.”
- One place, many variations. “Chicago,” “CHICAGO,” and “chicago” with a stray tab are three stored values for one region. A GROUP BY counts them incorrectly as three regions.
Each of these produces SQL that runs without an error, which is what makes the review hard. A reviewer who knows the business spots all three quickly, because they know Chicago is one place and know who to ask about VPN abuse.
Effective review of a semantic model requires:
- Technical foundation to read a definition as code and versioning every change.
- Strong business fluency and understanding of internal customers’ needs.
- Analytical understanding of non-additive metrics like ratios that can’t simply be added up.
- Heightened awareness of the shifting nature of the business to prevent drift.
Go from writing definitions to deciding them
AI will keep getting better at writing definitions. Deciding which ones are right will be the responsibility of those who understand the business, and their effectiveness determines AI accuracy and value for the organization.
AI is also evolving the data modeler’s job from writing definitions to reviewing them, and that means catching errors that the AI presents as correct. The modelers who understand this first will set the definitions every agent in their company reads and ultimately drive the greatest value from AI.
That’s what AtScale CoModel delivers. It drafts metrics, dimensions, and relationships for a person to review, so nothing reaches the semantic model without human approval. CoModel works inside the guardrails data teams already use: Git-governed, versioned, and promoted from dev to QA to production. And it remembers every correction a reviewer makes, so the same mistake doesn’t come back.
Want to see more of the BIRD-Interact benchmark results? Check out the full report here.