Anthropic’s data science and engineering team runs its internal analytics on Claude, and this week, they published the accuracy figures. Without a semantic layer, the answers were right 21% of the time. With one, accuracy improved to 95%, and some domains reached near 99%.
So they wrote a rule into Claude’s skills: before you guess an answer or write any SQL, check the semantic layer first. Their phrase for it was the “mandatory default path for every data question.”
It worked at 95% accuracy. But it only works for Anthropic, on Anthropic’s stack, for one set of questions. Moreover, 95% accuracy for analytics gets you fired. A 5% failure rate is simply not acceptable for enterprises that are making business-critical decisions every day. The version your company needs has to run everywhere: at 100% accuracy, for every tool, every database, every model. That’s the harder build, and it’s the one worth getting right.
The problem isn’t writing the query. It’s knowing which one is right.
Sporting a $965 billion valuation, Anthropic’s Claude is obviously great at writing code. But writing code is a different job from building an end-of-quarter report with AI. With code, there are many ways to get it right, and the computer grades the work for you. The code either runs or it doesn’t, the tests pass, or they fail, and you find out in seconds. You get it wrong, you fix it, you try again. Bugs are part of how code gets written.
A data question has no grader. There’s usually one right answer, and nothing tells you when the AI misses it, because a number built on the wrong definition looks exactly like the correct one. So the AI guesses, and nobody notices. But you can’t have a bug in a data question. You can’t fix a bug in a corporate earnings report. You can’t patch an FCC filing. You can’t apologize for getting your drug manufacturing quality assessment wrong.
So where do the wrong answers come from? Anthropic named three culprits:
- Ambiguity. Ask for “active users” and the agent has to guess what you mean. Which actions count as active? Do you include fraud accounts? Over what time window? Change the guess, change the number. Anthropic says “revenue” alone can map to forty plausible tables.
- Staleness. Tables and definitions change constantly, so the agent’s skills rot. Anthropic watched its own accuracy drift from 95% to 65% in a single month as the docs fell behind the data.
- Retrieval failure. Even when the right number exists and is labeled correctly, the AI can’t always find it, because there’s too much to sift through. Anthropic calls it a “million-field warehouse”: the answer is in there, buried in so much else that the AI walks right past it.
The answer is not more training data
To lift that 21%, Anthropic tried the obvious fix first: give the AI more to learn from. They handed the agent raw access to thousands of SQL files, dashboards, transforms, and notebooks, essentially every question the company had already answered correctly, and confirmed that it read them before answering. The results: accuracy moved by less than 1%. The right answer was sitting in those files about 80% of the time, and the agent had read it, yet it still got the question wrong. As they put it, “Our bottleneck wasn’t access to prior work, it was structure.”
Anthropic built a semantic skill, not a semantic layer
So Anthropic stopped making the agent search and started telling it where to look. What they built, though, isn’t really a semantic layer. It’s a set of semantic skills.
Their version is definitions written into a skill file: what a metric means, which table it lives in, which filters to apply, all spelled out in markdown that the agent reads. Ask for a defined metric, and it hands back one number, the same one every dashboard, spreadsheet, and report returns. Anthropic makes the agent check it first, and raw SQL is the fallback, used only when the skill comes up short. For one team on one warehouse, that’s plenty.
Take something simple like “active users.” Under the hood, that’s a filter: a rule for which rows count and which don’t. Anthropic calls these named filters “segments” and says that letting the AI write them by hand is its single biggest source of wrong answers.
A business runs on dozens of these rules: who counts as an active user, which accounts are fraudulent, which signups are real customers and not throwaway Gmail addresses. Get one of them slightly wrong, and the whole number is off, even when the query looks clean. Define each rule once in the semantic layer, or every agent reinvents it a little differently.
The Anthropic data analytics team even tried to shortcut the work and let an LLM write the metric definitions itself. It backfired, net-negative on their evals, because the model produced plausible definitions that smuggled in the exact ambiguity they were trying to kill. The rule they landed on: let Claude draft the documentation, but a human owns the definition. Governance is a human act. The model just makes it faster.
Hierarchies are how AI thinks
A hierarchy is how an AI reasons toward an answer, and it’s one of the biggest levers on whether that answer is right.
Think about finding one item in a giant grocery store. You don’t scan every shelf. You walk to the department, then the aisle, then the section, then the exact product. Produce, citrus, the bag of lemons. That’s a hierarchy: start broad, narrow down, drill in. It’s the same path an LLM takes through a question, which is why Anthropic’s own post calls standard dimensional modeling “just as important as they ever were.” Dimensional modeling is really about hierarchies.
Anthropic measured whether the answer was right. We measured what it costs. In a production benchmark with the commercial banking arm of a Tier 1 global bank, we put a semantic layer in front of the warehouse and raced it against an agent querying raw data. Routing through their AtScale semantic layer first cut compute by up to 21,000x and lifted accuracy from about 70% to 100%. The bank projected that stopping their LLM from guessing could save them $9 million for one set of five common questions.
Anthropic needs a semantic engine, not a set of LLM skills
Anthropic proved you need semantics in front of your AI, then built the version that fit one team with a skill file for one warehouse. That’s not a semantic layer. A real semantic layer is powered by a semantic engine. That’s what AtScale delivers: a universal semantic layer built on semantic models written in YAML that translates logical queries into physical queries, deterministically. Its dimensional engine resolves metric definitions across an unlimited number of permutations, serving the same answer to every tool and agent across a wide range of data platforms and frontier models.
Snowflake reached the same conclusion and selected AtScale to power CoCo and Cowork via Snowflake Semantic Views, enabling access to any tool, anywhere. When the company that builds the warehouse chooses a universal semantic layer to sit on top of, that’s the north star. Read the announcement.
A skill got Anthropic to 95% on one stack. A universal semantic layer delivers 100% accuracy, across every tool, model, and database. The model was never the bottleneck. The semantic layer is.
SHARE
Guide: How to Choose a Semantic Layer