Text-to-SQL Benchmark Study: 4KB Semantic Layer Adds 17-23 Percentage Points of Accuracy, Model Choice Doesn't Decide

Michael Rumiantsau and Ivan Fokeev published the ArXiv preprint Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models on April 28, 2026. The study poses a simple but powerful question: how much does the model “matter” versus how much does context (semantic layer) matter for text-to-SQL accuracy?

Experimental Setup

The authors test three frontier models: Claude Opus 4.7, Claude Sonnet 4.6, and GPT-5.4. The benchmark consists of 100 natural language questions translated into SQL queries over the Cleaned Contoso Retail Dataset hosted in ClickHouse. Each model runs two rounds: once without a semantic layer, and once with a 4 KB markdown document that describes “measures, conventions, and disambiguation rules” for the dataset.

Results

The results are surprisingly clean:

• Without semantic layer: 45.5%–50.5% accuracy across all three models
• With semantic layer: 67.7%–68.7% accuracy across all three models
• Improvement: +17 to +23 percentage points

Within each of the two setups, models are statistically indistinguishable. In other words, Opus 4.7 is not significantly better than Sonnet 4.6 or GPT-5.4 if they use the same context.

The Main Takeaway

The authors’ quote: “The presence of the semantic-layer document accounts for essentially all of the significant variance; model choice within tier does not.”

For enterprise practice, the message is clear: a better frontier model within the same tier will not replace better dataset documentation. A 4 KB markdown file with metric definitions, naming conventions, and homonym disambiguation rules delivers 17-23 percentage points — which is more significant than any model upgrade within tier.