Text-to-SQL study shows why executable queries can still give wrong answers

What the authors report

A Text-to-SQL preprint first submitted on 19 September 2026 was revised on 14 October. Its authors evaluated a LangGraph agent on 100 questions about Chicago crime data. They report 93% valid SQL but 60% execution accuracy under their chosen equivalence metric, and explicitly note no external baseline. These are findings from one engineering evaluation, not a general benchmark of database agents.

A successful execution is not enough

A database can execute valid SQL answering the wrong question. A filter may exclude relevant rows, an aggregation may use the wrong denominator or a date boundary may shift a reporting period. Our interpretation is that teams should evaluate syntax, execution and semantic correctness separately. A green query result means the database accepted a statement; it does not prove the business question was answered.

This matters for digital-currency dashboards and publishing analytics. If someone asks how many subscribers confirmed during a month, counting everyone created during that month gives a plausible but different answer. Such a result can look more convincing than an explicit error. Evaluation sets should include cases where a wrong query returns ordinary-looking numbers, so reviewers cannot succeed by noticing obvious failures alone.

A practical assessment approach

Before connecting an agent to operational data, prepare a synthetic database with known answers. Write expected results independently of the model. Include questions with missing definitions, such as whether canceled transactions count. A useful system should request clarification or expose its assumption, rather than silently choosing an interpretation. Score the answer against that contract and retain examples that remain ambiguous even to a human.

Use a restricted account and read-only copy for a trial. Keep generated statements, the schema available at the time and results used to compose answers. This record lets reviewers distinguish a language misunderstanding from stale context or a database limitation. It makes improvements easier to diagnose than an overall percentage alone. A later model or prompt change should rerun the same cases before additional data access is granted.

What the evidence supports

BijliCoin has not reproduced the study. A preprint is not peer-reviewed consensus, and the reported figures depend on its dataset, questions and scoring. The useful lesson is a design question rather than a model ranking: does the organization verify that a query answers the intended question? An agent exposing uncertainty can be more dependable than one always returning a polished explanation of whatever rows came back. Deployment decisions should retain that distinction even when execution success rates improve.

Sources

• Iterative LangGraph Text-to-SQL study version 2 — 2026-10-14
• Full study text — 2026-10-14