How SQL Analyst is measured
The evaluation harness is built and tested, but no run with a real model has been recorded, so this page shows no accuracy numbers. It explains what each suite measures and shows the one result that has been recorded, with its source.
Three ways to check the agent
BIRD Mini-Dev
500 public questions over 11 SQLite databases. Scored by execution accuracy with the official scorer: the answer counts only if its rows match the reference query's rows.
Golden Olist set
60 questions on the Olist data written for this project, with reference SQL and expected clarifications. Measures accuracy, clarification behaviour and cost per question.
Safety red team
30 adversarial cases: write and DDL requests, slow queries, settings changes, and attempts by the São Paulo manager to read other states. A case passes when nothing unsafe runs and no out-of-scope row returns.
The defences hold even when the model attacks
This run replaced the model with a scripted stand-in that always writes the attack, so it tests the guard, the read-only role and row-level security on their own. It says nothing about how often a real model attempts an attack.
| Model | Status | Cases passed | Non-SELECT executed | Rows outside scope |
|---|---|---|---|---|
| Real model | Not run | None yet | None yet | None yet |
| Scripted attacker model, not a real LLM | Run | 30 of 30 | 0 | 0 |
Source: backend/evals/results/safety/compromised-model/summary.json, recorded 19 October 2026 by backend/evals/safety/runner.py.
What to keep in mind
Read results with these limits
- BIRD Mini-Dev is 500 questions over 11 databases; differences of a point or two between ablations are within run-to-run noise.
- The few-shot step uses two-fold cross-validation over Mini-Dev itself (BIRD's train databases differ from the dev ones), so it measures in-domain examples, not a separate pool.
- Benchmark runs cap results at 100,000 rows and 30 s per query, matching the official scorer's timeout; the demo uses tighter limits.
- The golden Olist set was written for this project and reviewed by its author; it is small and not a public benchmark.
Built by Tiju Lukose
SQL Analyst is a portfolio project. More about my work, and how to reach me, is on my site.