Mubin Attar

Note

For text-to-SQL, measure execution accuracy

  • ~1 min read
  • #evals#sql
  • Source

Two SQL queries can be spelled completely differently and return exactly the same rows. So string-matching generated SQL against a reference — exact-match accuracy — punishes correct answers for cosmetic reasons.

The metric that matters is execution accuracy: run both queries against the database and compare result sets. That's what I'm building the golden-query evals around for DBWhisper, and it's why I'm running it against Spider, whose harness scores execution.

"The model felt right" is not a number. Until the eval runs, the honest state is in progress — which is exactly how it's shown on Evals, not dressed up as a result I don't have yet.

New writing, occasionally.

The odd essay or note when I ship something worth reading — no cadence, no spam, unsubscribe in one click.

Enter your email address, then select Subscribe.