- query executionHow this was measured — A SELECT-only, single-statement validator checks enrolled tables before execution through a least-privilege read-only connection. Queries outside the enrolled schema or without a schema index are rejected.
- read-only
- LLM providers behind one fallbackHow this was measured — OpenAI → OpenRouter → DeepSeek → Groq → Anthropic → Gemini; the first with credentials wins, Gemini's free tier is the final fallback.
- 6
- SQL dialects supportedHow this was measured — Postgres, MySQL, and SQL Server via SQLAlchemy 2.0 + ODBC, with per-dialect generation directives.
- 3
Context
Natural-language-to-SQL is easy to demo and hard to trust. Calling an LLM to write SQL takes an afternoon; the reason most such tools never reach production is that a model will happily emit DROP TABLE, a multi-statement payload, or a query against a table that doesn't exist. DBWhisper turns a plain-English question into SQL across Postgres, MySQL, and SQL Server, validates it, runs it through a read-only database connection, and explains the answer.
Constraints
Three constraints shaped the design. Safety must be structural— the read-only promise can't depend on the model behaving. Schemas are large — real databases have dozens to hundreds of tables, too many to paste into a prompt, so the agent has to retrieve the right ones. And it runs on a $0 stack (Vercel + a Hugging Face Docker Space + Neon Postgres), which rules out waste: small prompts, bounded loops, lean images.
System architecture
A split deploy in three tiers: a Next.js UI on Vercel, a FastAPI + LangGraph agent on a Docker Space, and the target database queried through a read-only user. Retrieval and conversation memory sit in Neon Postgres with pgvector.
System map
Tap to inspect
- 01
The browser client; sends a plain-English question over REST /query.
- 02
The agent loop — retrieves the right schema slice, generates SQL, validates, executes, and summarizes.
- Instead of
- Flask/Django request loops, or raw LLM calls with no structure
- Why
- LangGraph is a state machine for a multi-step job — retrieve schema → generate → validate → execute — with checkpoints persisted to Postgres for conversation memory. FastAPI gives async streaming, typed I/O, and clean middleware for the rate-limit and auth gates.
- Tradeoff
- Heavier runtime than raw httpx + Pydantic — paid back in typed tool contracts and state that survives a restart.
First provider with credentials wins; Gemini's free tier is the final fallback so it runs with zero paid keys.
- Instead of
- A single hardcoded provider
- Why
- OpenAI → OpenRouter → DeepSeek → Groq → Anthropic → Gemini, ordered by cost. Gemini's free tier is always appended as the last resort, so a quota wall on any one provider is invisible to the user. Every provider failure is logged, not just rate-limits.
- Tradeoff
- Six providers' credentials to manage, and slightly different response shapes to normalize.
Deterministic gate: SELECT-only, single statement, no DDL/DML, only enrolled tables — rejects when it can't prove safety.
- Instead of
- Trusting the model to write safe SQL; relying on the DB role alone
- Why
- A deterministic validator runs before every execution — whether the SQL came from the LLM or a user edit. sqlparse + regex reject DML/DDL/EXEC, multiple statements, SELECT INTO, and system-schema access, and check every table against the enrolled schema index. If the index is missing, it refuses.
- Tradeoff
- A heuristic parser isn't a formal proof — so it's one layer of defense-in-depth on top of a read-only DB user, timeouts, and row caps.
- 03
Semantic search over table summaries to find the candidate tables for the question — records which tables it touched.
- Instead of
- Stuffing the whole schema into every prompt
- Why
- Retrieval runs over documented table summaries, not raw DDL, so it stays accurate on large databases and keeps the prompt small. Top-4 tables per question; the agent is capped at 8 calls per tool per run so it can't loop.
- Tradeoff
- A one-time enrollment step per database to document + embed the schema.
Human-approved question→SQL pairs; the agent prefers adapting a close verified example over writing SQL from scratch.
- Instead of
- Generating every query from scratch, forever
- Why
- Approved (question → SQL) pairs are retrieved at generation time so the agent adapts a proven query instead of hallucinating one. Every approval makes the next answer for that database better.
- Tradeoff
- Needs a human to approve pairs — a deliberate quality flywheel, not free.
- 04
Table-documentation embeddings live in the same Postgres as everything else — retrieval searches the docs, not raw schema.
- Instead of
- A separate vector DB (Pinecone, Weaviate, Milvus)
- Why
- One managed Postgres holds embeddings, LangGraph checkpoints, metadata, and sessions — no second service to run, no ETL between stores. Embeddings are searched by cosine similarity with JSONB metadata filters on the same connection.
- Tradeoff
- pgvector's ivfflat index caps at 2000 dims, so the 3072-d Gemini embedding falls back to a sequential scan — fine at this scale, and worth it to avoid operating a second database.
LangGraph conversation memory, persisted per turn, so follow-ups have context.
- Instead of
- A stateless agent, or in-memory sessions lost on restart
- Why
- Conversation state (last few turns, tables touched, insights) is persisted per user + session, so a follow-up like "now break that down by month" has context — and survives a redeploy.
- Tradeoff
- A DB round-trip per turn to read + write the summary.
Postgres, MySQL, or SQL Server — queried through a read-only user, with a timeout and a row cap.
- Instead of
- One hardcoded dialect
- Why
- The same agent targets Postgres, MySQL, or SQL Server; a per-dialect directive in the system prompt keeps generated SQL dialect-correct. Execution runs through a read-only user with a 30s timeout and a 1000-row cap.
- Tradeoff
- Per-dialect prompt tuning — the alternative (dialect-blind SQL) shipped a real bug: T-SQL `TOP` emitted against Postgres.
Hover a node to preview · click one to see the decision behind it
Use the arrow keys, Home, or End to move between nodes. Press Enter or Space to pin a node's decision, and Escape to release it.
- Next.js UI: The browser client; sends a plain-English question over REST /query.
- FastAPI + LangGraph: The agent loop — retrieves the right schema slice, generates SQL, validates, executes, and summarizes. Why: LangGraph is a state machine for a multi-step job — retrieve schema → generate → validate → execute — with checkpoints persisted to Postgres for conversation memory. FastAPI gives async streaming, typed I/O, and clean middleware for the rate-limit and auth gates. Instead of: Flask/Django request loops, or raw LLM calls with no structure. Tradeoff: Heavier runtime than raw httpx + Pydantic — paid back in typed tool contracts and state that survives a restart.
- search_tables: Semantic search over table summaries to find the candidate tables for the question — records which tables it touched. Why: Retrieval runs over documented table summaries, not raw DDL, so it stays accurate on large databases and keeps the prompt small. Top-4 tables per question; the agent is capped at 8 calls per tool per run so it can't loop. Instead of: Stuffing the whole schema into every prompt. Tradeoff: A one-time enrollment step per database to document + embed the schema.
- verified queries: Human-approved question→SQL pairs; the agent prefers adapting a close verified example over writing SQL from scratch. Why: Approved (question → SQL) pairs are retrieved at generation time so the agent adapts a proven query instead of hallucinating one. Every approval makes the next answer for that database better. Instead of: Generating every query from scratch, forever. Tradeoff: Needs a human to approve pairs — a deliberate quality flywheel, not free.
- pgvector: Table-documentation embeddings live in the same Postgres as everything else — retrieval searches the docs, not raw schema. Why: One managed Postgres holds embeddings, LangGraph checkpoints, metadata, and sessions — no second service to run, no ETL between stores. Embeddings are searched by cosine similarity with JSONB metadata filters on the same connection. Instead of: A separate vector DB (Pinecone, Weaviate, Milvus). Tradeoff: pgvector's ivfflat index caps at 2000 dims, so the 3072-d Gemini embedding falls back to a sequential scan — fine at this scale, and worth it to avoid operating a second database.
- multi-LLM: First provider with credentials wins; Gemini's free tier is the final fallback so it runs with zero paid keys. Why: OpenAI → OpenRouter → DeepSeek → Groq → Anthropic → Gemini, ordered by cost. Gemini's free tier is always appended as the last resort, so a quota wall on any one provider is invisible to the user. Every provider failure is logged, not just rate-limits. Instead of: A single hardcoded provider. Tradeoff: Six providers' credentials to manage, and slightly different response shapes to normalize.
- validate_sql: Deterministic gate: SELECT-only, single statement, no DDL/DML, only enrolled tables — rejects when it can't prove safety. Why: A deterministic validator runs before every execution — whether the SQL came from the LLM or a user edit. sqlparse + regex reject DML/DDL/EXEC, multiple statements, SELECT INTO, and system-schema access, and check every table against the enrolled schema index. If the index is missing, it refuses. Instead of: Trusting the model to write safe SQL; relying on the DB role alone. Tradeoff: A heuristic parser isn't a formal proof — so it's one layer of defense-in-depth on top of a read-only DB user, timeouts, and row caps.
- Neon Postgres: LangGraph conversation memory, persisted per turn, so follow-ups have context. Why: Conversation state (last few turns, tables touched, insights) is persisted per user + session, so a follow-up like "now break that down by month" has context — and survives a redeploy. Instead of: A stateless agent, or in-memory sessions lost on restart. Tradeoff: A DB round-trip per turn to read + write the summary.
- Target DB: Postgres, MySQL, or SQL Server — queried through a read-only user, with a timeout and a row cap. Why: The same agent targets Postgres, MySQL, or SQL Server; a per-dialect directive in the system prompt keeps generated SQL dialect-correct. Execution runs through a read-only user with a 30s timeout and a 1000-row cap. Instead of: One hardcoded dialect. Tradeoff: Per-dialect prompt tuning — the alternative (dialect-blind SQL) shipped a real bug: T-SQL `TOP` emitted against Postgres.
Enrollment (once per database). A target database is introspected, each table is documented into structured sections (summary, columns, relationships, stats), and those chunks are embedded into pgvector. Retrieval later searches the documentation embeddings, not the raw schema — which is what makes it work on large databases at all.
Data flow
One question, end to end: the UI POSTs to /query; the agent runs search_tables to find candidate tables by semantic similarity, pulls only the needed section of each (columns / relationships / stats) so the prompt stays small, checks verified queries for a close human-approved example to adapt, then generates SQL. Generation passes through validate_sql before anything touches the database; the query executes under a timeout and row cap; the result is summarized back into natural language. A per-turn LangGraph checkpoint gives follow-ups context.
DBWhisper · the flagship
Inside the request path.
Explore how DBWhisper retrieves context, generates SQL, checks it, and returns results. This illustration uses sample data; published evaluations are linked separately.
pgvector similarity over embedded table docs pulls only the tables a question needs — never the whole schema into the prompt.
- search_tables · pgvector similarity
- Structured schema sections
- Verified NL→SQL examples
Example: Monthly revenue from January through April 2024
Key decisions
The decisions that make it trustworthy — each with the alternative it beat and the cost it accepted.
A deterministic, fail-closed SQL validator — not LLM trust
Instead of: trust the model's SQL, an LLM-as-judge safety check
It uses sqlparse (not fragile regex) to enforce SELECT/WITH-only, single-statement, no DDL/DML/EXEC, no information_schema/sys/pg_catalog — and, given a database, rejects any query touching a table outside the enrolled schema. If no schema index exists, it fails closed rather than open.
Trade-off accepted — Rejects some exotic-but-valid queries and adds an enrollment step — a false 'safe' is far costlier than a false 'unsafe'.
Defense in depth, not a single gate
Instead of: rely on the validator alone
Below the validator: read-only DB users (a probe warns if a connection looks writable), a query timeout + row cap, and log sanitization that masks passwords, API keys, and SQL literals. The warning probe does not replace correctly configured database permissions.
Trade-off accepted — More moving parts to maintain and test.
Multi-provider LLM fallback over single-vendor lock-in
Instead of: one provider (e.g. OpenAI only)
A priority chain (OpenAI → OpenRouter → DeepSeek → Groq → Anthropic → Gemini) picks the first provider with credentials, with Gemini configured as the final fallback. Requests still depend on available credentials, provider capacity, and quota.
Trade-off accepted — Six provider adapters to keep working behind one interface.
Retrieval over context-stuffing
Instead of: dump the whole schema into the prompt
Schemas are documented, embedded, and retrieved per question, keeping the prompt small and the hallucination surface (and token cost) low.
Trade-off accepted — An enrollment/embedding step per database.
What failed
Early on, the model would emit SELECT TOP n against Postgres — valid T-SQL, invalid everywhere else — so cross-dialect queries failed. The fix was a deterministic dialect directiveinjected before generation, stating the target dialect's rule explicitly (LIMIT for Postgres/MySQL, TOP for SQL Server) rather than hoping the model inferred it.
Cause — the model defaulted to SELECT TOP n regardless of the target database
Fix — inject a per-dialect directive before generation, and cap tool loops so a confused agent can't spin
Performance & cost
The controls and the measurements answer different questions. These are the configured execution boundaries, followed by the observed evaluation results.
| Metric | Value | Method |
|---|---|---|
| query execution | read-only | Fail-closed validator + read-only DB user; a query is rejected if it references a table outside the enrolled schema. |
| LLM providers | 6 | One fallback interface; Gemini free tier is the final fallback so it runs with zero paid keys. |
| SQL dialects | 3 | Postgres, MySQL, SQL Server via SQLAlchemy 2.0 + ODBC. |
| per-run guardrails | k=4 · ≤8 calls · 30s | Retrieval returns the top 4 tables; each tool is capped at 8 calls per run; execution runs under a 30s timeout, a 1000-row cap, and a 5000-char SQL limit. |
| infra cost | $0 | Vercel + Hugging Face Docker Space + Neon Postgres, all free-tier. |
Results (measured)
Two evaluation scopes: the complete pipeline on the golden set, and the generation core on a Spider dev subset. Each result states its method and limitations:
- 73% (101/139, dev)Spider · Execution accuracy
- Scoped Spider dev-split run of DBWhisper’s generation model (qwen/qwen3-32b at temp 0.1) with each database’s schema in context: 139 questions across 18 databases. Generated SQL is executed against the real Spider SQLite databases and compared by result set — ordered when the gold query has ORDER BY, multiset otherwise (standard execution match). 101/139 correct; malformed generations count as incorrect. 9 of the 148 sampled questions could not be scored after repeated provider throttling and are excluded, not counted either way. Measures the NL→SQL generation core; the deployed agent adds schema retrieval and a read-only validator on top.
- 82% exact · 100% fail-closedCustom golden-query set · Execution accuracy · fail-closed refusals
- 22 natural-language golden queries + 4 unsafe/out-of-scope prompts over a read-only Postgres store, run end-to-end through DBWhisper’s live pipeline (schema retrieval → generation → read-only validator → execute). 82% exact result-set match (18/22); 95% (21/22) when crediting correct answers that returned an extra column. All 4 destructive or out-of-scope prompts were refused fail-closed (4/4).
Operations
It runs as a split deploy with the agent in a Docker Space. Per-run guardrails keep it bounded: each tool result is cached per run, and a tool called more than eight times returns an abort hint that tells the model to stop looping and produce output. Logs are sanitized before write, and the validator's rules are covered by unit tests that gate CI (pytest + ruff, alongside a gitleaks secret scan).
def validate_sql(sql: str, db_flag: str | None) -> None:
stmt = _single_statement(sql) # exactly one statement
_require_read_only(stmt) # starts with SELECT/WITH; no DDL/DML/EXEC
_reject_system_tables(stmt) # no information_schema / sys. / pg_catalog.
if db_flag:
schema = _load_schema_index(db_flag) # fail closed if missing
_require_enrolled_tables(stmt, schema) # only tables we can vouch forWhat I'd do differently
Two things. I'd add a small golden-query eval per enrolled database so retrieval and generation quality are measured, not assumed — right now the verified-query flywheel is the feedback loop, but it isn't scored. And I'd move the six provider adapters behind a single typed capability interface earlier; retrofitting consistent retry/timeout behavior across them was more work than building it in from the start.
Evidence
The app is live and the source is public — the validator, the agent tools, and the multi-provider chain are all in the repository linked at the top of this page.
Changelog
Schema browser, a verified NL→SQL training flywheel, and the full marketing site.
Flagship web console — auto-generated charts, CSV export, sortable results, staged progress.
Production hardening: Argon2id auth, per-query tenancy, API-key + per-IP rate limits, a SQL allowlist, and optional Sentry.
Productionized from a prototype — Next.js frontend, dialect-agnostic schema extraction, and a live demo on Neon.