Tool-first agents: using RAG to fix what NL-to-SQL can't

Are you frustrated by waiting 3-4 minutes for an NL-to-SQL agent that spends tokens and returns no results? Asking for data you know exists, and getting back an empty table, again?

I was. Our supplier and benchmark pulls ran through a classic NL-to-SQL agent, and the experience was exactly that: long waits, burned tokens, empty results. Rebuilding it around a tool-first pattern took those pulls from ~3 minutes to ~30 seconds and, more importantly, made the empty-result problem mostly disappear. Here's the pattern, because it transfers to almost any agent that has to query real enterprise data.

Why NL-to-SQL fails on real warehouses

NL-to-SQL demos beautifully and degrades in production for three compounding reasons:

1. The model guesses filter values it has never seen. Your user asks for "Acme's spend in packaging." The warehouse knows that supplier as ACME GMBH & CO. KG, and the category as Packaging & Containers > Corrugated. The model writes a syntactically perfect query against values it invented, and gets zero rows. Not an error. Zero rows.

2. Empty results are silent failures. A bad query throws; a mis-filtered query returns nothing. The agent can't tell "no data exists" from "I guessed the spelling wrong," so it retries with another guess. Each retry is another round trip through a large schema prompt. That's where your 3-4 minutes and your token bill go.

3. Schema sprawl eats the context window. Real lakes have hundreds of tables and columns with names only the data engineer loves. Stuffing schema into the prompt scales linearly in cost and worse in accuracy.

The root cause isn't the model. It's the interface: free-form generation against a surface the model can't observe.

The pattern: retrieve, then constrain, then decide

Invert the order of operations. Don't let the agent write a query until the values in that query are real.

Step 1: index the filterable values, not just the documents. Build retrieval indexes over the actual distinct values of your filter columns: supplier names, category labels, regions, business units. This is RAG, but pointed at column values instead of wiki pages. A lexical and embedding hybrid works well because entity names fail differently in each.

Step 2: resolve entities before querying. "Acme's packaging spend" first hits the index and comes back with candidates: ACME GMBH & CO. KG, Acme Industrial Ltd. The agent picks from a short list of values that provably exist. I think of this as constrain-then-decide: generation chooses from retrieved candidates; it never invents.

Step 3: replace raw SQL with typed tools. The agent doesn't emit SQL strings; it calls get_spend(supplier, category, period), a tool whose parameters are validated against the resolved entities, and whose body is SQL someone competent wrote once. Tool contracts give you three things free-form generation never will: validation at the boundary, deterministic and replayable outputs, and an API surface other products can call. Our other internal tools ended up consuming the same capabilities the agent uses: one harness, many consumers.

What changed

• Supplier-list and benchmark pulls: ~3 minutes → ~30 seconds, mostly from eliminating retry loops.
• Empty-result rate: from "constant support complaint" to rare, and when it happens, it now means the data doesn't exist.
• The retrieval layer became reusable infrastructure: every new use case starts from resolved entities instead of raw schema.

When NL-to-SQL is still fine

Ad-hoc exploration by an analyst who reads the SQL before running it. Small, stable schemas. Internal prototyping. The pattern above earns its complexity when non-technical users depend on answers being right, at scale, without supervision.

Adopting it: a short checklist

1. List the 5 to 10 columns users actually filter on. Index their distinct values.
2. Add an entity-resolution step that returns candidates with confidence, not a single guess.
3. Wrap your top queries as typed tools with validated parameters.
4. Log every empty result. It's now signal, not noise.
5. Only then let an agent orchestrate the tools.

The general lesson: when an agent is failing, look at the interface before the intelligence. Most "dumb agent" problems are observability problems: the model can't see the thing you're asking it to reason about.