Every sourcing conversation starts with three questions: what should this cost, who else should we buy from, and what have companies like us saved. Each took an analyst a day or more to answer, stitched together from old decks and tribal memory. Benchmarks arrived without context (no cohort, no sample size) and clients rightly pushed back.
The first attempt at an AI answer made it worse in a new way: a natural-language-to-SQL agent that guessed filter values, hallucinated categories that didn't exist, and burned three minutes of tokens to return zero rows.
A tool that answers 'are we overpaying, and who else could we buy from?' in minutes instead of a day. Every buying decision starts with three questions: what should this actually cost, who else could we buy it from, and what have similar companies saved by doing this? Answering them used to take an analyst a full day of digging through old files and memory, and the firm's first AI attempt made it worse by confidently inventing answers that were wrong. This answers all three in minutes, backed by real evidence drawn from hundreds of companies, never guesswork. It's the difference between telling a client 'I think you're overpaying' and showing them 'here's proof, based on what 200 comparable companies actually did.' I led the build, including scrapping the first approach that failed and replacing it with one that works: agents that retrieve valid, evidence-backed paths first, then run deterministic query templates, instead of writing SQL they can get wrong.
Make the corpus trustworthy first
Ingestion enforces hard invariants: no client-year double counting across dataset versions, plus completeness filters on partial extracts. Benchmarks are only as good as the discipline underneath them.
Compute ranges the defensible way
Peer percentiles aggregate to client level before any distribution is computed, so one client with thirty projects can't masquerade as thirty peers. Every range ships with its cohort and sample size.
Index what's askable
Instead of letting an agent guess filters, the platform indexes the valid analytical paths through the data: real categories, real cohorts, each with evidence attached. The agent's first move is always retrieval, never generation.
Two tools, no free-form SQL
Each agent gets exactly two tools: search the paths, then pull through a deterministic SQL template with validated parameters. Empty results stopped meaning 'bad guess' and started meaning 'the data isn't there.'
Answers become artifacts
Results return as schema-tagged sections that render into briefs, shortlists, and Excel exports: replayable, auditable, and consumable by other internal tools through the same APIs.
Realized beats projected
The savings library stages every initiative by how far it actually got. Showing clients what peers executed, not what consultants once proposed, is what ended the benchmark arguments.
Retrieval-first agents
The pivot from NL-to-SQL to retrieve-then-decide took pulls from ~3 minutes to ~30 seconds and near-eliminated empty results. The pattern now underpins every agent on the platform.
Anonymity in depth
Identity columns are stripped before the model sees rows, prompts forbid naming, and a final sanitizer scrubs answers. Benchmarking confidentiality can't hinge on one layer behaving.
Schema-tagged payloads
Outputs carry their own column schema, so the frontend parses structure instead of hard-coding field names. Data evolves; parsers survive.