Every procurement engagement starts the same way: five to seven weeks of data cleanup before a single insight ships. The deliverables are standardized (spend cube, category story, supplier shortlist) but the inputs never are: fragmented extracts, drifting taxonomies, twelve languages, currencies, and column conventions per client.
The first AI pipelines fixed pieces of this, but only the people who built them could run them. The bottleneck just moved from cleanup to the specialists.
The AI engines that clean a client's messy data automatically, plus the platform that put them in every consultant's hands. Every consulting project begins with weeks of tedious cleanup before anyone can do real analysis. I built the engines that do that work: one sorts billions of rows of a company's spending into the right categories so it can finally be analyzed; another untangles messy supplier and material records into clean ones (the harmonization tool is part of this); and underneath sits search infrastructure I built so a machine can actually understand what each supplier and material is, at massive scale. Then I built the self-serve platform that put all of these in every consultant's hands: upload your raw files, click once, get clean client-ready results a few hours later, no technical skill needed. Work that used to bottleneck on a few specialists now runs for the whole firm on demand.
Cleaning the data is the heart of this, and the engines above do most of it. But a client signs their name under a number, not a tidy file, so I am building a thin layer on top of the pipelines that closes the loop between the machine and the people who vouch for its work. After a run, the machine's answer is frozen as evidence and an analyst's correction sits beside it as a separate, attributable layer, all in one place: raw extract, AI-processed result, and human-validated result living in the same store. That last part is the real point. Those human validations are not just a one-time sign-off; they are the signal that feeds a supplier truth record and a client-by-client knowledge base, which over time refine the pipelines underneath and give us something honest to run the models against. It is work in progress, not a finished system, but the shape is a flywheel: the more people validate, the smarter the engine gets.
Dedupe to the work that's truly unique
A spend file's tens of thousands of lines hide only a few thousand distinct supplier-and-item combinations. The engine tidies up names and descriptions, finds those unique combinations, and categorizes each one once, then copies the answer back to every line it appears on.
Build a profile of who each supplier really is
Before categorizing anything, the engine works out what a supplier actually does (whether it makes, resells, or services) and what it primarily sells, using public information and the supplier's own highest-spend items as evidence. This 'who is this supplier' picture is the anchor for every decision that follows.
Define every category, including what it is not
For the client's own three-level category tree, the engine writes a precise definition of each category: what belongs, a couple of examples, and crucially what does not belong, so similar-sounding categories stop getting mixed up.
Shortlist the few categories that could fit
Rather than scan an entire, often huge, category tree for every line, the engine first narrows it to a short list of the most plausible candidates for that specific item. Smaller, focused choices are faster and more accurate.
Decide with a reason and a confidence level
For each line the model weighs the supplier profile, the supplier's top items, and the shortlisted definitions, reasons through the evidence step by step, and picks the single best-fit category, recording a short written reason and a high, medium, or low confidence. When the line text is vague, it leans on what the supplier actually does.
Validate, retry, and run at scale
Every answer is checked against the client's real category tree; anything that does not pass is retried automatically. The whole file runs in batches with retries, so even very large files finish reliably and land as one client-ready package.
Review and edit in place, then export something a client can stand behind
After a pipeline run, the analyst gets one review document per run instead of a raw workbook to wrangle. They work the calls the machine doubted, edit the category in place, and produce a finalized, SME-validated export that a client can actually open. The machine's call is frozen as evidence and the human's correction sits beside it as a separate layer, so the export can always say what was automated and what a person changed. This export part is built today.
Keep raw, AI-processed, and human-validated results in one store
The whole point of the layer is that it does not split the work across three systems. The immutable AI baseline, the human override, and the validation status all live side by side in one document per run. Reading resolves human-over-machine on the fly: show the correction where one exists, the machine's answer everywhere else. One cohesive store means the validation actually compounds instead of getting lost in a pile of edited spreadsheets.
Turn validations into a truth record and a knowledge base that refine the engine
This is the part I am building toward, and I want to be honest that it is partly designed rather than fully wired. The idea is that every human correction appends to a supplier truth record and a client-by-client knowledge base, so the pipelines get a growing, validated reference to categorize against and to run evals against. The validation step stops being a cost and becomes the thing that makes the next run better. Right now the single cohesive store and the attributable human layer are real; auto-feeding those validations back into the engine is the next thing to land.
Build it for non-technical analysts, not engineers
The layer is deliberately one system a consultant with no technical background can use: review a queue of doubtful calls, fix them, sign off in the unit the business cares about, export. The harder machinery, the pipelines and the search infrastructure underneath, stays out of their way. The goal is to let analysts work with very large data on one coherent surface instead of needing a specialist to run anything.
Context over keywords
The biggest accuracy lever is reading what the supplier genuinely does, not just the words in a line. A cryptic line like 'valve, flow control, steel body' is categorized against a supplier whose real business is fittings, so it lands in the right place even when the description alone would mislead.
Shortlist before deciding
Narrowing a large taxonomy to a handful of candidates before the model commits keeps every decision small, fast, and checkable. Focus beats asking the model to reconsider the entire tree on every single line.
A confidence and a reason on every line
Each categorization carries a high, medium, or low confidence and a one-line rationale. Reviewers then spend their time only where the model flagged doubt, and every answer can be defended to a client instead of taken on faith.
Dedupe before you spend
Each unique supplier-and-item combination is categorized once and applied everywhere it appears, cutting model cost on a typical spend file by roughly 95%. The cheapest call is the one you don't make twice.
Checkpointed, resumable runs
Large jobs fail for boring reasons: rate limits, evictions. Row-level checkpoints and automatic retries mean a failure costs minutes, not a restart from zero, which is what makes overnight-scale runs trustworthy.
Keep raw, AI, and human results in one store so validation compounds
The easy build is three places: the raw extract somewhere, the model output somewhere else, and the corrected file on someone's laptop. That demos fine and quietly throws away the most valuable thing. I kept the immutable AI baseline, the human override, and the validation status in one document per run, resolved human-over-machine on read. The machine's answer is frozen as evidence and the correction is a separate, attributable layer, never an overwrite. The tradeoff is more discipline up front (two layers to reconcile instead of one editable grid), but it is what lets the human work feed back into the engine instead of evaporating.
Treat SME corrections as the training and eval signal, not just a sign-off
The tempting framing is that review is the last mile: a human checks the file and ships it. The more useful framing is that the correction itself is the asset. A machine-said-X, human-changed-it-to-Y pair is the cleanest signal you will get for refining the categorization, and a growing set of validated answers is what you run evals against. So I am designing the layer so validations append to a supplier truth record and a client knowledge base that the pipelines read from over time. I want to be straight that this feedback path is designed and partly built, not fully wired yet, but it is the reason the layer exists at all: it is a refinement loop on the engine, not a review screen bolted on the end.
Make it usable by analysts, not just by the people who built the engine
The first pipelines only the builders could run, so the bottleneck moved from cleanup to specialists. The judgment here is to spend effort making the loop-closing layer something a non-technical analyst can drive end to end (review, edit, validate, export) on one store, rather than exposing the raw machinery. That costs flexibility, an analyst gets a guided surface, not a query console, but it is what turns the engine from a specialist tool into something the whole team can actually use on real client data.