Inferred-Truth AI Won't Survive a Regulator's Question

Most of my career has been in regulated financial services. Wealth, investments, advice. Six of those years running a contact centre where every customer call was a regulated interaction. So when I see AI vendors pitching “your AI just reads everything you have” to financial services buyers, I have a particular reaction.

It works fine until somebody asks where the answer came from.

The scene

Picture a customer asking your AI assistant about a fee. Maybe the management fee on a particular product. Maybe whether their account qualifies for a tier discount. Maybe whether a specific transaction attracts an exit fee.

Your AI synthesises from everything it has. The PDS (the product disclosure statement) from a couple of years back. A SharePoint page somebody updated last month. A Teams thread where two product managers debated a fee change in March. A Confluence doc that shows the new fee structure but hasn’t been signed off by Compliance yet.

The AI gives a confident answer. The customer takes the answer. Acts on it.

Eight months later, somebody on your remediation team is reading that conversation, trying to work out which version of the truth the customer was given. Trying to explain to the regulator how the answer got produced. Trying to map the chain of evidence.

There isn’t one. The AI inferred. It doesn’t know which document it weighed most. It doesn’t know what was deprecated. It doesn’t know what was approved.

This is the inferred-truth problem. We’ve written about it before. But in regulated work it isn’t a productivity problem. It’s a compliance problem.

Why regulated work is different

In regulated industries, approximately right is wrong.

If a customer asks about a fee, there’s one correct answer. Not the average of three documents. Not the most-cited version across the corpus. The specific, current, approved fee schedule for that product, that customer, that day.

The whole apparatus of regulated work exists because “we got it roughly right” doesn’t pass. There are approvers. Review dates. Version controls. Disclosure language signed off by Compliance. Advice scripts approved by Risk. The reason this apparatus exists is that customers act on the answer, and somebody has to be accountable for it being correct.

Now imagine that apparatus reading the output of an inferred-truth AI assistant. The AI synthesised from sources of varying authority and varying age. Nobody approved its output. Nobody can name the version it used. Nobody can produce the chain of evidence.

This isn’t a future problem. It’s happening now in businesses that have plugged AI assistants into their content stores without thinking about what those assistants actually do.

Four things inferred truth can’t do for regulated work

Authorship. Who wrote this answer? When? With what approval? Inferred-truth AI can tell you which documents it consulted, but the answer itself has no author. It’s a stitched-together synthesis. In a regulated context, an unapproved answer reaching a customer is a problem regardless of whether a human or an AI delivered it.

Versioning. Which version of the policy did the AI use? Inferred-truth indexes don’t distinguish between the approved current version and the draft sitting next to it. They don’t know which PDS supersedes which. They certainly don’t know that the version from six months ago is now wrong because the fee changed.

Audit trail. When the regulator asks how a customer ended up with a particular answer, you need a chain. The article the answer came from. The approver who signed it off. The review date. The change log. Inferred truth produces an answer and a list of source links. Not a chain. A scatter.

Contradiction. This is the worst one. Inferred-truth AI happily cites the new fee structure and the old fee structure in the same response, especially if both documents still exist in the index. It will quote the deprecated FAQ next to the current one. It doesn’t know the difference. The customer reading the answer doesn’t either.

Why a substrate works for this

A substrate is the inverse pattern. Instead of synthesising across everything you have, it reads from a canonical layer that humans curate.

The canonical layer has properties inferred truth doesn’t.

There is one approved version of each article. When the policy changes, the article gets updated, the previous version becomes a record in the version history, and the AI can only call the current version. There’s an owner. There’s a review date. The article tells you when it was last looked at, and by whom. The AI cites the article it used, by name and ID, not a vague list of “sources.” When something changes, you update one place, and every AI assistant, every widget, every integration calls the same updated source from that point forward.

That’s a chain. That’s auditable. That’s what a regulator can read.

On the productivity vendors

Microsoft Copilot reads your SharePoint and Teams. Glean indexes your Slack and your wikis. Both are pitched as “your AI gets smarter the more content you have.” That’s a productivity story. It’s reasonable for productivity work. It’s the wrong shape for regulated work.

The vendors aren’t lying. The tools do exactly what they say. They infer answers from the content you have. The mismatch is between the answer they produce and the answer regulated work actually needs.

If you’re in a regulated business, the question isn’t “how clever is the AI.” It’s “can we defend the answer.”

The honest admission

A substrate doesn’t make compliance free. Somebody still has to write the canonical articles. Somebody still has to approve them. Somebody still has to review them on a sensible cycle. The work doesn’t go away. It moves into the right place.

The pitch isn’t “AI replaces compliance.” It’s “AI becomes defensible.” The compliance work that already has to happen for human staff also serves the AI. One curation effort, two audiences. Your customer service agents read approved content. Your AI assistant reads the same approved content. Same answers. Same trail.

Substrate doesn’t lower the bar. It just stops the AI from sneaking under it.

Where this is going

The pitch from the inferred-truth crowd is going to keep getting louder. The promise of AI that just figures things out, no curation needed, will keep being attractive because it sounds like no work.

In regulated industries, the bar for “no work” is the same as the bar for “no audit defence.” It’s a question of when, not if, somebody in the business will need to explain how an AI produced a particular customer-impacting answer.

The AI era doesn’t lower the compliance bar. It raises it. Every customer interaction is now potentially mediated by something that talks confidently and shows no working. The substrate is how you meet that bar.

If you’re in a regulated business and you’re evaluating AI right now, the question to ask isn’t whether the AI is smart enough. It’s whether the chain of evidence behind every answer is defensible. If it isn’t, smart doesn’t help you.

The KnowledgeScout Team