Banner: Govern the decision, not the architecture. A Playbook from IQ4hire.
AI governance · Part 2

Govern the decision, not the architecture

How hard to watch an AI system depends on the decision it touches and how far the damage travels, not on whether it learns or follows rules.

Part 2 of the series AI governance. Each part stands alone.

How hard you have to watch an AI system depends on the decision it touches and how far the damage travels when it is wrong. It does not depend on whether the system learns or follows rules. Govern the decision, not the architecture.

Part 1 argued that the honest way to price an AI purchase is to ask which part does the work: learning from data, or a rule a person wrote. Answer that and you know what you bought. Now I have to take something away from you, because the same distinction leads straight into the most expensive mistake in AI governance.

The trap is natural. Having learned to tell learning from rules, you watch the learning systems closely because they are unpredictable, and relax about the rule-based ones because you can read every line. That feels rigorous. It is backwards often enough to ruin you.

The hood sets your price. It does not set your risk.

What is under the hood, learning or rules, determines what you paid and which controls are available to you. Part 3 covers that. How hard you need to watch a system has almost nothing to do with its architecture. It depends on the decision the system touches and how far the damage travels when it is wrong.

Two cases make this concrete, and neither is the AI you are worried about.

On August 1, 2012, Knight Capital deployed new trading software. A piece of dormant code reactivated, and the system began sending erroneous orders into the market. There was no model and no learning, only deterministic software doing what it was told. According to the SEC, the system sent millions of orders over about 45 minutes. Knight reported a realized loss of about $440 million, and the firm nearly went under. Every line was readable. None of that mattered, because the decision it controlled, sending orders to a live market, had an enormous blast radius and almost no time for a human to step in.

The second case is Robodebt, the Australian government's automated debt-recovery scheme. It used a crude averaging method to calculate that people owed money, then issued the notices automatically. Again the logic was rules, not learning, and readable. A Royal Commission reported in 2023 that the income averaging at its core was unlawful, and the scheme had by then raised hundreds of thousands of wrongful debts. A human reviewing each case would have caught the harm. It came from letting a simple rule make a high-stakes decision about individuals at scale.

A naive risk framework would have waved both systems through: transparent, deterministic, explainable. Both were catastrophes. Predictability is not safety. A system you can fully predict can be fully, predictably wrong, and if it is wired to a decision with a wide blast radius, predictability means you will be wrong faster and at scale.

Ask the decision question

For any system you are about to deploy, ask: what decision does this make or shape, and who absorbs the damage when it is wrong?

A learning model that picks which blog headline to show gets light oversight, because the blast radius is trivial, even though it is the "scary" learning system. A rule-based engine that flags people for benefit clawbacks, denies claims or routes money gets heavy oversight, because the blast radius is people and dollars, even though it is the "safe" deterministic one. The architecture told you the price. The decision tells you the watch level. They are different axes, and mixing them up is how an organization ends up scrutinizing a recommendation engine while a spreadsheet macro quietly decides who gets a loan.

This is also why "we don't really use AI here" is not the comfort people think it is. The most dangerous automated decisions in most enterprises were made by rules nobody calls AI: eligibility logic, pricing tables, routing scripts, thresholds buried in a system of record fifteen years ago. They have blast radius and no oversight. Nobody filed them under "AI," so nobody governs them.

The question to put to any vendor

Put one question to any vendor, in any category, whatever is under the hood: show me the decisions this system makes without a human in the loop, and tell me what happens downstream when each one is wrong.

If they answer cleanly, you can size the oversight. If they retreat to "it is highly accurate" or "the model is state of the art," they have answered the price question and dodged the risk question, and you have learned which one they understand.

A note for the demo room

In the pitch, the vendor will sell you the engine: the neural networks, the training data, how the model learns. Take it seriously, because it tells you what you are paying for. But how powerful a model is says nothing about how much damage it can do. The engine is the price tag, not the risk.

A powerful model that only flags something for a human to check is a small risk. A simple one wired to lock an account, block traffic or move money by itself is a large one. The brochure will not draw that line. You have to.

Where it lands

The architecture sets the price. The decision, and how far its damage travels, sets how hard you watch. Let the vendor keep the engine and govern the decision. Part 3 asks whether you can build the oversight at all.

Sizing oversight by decision is part of every AI evaluation we run. See how we evaluate AI.

Sources

← All Insights