Banner: When you cannot govern what you should. A Playbook from IQ4hire.
AI governance · Part 3

When you can’t govern what you should

Four questions tell you whether you can govern an AI system at all. If you cannot govern the inside, shrink what it is allowed to touch.

Part 3 of the series AI governance. Each part stands alone.

You can know exactly which system needs oversight and still find you cannot build the controls. Four questions tell you whether you can govern it. If you cannot govern the inside of the system, you shrink the box until obedience to a mistake is survivable.

Parts 1 and 2 gave you two questions. What part is doing the work? That prices the purchase. What decision does it touch, and how far does the damage travel? That tells you how hard to watch it. Together they let you walk into a vendor meeting and separate the system that needs a guardrail from the one that needs a leash.

Now the uncomfortable part. You can know which system needs the leash and discover there is no collar to attach it to. Governance has a feasibility problem, and most buyers find it only when the system is in production and the controls they assumed turn out not to be buildable. Here the architecture comes back, not as a measure of risk but as the hard boundary on what you can do about it.

The trap in one line: policy is a wish, and architecture is the boundary. You can write any oversight requirement into a contract. Whether you can enforce it depends on properties of the system that were decided long before you arrived, and on whether the vendor built the surfaces you need.

Two properties do most of the work

Reproducibility is whether the same input reliably produces the same output. A rule-based system is reproducible by construction. A learning system may or may not be, depending on how it was built and configured: temperature, retrieval, model versioning, and whether the weights are frozen or drifting underneath you.

Explainability is whether anyone can say why the system produced the output it did. Some reproducible systems are perfectly explainable, and some are reproducible black boxes. Some learning systems can surface a rationale, and many cannot.

These are two axes, and where a system lands decides which controls are on the menu. Reproducible and explainable: you can audit it like any other deterministic process. Reproducible but unexplainable: you can pin its behavior and test it exhaustively, even if you cannot narrate it. Explainable but not reproducible: you get a story, but cannot be sure of getting the same story twice. Neither: you are governing a weather system, and your only real controls are at the edges, which are what you let it touch and what you let it do.

That last quadrant is where most of the anxiety about AI lives, and a paradox bites there. The property that makes a learning system valuable is the same property that makes it hard to control. You bought it because it adapts, and adaptation is non-reproducibility in a nicer suit. You cannot ask a system to learn and also demand it never surprise you.

A correction to Part 1

Some readers took the wrong lesson from Part 1. I said much of what we call AI might be rules in a better suit, and a few heard permission: if it is only rules, I do not have to govern it. That is the opposite of what I meant.

"Just rules" does not shrink the blast radius. It shrinks the drama. The rule-based system sits in the reproducible-and-explainable corner, which makes it the most governable thing you own. The learning system everyone is nervous about sits in the weather-system corner. One you can govern completely and the other you can barely govern at all.

Now watch where attention goes. It runs to the weather system, because that one frightens people, and walks past the rule engine, because nobody is afraid of a lookup table. The system you could fully control ships with the brakes off and nobody watching, while the one you can barely control collects all the oversight in the building. The most expensive automated failures on record were never the systems anyone feared. They were boring, reproducible code doing exactly what it was told.

Even a perfectly reproducible, perfectly explainable system is not safe, because it can be obedient and still execute a catastrophe. Knight Capital, from Part 2, was obedient, and so was Robodebt. Obedience to a bad instruction or bad data is not control. The real question is not "can I make it behave?" It is "have I bounded what it is allowed to touch and do, so that obedience to a mistake stays survivable?"

Four questions before you trust it with a decision

Provenance. Can I reconstruct, after the fact, what this system saw and what it did? With no durable record of inputs and outputs, you cannot investigate, prove compliance or learn from the failure. You can only repeat it.

Reproducibility. Can I pin this system's behavior well enough to test it, or does it shift every time the vendor pushes a model update? If you cannot freeze it, you cannot certify it.

Explainability, to the standard the decision requires. Not "can a data scientist reverse-engineer it," but "can I give the affected person, the regulator or the court the explanation they are owed?" The blast radius from Part 2 sets the standard, not what the vendor finds convenient.

Bounded authority. What is this system permitted to touch, move, send or decide without a human, and can I make that boundary smaller? This is the one control that works in every quadrant, including the weather-system corner. When you cannot govern the inside of the box, you govern the size of the box.

Look at what those four are. Provenance is a logging and observability problem. Reproducibility is a model-configuration and version-control problem. Explainability is partly architecture and partly process. Bounded authority is identity, least privilege, segmentation and data boundaries: zero-trust thinking applied to a non-human actor. None of them is the AI. Every one is plumbing.

The market gets that backwards. The hard problem in AI governance was never the model. Controlling one consequential system pulls in identity, observability, configuration management, data boundaries and a compliance wrapper to tie them together, and almost no vendor sells all of it. The complexity is in the integration nobody owns. It is invisible until something goes wrong, and it is the work worth accounting for before you sign.

Where it lands

Part 1: figure out what you are paying for. Part 2: figure out how hard to watch it. Part 3: figure out whether you can govern it at all, and if you cannot govern the inside, shrink the box until obedience to a mistake is something you can survive. Bounded authority works in every case.

It matters most when AI stops talking and starts doing: calling tools, spending budget, acting on your behalf. An agent with broad access and a wrong instruction does not rebel. It obeys. Checking provenance, reproducibility, explainability and authority before you sign is part of every AI evaluation we run. See how we evaluate AI.

← All Insights