In November 2021 Zillow shut down its home-buying business, Zillow Offers. It had written down about $304 million on houses bought for more than it could sell them for. The unit leaned heavily on automated price estimates, and when the housing market turned, the organization around those estimates did not pull back fast enough. That is the gap an AI operating model exists to close: the habits, roles, rules and feedback loops that keep an AI system useful and safe for as long as it runs, and the numbers that tell you when it has stopped being either.
Most AI failures happen after launch
AI systems learn patterns from the past and act in the present. The world keeps moving: behaviors shift, incentives change, and what worked last quarter quietly stops working. So "Did we ship AI?" is the wrong question. The useful one is "Can we keep it performing while things change?"
It helps to separate three layers. Model capability is what the algorithm can do. System behavior is what actually happens once people use it. Organizational control is how you detect issues, decide what to do and enforce boundaries. The operating model manages the space between those three, and most trouble lives there. The model itself is often fine. What fails is that nobody owns the moment its outputs start meaning something different than they did at launch.
Finance, security and quality assurance already work this way. Nobody "ships" a finance function and walks away; it runs on closes, audits and controls. An AI system that makes or shapes decisions deserves the same treatment, and the rest of this piece describes what that looks like in practice.
Write the decision down, then what must never happen
Most AI roadmaps are drawn like software delivery: define, build, launch. Operating an AI system starts earlier, with intent. AI adds clarity only when it supports a specific decision under uncertainty. "Improve efficiency" is not a decision. "Prioritize inspections so the riskiest sites are visited first" is.
A strong decision statement has four parts: an input boundary (which signals are allowed), an output boundary (what the system produces), an action mapping (what happens next) and a named owner who answers for outcomes. If you can’t fit that into one paragraph, you’re still at the hype stage.
Constraints come next: the things that must never happen. In lending, that might be no automated denials without human review. In healthcare operations, no prioritization that bypasses safety protocols. In hiring, no ranking that uses protected characteristics or their proxies; Reuters reported in 2018 that Amazon scrapped an experimental recruiting model after it learned to penalize résumés that mentioned women’s organizations. In public benefits, a clear appeal path for any adverse decision.
Constraints must be testable, because what you can’t test you can’t enforce. They also have to cover what the system says as well as what it decides. In Moffatt v. Air Canada (February 2024), British Columbia’s Civil Resolution Tribunal held the airline responsible for wrong refund information given by its website chatbot and ordered it to pay $812.02. The sum was small; the principle was not. The company owns its system’s answers, so "the chatbot may not state policy it can’t cite" is a constraint worth writing down.
Logs are the system’s memory
Without logs, every diagnosis is a guess. At a minimum, record six things for each decision:
- the input snapshot, meaning what the system saw at decision time;
- the output it recommended or predicted, with its confidence or uncertainty signal;
- the human action: accepted, edited or overridden, with a coded reason;
- the downstream outcome, recorded when it becomes known;
- context tags such as segment, region, channel and case type;
- the model and policy version that produced the output.
That is the difference between "we think it got worse" and "we know why it drifted." The context tags earn their keep in a specific way: an aggregate metric can look flat while one segment falls apart. Say 80% of cases come from region A and 20% from region B, and accuracy is 90% in both, so 90% overall. If region B drops to 75% while A edges up to 93.75%, the blended figure is still exactly 90%. The dashboard shows nothing, and one in four region-B decisions is now wrong.
Version tags matter for a related reason. A common production failure is training-serving skew, where the data a model sees in production differs from what it was trained on. You can only spot that skew if you can line up predictions, inputs and model versions after the fact.
Four roles, or a blame carousel
Most AI mistakes turn out to be role mistakes. A stable operating model assigns four roles. In a smaller organization one person can cover several, but each role has to exist.
The System Owner is accountable for outcomes long after delivery. The Model Steward watches drift, quality and retraining triggers. The Risk/Governance Lead audits, approves changes and handles incidents. Frontline Operators use the system daily and supply ground-truth feedback. If these roles aren’t explicit, AI "ownership" becomes a blame carousel.
The operator role is the one most often designed badly. Lisanne Bainbridge’s 1983 paper "Ironies of Automation" pointed out that the more reliable automation becomes, the less practice humans get at the moments when they must step in. Parasuraman and Manzey’s 2010 review in Human Factors documented automation bias: people over-trust a system’s suggestions and miss its errors. Europe’s AI Act now writes this into law for high-risk systems: its Article 14(4)(b) requires that people overseeing them remain aware of the tendency to over-rely on the output.
In operating terms, that means you measure whether humans actually disagree. An operator team that accepts 99.5% of suggestions over three months is either supervising a near-perfect system or rubber-stamping. Sampling audits tell you which. Reward operators for the issues they catch as well as for speed, or the override rate will drift toward zero for the wrong reason.
Change is a controlled intervention
When you change an AI system, you change how people and the system work together. So roll out updates in stages (shadow → assist → partial automation → broaden). In shadow mode the model runs but nobody acts on it. In assist mode a person still makes the call. Before each step, agree what success looks like and what counts as harm. Have a way to roll back, and tell users what is changing. Skip these steps and problems stay hidden until the damage is done.
Governance is where trust is won or lost, because it deals with exceptions. People need to know who to call and when. Override reasons should fit a few simple categories. Someone should review a random sample of cases as routine. Incidents need a severity scale and a playbook. In practice, governance is whatever happens on a Tuesday when something goes wrong.
Learning makes improvement routine. Put the checks on a calendar and keep them boring. Each week, look for drift and odd spikes. Each month, review the quality of decisions, not just raw accuracy. Each quarter, test how the system holds up when conditions change sharply. If improvement depends on a few experts happening to notice problems, it won’t scale.
Five health signals, each with a threshold
Accuracy is rarely the number that decides things in production. Watch five signals instead. How often people override the system, and whether that rate is stable. Where people keep rejecting its outputs. Whether results hold in every segment, not only on average. How long it takes to notice a problem and to fix it. And how many appeals or complaints come in on high-stakes decisions. A system can be accurate on average and still be unsafe to run.
Alerts need thresholds, or they become noise. Suppose operators review 400 decisions a week and override 12% of them. With that many cases, the rate moves by about 1.6 points from week to week by chance alone. So a reading above 15% is unusual. A sensible rule: look into it when the override rate stays above 15% for two weeks in a row. One bad week is often noise. Two rarely are. A drop matters as much as a rise, because an override rate falling toward zero can mean operators have stopped checking.
Each signal also needs an owner on the analytics side, and that owner is rarely the team that built the model. The builders own test metrics and retraining. Someone closer to the business has to own the production dashboard. That person defines what counts as an override and when an outcome is final. They also match each prediction to its outcome, which may arrive weeks later, and they set the alert thresholds. When nobody does that matching, the outcome column stays empty. The dashboard slowly turns into an activity report.
How fast you notice a problem and fix it sets the size of the damage. If a drifted model affects 2% of 5,000 daily decisions, that is 100 bad decisions a day. Detect in two days and roll back in one, and roughly 300 decisions need review. Detect in three weeks and fix in another two, and the count reaches 3,500. That is why a system you can pull back in a day tolerates far more drift than one that takes a quarter to bring back into bounds.
Four deployments, four sets of design choices
The examples below are composites, but each shows how the same signals turn into design choices.
City permitting that cuts backlog without creating bias. A municipality uses AI to triage building permit applications. The goal is to route applications to the right reviewers and flag likely missing documents early. The choices that matter: applicants see what triggered a "needs more info" status; reviewers override the triage with a coded reason; monthly audits check for unequal delays across neighborhoods; and the system drops to assist mode whenever policy changes. Service quality scales without silently reshaping who gets access.
Field maintenance that avoids brittle automation. An industrial company predicts failure risk to reduce unplanned downtime. The model predicts; the operating model decides what happens next. Thresholds are tied to the cost of failure, so a $2 million turbine and a $5,000 pump trigger different actions. High-impact work orders get a manual check. The feedback loop records maintenance actions and later failures, treating "avoided failures" as an estimate that needs a comparison method. Retraining is triggered by new equipment and usage patterns. Schedules stay tied to what the equipment actually does.
Insurance claims triage that protects edge cases. AI routes simple claims to fast handling and complex ones to specialists. It never auto-rejects; it only routes and suggests documentation. A protected "customer harm" metric tracks re-open rate, complaints and escalation time. A red-team review focuses on rare, high-impact corner cases, and customer letters never hide behind "the algorithm decided." Most claimants get faster service, and the hard cases still reach a person.
Internal learning platforms that avoid recommendation traps. A large organization recommends training paths, and the risk is reinforcing inequality: high performers get more opportunities, everyone else gets stale content. The fixes are diversity constraints that favor breadth, outcome tracking beyond clicks (role mobility, performance), periodic resets that surface new domains, and human mentoring as a second channel for high-stakes roles. Without them, the recommender keeps promoting whoever it promoted last quarter.
Small, reversible features need a lighter version
Small teams push back here, with reason: four roles, a change calendar and sampling audits are heavy machinery for a model that suggests email subject lines. The cost is real. Weekly reviews and audits can easily take a day of someone’s time per week, and for a low-stakes feature that day buys very little.
They are right when three conditions are true together: a wrong output is cheap and reversible, a human sees every output before it matters, and volume is low enough that problems surface on their own. A drafting assistant for internal documents usually passes all three.
So tier the operating model by stakes. If outputs affect money, health, access to services or legal standing, run the full model from day one. If they are reversible suggestions reviewed by a person, a one-page decision charter, version logging and a quarterly review are enough. Move a system up a tier the moment it starts acting without review.
A first month that needs no reorg
Write a decision charter. One page naming the decision supported, allowed and forbidden inputs, the intended action path, constraints and fallback modes, and the owner with escalation routes. It forces clarity before code is written.
Build the minimum audit trail early. Even if the first version is a manual export, get the logging right. Retrofitting auditability later is expensive and politically painful.
Set a change calendar. Put model reviews and policy reviews on a schedule. If you wait until something breaks, your "operating model" is really an incident-response habit.
Train users on when not to use the system. Trust improves when people know the boundaries. "Use it for X, don’t use it for Y" is more credible than pretending the system is always right, and the AI Act’s Article 4 asks providers and deployers to take measures to support AI literacy among their staff, the softer wording set by the Digital Omnibus, Regulation (EU) 2026/1744, in force since 27 July 2026.
Earn automation with evidence. Strategy defines where AI should create value; the difference between an AI strategy and an AI operating model is that the second keeps it reliable, safe and improving once it is in production. Automate only when outcomes are measurable, harms are bounded, uncertainty is visible and rollback is practical. Otherwise start in assist mode, and move up only after, say, three months of override rates and outcome data inside agreed bounds.
The difference between running a system and merely deploying it rarely shows in a normal week. It shows when something moves, say when override rates in one region double over seven days. In an organization that runs the system, someone sees the signal in a specific log, someone owns the call to drop back to assist mode, rollback takes a known number of hours, and operators and affected customers hear about it from a named person. In one that only deployed it, the same questions get answered with a vendor’s name, a dashboard nobody checks or "the data team." If nobody can be named today as the person who makes that rollback call, the system is deployed, not operated.