I first noticed something off in 2022. A tier-one bank’s AI reported zero issues across a full quarter of transaction monitoring. The risk committee reviewed the numbers, saw the clean run, and moved on. False positives were down. Processing time had dropped by sixty percent. By every measure they had, the model worked exactly as designed.
A mid-level analyst thought something felt wrong.
Not a hunch without reason. She had fifteen years in correspondent banking. Fifteen years watching money move through nested accounts, cross-border flows, paper entities that seemed solid one day and vanished the next. She read those rhythms like a cardiologist reads an ECG, not just the spikes, but the silences that shouldn’t be that quiet., –
The Situation
She flagged a counterparty. When I asked what had triggered her concern, she hesitated. The honest answer? She couldn’t fully explain it. The flows looked normal. The entity had paperwork. The model had processed and cleared it without a hitch. No single data point stood out.
What she had was a shape. A pattern she’d seen before, not identical, but close enough that fifteen years of experience made her notice. The counterparty was structuring: deliberately breaking transactions into pieces to stay below automated detection limits. The method wasn’t new, but the setup, the jurisdictions, the counterparty type, the timing, fell outside the model’s training data. It had never encountered this exact arrangement, so it said nothing.
Here’s what haunts me about that year. The model wasn’t broken. It did exactly what it was built to do, precisely. The risk committee wasn’t careless. They reviewed the outputs they were given. The governance process looked correct from every angle it could see. That’s the problem. The system had no visible failure mode for the people responsible for spotting failures. It had no red light. It only had the absence of one, and the organisation had, over time and without anyone saying it aloud, learned to treat that silence as safety., –
What Actually Failed
The first failure wasn’t the model. It was the way we thought about what the AI was doing.
Some AI risk governance treats model metrics as a stand-in for real-world coverage. False positives. Processing speed. Accuracy on test data. These numbers matter. They tell you something real. But they don’t tell you what the model has never seen. A model trained on past transactions will catch patterns it knows. It won’t notice when the world changes, when a new structuring trick emerges, when a rarely used jurisdiction becomes a conduit. It can’t warn you about its own blind spots. That’s not a flaw in design. It’s how these systems learn.
The second failure was subtler. When AI metrics look good for long stretches, organisations adjust their human oversight accordingly. Senior analysts spend less time on cleared transactions. Review processes thin out around the automated layer. That makes sense, you wouldn’t manually check every calculation a spreadsheet makes. But the analogy is wrong. A spreadsheet follows rules consistently. An AI model learns patterns from data, and the patterns it never saw are the ones it will never find. The oversight we’re cutting back is exactly the oversight we need to catch what the model misses.
The third failure is the one that stays with me. It’s what happened to the analyst’s instinct inside the organisation before that moment. She had flagged things before that didn’t turn into confirmed issues. That’s how real pattern recognition works, not every signal leads to a finding. In some places, repeated flags without outcomes become a professional liability. Analysts learn to adjust their instincts to match what the model approves. The pressure, unspoken but real, pushes toward alignment with the machine. When the machine says nothing is wrong, insisting something is feels risky. Organisations can quietly erode that confidence over time without meaning to., –
What This Means for Your Organisation
If you run AI-assisted transaction monitoring, or any AI-assisted risk function in a regulated setting, the question isn’t whether your model performs well. It’s whether your oversight is built around what the model cannot see, or whether it’s been quietly reshaped around what the model can process. Those are not the same structures.
The institutions getting this right aren’t choosing between AI capability and human judgment. They’re being precise about what each can actually do. AI handles volume and applies learned patterns at a scale no team can match. Experienced analysts spot anomalies outside those patterns, not because they’re better than the model, but because they’re different from it in the ways that count. The gap between the model’s world and reality isn’t something you close by improving the model. You close it by staffing it., –
The best AI risk control system I’ve seen wasn’t the one with the strongest model. It was the one that knew, without doubt, where the model ended and what had to happen next.
A system that understands its limits is still a system. One that doesn’t is a liability wearing impressive metrics.