Explain It or Kill It: Why Agent Explainability Is Now a Board Level Issue in Financial Services

It is 2:14 in the morning. A payments reconciliation agent at a large bank starts pulling customer account records it has never touched before. Within seconds, a runtime control flags the behavior, the agent is quarantined, and a kill switch severs its access to every downstream tool.

The control worked. Now comes the hard part. By 8:00 AM the Chief Risk Officer, the CISO, internal audit and, eventually, an examiner will all ask the same question: why? Why did the agent do what it did, why was it stopped, and can you prove it?

For AI leaders in financial services, stopping a bad agent is table stakes. Explaining the stop, with evidence that holds up to a regulator, is the real test. That is what agent explainability means in 2026.

Regulators and Bank Leaders Are Asking for Explainability

The concern is no longer theoretical. Supervisors on both sides of the Atlantic, and the people who run the largest banks, are naming agent explainability as a gap when considering releasing autonomous fleets of agents to production environments.

FINRA put it in writing. Its 2026 Annual Regulatory Oversight Report lists auditability as a core AI agent risk:

"Multi-step reasoning or complex chains of agent actions may be difficult to reconstruct, complicating auditability."

The same report flags "autonomy and scope creep," where agents "may take actions that exceed the user's actual or intended scope or authority," and urges firms to track agent actions and decisions and to establish guardrails that constrain agent behavior.

The Bank of England is talking about kill switches. Speaking at the ECB Forum on Central Banking on June 30, 2026, Deputy Governor Sarah Breeden said:

"Our frameworks were not built to contemplate autonomous agents, and relying on a human in the loop for all agent actions is unlikely to be realistic."

She raised circuit breakers and kill switches as possible safeguards, and warned that agents whose "objectives drift from original goals" could amplify volatility in stress.

US bank model risk guidance left agents out. When the Federal Reserve, OCC and FDIC issued SR 26-2 on April 17, 2026, replacing SR 11-7, the guidance expressly omitted generative and agentic AI from its scope, calling them "novel and rapidly evolving" and promising a future request for information. That is not a pass. As DefenseStorm notes, banks must still govern agents under their existing risk programs, which means building the evidence themselves.

Bank leaders feel the same pressure. Former Goldman Sachs CEO Lloyd Blankfein told a16z in May 2026 that the danger of AI agents is "not because it's smarter than us... but because we don't have the ability to test whether it's right or not," adding that a single piece of software "could go out and do 70,000 transactions." At JPMorgan Chase, consumer bank CIO Gill Haus described the current posture plainly: "We don't let it loose in that sense. We have a human in the middle."

The survey data shows how wide the gap is:

And the clock is running in Europe. Under the EU AI Act omnibus agreement, high risk systems such as credit scoring must comply by December 2, 2027, bringing logging, transparency and human oversight obligations with them.

Explainability for Agents Is Three Questions, Not One

For a traditional model, explainability meant answering why a score came out the way it did. Agents plan, call tools, move data and take actions across systems. So explainability now has to answer three questions, each with evidence.

1. Why did the agent do what it did? What was the goal it was given, what inputs and context did it see, which tools did it call, in what order, and on whose authority?
2. Why was it quarantined? Which runtime control fired, what threshold or policy was crossed, and what exactly did the agent do that crossed it?
3. Why was the kill switch activated? Who or what made the call to halt the agent, at what moment, what was cut off, and what downstream actions were prevented as a result?

If any of those answers begins with "we think" or "the logs suggest," the institution does not have explainability. It has a story. Examiners, auditors and boards want a record.

Anatomy of an Investigation: Identity Plus Tracing

Two capabilities turn a 2:14 AM quarantine into a clean answer by 8:00 AM: agent identity registration and agent tracing. Identity tells you who the agent is and what it was allowed to do. Tracing tells you what it actually did. The gap between the two is the finding.

agent investigation flow · identity and trace meet at the runtime control

The runtime control sits between what the agent may do and what it did; every Intervention it triggers feeds the Evidence bundle reviewers read.

Step 1: Start from a registered identity

Every agent is registered before it ever runs, like a new employee getting a badge. The registration records its owner, business purpose, the models it may use, the tools and data it is entitled to, and the runtime controls that apply to it. In our example, the reconciliation agent was registered by the payments operations team with read access to transaction ledgers only. Customer account records were never in scope.

Without that registration, investigators cannot tell an agent acting outside its authority from one acting as designed. FINRA's concern about agents that "exceed the user's actual or intended scope or authority" is only answerable if the intended scope was written down first.

Step 2: Replay the trace

Agent tracing captures every step of the run as a linked chain: the prompt and context the agent received, each reasoning step, each tool call with its inputs and outputs, and each data access. This is exactly the "complex chain of agent actions" FINRA says is hard to reconstruct. With tracing, nothing needs reconstructing. The chain is already there.

Replaying the trace, investigators see the agent received a reconciliation task, hit a mismatch, and then called a customer lookup tool to "resolve the discrepancy." That tool call was the first step outside its registered entitlements.

Step 3: Explain the intervention

The trace shows the moment Agent Behavior Controls compared the tool call against the agent's registered identity and flagged it. The Intervention log records which policy fired, the quarantine that followed, and the kill switch that revoked the agent's credentials and tool access. It also records what was prevented: 1,400 queued lookups that never executed.

Step 4: Hand over the evidence bundle

The investigation closes with an Evidence bundle drawn from Immutable Audit Logs: the agent's registration, the full trace, the control decision, the intervention timeline and the remediation. That bundle is what goes to the CRO, internal audit and, if needed, the examiner. It answers all three questions with records, not recollection.

The root cause in our example turns out to be a newly added tool in a shared tool library that no one had scoped to the agent's role. The fix is a registration change and a Component Input/Output Control on that tool, not a rewrite of the agent.

Six Questions AI Leaders Should Ask Before the Next Agent Goes Live

If your team cannot answer yes to each of these today, the first quarantine will expose it.

1. Is every agent registered with an owner, a purpose, and explicit entitlements to models, tools and data?
2. Is every run traced end to end, including reasoning steps, tool calls and data access, and linked back to that registered identity?
3. Do runtime controls act in real time, comparing each action to the agent's entitlements rather than reviewing outputs after the fact?
4. Is every Intervention explainable, with the policy that fired, the moment it fired, and what was prevented?
5. Are the logs immutable, so the record an examiner sees is the record that was written at 2:14 AM?
6. Can you produce an Evidence bundle in minutes, not weeks, in a form your CRO, auditors and regulators can read?

Notice what is missing from that list: a human approving every step. As Breeden said, that is "unlikely to be realistic" at scale. Explainability is what lets institutions move past human in the middle without losing accountability.

How Prediction Guard Approaches It

Prediction Guard is a sovereign AI control plane built for exactly this moment. It gives regulated institutions operational control of their agents, running inside their own environment so sensitive data and audit records never leave their boundary.

• Agent identity registration so every agent has an owner, a scope and a policy before it runs
• Agent tracing across models, tools and data, in one control plane rather than stitched together from vendor logs
• Runtime controls, including Agent Behavior Controls and Component Input/Output Controls, that enforce policy at the moment of action
• Interventions that quarantine an agent or trigger a kill switch automatically, with the reason recorded
• Immutable Audit Logs and Evidence bundles that turn every intervention into an answer your regulator can verify

Stopping a rogue agent is the easy part. Explaining it is what earns the right to deploy the next one. Talk to our team to see an agent investigation end to end.

Test It in Your Own Environment

Sources

• FINRA's 2026 Regulatory Oversight Report: Continued Focus on Generative AI and Emerging Agent Based Risks, Debevoise & Plimpton
• Agentic AI may require regulatory reform, BoE's Breeden says, Reuters via WDEZ, June 30, 2026
• MRM Guidance SR 26-2 in the Era of AI, Charles River Associates, July 2026
• There's No Guidance for AI in Banking Yet, DefenseStorm
• Lloyd Blankfein just put his finger on why even Goldman Sachs is wary of AI agents, Fortune, May 13, 2026
• How Big Banks Plan To Make AI More Accessible And Explainable In 2026, Forbes
• Banking insights: 2026 AI Impact Survey Report, Grant Thornton
• Avalara Survey: Finance Leaders are Racing to Deploy AI Agents Before Governance is Ready, Avalara, July 21, 2026
• EU AI Act Omnibus Agreement: Postponed High Risk Deadlines and Other Key Changes, Gibson Dunn