The Core Point
In the first three articles in this series, I considered why AI governance has become a board-level issue, why risk should be assessed by consequence rather than technology and how security can support commercial trust and growth.
This fourth instalment addresses the next operational question:
What happens when an AI-enabled process fails?
Assuming that AI models will remain stable, available and predictable is not a credible resilience strategy. Relying solely on the provider, future model improvements or manual intervention is equally insufficient.
Operational resilience requires organisations to understand which important outcomes depend on AI, how those dependencies could fail and how those outcomes will continue to be delivered.
The objective is not to prevent every disruption.
It is to plan for failure, detect when something has gone wrong, respond effectively and minimise the risk of material or poorly controlled harm.
Start with the Outcome, Not the Uptime
AI resilience is often discussed through technical measures:
model availability
API response times
infrastructure redundancy
recovery targets
supplier service levels
Those measures matter, but they do not tell the whole story.
A model can remain available while producing unreliable outputs.
An API can meet its service level while returning incomplete or inappropriate recommendations.
A platform can remain online while the data feeding it is stale, corrupted or incomplete.
The system may be available.
The outcome may still be wrong.
Start with the business service or operational outcome:
What must continue to be delivered?
Who could be affected if it is disrupted?
How much disruption can be tolerated?
Where has AI become part of the dependency chain?
What happens if the AI capability becomes unavailable or untrustworthy?
This outcome-led approach aligns well with both EU and UK regulatory expectations, even where the legal frameworks address AI and resilience from different perspectives.
Different Regulations, Similar Expectations
DORA, the EU AI Act and the UK regulatory framework do not impose a single, uniform set of AI resilience requirements.
They do, however, converge around several practical expectations:
clear accountability
risk and dependency identification
proportionate controls
testing
monitoring and record-keeping
third-party management
incident response
evidence that controls operate effectively
The terminology and scope vary. The underlying governance disciplines are increasingly consistent.
DORA: Bring AI into the ICT Risk Environment
DORA has applied since 17 January 2025 to EU financial entities within its scope. It covers ICT risk management, ICT-related incident management and reporting, digital operational resilience testing and ICT third-party risk.
For an AI-enabled service, the practical questions may include:
Does the capability support a critical or important function?
Which models, APIs, gateways, data services and providers does it depend on?
Has concentration risk been assessed?
Could an AI-related failure constitute an ICT-related incident?
Have continuity, recovery and exit arrangements been tested?
Can the organisation reconstruct what happened?
Where an AI capability relies on third-party ICT services, the contractual arrangements fall within DORA’s third-party risk framework and register-of-information requirements.
This does not mean creating a separate resilience regime for AI.
The practical implication is that relevant AI dependencies should be incorporated into the organisation’s existing ICT risk, incident, testing and third-party control environment.
For example, an insurer using an external model to prioritise claims should understand not only the model’s performance, but also:
the supporting provider and data dependencies
whether the service supports a critical or important function
the effect of provider failure or output degradation
the ability to continue claims processing
the contractual and technical feasibility of switching or exiting
A named alternative provider is not necessarily a workable fallback. Data portability, model behaviour, integration effort and control equivalence may all affect whether the alternative can be used within the required timeframe.
The EU AI Act: Resilience Extends Beyond Availability
The EU AI Act focuses more directly on the AI system and its lifecycle.
For high-risk AI, its control framework includes risk management, data governance, record-keeping, transparency, human oversight, accuracy, robustness, cybersecurity and lifecycle monitoring. These disciplines are directly relevant to whether failures can be detected, investigated and corrected.
An AI system can fail without becoming unavailable.
For example:
outputs may change after a model update
a system may begin producing unreliable or discriminatory results
human reviewers may stop challenging its recommendations
an agent may exceed its intended authority
monitoring may fail to detect a material change in behaviour
The AI Omnibus entered into force on 27 July 2026. Following the revised timetable, the rules for Annex III high-risk AI systems apply from 2 December 2027, while those for high-risk AI embedded in products covered by Annex I product-safety legislation apply from 2 August 2028.
Separately, the transparency obligations in Article 50 apply from 2 August 2026 to certain systems, including specified interactive and generative AI uses.
Those application dates do not remove the need for preparatory work.
Logging, monitoring, human oversight, incident processes and the ability to reconstruct decisions are difficult to add retrospectively once a system has become embedded in an operational process.
The UK Position: Existing Rules Still Apply
The UK financial-services approach remains largely technology-neutral and outcomes-focused. The Government’s July 2026 Financial Services AI Adoption Plan records industry support for applying the existing regulatory framework rather than introducing a separate general AI regime for the sector.
Depending on the organisation and use case, relevant requirements may include:
operational resilience
Consumer Duty
model risk management
data protection
equality law
third-party risk
senior-management accountability
These frameworks do not apply identically to every organisation. Scope matters.
Operational Resilience
Under the FCA’s operational-resilience framework, firms within scope must identify important business services and set impact tolerances. By 31 March 2025, they were required to have completed the mapping and testing needed to remain within those tolerances and to have made the necessary investments to do so consistently.
Applied to AI, this may involve testing whether:
a claims service can continue if AI triage fails
customer support can operate safely without generated responses
fraud teams can manage increased manual volumes
important decisions can still be made if a model or data source becomes unavailable
a critical workflow can continue following a provider-side change
fallback arrangements remain credible at production volumes
If an AI service processes 20,000 cases and the previous manual process handled 200, stating that the team can revert to manual processing is not a sufficient resilience plan.
Consumer Duty
For firms within scope, the Consumer Duty requires them to act to deliver good outcomes for retail customers, avoid foreseeable harm and support consumer understanding and access to effective support.
An AI-enabled customer journey may therefore create an issue even where the system remains technically available.
A chatbot that gives inaccurate information about eligibility, exclusions or product terms could affect consumer understanding and support. The firm would need to consider:
how the issue is detected
which customers received inaccurate information
whether earlier interactions can be reconstructed
how the information will be corrected
whether the automated service should be restricted
whether customers should be transferred to human support
The relevant measure is not simply whether the chatbot was online. It is whether the customer outcome remained acceptable.
Model Risk Management
For specified UK-incorporated banks, building societies and PRA-designated investment firms with internal-model approval, PRA Supervisory Statement SS1/23 establishes principles covering model identification and classification, governance, development and use, independent validation and model-risk mitigants. It also recognises the risks arising from AI and machine-learning techniques where they fall within the firm’s model environment.
This reinforces a broader point: model resilience is not limited to technical recovery.
It includes understanding whether the model remains fit for purpose, whether changes are controlled and whether its outputs are being used appropriately.
Data Protection and Automated Decisions
The Data (Use and Access) Act 2025 widened the circumstances in which significant decisions may be based solely on automated processing of non-special-category personal data.
Safeguards remain mandatory. These include providing information about the decision and enabling the individual to make representations, obtain human intervention and challenge the outcome. Stronger restrictions remain where special-category data is involved.
From a resilience perspective, this means the organisation may need to:
identify which decisions were affected
explain how the decision was reached
preserve the relevant records
provide effective human reconsideration
correct or reverse an inappropriate outcome
Equality Law
The Equality Act 2010 remains relevant where an AI-enabled process results in unlawful direct or indirect discrimination.
For example, if a recruitment model begins ranking candidates differently after an update, the issue is not merely whether the system is available.
The organisation should be able to establish:
whether the change was detected
whether earlier decisions can be reconstructed
whether particular groups were disadvantaged
whether human oversight was effective
whether recruitment can continue safely while the issue is investigated
AI Failure Is Not Just an Outage
A useful resilience assessment should consider several types of failure.
Availability failure: The model, provider, gateway or supporting service cannot be accessed.
Output failure: The system produces plausible but materially incorrect information.
Data failure: Inputs become incomplete, stale, corrupted or inappropriate.
Behavioural change: A model, prompt, configuration or provider update changes expected outputs.
Manipulation: Prompt injection or malicious content alters the system’s behaviour.
Loss of control: An agent accesses an inappropriate system or performs an unintended action.
Supplier failure: A provider experiences an outage, control failure, legal restriction or material service change.
Control failure: Monitoring, logging or human oversight exists in design but does not operate effectively.
Each failure mode may require a different response.
A second model provider may help with availability.
It will not necessarily address poor data, discriminatory outputs, compromised instructions or ineffective oversight.
Plan for Graceful Degradation
A human in the loop is not automatically a credible fallback.
Manual intervention works only where people retain the skills, information, capacity and authority needed to operate without the AI capability.
Graceful degradation may instead involve:
prioritising the most important cases
moving from automated action to recommendation only
introducing additional approval
reverting to a simpler rules-based process
using a tested alternative provider
restricting system permissions
suspending a non-essential feature
communicating clearly with affected customers
The fallback does not need to reproduce the normal service in full.
It needs to protect the important outcome and keep disruption within the organisation’s tolerance.
Test the Failure, Not Only the Demonstration
AI testing often establishes that a system performs under expected conditions.
Resilience testing should examine what happens when those conditions no longer apply.
Useful scenarios include:
the primary model provider becoming unavailable
a model update changing expected behaviour
source data becoming incomplete or corrupted
a material increase in incorrect outputs
manipulated content reaching an AI agent
monitoring or logging becoming unavailable
staff having to operate without the AI capability
several connected suppliers failing at the same time
Testing should establish whether:
ownership is clear
escalation routes work
the issue can be detected
affected decisions can be identified
fallback arrangements are credible
evidence is preserved
recovery occurs within tolerance
DORA testing, EU AI Act lifecycle controls and UK operational-resilience exercises should not become disconnected activities.
A well-designed scenario may provide useful evidence against several obligations.
Four Practical Examples
AI-Assisted Recruitment
A recruitment model begins ranking candidates differently after an update.
The organisation needs to detect the change, reconstruct affected decisions, evaluate potential discrimination and determine whether recruitment can continue safely.
AI-Enabled Customer Support
A chatbot remains available but starts giving inaccurate information about eligibility or product terms.
The organisation needs to identify affected customers, correct the information and decide whether the service should be restricted or transferred to human support.
AI-Supported Financial Operations
A model used to prioritise claims or fraud alerts becomes unavailable during peak demand.
The organisation needs tested fallback arrangements that account for the real volume of work, not an assumption that the previous manual process can absorb it.
Agentic Processing
An AI agent can update records, trigger payments or send communications.
The assessment must cover more than model accuracy. It should include:
system permissions
action and transaction limits
segregation of duties
logging and monitoring
approval requirements
the ability to stop or reverse actions
What the Board Needs to See
Boards do not need every model-performance statistic.
They need visibility of material dependency and preparedness.
Useful reporting might include:
important services materially dependent on AI
high-risk uses without tested fallback arrangements
concentration in critical providers
incidents and significant near misses
unresolved resilience gaps
use cases operating outside tolerance
unclear executive ownership
The board-level questions are straightforward:
Where could AI failure cause material or intolerable harm?
Do we understand the full dependency chain?
Have credible failure scenarios been tested?
Can important outcomes continue without the AI capability?
Who is accountable for closing the gaps?
The Practical CISO Agenda
The CISO should not become the default owner of every AI-enabled service or business process.
Accountability for the outcome should remain with the relevant executive owner.
The CISO’s role is to connect the disciplines and ensure that technical failure is translated into business risk.
That includes helping the organisation:
identify material AI dependencies
align AI controls with existing obligations
challenge weak fallback assumptions
test supplier and concentration risk
integrate AI incidents into established response processes
produce evidence for executives, boards, customers and regulators
The role is part translator, part challenger and part enabler.
The CISO should be able to explain why a capable AI system may still create an unacceptable operational dependency.
Equally, the CISO should avoid treating every AI use case as critical infrastructure.
That judgement is where practical governance adds value.
Resilience Is Where Governance Is Tested
AI governance is often judged by the quality of the policy, assessment or approval process.
The practical test comes when the system fails, changes behaviour or produces an outcome the organisation did not expect.
Can the problem be detected?
Can the organisation contain it?
Can important outcomes continue?
Can affected decisions be identified?
Can the organisation explain what happened?
Can it recover within tolerance?
The objective is not to make AI failure impossible. It is to plan for it, detect it, respond to it and minimise harm.
My AI GRC journey may look quite different from yours, and I would welcome your perspective.
What has worked, what has not and what have you learned along the way?
Please share your successes, misses and lessons in the comments.
Part 5 of this series will look at why policies fail and how practical standards, clear ownership and measurable controls turn governance intent into day-to-day behaviour.
Personal commentary. Views are the author’s own.