Governing Agentic Systems in Practice
The Core Point
In the first six articles in this series, I examined AI governance as a board-level issue, consequence-led risk tiering, security as a commercial enabler, operational resilience, the operating model beneath policy and, most recently, how organisations can assure themselves that governance is actually working. All the em-dashes are mine.
I had originally expected the series to end with the CISO’s role.
It has expanded again.
The reason is agentic AI.
Moving from AI that recommends or generates content to AI that can take actions materially changes the governance problem. Once a system can call tools, update records, send communications, initiate transactions or interact with other systems, governance has to address not only what the model says, but what it is permitted to do.
I’ve been exploring how we can put sufficient guardrails around agentic activity without neutering its transformative potential — allowing low-risk activity to flow, moderating the risky edge cases, and retaining enough evidence to understand and prove what happened.
That balance matters.
Too little control creates obvious risk.
Too much control can reduce an agent to an expensive suggestion engine.
The objective is therefore not to remove autonomy.
It is to make autonomy deliberate, bounded, observable and proportionate to the potential consequence.
Recommendation, Decision and Action Are Different Things
One useful starting point is to distinguish three levels of AI involvement.
Recommendation
The AI proposes something, but a person or deterministic process decides what happens next.
Examples might include:
suggesting a response to a customer
identifying potentially fraudulent activity
recommending a supplier risk rating
proposing remediation actions
drafting a contract clause
The principal governance questions concern output quality, data use and effective review.
Decision
The AI selects an outcome within defined parameters.
For example:
determining which cases receive enhanced review
selecting a routing path
prioritising remediation
deciding whether a transaction meets predefined criteria
The governance requirement is stronger because the model is now influencing what actually happens.
Action
The AI can execute.
It may:
update a record
send a communication
call another application
create or disable an account
modify a configuration
trigger a payment
purchase a service
initiate another automated workflow
At this point, inaccurate output is no longer the only concern.
Authority itself becomes a control surface.
The question changes from:
Is the model producing a reasonable answer?
to:
What is this system authorised to do when it believes that answer is correct?
Autonomy Is Not Binary
The debate around agentic AI can become unnecessarily binary.
At one extreme:
Let the agent operate autonomously and trust the model.
At the other:
Never allow AI to make or execute a decision.
Neither is particularly useful as a general governance principle.
A more practical approach is bounded autonomy.
An agent might be permitted to:
perform low-value reversible actions automatically
recommend higher-impact actions
execute actions below defined thresholds
request approval where consequence increases
stop when information is incomplete
escalate unusual or conflicting conditions
operate only against approved systems and data
lose permissions when defined conditions are breached
The amount of autonomy should follow the potential consequence.
A £20 operational purchase is not the same as a £2 million transfer.
Rescheduling an internal meeting is not the same as terminating an employee account.
Drafting a customer response is not the same as sending legally significant advice without review.
The governance model should recognise those differences.
Guardrails Should Constrain Authority, Not Just Output
Many AI controls concentrate on the model’s response:
prompt filtering
content moderation
accuracy checks
retrieval controls
hallucination detection
These are useful.
But once an agent can act, control also needs to sit around the authority being exercised.
That may include:
which tools the agent can call
which systems it can access
what information it can retrieve
which records it can modify
maximum transaction values
rate limits
permitted counterparties
geographical restrictions
approved action types
time-based restrictions
segregation of duties
whether actions are reversible
which activities require human approval
The agent should not simply inherit the permissions of the person who deployed it.
Nor should it receive broad access because that makes integration easier.
The authority granted to the agent should reflect what it genuinely needs to perform its intended role.
Deterministic Controls Still Matter
Probabilistic systems and deterministic controls are not opposites.
They can be complementary.
An agent may use probabilistic reasoning to decide what it wants to do while deterministic controls govern whether that action is permitted.
For example:
The agent determines that a supplier payment should be released.
The surrounding control environment might then enforce:
payment below £5,000
approved supplier only
invoice matched
no recent bank-account change
appropriate purchase order exists
no sanctions or fraud alert
action within normal operating hours
If those conditions are met, execution may proceed.
If they are not, the transaction can be routed for review.
This is where deterministic controls can provide considerable value.
But deterministic does not mean infallible.
The rule may be wrong.
The threshold may be inappropriate.
The data feeding it may be inaccurate.
The organisation still needs:
ownership
testing
change control
monitoring
exception management
assurance
The control layer itself becomes part of the governed system.
Moderation Rather Than Blanket Approval
One of the most interesting challenges is deciding where intervention should occur.
If every agent action requires human approval, the agent may deliver little meaningful automation.
If nothing requires approval, the organisation may be delegating more authority than it understands.
A useful middle ground is a moderation layer.
The objective is to allow expected, low-consequence behaviour to proceed while identifying activity that needs additional scrutiny.
That could be based on:
action value
confidence
data sensitivity
system privilege
unusual behaviour
deviation from normal patterns
external impact
irreversibility
customer consequence
regulatory significance
combination of several individually modest risk factors
For example, an agent might routinely:
schedule meetings
retrieve approved information
classify documents
create draft records
without intervention.
But additional approval might be required if it attempts to:
send an external communication
change payment details
access a restricted dataset
delete information
modify privileged access
initiate a high-value transaction
The important point is that the moderation mechanism should be risk-sensitive, not simply a universal human checkpoint.
Human Intervention Should Be Designed, Not Added
“Put a human in the loop” remains one of the most common responses to AI risk.
As discussed earlier in this series, that is not automatically an effective control.
For agentic systems, the problem becomes even more pronounced.
If an agent performs hundreds or thousands of actions, requiring a person to approve each one may lead to:
delays
automatic acceptance
alert fatigue
workarounds
loss of the productivity benefit
Human intervention should therefore be placed where judgement genuinely adds value.
That may include:
unusual transactions
high-impact irreversible actions
conflicting evidence
low-confidence decisions
customer complaints
policy exceptions
privileged activity
actions outside normal operating patterns
The human reviewer also needs enough context to make the decision.
An approval screen saying:
Agent recommends proceeding. Approve?
is not meaningful oversight.
The reviewer may need to see:
what the agent is trying to do
why it reached that conclusion
which data it relied upon
which control triggered the review
the potential consequence
relevant alternatives
Human oversight should improve the decision rather than merely provide a name against it.
Separate Planning from Execution
Another useful architectural pattern is separating the agent’s ability to plan from its ability to execute.
An agent may be permitted to determine:
the objective
required steps
relevant systems
information needed
proposed actions
without being able to perform every action directly.
Execution can then occur through a controlled tool layer.
This creates an opportunity to enforce:
identity
permissions
validation
transaction limits
schema checks
logging
approval requirements
policy rules
The model can retain useful autonomy while the organisation retains control over what reaches production systems.
For higher-consequence activity, this separation can be particularly valuable.
Identity Matters More When AI Can Act
An agent taking actions should have an attributable identity.
It should be possible to determine:
which agent acted
on whose behalf
under which authority
which credentials were used
which tools were called
what permissions existed at the time
what data was accessed
which action resulted
Agents should not operate through poorly controlled shared credentials.
Nor should they automatically inherit unrestricted user privileges.
Workload identities, scoped permissions and strong auditability become important controls.
For higher-risk use cases, organisations may also need to consider whether an agent should be allowed to delegate authority to another agent or invoke additional tools dynamically.
An agent creating more agents or acquiring additional capability can materially change the original risk assessment.
Reversibility Changes the Risk
Not every automated action needs the same level of control.
Whether an action can be reversed is an important consideration.
An agent that:
Moves an internal document into the wrong folder
creates a different exposure from one that:
Permanently deletes regulated records.
Similarly:
Drafting a payment instruction
is materially different from:
Transmitting the payment.
The governance model should consider:
reversibility
recovery time
financial consequence
affected parties
ability to identify all downstream effects
Where actions are easily reversible, the organisation may reasonably permit more autonomy.
Where actions are irreversible or difficult to unwind, stronger preventative controls are likely to be appropriate.
Build a Practical Stop Mechanism
Agentic systems need a practical way to be restricted.
That may include:
revoking credentials
disabling specific tools
reducing permissions
stopping the agent
switching to recommendation-only mode
restricting transaction limits
blocking external communications
isolating the workflow
The ability to stop an agent should be tested.
A theoretical kill switch that nobody knows how to use or that depends on the same compromised workflow provides little confidence.
The organisation should know:
who can invoke it
what happens when it is invoked
which actions continue
whether in-flight transactions complete
how normal operation is restored
how evidence is preserved
This is operational resilience applied to autonomous activity.
Observability Becomes a Governance Requirement
Traditional application logs may not tell the whole story of an agentic workflow.
For a material action, the organisation may need to reconstruct:
what objective the agent was given
what information it retrieved
what tools it invoked
which intermediate decisions it made
which controls were applied
what action was requested
whether moderation occurred
who approved or overrode it
what finally happened
This does not necessarily mean retaining every token or internal model state.
That may be impractical, unnecessary or inappropriate.
It means retaining enough evidence to explain the material activity and demonstrate that the required controls operated.
The level of evidence should be proportionate to the potential consequence.
Evidence Should Prove More Than Activity
Part 6 focused on assurance: how do we know governance is working?
Agentic systems make that question even more important.
A log showing that:
Agent A called API B at 14:03
proves that an action occurred.
It does not establish:
whether the action was authorised
whether the permissions were appropriate
whether the moderation control operated
whether a required approval occurred
whether the action remained within risk tolerance
Useful assurance may therefore require evidence linking:
intent → authority → control → action → outcome
For example:
Agent proposed payment → payment within delegated limit → deterministic checks passed → no moderation trigger → action executed → transaction logged.
Or:
Agent proposed privileged access change → high-risk threshold triggered → human review required → request rejected → no change executed.
That is considerably more useful than a raw technical log.
Test the Agent at the Boundaries
Agentic assurance should not test only expected behaviour.
It should test what happens at the edge of the permitted operating envelope.
Examples might include:
transaction just below and above an approval threshold
attempt to access an unapproved system
conflicting instructions
incomplete source data
malicious content in a retrieved document
unexpected tool response
failed downstream transaction
attempt to repeat an action
revoked permission
unusual volume
model-provider degradation
human reviewer unavailable
The objective is to establish whether the agent:
proceeds
pauses
retries
escalates
fails safely
records sufficient evidence
Testing the normal workflow shows that the system works.
Testing the boundary shows whether the governance works.
Watch for Authority Creep
Agentic systems may start with tightly constrained permissions.
Over time, requests appear:
Can it access this system as well?
Can we increase the limit?
Can it send the message automatically?
Can we remove the approval step?
Each change may look modest.
Collectively, they can create substantial authority creep.
Governance therefore needs reassessment triggers for changes such as:
new tools
new systems
additional data
higher transaction limits
new user groups
autonomous execution
changed model
changed purpose
new downstream actions
The risk is not only model drift.
It is authority drift.
Third Parties Complicate the Control Boundary
Many organisations will not build their own agentic platforms.
They will consume agents embedded within:
SaaS platforms
productivity suites
cloud services
security products
finance systems
customer-support platforms
That raises additional questions:
What can the supplier’s agent access?
Which actions can it perform?
Can permissions be restricted?
What evidence is available?
Can the organisation configure approval thresholds?
How are provider-side changes managed?
Can autonomous features be disabled?
Who is responsible when an embedded agent interacts with another supplier?
Supplier assurance therefore needs to consider agent authority, not simply whether the product “uses AI”.
What the Board Needs to Understand
Boards do not need detailed agent architecture.
They do need visibility where significant authority has been delegated to AI.
Useful reporting may include:
material agentic use cases
highest levels of autonomous authority
use cases capable of financial or externally consequential actions
significant exceptions
autonomous activity outside approved conditions
incidents and near misses
material changes in agent permissions
weaknesses in monitoring or evidence
high-risk actions requiring human intervention
assurance results
The useful board question is not:
How many AI agents do we have?
It is:
Where have we delegated meaningful authority to AI, and are we confident the boundaries around that authority are working?
A Practical Agentic Governance Checklist
For a material agentic use case, I would want clear answers to at least the following:
What outcome is the agent intended to achieve?
Who is accountable for that outcome?
Which systems and data can it access?
What actions can it take?
What is explicitly prohibited?
Which actions can proceed autonomously?
Which conditions trigger moderation or human review?
What transaction, volume or privilege limits apply?
Are actions attributable to an identifiable agent or workload identity?
Which actions are reversible?
How can the agent be restricted or stopped?
What happens when required information is unavailable or contradictory?
How are model, tool and permission changes governed?
What evidence is retained?
How do we test that these controls operate in practice?
If those questions cannot be answered, the organisation may not yet understand the authority it has delegated.
Autonomy with Evidence
Agentic AI presents a genuine governance challenge because the capability that creates the value — the ability to act independently — is also what changes the risk.
The answer should not automatically be to remove that capability.
Nor should it be to rely on the model behaving as expected.
The more useful objective is controlled autonomy:
allow routine low-risk activity to flow
moderate the risky edge cases
apply deterministic controls where they add value
require human judgement where consequence warrants it
restrict authority and permissions
retain practical stop mechanisms
monitor changes in behaviour and authority
preserve enough evidence to reconstruct material actions
assure ourselves that those controls actually work
The challenge is to keep enough guardrails around agentic activity without neutering its transformative potential.
That requires more than policy.
It requires architecture, governance, moderation and evidence working together.
And perhaps most importantly, it requires organisations to be explicit about the question at the centre of agentic governance:
How much authority are we prepared to delegate — and what would give us confidence that the boundaries around it are working?
I would be interested in how others are approaching this balance.
Where are you comfortable allowing autonomous action today, and where would you still insist on intervention?
Part 8 will close the series by examining the modern CISO’s role in AI governance: where Security should lead, where it should enable and where accountability should remain with the appropriate business owner.
Personal commentary. Views are the author’s own.