Governing the agents: Leadership dilemmas in deploying agentic AI in a cyber defence project team

Learning objectives and key issues

The case is organised around three core themes, each with associated learning objectives:

The problem

It was late on a Thursday evening, and Maya Mercer had stayed behind long after the day shift of the security operations centre (SOC) had gone home. On the desk sat the slide pack for the following week’s risk committee. The recommendation came down to a single sentence: that Meridian’s AI incident-analysis agent be authorised to move from ‘recommend’ to ‘act’ – allowing it to take limited containment actions itself, without a human analyst approving each one. Two years earlier the idea would have been unthinkable. But the three agents the team had built were performing well: triage was faster, false positives were down, and analysts finally had time to hunt. The executive was enthusiastic, talking openly about becoming a ‘frontier’ organisation. And yet Mercer hesitated. Meridian ran critical national infrastructure; a wrong automated containment action could disrupt a service that millions relied on. The regulator was asking sharper questions about AI accountability every quarter. The team had only just begun to trust the agents – and some still quietly feared for their jobs. Threat actors, meanwhile, were rumoured to be building agents of their own. Should Mercer sign the recommendation and push towards the frontier, or hold the agents at human-in-the-loop oversight for another year and keep building the foundation? The committee would need an answer on Monday.

Background

Meridian Water & Power is a (fictional) United Kingdom operator of critical national infrastructure, supplying water and electricity to several million households and businesses across a large region. Like its real-world counterparts, Meridian runs a mix of modern digital systems and older operational technology, sits within a tightening regulatory regime, and presents an attractive target to financially motivated criminals and hostile states alike. A successful attack could mean not just data loss but disruption to an essential service. Its security operations centre therefore operates around the clock.

Maya Mercer had spent more than 15 years in cyber security, moving from hands-on analyst and detection-engineering roles into security leadership. By the time of the case, Mercer led the cyber defence function at Meridian, with accountability for the SOC, detection engineering, threat intelligence and incident response. The role carried a familiar tension for security leaders: personal accountability for decisions that increasingly depended on systems too complex for any one person to inspect in full.

Like many security teams, Meridian’s had grown by accretion. Years of procurement decisions – each sensible in isolation – had left a fragmented architecture of tools, detections and alerts. Collaboration between those who operated controls and those who implemented them was limited, and there was no clear, shared view of how the controls interconnected. The SOC suffered from a high false-positive rate, which drove analyst burnout and meant skilled, expensive people spent their days triaging noise rather than hunting threats.

‘The thing that kept me up at night was not the sophisticated attacker,’ Mercer would later reflect. ‘It was watching good analysts burn out clearing false positives, knowing that the time we lost there was time we could never spend getting ahead of the threat.’

Mercer’s working hypothesis was that time saved in the SOC would deliver the greatest gains across the whole of cyber defence, and could be redirected to higher-value work such as proactive threat hunting. That hypothesis is what set the team on the road to agentic AI.

Meridian’s first steps: A false start

The team began with a security-specific AI assistant aimed at repetitive tasks: summarising incident tickets, drafting reports, analysing threat intelligence and suggesting next steps. Early results were modest. The tool had been introduced in a stand-alone fashion, without being embedded into workflows and without clear usage guidelines, baselines or problem statements. Usage became inconsistent and dependent on individual analyst preference.

‘We had been sold plug-and-play,’ Mercer recalled. ‘What we learned was that none of the vendor tools solved an end-to-end problem. That work falls to the people who own the process. If you just plunk a tool in, you flunk.’

The breakthrough came when the team applied the assistant to a specific, high-impact pain point: the detection rule base. Rules were inconsistently named, poorly governed, difficult to search and lacked clear mapping to industry regulatory frameworks (e.g. MITRE ATT&CK). Using the assistant, the team accelerated its migration to detection-as-code, improving efficiency and code quality far faster than manual effort alone. The lesson the team drew was deceptively simple: always start by asking what problem the AI is solving, not what technology is available.

Building agents as teammates

Building on that foundation, the team developed three custom multi-agent systems, each designed around human-in-the-loop principles. Full autonomy was explicitly not the goal; humans remained central to high-stakes decisions and oversight. The first, an incident-analysis support system, orchestrated AI-powered triage, deep analysis across data sources, limited threat hunting and structured recommendations – including call-out decisions – for human analysts. The second, a detection-engineering agent, managed the full rule lifecycle: analysis, code validation, optimisation, overlap checks, ATT&CK mapping and threat-intelligence integration, with analysts able to converse with it directly. The third, a threat-intelligence agent, automatically ingested, normalised and operationalised intelligence, generated hypotheses, ran targeted hunts, assessed detection coverage and created tasks for proactive teams – linking directly to the other two.

A profound shift occurred when the team stopped treating these agents as tools and began treating them as active team members. The agents generated content, made decisions, executed logic continuously and interacted with one another, much like humans on shift. This mindset reshaped the team’s practices: agents were given ‘job descriptions’ with clear objectives, defined collaboration points, access controls and even performance-improvement mechanisms when they underperformed. Development mirrored onboarding a new hire – coaching, feeding context, correcting and building trust gradually. Model upgrades felt like promotions: an agent would suddenly gain new capabilities, prompting the team to revisit its permissions, role and place in the architecture.

Crucially, the team protected the conditions that made this possible. Rather than treating agent development as a side-of-desk activity, Mercer dedicated two team members full-time to designing and building the systems, with sandbox environments and time for deep, uninterrupted experimentation. Open, transparent communication about the initiative reduced suspicion and drew ideas from the wider team. Mercer came to regard this protected time as the single most important factor behind the quality of the agents.

Guardrails, assurance and getting comfortable

Before agents could make decisions in live cyber defence workflows, Mercer needed to be personally confident that they were safe, auditable and appropriately controlled. Security was treated as a foundational requirement rather than an afterthought. The team implemented multi-layered controls spanning access-control infrastructure, inter-service authentication, content-safety guardrails and application-level role-based access controls, drawing on the native security primitives of its technology stack.

Particular attention went to authentication – especially for agent APIs – identity management for inter-agent and agent-to-tool communication, and the governance of secrets and tokens. Access to external systems introduced further considerations. The team established auditable logging of agent decisions to create a verifiable evidence trail, applied content filtering to mitigate prompt injection and jailbreaks, and built in failover logic for safety-critical scenarios. To augment these controls, it introduced independent verification and validation layers: deterministic pre-checks before any AI-driven decision, static risk assessments, conservative downgrade logic, and quality and confidence scoring on machine-generated detections. Purple-teaming exercises and human-in-the-loop validation provided independent assurance that the controls performed against relevant threat behaviours.

The human dimension was addressed through a dedicated ‘People Workstream’ focused on governance, assurance, job-description updates and time-reallocation planning. Mercer was candid with the team about job anxiety, offering a now-familiar framing: AI would not take their jobs, but a colleague who used AI might. For high-risk decisions, such as call-out recommendations, the team required documented rationale, independent verification, confidence scoring, parallel running with humans, and shift-handover summaries of agent decisions. Trust was built like the onboarding of a new colleague – gradual, evidence-based and never assumed from day one.

Measuring the value

Mercer insisted that AI enablement deliver a measurable return. The team retained its traditional SOC metrics and added agent-specific ones that reflected the value delivered by each system. Costs – prompt usage, tooling and engineering time – were tracked against these gains to give a holistic view of return on investment, and the team deliberately documented its builds and shared knowledge to avoid single points of failure. Exhibit 1 summarises the principal metrics.

OPEN IN VIEWER

Exhibit 1.

Selected SOC and agent-specific metrics.

Metric	Definition/explanation
Average triage time	Time from incident alert to initial analysis and categorisation. Faster triage reduces analyst backlog and burnout.
False-positive alerts processed	Volume of alerts incorrectly flagged that still require human review. Lower numbers of alerts free analysts for higher-value work.
Mean time to respond (MTTR)	Average time from detection to resolution or mitigation – a core industry benchmark for response effectiveness.
Detection rule optimisation	Efficiency and quality of refining, standardising and improving the detection rule base.
False-positive rate in new rules	Proportion of newly created or optimised rules that generate incorrect alerts.
Speed of ingesting/deploying rules	Time required to analyse, validate and activate new or updated detection rules.
Proactive threat hunts initiated	Number of targeted, hypothesis-driven searches for hidden threats.
Intelligence-to-action time	Duration from ingesting new threat intelligence to operational response or rule update.
Detection coverage gaps closed	Previously under-covered areas of the threat surface that are now addressed.
Manual work saved	Cumulative analyst hours freed from repetitive tasks across the SOC.
Unnecessary call-outs avoided	Instances where the incident-analysis agent correctly decides NOT to escalate a false positive to a human. Directly measures reduced alert fatigue.
Employee engagement (workload & satisfaction)	Internal survey results on perceived workload, job satisfaction, burnout risk and morale.

The decision: How far, and how fast?

Nearly two years after the first experiment, the results were undeniable: faster triage, fewer false positives, measurable reductions in manual work, and analysts with time to hunt. The executive’s appetite for ‘frontier’ status was growing, and the roadmap had always pointed towards a next phase of automated containment. Yet, the same factors that made the agents valuable also made expanding their autonomy consequential.

Mercer weighed competing pressures. Extending autonomy promised further efficiency and signalled ambition to the board, but Meridian operated critical national infrastructure, where a mistaken automated action carried real-world consequences and the regulator’s expectations on AI transparency and accountability were tightening. The team’s trust in the agents, though growing, was still young, and the workforce’s anxieties had not fully settled. Hostile threat actors were beginning to field agents of their own, raising the prospect of AI-versus-AI dynamics that current guardrails had not been designed to withstand.

Mercer had come a long way and did not want to stall the momentum the team had earned. The question for the risk committee was no longer whether agentic AI could add value – that had already been demonstrated – but how far, and how fast, a responsible leader should extend its use.

Suggested discussion questions

The questions below are grouped by theme and audience so that instructors can select those best suited to their cohort. The first set suits general digital and AI transformation courses; the second is pitched at cybersecurity and security-management audiences; the third explores trust and collaboration. Instructors may mix across the groups depending on the background of their students.