Module 14: Metrics, Reporting, and Continuous Improvement in DORA Programs

Step 1 – Why Metrics Matter in DORA (and What ‘Good’ Looks Like)

Under the Digital Operational Resilience Act (DORA), in force since January 2023 and applicable since 17 January 2025 (about a year ago relative to today), financial entities must prove that their digital operational resilience framework is effective and improving over time.

Metrics and reporting are the backbone of that proof. For an advanced DORA program, you need three layers of measurement:

1. Compliance metrics

Are we meeting DORA’s explicit requirements?

• % of ICT incidents reported to the competent authority within required timelines
• % of critical ICT third‑party contracts including DORA‑aligned clauses
• Coverage of mandatory policies (e.g., ICT risk management, incident management, testing)

1. Operational resilience KPIs (Key Performance Indicators)

How well is our ICT estate performing in supporting business services?

• Service availability / uptime for critical business services
• Mean Time to Detect (MTTD) and Mean Time to Recover (MTTR) for incidents
• % of successful execution of critical processes during major incidents

1. Risk‑oriented KRIs (Key Risk Indicators)

How much digital operational risk are we running? Is it trending up or down?

• Number of critical vulnerabilities older than X days
• Dependency concentration on a single cloud provider
• Volume and severity of control test failures over time

Goal of this module: by the end, you should be able to:

• Design a metrics and reporting framework that links DORA requirements to resilience outcomes.
• Communicate these metrics upwards (board, C‑suite) and outwards (regulators, supervisors).
• Embed continuous improvement loops so that each incident, test, and audit measurably strengthens your DORA operating model.

Keep in mind: the European Supervisory Authorities (ESAs) continue to publish Regulatory Technical Standards (RTS) and Guidelines that refine expectations on incident reporting, testing, and third‑party risk. Your metrics framework must be adaptable to these updates.

Step 2 – Building a DORA Metrics Hierarchy (from RTS to Dashboard)

To avoid a chaotic list of metrics, construct a top‑down hierarchy that starts from DORA obligations and ends with concrete indicators.

1. Map DORA Articles to Measurement Domains

Typical domains (non‑exhaustive):

• ICT Risk Management (Art. 5–15)
• ICT‑related Incident Management & Classification (Art. 17–23)
• Digital Operational Resilience Testing (Art. 24–27)
• ICT Third‑Party Risk Management (Art. 28–44)
• Information Sharing (Art. 45)

2. For Each Domain, Define 3–5 Strategic Questions

Example for ICT incident management:

• How quickly do we detect and contain critical incidents?
• Are we classifying and reporting incidents correctly under the latest RTS?
• Are the root causes of major incidents being eliminated over time?

3. Translate Questions into KPIs and KRIs

Use a simple template:

• Name: e.g., Critical Incident Mean Time to Recover (MTTR)
• Type: KPI (performance) or KRI (risk)
• Unit: minutes / hours / % / count
• Scope: critical business services, entire ICT landscape, specific provider
• Data source: ITSM tool, SIEM, GRC platform, CMDB, etc.
• Frequency: real‑time, daily, monthly, quarterly
• Thresholds / Limits: target, warning, breach
• Owner: responsible role (e.g., Head of IT Ops, CISO)

4. Align with Business Services and Impact Tolerances

DORA implicitly pushes you to think in terms of critical or important functions. Metrics should be:

• Service‑centric: e.g., availability of payments processing rather than generic server uptime.
• Impact‑oriented: linked to customer harm, market disruption, or safety of client assets.
• Proportional: more granular for systemically important entities; simpler for small firms.

This hierarchy forms the blueprint for the rest of the module: we’ll now fill it with concrete measures for incidents, testing, third‑party risk, and remediation.

Step 3 – Concrete KPIs & KRIs for Digital Resilience

Below is a sample metric catalogue for a mid‑sized bank. Focus on structure, not memorization.

A. ICT Incident Management Metrics

Performance KPIs

• MTTD (Mean Time to Detect) – Critical Incidents
• Definition: Average time between incident occurrence and detection for DORA‑reportable incidents.
• Target: < 15 minutes.
• Why it matters: Long MTTD implies weak monitoring or alert fatigue.

• MTTR (Mean Time to Recover) – Critical Services
• Definition: Average time from incident start to full restoration of critical services.
• Target: Aligned with business impact tolerances (e.g., 2 hours for retail payments).

• % of Incidents Classified According to RTS Taxonomy
• Definition: Incidents tagged with the correct category, cause, and impact per latest ESA incident classification standards.
• Target: 100% for DORA‑reportable incidents.

Risk KRIs

• Number of DORA‑reportable Incidents per Quarter
• Trend analysis: Is the number rising (worsening risk) or rising because of better detection?
• Must be interpreted with context (e.g., major technology transformation).

• % of Major Incidents with Repeat Root Cause within 12 Months
• High value indicates poor effectiveness of corrective actions.

B. ICT Testing & Control Effectiveness Metrics

Performance KPIs

• Annual Coverage of Critical Services by Scenario‑based Testing
• Definition: Proportion of critical/important functions tested against severe but plausible scenarios.
• Target: 100% over a defined cycle (e.g., 1–3 years, depending on RTS).

• Penetration Test Defect Closure Rate
• Definition: % of high/critical findings closed within agreed SLA (e.g., 30 days).
• Target: > 95%.

Risk KRIs

• Number of Open Critical Vulnerabilities > X Days (e.g., 30)
• Indicates exposure window to known high‑risk vulnerabilities.

• Control Failure Rate in Key ICT Processes
• E.g., % of change requests implemented without formal approval.

C. Third‑Party / Cloud Risk Metrics

Performance KPIs

• % of Critical ICT Third‑Party Contracts with DORA‑Compliant Clauses
• Scope: Data location, access, audit & inspection rights, exit strategies, incident notification SLAs, etc.
• Target: 100% for newly signed and renewed contracts; migration plan for legacy contracts.

• Average Time for Third‑Party Incident Notification
• Definition: Time between incident occurrence at provider and notification to your entity.
• Compare against contractual SLAs and DORA’s reporting timelines.

Risk KRIs

• Concentration Risk Index for Cloud Providers
• Simple proxy: % of critical workloads hosted by a single provider.
• More advanced: Herfindahl‑Hirschman Index (HHI) across providers.

• Number of Critical Findings from Third‑Party Audits / On‑Site Inspections
• Trend and severity over time.

When designing your own catalogue, force yourself to answer: What decision will this metric inform? If you cannot answer convincingly, the metric is probably noise.

Step 4 – Design a Metric from Scratch (Thought Exercise)

You are the DORA program lead for a payments institution heavily reliant on one cloud provider. The board is worried about concentration risk and recovery from a cloud outage.

Task: Draft one KPI and one KRI that you would present to the board. Use the template below.

1. KPI – Performance of Recovery from Cloud Incidents

• Name:
• Definition (what exactly is measured?):
• Unit (e.g., minutes, %, count):
• Data Source (e.g., incident management tool, cloud monitoring):
• Frequency (e.g., monthly, quarterly):
• Target / Threshold:
• Owner:

1. KRI – Cloud Concentration Risk

• Name:
• Definition:
• Unit:
• Data Source:
• Frequency:
• Limit / Risk Appetite (what level is too high?):
• Escalation Rule (what happens if breached?):

Reflect:

• Would these metrics enable the board to challenge management?
• Could they be used by supervisors to assess whether your cloud exit strategy and resilience testing are realistic?
• Are you mixing up performance (how well you respond) and risk (how exposed you are)?

Write down your answers and refine them until each metric is:

• Specific (no ambiguity in definition)
• Actionable (breaches trigger clear responses)
• Comparable over time and across business units.

Step 5 – Metrics for ICT Incidents, Testing, and Remediation

DORA and its RTS on incident reporting and testing expect traceability from:

> incident or control failure → classification → root cause → remediation → re‑testing → closure

Your metrics should therefore cover the full lifecycle.

1. Incident Lifecycle Metrics

• Detection Quality
• % of critical incidents detected by automated monitoring vs. user reports
• False positive rate of monitoring alerts (too many → alert fatigue; too few → blind spots).

• Response & Communication
• % of major incidents with communication to affected customers within defined timeframe
• Adherence to internal escalation timelines (e.g., N minutes to notify CISO, COO).

• Regulatory Reporting
• % of DORA‑reportable incidents submitted within RTS deadlines (initial, intermediate, final reports).
• Number of regulator follow‑up queries per incident (proxy for report quality).

2. Testing & Scenario Coverage Metrics

Link your testing program (e.g., threat‑led penetration testing, scenario testing, failover exercises) to metrics:

• Coverage
• % of critical business services covered by at least one severe‑but‑plausible scenario test in the last X years
• % of critical third‑party dependencies included in major resilience exercises.

• Effectiveness
• % of tests where recovery objectives (RTO/RPO, maximum tolerable downtime) were met
• Number of critical findings per test and average time to remediate.

3. Remediation & Control Improvement Metrics

DORA supervisors will be skeptical if the same issues reappear. Track:

• Remediation Timeliness
• % of high/critical issues closed within SLA (e.g., 30/60/90 days).
• Average age of open high‑risk findings.

• Sustainability of Fixes
• % of issues that reoccur within 12–24 months (indicates superficial fixes).
• Number of incidents linked to previously identified but unremediated vulnerabilities.

• Control Maturity
• Use a simple 1–5 maturity scale (Initial → Optimized) for key control domains (e.g., backup & restore, change management) and track year‑on‑year improvement.

Together, these metrics operationalize the idea that every incident and test is a learning opportunity, not just a compliance burden.

Step 6 – Quick Check: Are These Good DORA Metrics?

Evaluate a proposed metric for a DORA program.

Step 7 – Board, Management, and Regulator Reporting

Different stakeholders need different views of the same underlying data.

1. Board‑Level Reporting

Boards care about risk appetite, impact on critical services, and accountability, not raw technical detail.

Key principles:

• Aggregation & trend: 3–7 top‑level indicators per domain (incidents, testing, third‑party, cyber).
• Narrative: Explain why metrics changed and what management is doing.
• Risk appetite linkage: Explicitly show where metrics are within, at, or beyond risk appetite.

Example board dashboard items:

• Heatmap of critical business services vs. resilience status (green/amber/red).
• Trend of DORA‑reportable incidents by severity and root cause category.
• Cloud concentration index vs. board‑approved limit.
• Testing coverage of critical services and major findings outstanding.

2. Executive / Senior Management Reporting

Executives need more granular operational detail to manage trade‑offs.

Examples:

• Metrics by business unit or region.
• Drill‑down into top recurring root causes (e.g., change management, capacity, third‑party issues).
• Cost vs. resilience indicators (e.g., incremental cost of redundancy vs. reduction in downtime).

3. Reporting to Supervisory Authorities

Supervisors expect:

• Formal incident reports in line with the latest RTS (structure, fields, timelines).
• Evidence of lessons learned and follow‑up actions after major incidents.
• Documentation of testing programs, methodologies, and results, especially for advanced tests (e.g., threat‑led penetration testing).
• Clear governance evidence: minutes of board discussions, risk appetite statements, and decisions based on metrics.

Your challenge as a DORA leader is to re‑use the same data but tailor visualization, granularity, and narrative to each audience.

Step 8 – Draft a One‑Slide Board Dashboard (Thought Exercise)

Imagine you have only one slide to brief the board on DORA metrics this quarter.

Task: Sketch the content of that slide (mentally or on paper) using the structure below.

1. Top‑Left: Overall Resilience Status

• A simple traffic‑light summary for 3–4 domains:
• ICT incidents
• Testing & exercises
• Third‑party risk
• Vulnerability management

1. Top‑Right: Key Metrics vs. Appetite

• Choose 3–5 metrics, e.g.:
• Number of DORA‑reportable incidents (trend last 4 quarters)
• Cloud concentration index vs. limit
• % of critical services tested in last 12 months
• % of critical vulnerabilities > 30 days

1. Bottom‑Left: Major Issues and Actions

• List 3 bullets:
• Key issue
• Root cause
• Management action & target date

1. Bottom‑Right: Ask from the Board

• E.g., approval for investment, risk appetite adjustment, or endorsement of a remediation plan.

Reflect:

• Are you overloading the board with detail?
• Does every element on the slide support a decision or oversight responsibility?
• Would a supervisor, reading the slide, see clear evidence of active board engagement on DORA?

Step 9 – Continuous Improvement: Closing the Loop

DORA expects iterative strengthening of your ICT risk framework, not a one‑off project. Metrics should feed into formal feedback loops.

1. Lessons‑Learned Loop

For each major incident or failed test:

1. Capture

• What happened?
• Which controls failed or were absent?
• How did metrics behave (e.g., MTTD, MTTR, capacity utilization)?

1. Analyze

• Root cause analysis (technical, process, human, third‑party).
• Were risk appetite and thresholds realistic?
• Were early‑warning KRIs triggered but ignored?

1. Improve

• Update controls, procedures, and training.
• Adjust metrics (new KRIs? different thresholds?).
• Feed changes into testing scenarios and business impact analyses.

2. Audit and Assurance Loop

Internal audit, external audit, and independent reviews provide:

• Challenge to your metrics design (are they biased or incomplete?).
• Validation of data quality and reporting accuracy.
• Recommendations that should be tracked with the same remediation metrics (timeliness, recurrence).

3. Change & Transformation Loop

Major technology or organizational changes (e.g., cloud migration, core banking replacement) should:

• Trigger a re‑assessment of KRIs (new dependencies, new failure modes).
• Lead to updated stress scenarios and test plans.
• Be accompanied by temporary risk indicators (e.g., change‑related incident rate) until the new state stabilizes.

A mature DORA program can show a regulator concrete evidence that:

> Metrics → Insights → Decisions → Changes → Better Metrics → Improved Outcomes.

Step 10 – Staying Aligned with RTS and Future EU Developments

Since DORA entered into force (2023) and became applicable in early 2025, the ESAs have been issuing Regulatory Technical Standards (RTS), Implementing Technical Standards (ITS), and Guidelines that refine what data must be collected and reported.

Your metrics framework must be maintainable under regulatory change.

1. Design for Flexibility

• Use a central data model for incidents, assets, providers, and controls.
• Avoid hard‑coding regulatory fields in multiple tools; instead, map RTS requirements to this central model.
• Maintain a data dictionary that documents:
• Field name and definition
• Source system
• Regulatory references (e.g., DORA article, specific RTS/ITS paragraph)

2. Monitor the Regulatory Pipeline

Track updates not only to DORA RTS/ITS but also to related EU legislation, for example:

• NIS2 Directive (network and information systems security) – especially where your entity is also an essential/important entity under NIS2, to avoid duplicate or inconsistent metrics.
• EU Cybersecurity Act and related certification schemes that may introduce new assurance expectations.
• Sectoral guidance from the ECB, EBA, ESMA, and EIOPA on outsourcing, cloud, and ICT risk.

3. Versioning and Backward Compatibility

When RTS or guidance change:

• Version your metrics: keep historical definitions so time‑series remain interpretable.
• Document breaks in series (e.g., incident classification schema changed in 2025 Q3).
• Use mapping tables to reconcile old and new categories where feasible.

The objective is to be able to show, at any supervisory review:

• How your metrics currently align to the latest RTS and guidance.
• How and why they have evolved over the last few years.

Step 11 – Flashcard Review: Key Terms and Ideas

Flip through these cards to consolidate your understanding of DORA metrics and continuous improvement.