SkarpChapter 9 of 11
Designing Technical and Organizational Controls for Compliance
Turn legal requirements into concrete design choices: from data governance and documentation templates to human oversight protocols and cybersecurity controls that satisfy AI Act expectations.
From AI Act Obligations to Concrete Controls
Module Goal
You will practice turning high-level EU AI Act duties into concrete technical and organizational controls that work in real AI projects.
Current Context (2026)
The AI Act is adopted and phasing in. High-risk, GPAI, and foundation models must implement risk management, data governance, documentation, oversight, and security controls.
Translation Pattern
Use a simple pattern: identify the legal obligation, define a control objective, choose specific measures aligned with standards, then document and monitor.
Running Example
We will revisit a case: a bank deploying a high-risk credit-scoring model built partly on a general-purpose foundation model from a large provider.
Step 1: Build an AI Risk Management Loop (Aligned with ISO 42001 & NIST AI RMF)
Continuous Risk Management
The AI Act expects continuous risk management. Use ISO/IEC 42001 and NIST AI RMF to build a loop instead of a one-time checklist.
Govern & Map
Govern: define roles and policies. Map: describe the system, classify its AI Act category, and identify affected rights and interests.
Measure & Manage
Measure: define metrics and tests for accuracy, robustness, fairness. Manage: choose risk treatments and implement controls like extra review or monitoring.
Monitor & Improve
Monitor logs and metrics, review incidents, and update controls. In ISO 42001 this forms part of your AI Management System (AIMS).
Running Example: Bank
A bank’s credit-scoring model is high-risk. A Model Risk Committee oversees fairness metrics, drift tests, and monthly performance reviews.
Exercise: Map One Obligation to a Control
Activity: Take a single AI Act-style obligation and turn it into a control.
Prompt:
Imagine you are designing a high-risk AI system used by a public authority to allocate access to housing benefits. One obligation under the AI Act is to “ensure appropriate human oversight” so that the system does not undermine fundamental rights.
- Write down one concrete organizational control you would introduce.
- Example pattern: "For decisions above risk level X, a trained human reviewer must ..."
- Write down one concrete technical control that supports this oversight.
- Example pattern: "The system interface must display ... to the reviewer before they confirm a decision."
- Check your ideas against this quick checklist:
- Does your control make it clear who is responsible for final decisions?
- Does it give humans enough information to challenge or override the AI?
- Is it realistic in the working environment (time, training, tools)?
Pause for 2–3 minutes and draft your two controls in a notebook or notes app.
After you are done, compare to these sample answers:
- Organizational: "All benefit refusals must be confirmed by a caseworker who has completed AI oversight training and has access to full case details and model reasoning aids."
- Technical: "The decision dashboard shows the model’s key input factors, confidence score, and a one-click option to override the AI recommendation with a free-text justification field."
Step 2: Data Governance and Data Quality Controls
Why Data Governance Matters
The AI Act demands high-quality, relevant, and representative data, with documented practices. GPAI and foundation models must show how they manage data risks at scale.
Inventory & Quality
Keep a register of all training, validation, and test datasets, with sources and legal bases. Define quality criteria and test them before training.
Bias & Minimization
Check representativeness across key groups. Minimize personal data, apply pseudonymization, access controls, and retention limits.
Change Control
Treat dataset changes like code changes: version, approve, and re-validate when data sources or scopes change significantly.
Running Example: Bank
The bank keeps a dataset register, runs quality and fairness checks, excludes sensitive features, and re-validates models after major data refreshes.
Practical Template: Minimal Data Governance Register
Here is a simple JSON-style structure you could use as a starting point for a dataset register. In practice this would sit in a database or data catalog tool.
```json
{
"datasetid": "credithistory_v3",
"purpose": "Training high-risk credit scoring model",
"aisystemsusing": ["creditscoringv5"],
"sourcesystems": ["corebanking", "online_portal"],
"collection_period": {
"start": "2021-01-01",
"end": "2024-12-31"
},
"data_subjects": "Retail customers in EU",
"personal_data": true,
"legalbasisgdpr": ["contract", "legal_obligation"],
"containsspecialcategories": false,
"dataqualitychecks": {
"missing_values": "< 1% per key feature",
"duplicates_removed": true,
"outlier_strategy": "winsorization at 0.5%"
},
"bias_checks": {
"groupstested": ["gender", "ageband", "region"],
"metrics": ["approvalrate", "defaultrate"],
"last_run": "2026-06-15",
"known_limitations": "Fewer historical loans in new regions"
},
"securitycontrols": ["pseudonymizedids", "rolebasedaccess"],
"retention_policy": "5 years from collection, then aggregated",
"owner": "datagovernanceteam@bank.example"
}
```
Step 3: Human Oversight Design Patterns
Effective Oversight
Oversight must be real, not symbolic. It should prevent or minimize risks and allow humans to override or intervene when needed.
Oversight Patterns
Use patterns: Human-in-the-loop (AI suggests, human approves), Human-on-the-loop (AI acts, human monitors), Human-in-command (humans can set constraints and shut down).
Define Oversight Details
For each pattern, specify who oversees, what actions they can take, when they intervene, and what information they see.
Oversight Documentation
Create a use and oversight manual, warnings about misuse, and training materials for overseers. Regulators will look for these.
Running Example: Bank
The bank uses human-in-the-loop oversight for large or low-confidence loans. Credit officers see explanations and can override with justification.
Step 4: Robustness, Accuracy, and Cybersecurity Controls
Robustness & Accuracy
Set explicit targets and error ranges. Use independent validation, stress tests, and red-teaming (especially for GPAI and foundation models).
Cybersecurity Controls
Align with ISO 27001: secure development, access control, encryption, and protection against data poisoning, model theft, and prompt injection.
Resilience & Fallback
Define safe fallback modes and health checks. If the AI fails, the system should revert to manual or degraded but safe operation.
Running Example: Bank
The bank sets performance thresholds, secures the model registry, monitors for attacks, and queues applications for manual review if the model fails.
Quick Check: Robustness and Security
Test your understanding of robustness and cybersecurity controls.
Which combination best reflects AI Act-aligned robustness and security controls for a high-risk AI system?
- Only perform accuracy testing once before deployment; rely on firewalls for security.
- Define performance targets, run stress tests, control access to models, and monitor for AI-specific attacks.
- Focus on user training and accept that technical robustness is mostly out of your control.
Show Answer
Answer: B) Define performance targets, run stress tests, control access to models, and monitor for AI-specific attacks.
The AI Act expects defined performance targets, robustness testing, and cybersecurity measures. That includes stress tests, strict access control to models, and defenses against AI-specific attacks like data poisoning or prompt injection.
Step 5: Logging, Monitoring, and Incident Management
Why Logging Matters
Logs make AI Act obligations visible. They support internal governance and allow investigations when issues arise.
What to Log
Log inputs/outputs (with safeguards), model versions, key decision variables, user actions, and security events.
Monitoring & Alerts
Use dashboards for performance, fairness, and security metrics. Set alerts for unusual patterns or degradation.
Incident Management
Define AI incidents, escalation paths, and links to existing security and privacy response processes.
Running Example: Bank
The bank logs each score and override, monitors group approval rates, and uses an AI incident playbook for serious issues.
Key Terms Review
Flip the cards to reinforce the core concepts you have used in this module.
- AI Management System (AIMS)
- An organizational framework, described in ISO/IEC 42001, for systematically managing AI risks and controls across their lifecycle, integrated with existing management systems.
- Human-in-the-loop (HITL)
- An oversight pattern where the AI provides recommendations but a human must review and approve or reject them before action is taken.
- Data lineage
- Documentation of where data comes from, how it is transformed, and which AI systems use it, enabling traceability and accountability.
- Red-teaming (for AI)
- Structured testing in which internal or external teams try to make an AI system behave harmfully or unsafely, to discover vulnerabilities and improve defenses.
- AI incident
- An event where an AI system causes or risks causing significant harm, such as safety issues, serious discrimination, or security and privacy breaches.
Final Scenario Quiz: Putting It All Together
Apply what you have learned to a short end-to-end scenario.
A hospital deploys a high-risk AI system to prioritize patients for intensive care. Which option best shows a coherent set of technical and organizational controls aligned with the AI Act?
- Train on whatever data are available, rely on doctors to spot issues, and keep logs only for billing.
- Use a dataset register and bias checks, define human-in-the-loop review for borderline cases, log decisions and overrides, and monitor performance and fairness over time.
- Focus on strong network security and ignore data quality or oversight, because doctors remain legally responsible.
Show Answer
Answer: B) Use a dataset register and bias checks, define human-in-the-loop review for borderline cases, log decisions and overrides, and monitor performance and fairness over time.
Option 2 combines data governance, human oversight, logging, and monitoring, matching AI Act expectations for high-risk systems. Options 1 and 3 miss critical controls like data quality, structured oversight, and systematic monitoring.
Key Terms
- AI incident
- An event in which an AI system causes or risks causing significant harm, such as safety issues, serious discrimination, or major privacy or security breaches.
- Red-teaming
- Adversarial testing where a team deliberately tries to cause harmful or unsafe AI behavior to identify and fix weaknesses.
- Data lineage
- End-to-end tracking of data origins, transformations, and uses, enabling traceability for AI training, validation, and testing datasets.
- High-risk AI system
- Under the EU AI Act, an AI system in specific sensitive areas (such as credit scoring, employment, healthcare, or education) that must meet strict requirements on risk management, data governance, oversight, and more.
- Human-in-command (HIC)
- An oversight pattern where humans keep ultimate control over strategic decisions and can define objectives and constraints for AI systems.
- Human-in-the-loop (HITL)
- An oversight pattern where humans must review and approve AI recommendations before they are acted on.
- Human-on-the-loop (HOTL)
- An oversight pattern where AI acts automatically but humans monitor its behavior and can intervene or shut it down.
- AI Management System (AIMS)
- A structured set of policies, processes, and roles for managing AI risks and controls throughout the AI lifecycle, as defined in ISO/IEC 42001.
- NIST AI Risk Management Framework (AI RMF)
- A framework from the US National Institute of Standards and Technology that organizes AI risk management into Govern, Map, Measure, and Manage functions.
- General-purpose AI (GPAI) / Foundation model
- A model trained on broad data at scale, capable of performing a wide range of tasks. The AI Act sets special obligations for their providers, especially for powerful models.