SkarpChapter 5 of 25
The Compliance Design of High-Risk Systems
The recitals turn high-risk status into a lifecycle engineering and governance discipline. Risk management, data quality, logging, human oversight and resilience are treated as mutually supporting controls rather than isolated paperwork duties.
Recital (64): Compliance Is Cumulative, Not Optional
Why requirements apply
Recital (64) links high-risk requirements to risk mitigation and trustworthiness. Providers should account for intended purpose, context of use, state of the art, proportionality, and effectiveness.
Compliance accumulates
A product may fall under more than one Union harmonisation legal act. It can be made available or put into service only after compliance with all applicable legislation.
AI rules complement sectoral rules
Machinery or medical devices with AI can create risks that sectoral health-and-safety law does not address because those risks are specific to AI systems.
Integration is allowed, omission is not
Providers may combine AI testing, reporting, information, and documentation with existing sectoral processes. This flexibility must not undermine full compliance with every applicable requirement.
Recital (65): Risk Management Across the Lifecycle
The exact lifecycle principle
"The risk-management system should consist of a continuous, iterative process that is planned and run throughout the entire lifecycle of a high-risk AI system."
Review, update, document
The process should be regularly reviewed and updated for continuing effectiveness. Significant decisions and actions should be justified and documented.
Foreseeable misuse matters
Risk identification includes reasonably foreseeable misuse: predictable human uses that may sit outside the stated intended purpose or instructions for use.
Choose and explain measures
Providers should select the most appropriate measures in light of the state of the art, document and explain choices, and involve experts or external stakeholders when relevant.
Risk-Lifecycle Design Exercise
Scenario: AI-assisted emergency-department triage
A provider develops a high-risk AI system that prioritises incoming patients for clinical review. Its intended purpose is to help staff identify patients who may need rapid attention. During real use, staff begin treating its priority score as an automatic instruction rather than a support signal.
Work through the following prompts before revealing your answer:
- Lifecycle: At which stages should the provider run risk management: only before release, or before release and throughout deployment?
- Foreseeable misuse: Is staff over-reliance outside the intended purpose but still a use that may be reasonably expected from readily predictable human behaviour?
- Mitigation: What could be documented in the instructions for use about circumstances in which the score can create health, safety, or fundamental-rights risks?
- Evidence trail: Which decisions should be justified and documented when the provider selects mitigation measures?
Suggested analysis
Under Recital (65), risk management should run continuously and iteratively throughout the system lifecycle. Treating the score as an automatic instruction can plausibly be reasonably foreseeable misuse if it follows predictable human behaviour in this operational setting. The provider should identify and mitigate the risk, record its reasoning, and include relevant known or foreseeable risk circumstances in the instructions for use. The recital does not say that the provider must retrain the system for every foreseeable misuse; it says additional training is not required for that purpose, though providers are encouraged to consider it when necessary and appropriate.
Design insight: a compliance record should show the connection between the risk identified, the measure selected, the rationale for selecting it, and the information supplied to the deployer.
Recitals (66)-(70): Data Governance, Privacy, and Bias
Mutually supporting controls
Recital (66) connects risk management, data quality, documentation, logging, transparency, human oversight, robustness, accuracy, and cybersecurity as necessary risk-mitigation requirements.
The exact data-quality standard
"Data sets for training, validation and testing, including the labels, should be relevant, sufficiently representative, and to the best extent possible free of errors and complete in view of the intended purpose of the system."
Bias is dynamic
Data should have appropriate statistical properties for intended users or groups. Historical bias and deployment-created bias can be amplified, especially when outputs become inputs through feedback loops.
Privacy across the lifecycle
Recital (69) applies data minimisation and data protection by design and by default. Measures may include anonymisation, encryption, and bringing algorithms to data rather than copying data.
Exceptional special-category processing
For strictly necessary bias detection and correction, Recital (70) says providers should exceptionally "be able to process also special categories of personal data, as a matter of substantial public interest", subject to safeguards and conditions.
Quiz: Data Quality and Foreseeable Misuse
Choose the answer that most closely follows Recitals (65) and (67).
Which approach best reflects the source text for a provider whose high-risk AI outputs affect later training inputs?
- Use any historically available data if the model's overall accuracy is high.
- Consider whether feedback loops may perpetuate or amplify bias, and use data governance practices suited to the intended purpose and affected groups.
- Ignore foreseeable misuse because only uses expressly listed in the instructions for use matter.
- Avoid privacy-preserving techniques because data sets must be completely error-free.
Show Answer
Answer: B) Consider whether feedback loops may perpetuate or amplify bias, and use data governance practices suited to the intended purpose and affected groups.
Recital (67) specifically identifies feedback loops as a setting in which biases can affect future operations and amplify discrimination. It also requires purpose-relative relevance, representativeness, and data governance. Recital (65) requires attention to reasonably foreseeable misuse, not only expressly intended use. The source also says the completeness and error-freedom expectation should not affect privacy-preserving techniques.
Recitals (71)-(72): Traceability and Usable Transparency
What technical documentation should contain
Recital (71) identifies characteristics, capabilities, limitations, algorithms, data, training, testing, validation, and the risk-management system. Documentation should be clear, comprehensive, and kept up to date.
The automatic-log requirement
"high-risk AI systems should technically allow for the automatic recording of events, by means of logs, over the duration of the lifetime of the system."
Documentation and logs do different jobs
Documentation explains system design and compliance evidence. Logs record events over operational time, supporting traceability, monitoring, and post-market monitoring.
Instructions enable informed use
Instructions should explain capabilities, limitations, risks, deployer actions affecting performance, predetermined conformity-assessed changes, and human-oversight measures in an understandable language.
Recital (73): Human Oversight That Can Actually Intervene
Oversight is designed in
Recital (73) expects providers to identify appropriate human-oversight measures before placing the system on the market or putting it into service.
A capable human role
Where appropriate, oversight means operational constraints the system cannot override, responsiveness to the operator, and people with the competence, training, and authority to act.
Intervention and stopping
The system should, as appropriate, guide the overseer on whether, when, and how to intervene, avoid negative consequences, or stop the system when it fails to perform as intended.
Enhanced biometric safeguard
"no action or decision may be taken by the deployer on the basis of the identification resulting from the system unless this has been separately verified and confirmed by at least two natural persons."
Recitals (74)-(78): Accuracy, Robustness, and Cybersecurity
Declare performance expectations
Recital (74) says systems should perform consistently across their lifecycle at an appropriate accuracy, robustness, and cybersecurity level. Expected performance metrics should be declared in instructions for use.
Robustness includes safe interruption
Recital (75) covers errors, faults, inconsistencies, and unexpected situations. Measures may include fail-safe plans that safely interrupt operation during anomalies or outside predetermined boundaries.
The exact cybersecurity principle
"Cybersecurity plays a crucial role in ensuring that AI systems are resilient against attempts to alter their use, behaviour, performance or compromise their security properties"
AI-specific attacks
The recital identifies data poisoning, adversarial attacks, membership inference, and vulnerabilities in AI digital assets or underlying ICT infrastructure as relevant cybersecurity concerns.
Two regulatory pathways can interact
Recitals (77)-(78) describe how cybersecurity conformity under the horizontal products-with-digital-elements regime can demonstrate AI cybersecurity compliance, while preserving special assessment treatment for specified important and critical products.
Quiz: Logging, Oversight, and Resilience
Select the statement that accurately combines the requirements described in Recitals (71), (73), and (76).
Which statement is correct?
- Logs are optional narrative summaries prepared only after an incident, and biometric matches can directly determine deployer action.
- A provider should enable automatic event recording over the system lifetime; certain biometric-identification actions require separate verification and confirmation by at least two natural persons; cybersecurity measures should address AI-specific attacks such as data poisoning.
- Human oversight requires every decision made by every high-risk AI system to be confirmed by two natural persons.
- Cybersecurity concerns only the underlying network and never training data or trained models.
Show Answer
Answer: B) A provider should enable automatic event recording over the system lifetime; certain biometric-identification actions require separate verification and confirmation by at least two natural persons; cybersecurity measures should address AI-specific attacks such as data poisoning.
Recital (71) calls for technical capacity for automatic event recording by logs over the system lifetime. Recital (73) creates the two-natural-person safeguard for identification resulting from certain biometric identification systems, not every high-risk system. Recital (76) expressly includes data poisoning and attacks on trained models among AI-specific cybersecurity concerns.
Flashcards: The High-Risk Compliance Architecture
Flip each card, then explain how the term connects to another control in the lifecycle design.
- Continuous, iterative risk management
- A process planned and run throughout the entire lifecycle of a high-risk AI system, regularly reviewed and updated for continuing effectiveness.
- Reasonably foreseeable misuse
- Use outside the intended purpose or instructions that may nevertheless be reasonably expected from readily predictable human behaviour in the system's particular context.
- Data quality standard
- Training, validation, and testing data sets, including labels, should be relevant, sufficiently representative, and to the best extent possible free of errors and complete for the intended purpose.
- Feedback loop
- A situation in which AI outputs influence inputs for future operations, allowing bias to gradually increase, perpetuate, or amplify discrimination.
- Technical documentation
- Clear and comprehensive, up-to-date information on characteristics, capabilities, limitations, algorithms, data, training, testing, validation, and the risk-management system.
- Automatic logs
- High-risk AI systems should technically allow automatic recording of events by logs over the system's lifetime.
- Human oversight
- Provider-identified measures that enable natural persons with competence, training, and authority to oversee, intervene in, or stop the system where appropriate.
- Enhanced biometric oversight
- No deployer action or decision based on an identification result unless it has been separately verified and confirmed by at least two natural persons, subject to the stated proportionality exception.
- Technical robustness
- Resilience against harmful or undesirable behaviour caused by system or environmental limitations, including errors, faults, inconsistencies, and unexpected situations.
- AI-specific cybersecurity risks
- Risks including data poisoning, adversarial attacks, membership inference, and exploitation of vulnerabilities in AI digital assets or underlying ICT infrastructure.
Key Terms
- logs
- Automatically recorded events over the lifetime of the high-risk AI system, supporting traceability and monitoring.
- deployer
- The actor using the high-risk AI system and receiving information and instructions intended to support correct and informed use.
- provider
- The actor described in the recitals as establishing the risk-management system, selecting mitigation measures, maintaining documentation, and supplying instructions for use.
- data poisoning
- An AI-specific cyberattack example in which an attacker targets training data sets.
- fail-safe plan
- A technical solution enabling safe interruption of operation when anomalies occur or operation moves outside predetermined boundaries.
- human oversight
- Measures enabling natural persons to supervise use, make informed interventions, avoid negative consequences, or stop a system where appropriate.
- intended purpose
- The purpose used throughout the recitals to frame appropriate data, risk assessment, accuracy, robustness, and transparency.
- adversarial attack
- An AI-specific attack example targeting a trained model or its behaviour through crafted inputs.
- technical documentation
- The maintained compliance information needed to assess a system and facilitate post-market monitoring.
- reasonably foreseeable misuse
- A use not directly covered by intended purpose or instructions that can nevertheless be reasonably expected from readily predictable human behaviour in the particular context.
- special categories of personal data
- Sensitive personal-data categories that Recital (70) addresses in the exceptional, strictly necessary context of bias detection and correction, subject to safeguards and conditions.
- data governance and management practices
- Practices supporting high-quality training, validation, and testing data, including appropriate handling of personal-data collection purposes and bias risks.