Skip to main content
Barry Duffy | CalypsoAI’s Director of Commercial Product


On April 21st, the European Commission issued a framework for regulating AI systems. The proposal provides a nuanced regulatory framework for the use of AI systems. The draft language from the EU’s executive commission proposes taking a four-level “risk-based approach” to quantify AI trust and accordingly regulate systems. 

AI Legislation
The European Union’s risk-based regulatory framework on AI, from the European Commission’s “Regulatory framework proposal on Artificial Intelligence.”

Unacceptable risks include AI systems that manipulate human behavior to circumvent users’ free will (e.g. government ‘social scoring’ systems, and toys using voice assistance to encourage dangerous behavior of minors) and the use of AI surveillance in public by law enforcement (certain exemptions are available here). Unacceptable risks will be banned under new legislation.

The High Risk category covers AI systems that pose significant risks to the health and safety or fundamental rights of persons (e.g. AI applications in robot-assisted surgeries, or autonomous systems that determine which citizens qualify for a loan). Systems in this category will be compelled to have a robust risk management framework.

“The proposal lays down a solid risk methodology to define “high-risk” AI systems that pose significant risks to the health and safety or fundamental rights of persons. Those AI systems will have to comply with a set of horizontal mandatory requirements for trustworthy AI and follow conformity assessment procedures before those systems can be placed on the Union market. Predictable, proportionate, and clear obligations are also placed on providers and users of those systems to ensure safety and respect of existing legislation protecting fundamental rights throughout the whole AI systems’ lifecycle.”

European Commission, Proposal for a regulation laying down harmonised rules on artificial intelligence

Limited risk systems (e.g. chatbots) are subject only to minimal transparency obligations, while minimal risk systems (e.g. spam filters) are free to use.


Looking to bring AI in line with all peoples and processes covered by the EU Charter of Fundamental Rights, including the rights of the consumer, the Commission’s proposed AI legislation takes aim at highly regulated activities. For instance, the proposed legislation specifically references AI systems utilized by the financial sector for credit risk scoring, as well as the algorithms used by hiring managers to promote or terminate employees. Thus the regulation targets those applications of AI systems that have the potential to directly impact human life. 

At present, insurance activities are not explicitly referenced in the EU’s AI legislation. However, as part of the EU’s continuous regulatory monitoring regime, one can expect that AI systems utilized in the insurance sphere shall be regulated under these guidelines in the near future. AI is increasingly used across the insurance industry, from determining one’s accessibility to insurance coverage, the cost of one’s coverage, through the claim determination process. Thus, it is vital for insurance companies to look ahead, ensuring their AI is auditable, secure, and reliable.


Leaders in the insurance sector and other highly regulated industries seeking to accelerate AI across their organization, while adhering to the EU’s AI framework, must adopt a robust model risk management system to ensure that AI is trustworthy and secure.

“High-risk AI systems will be subject to strict obligations before they can be put on the market:

Providers must ensure their AI is transparent, enabling users to understand and “control how the high-risk AI system produces its output.” High-risk AI users must also provide disclosures such as the general logic and design choices of the system, the potential circumstances where the AI could create risks to safety and rights, descriptions of the system’s training data, and “the level of accuracy, robustness, and security.”

A key obligation of providers is to ensure resilience against “attempts to alter [an AI systems] use or performance by malicious third parties intending to exploit system vulnerabilities.”

Companies that fail to comply with the proposed rules face fines of up to 6% of their global turnover or 30 million euros ($36 million), whichever is the higher figure. 


By implementing a robust Model Risk Management (MRM) process backboned by a product like CalypsoAI’s VESPR, mission owners can ensure compliance and accelerate the development and deployment of AI/ML models.

VESPR provides advanced AI testing capabilities with a streamlined workflow to ensure that every machine learning algorithm put into production has been verified, maintaining the integrity, trust, and security of the algorithms. The systems’ record keeping allows for a deep audit of all artifacts considered and decisions made throughout the system’s lifecycle. The end result is a set of trustworthy AI systems that you can have high confidence in and are easy to understand.

Barry Duffy on EU's AI Legislation

Barry has designed and delivered software solutions for some of the world’s largest insurers. For over 15 years, his work has spanned early engagement, pre-sales, design, delivery, and go-live support. Before joining CalypsoAI in 2021, he was the Global Product Manager for the FINEOS Claims system.