# Malicious use of AI: What the AI Act really protects and its blind spots

> The AI Act offers only incomplete protection against the malicious use of artificial intelligence: some risks are strictly regulated, while others escape regulation entirely. Companies must therefore adopt a broader risk management approach, integrating threats not covered by the AI Act, to ensure robust compliance and global security.

- Tags: usage malveillant ia ai act, risques ia malveillante, conformité règlement ai act, deepfakes ai act, cyberattaques intelligence artificielle, gestion des risques ia, concentration du pouvoir ia, surveillance étatique ia
- Main keyword: usage malveillant IA AI Act

## Key points

- Out of the 9 main sub-risks of malicious AI use, the AI Act addresses only one comprehensively.
- Only mass surveillance and oppression are subject to explicit prohibitions in the text.
- Threats related to disinformation, abusive content, fraud, and cyberattacks are addressed only partially or via other European legislation.
- Biological weapons, malicious autonomous AI, autonomous weapon systems, and the concentration of power fall entirely outside the scope of the regulation.
- To master all risks, companies must expand their approach beyond strict compliance with the AI Act.

## Introduction

In February 2026, the Real Instituto Elcano published an in-depth study questioning the actual capacity of the AI Act to regulate the most dangerous uses of artificial intelligence. Paula Oliver Llorente analyzes in detail **nine sub-risks linked to the deliberate use of AI to cause harm**. The assessment is clear: the protection offered by the AI Act is very uneven. For companies subject to **Regulation (EU) 2024/1689**, this situation has direct consequences on their risk management.

## Content

## Defining malicious use of AI

Malicious use of artificial intelligence is characterized by the intentional exploitation of AI systems to harm the safety of individuals, groups, or society as a whole. What distinguishes this use from errors or accidents is the deliberate intent to cause harm.

This notion should not be confused with *malicious abuse*, which consists of taking advantage of internal system flaws (e.g., adversarial attacks) rather than diverting their capabilities to harm others.

Based on institutional reports, real-world incidents, and analyses by international bodies, nine major categories of risks have been identified:

1. **Biological weapons and chemical threats**: Using AI to create pathogens, orchestrate biological attacks, or provide instructions for the manufacture of existing weapons.
2. **Malicious autonomous AI**: Development and deployment of autonomous systems pursuing destructive goals (e.g., ChaosGPT), capable of acting without human oversight.
3. **Disinformation and persuasive AI**: Massive production of deceptive content, personalized manipulation exploiting cognitive weaknesses, and foreign influence campaigns.
4. **Abusively generated content**: Creation of non-consensual intimate imagery (NCII), AI-generated child sexual abuse material (CSAM), voice impersonation, blackmail, and reputational damage.
5. **Fraud, scams, and social engineering**: Use of AI systems (WormGPT, FraudGPT) to generate phishing campaigns, identity theft, or fraudulent chatbots.
6. **Offensive cyberattacks**: Automation of malware creation, identification of security vulnerabilities, multilingual phishing, and the democratization of attack tools.
7. **Autonomous weapon systems and military use**: Use of drones and AI weapons capable of targeting and attacking without human intervention.
8. **Concentration of power**: Use of AI by governments or corporations to strengthen their dominance, stifle dissent, or monopolize AI resources.
9. **State surveillance and oppression**: Mass surveillance, use of predictive policing, censorship, and targeted repression of minorities using AI.

## Mapping the AI Act's coverage

The Real Instituto Elcano study compares the regulation's provisions against these nine sub-risks. Here is a synthetic overview of the regulatory coverage:

**Legend:** 🔴 Not covered — 🟡 Partial or indirect coverage — 🟢 Full coverage

- **Biological and chemical weapons** 🔴: Only general risk management and incident notification requirements for systemic risk GPAI models apply. Protection relies primarily on international conventions.
- **Malicious autonomous AI** 🔴: Similar regulatory vacuum. Mitigation is limited to systemic GPAI risk management and the human oversight obligation for high-risk systems. No specific rules on the creation of destructive AI agents.
- **Disinformation and persuasive AI** 🟡: The AI Act prohibits certain manipulative techniques (Art. 5) and mandates deepfake labeling (Art. 50), but personalized manipulation via chatbot is not explicitly targeted. The Digital Services Act (DSA) partially complements this protection.
- **Abusive content** 🟡: Indirect prohibitions (exploitation of vulnerabilities, Art. 5), but severe forms like non-consensual intimate imagery (NCII) or AI-generated CSAM are not addressed. Deepfake labeling offers limited protection. The directive on non-consensual intimate imagery acts as a supplement.
- **Fraud and social engineering** 🟡: Transparency obligations reduce the effectiveness of scams, but there is no direct prohibition of AI fraud tools.
- **Offensive cyberattacks** 🟡: Previous European legislation already criminalizes cyberattacks. The AI Act mainly aims to protect high-risk systems against adversarial attacks (malicious abuse). The Cyber Resilience Act complements this framework.
- **Autonomous weapon systems and military use** 🔴: The regulation explicitly excludes these systems (defense and national security fall under Member States). Only dual-use systems are concerned. This is a major gap.
- **Concentration of power** 🔴: The AI Act partially limits state use of AI via restrictions on predictive policing, but does not address the concentration of power by corporations (data access, cloud infrastructure, computing power). The Digital Markets Act does not cover AI specifics.
- **State surveillance and oppression** 🟢: This is the most strictly regulated area. The AI Act prohibits social scoring (Art. 5), predictive policing, certain uses of biometrics, and biometric categorization. This choice reflects the political sensitivity of the subject in Europe.

## Why are these gaps intentional?

The lack of coverage for certain risks is not accidental. European legislators opted for an approach that avoids overlapping standards.

The AI Act is part of a set of existing European texts. Criminal acts such as the creation of biological weapons, fraud, or cyberattacks were already regulated before the rise of AI. Policymakers deemed it unnecessary to create regulatory duplicates for offenses that remain the same, regardless of the tool used.

In practice, many risks are covered by complementary texts:

- **Disinformation**: Digital Services Act (DSA)
- **Abusive content**: Directive on non-consensual intimate imagery
- **Cyberattacks**: Cyber Resilience Act
- **Concentration of power**: Digital Markets Act (DMA)
- **Biological weapons**: International conventions
- **Autonomous weapons**: International initiatives (UN)

This internal logic aims to avoid over-regulation and simplify compliance. However, it poses a problem of exportability: outside of Europe, the AI Act risks losing its effectiveness if the complementary texts are not adopted.

## Cross-cutting limitations of the framework

Beyond coverage by risk type, two structural weaknesses reduce the AI Act's effectiveness against malicious use.

### Private use falls outside the scope

Individual and non-professional uses of AI systems are not targeted by the regulation. The text relies on the responsibility of developers and providers to limit downstream risks. Malicious individuals can therefore act without being directly concerned, unless criminal law intervenes. However, AI facilitates and amplifies this type of use, making the threat of criminal sanctions less of a deterrent.

### The notion of "reasonably foreseeable misuse" remains vague

The AI Act requires providers to consider not only the intended use but also the misuses they can anticipate (Article 9 and Article 51 for systemic risk GPAI). However, this notion remains ambiguous and leads to varied interpretations. Some companies, like OpenAI, have already invoked this ambiguity to disclaim responsibility in the event of dangerous use of their product.

## Consequences for companies

For [providers and deployers](https://compaia.eu/obligations) of AI systems, these findings imply concrete adjustments in risk management.

### Risk analysis must go beyond the text of the regulation

Compliance with Article 9 requires identifying "known and reasonably foreseeable" risks. Malicious uses must therefore be integrated into the technical documentation (Annex IV, Section 5), even if they fall under other legislation.

### Transparency, the first line of defense

For the four partially covered sub-risks, the regulation's transparency obligations (deepfake labeling, signaling human-machine interactions, informing users) constitute the main bulwark. Their rigorous application is essential, even if it is not enough to stem determined malicious use.

### Systemic risk GPAI in a key position

Providers of systemic risk GPAI models (Article 51 et seq.) bear increased responsibility: risk management, adversarial testing, and incident notification constitute the only safety net for four of the nine identified sub-risks. The quality of their implementation determines the security of the ecosystem.

> Are you developing or deploying an AI and want to assess your exposure to malicious use risks? The [free compaia diagnostic](https://compaia.eu/diagnostic) allows you to obtain a map of your obligations in less than three minutes.

## The European ambition called into question

The AI Act was designed as an international model for AI governance, embodying the famous "Brussels effect." The Council of the EU has indeed presented this text as a future global reference.

However, the analysis reveals that this status is weakened by coverage gaps: leaving biological weapons, destructive autonomous AI, and the concentration of technological power outside the direct scope weakens the model's credibility.

The **Digital Omnibus**, proposed in November 2025, accentuates this fragility: it postpones the application of certain obligations for high-risk systems, introduces transition periods for GPAI marking, and expands access to training data. These developments could facilitate the spread of disinformation and social engineering systems.

For European companies, the situation is complex: the [August 2026 deadlines](https://compaia.eu/echeancier-ai-act) are approaching, but the regulatory framework remains unstable.

## Practical advice for organizations

Given this context, three recommendations are essential:

1. **Include malicious use risks in the Article 9 analysis**: It is crucial not to limit oneself to intended uses. It is advisable to explicitly document malicious use scenarios and mitigation measures, even for risks covered by other texts.
2. **Monitor the entire regulatory landscape**: The AI Act is only one piece of the puzzle. It is necessary to identify and integrate the requirements of complementary texts (DSA, Cyber Resilience Act, NCII directive, DMA) into the compliance strategy.
3. **Anticipate regulatory developments**: Article 112 provides for periodic revisions. The list of high-risk systems (Annex III) can evolve through delegated acts. It is therefore essential to follow regulatory news, particularly regarding personalized manipulation and AI-generated content.

Compliance with the AI Act is a foundation, but it is not enough on its own. Companies that adopt comprehensive risk management, integrating malicious uses beyond the scope of the regulation, will be better equipped to face regulators and the real threats that AI can amplify.

## Official source
- [Source](https://www.aiacto.eu/fr/blog/usage-malveillant-ia-ai-act-couverture-risques)