Security in the AI era: attacks will not wait for your PoC

Friday, 18:40. A mail goes out as “the CEO”. The tone matches the house style. The invoice link is a little too urgent. Two people click. Nobody breached the server. Someone pasted three real emails into a model and asked it to write like him.

This is not science fiction. It is not “all of cyber just flipped” either. Public reports disagree a little, which is useful: it kills the single slide.

The useful question for a mid-market firm is not whether AI is dangerous. It is: is the SI still designed for a slow human, while the attack is not? If that answer is fuzzy, a fixed-price assessment is for that. Not a white paper.

What the figures say — and what they do not

People mix three things: phishing written by a model, attacks on AI systems, and the agent you wired into the ERP. Those are three jobs.

• AI as a weapon Copy, voice, malware, faster. The SI did not change. The volume can.
• AI as a target Poisoned models, PyPI packages, coding-assistant rule files. Software supply chain, 2025 edition.
• AI living in the SI Copilot, agent, MCP. An account that acts. Often too many rights, too few traces.

The ENISA Threat Landscape 2025 covers 4,875 incidents (1 July 2024 – 30 June 2025). Phishing remains the main intrusion path, about 60%. And this, drier:

By early 2025, more than 80% of observed social-engineering activity worldwide was already AI-supported.

That is ENISA, from open sources. It is not “80% of every attack on earth”. It is social engineering. The point is to stop waiting for broken English as the tell.

IBM’s Cost of a Data Breach 2025 is more of a ledger. Global average breach cost: USD 4.44 million. Phishing is the most frequent initial vector (16%), around USD 4.8 million per incident. 16% of breaches already involved attackers using AI (phishing, deepfakes). On the defender side: 97% of AI-related incidents involved systems without proper access controls. 63% of organisations had no policy to govern AI or stop shadow AI. High shadow AI adds about USD 670,000 to breach cost versus little or none.

Verizon’s DBIR 2025 refuses the novel. On a partner sample of malicious email, the AI-assisted share doubles in two years, from about 5% to 10%. They say it outright: not a revolution everywhere. Industrialisation in progress.

The three sources are not fighting. They do not measure the same thing. A CIO who quotes only one picked a slide.

• > 80% AI-supported social engineering (ENISA, early 2025)
• 60% Intrusions via phishing (ENISA, 2024–25 set)
• 16% Breaches where attackers used AI (IBM 2025)
• 97% AI incidents without proper access controls (IBM)
• ~5% → ~10% AI-assisted malicious email (Verizon, 2 years)
• 94% AI as the top driver of cyber change (WEF Outlook 2026, survey)

The World Economic Forum’s 94% is executive opinion, not attack volume. Keep it for what it is: the topic reached the board, not only the SOC.

The attack does not need to be “smart”

Demos of self-rewriting malware travel well. The real gain is flatter, and worse.

Phishing-as-a-service kits already cloned login pages. ENISA names platforms (Darcula, Lucid, FlowerStorm), including iMessage and RCS. AI adds copy, voice, face. The kit was already there.

IBM notes a phishing email that took 16 hours to polish now takes about five minutes. That is not a zero-day. It is less human time. At campaign scale, volume moves. Network physics does not.

The opposite reflex is “we have a phishing filter”. It catches clumsy French. It catches less the mail that reuses the real tone of finance, with the real project name. “Don’t open .exe attachments” is not enough. You need business paths: dual control on an IBAN, a channel outside mail for urgent transfers. Boring. It works.

Stand-alone malicious AI systems show up in ENISA from early 2025 — not only a jailbroken chatbot. Mandiant / Google in M-Trends 2026 describe malware that queries an LLM at runtime to hide, and stealers that hunt AI CLI tools already on the box (GitHub, NPM tokens). Once inside, the attack uses your copilot.

The agent you deployed

This is the bit people underweight, because it has an “innovation” stamp.

A chatbot that summarises Confluence is a debate. An agent that opens a ticket, reads stock, sends mail is a service account. Often created for the demo, with an admin key, “we’ll tighten after the steering committee”. After the committee, nobody tightens.

We already wrote this for MCP: without bounded tools the model only has the clipboard. With tools that are too wide, it has the SI. The middle exists. You design it. You do not prompt it.

1. Business pastes a file
2. Unscoped AI SaaS
3. Prompt + attachments
4. Unknown region
5. No log

This is not an APT. It is a copy-paste leak. IBM calls it shadow AI. The CNIL calls it loss of control.

1. Named identity
2. Bounded agent / tool
3. Right per action
4. Log
5. Human stop

If you cannot tell who called “export customers” on Tuesday at 21:00, you do not have an agent. You have a hole.

Coding assistants have their own flavour: ENISA mentions a Rules File Backdoor — malicious instructions in the rule files the assistant reads. Plus slopsquatting: package names a model invented, then registered by someone else. That is not “developers are bad”. Code review was not designed for files nobody typed.

An ERP generated by AI with a key in the front-end: we have seen it. The agent only speeds up the same omission.

Five years: no official volume, three moves

People ask “and 2031?”. The honest answer annoys boards that want a pie chart.

Nobody — ENISA, IBM, Verizon — publishes an AI-attack count at five years. “×10 by 2030” charts are almost always vendor studies, not public observatories. We do not repeat them.

What we can write without cheating:

1. Unit cost falls. Five minutes instead of sixteen hours is capacity. Even if click-through stays flat, attempt volume can move.
2. Agents leave the lab. 2025: stand-alone malicious systems observed. 2026: the CNIL’s note on agentic AI. On the attacker side, five years is not Skynet. It is one operator chaining recon, mail and a voice follow-up without three interns.
3. The law catches up, slowly. The EU AI Act phases in through 2027 for part of the high-risk duties. That is a calendar, not a firewall. If you scoped nothing in 2026, you will document in a rush.

The WEF Global Cybersecurity Outlook 2026 (survey: 94% see AI as the top driver of change; 87% name AI-related vulnerabilities as the fastest-growing risk in 2025) describes a race, not a tally. 77% of organisations already say they use AI to defend. The race is not “they have AI, we don’t”. It is: they industrialise; you have a PoC and a shared admin account.

The reasonable five-year picture is not apocalypse. More campaigns, cleaner copy, more executive deepfakes, more agents that err — or that you over-authorised — and more fines if personal data travelled with no legal basis. Attack volume will follow generation cost. Your surface follows how many tools you wired without a review.

What an audit should look at (not a port scan)

A scan runs on a Tuesday. A security audit “in the AI era” is closer to an IT project audit: facts, rights, run.

1. Which copilots and agents are in production? Names, vendors, region, data they see. Including the ones IT did not buy.
2. Which rights? Read, write, export. A “support” agent does not export the customer file.
3. Who stops it? A kill switch. A human. Hours. If the answer is “the intern with the key”, that is not a run.
4. Where are the secrets? `.env` in the repo, tokens in the browser, webhooks in an open Slack: we have found them. AI only helped put them there.
5. What GDPR evidence? Purpose, retention, processor, transfer. Not a cookie banner. A processing.

Human-scale SRE / DevSecOps means that: logs, secrets, a release you dare to ship. AI does not add a magic layer. It adds accounts and data egress.

If you cannot answer the five questions on one page, you do not have an “AI strategy” delay. You have a hole. The fixed-price assessment is how the page gets written. Most stay under €2,000. Scope is stated first.

AI Act, GDPR, CNIL: what already holds, without waiting for 2027

“We’ll see at the AI Act” is a bad excuse. The GDPR applies now. An agent that reads HR files does not wait until 2027 to become processing.

On 20 July 2026 the CNIL and CIANum published an exploratory note on agentic AI. Useful points, without consultancy fog:

• The agent acts in the user’s place, with memory, several services, a sometimes opaque chain. Risk of loss of control over personal data.
• No special regime. GDPR + AI Act. The CNIL recalls the AI Act already applies to agentic systems, even without an “agents” chapter.
• Full application of parts of the high-risk AI Act is aimed around 2027. That is not a stay for DPIAs.
• EDPB / Commission guidelines on GDPR ↔ AI Act are expected by end-2026. Until then, do not freeze projects. Document: purpose, minimisation, rights, processors, region.

Transfers outside the EU, US models, logs at the vendor: classic, just more frequent because the business pastes a PDF “to see”. IBM’s shadow AI and the CNIL’s “loss of control” are the same photograph.

Civil liability across the chain (vendor, integrator, deployer) stays blurry — the CNIL says so, after the 2025 withdrawal of the AI liability directive. That is not a reason to log nothing. It is another reason to know who did what.

Where to start

Not with a tool. With a list.

The three copilots actually used. The files that went in. Agents in test. Accounts. One human allowed to cut everything. Only then: harden, bound, or stop.

If the list does not exist, it comes out of an assessment. That is the deliverable. Not a score. A decision: leave it, bound it, re-host it, freeze it.

AI did not make security “new”. It made the fudge more expensive, and more visible. Friday’s mail will arrive anyway. The question is whether it finds an SI still designed for 2019 — or a run someone owns.