AI Reached the Scammers First

For about twenty years, most organisations taught staff the same lesson about phishing. Look for bad spelling. Watch for clumsy grammar. Be suspicious of anything that reads as though it were translated by a machine. It was practical advice, it was easy to remember, and it worked often enough that people believed in it.

That advice has quietly stopped being reliable, and the reason is not subtle. Generative artificial intelligence gave attackers a fluent, patient writer that never makes the mistakes the training was built around. The United Kingdom's National Cyber Security Centre, in its assessment of the near-term impact of AI on the cyber threat, concluded that the clearest capability uplift for attackers is in social engineering: convincing lures and interactions produced without the translation, spelling and grammatical errors that used to give phishing away. AI did not invent phishing. It removed the flaws defenders had come to depend on.

Grammar was never a real control

It is worth being honest about why the old advice worked, because that explains why it stopped. Poor spelling was never a property of malicious email. It was a property of attackers who did not speak the target's language well, or who did not care to polish a message they were sending to a hundred thousand people at once. Grammar was a proxy for effort and origin, not for intent.

Proxies fail when the thing they stand in for changes. The moment producing a clean, idiomatic, on-brand message costs an attacker nothing, spelling tells you nothing about whether the message is safe. A large language model will happily write a flawless request for an urgent payment, in the house style of a specific finance team, in whatever language and register the attacker asks for. The message reads exactly like the legitimate ones, because it was generated to.

What AI actually changes for the attacker

The uplift is not a single dramatic new capability. It is a set of small frictions removed, and the sum of them matters more than any one.

Research is faster. Public company pages, staff directories, LinkedIn posts, press releases and old breach data can be gathered and summarised in minutes, giving an attacker the names, roles, reporting lines and current projects needed to make a lure specific. Personalisation is cheaper. Instead of one template sent to everyone, an attacker can produce a distinct, tailored message for each recipient, referencing the right supplier, the right invoice cycle, the right internal jargon. Volume and quality stop trading against each other. Historically, a convincing spear-phishing email took human effort, so attackers reserved that effort for high-value targets. When the effort approaches zero, spear-phishing quality can be applied at commodity-phishing scale.

The NCSC's assessment makes a related point that often gets lost. This uplift helps attackers at every skill level, and it is the least capable ones who gain the most, because AI supplies the language ability and the research they previously lacked. The floor rises. A larger number of people can now send a message that would have needed a skilled operator a few years ago.

Spelling was never a property of malicious email. It was a property of attackers who did not speak your language well. Once that stops being true, the tell stops being useful.

Why awareness training built on obvious examples is ageing badly

Most security awareness programmes still teach recognition. They show staff examples of phishing, point at the give-aways, and ask people to spot the next one. When the give-aways were reliable, this produced measurable improvement. The problem now is that the canonical bad example, the one with the mangled grammar and the obviously wrong sender name, is the kind of attack least likely to reach a well-defended inbox, and least representative of the messages that do.

This does not make awareness training worthless. It means the thing it trains has to change. Teaching people to judge a message by how it reads is teaching a skill that attackers have specifically neutralised. Teaching people to pause on the shape of a request, an unexpected change to payment details, a first-time sender asking for something sensitive, a message that pushes them to act outside a normal process, is teaching something an attacker cannot write their way around. The lesson moves from "does this look wrong?" to "does this request make sense, coming from this person, through this channel, right now?"

What defenders should rely on instead

If the readable surface of an email no longer separates real from fake, detection has to lean on the signals that do. None of these are new, but their relative importance has shifted.

Sender identity and history matter more than tone. Whether the domain is authenticated, whether it was registered last week, whether this person has ever emailed this recipient before, and whether the reply-to address quietly differs from the display name are all harder to fake than prose. The destination of a link matters more than the wording around it, because a fluent message can still point to a credential-harvesting page. Intent matters: a request to move money, change bank details, share credentials or bypass a verification step carries risk regardless of how politely it is phrased. And behaviour matters, because a genuine account behaving unusually, or a request that breaks the recipient's normal pattern, is a signal that no amount of good writing can hide.

The through-line is context. Any single one of these signals can mislead. Taken together, and weighed against what is normal for the specific sender and recipient, they describe whether a message deserves trust far better than spelling ever did. This is the same argument we make about why modern phishing emails often look completely routine, and it is central to how business email compromise succeeds with no malicious payload at all.

Defenders can use AI too, without pretending it is magic

It would be one-sided to describe AI purely as an attacker's tool. The same language understanding that helps an attacker write a convincing lure helps a defender read one. A system that can weigh the intent of a message, compare it against a sender's history, follow where a link resolves and explain why the combination looks wrong is doing the kind of judgement that used to require a human analyst, at a speed and scale humans cannot match.

The honest caveat is that this is assistance, not a verdict machine. AI can be wrong, it can be manipulated, and a system that produces a confident label with no evidence behind it is not much use to the analyst who has to defend the decision. The value is in analysis that shows its working: this sender is new, authentication failed, the link resolves somewhere unrelated, the request is financial, and here is why that combination is being flagged. Osiris is built to analyse email that way, weighing sender identity, intent, links and the recipient's exposure together, rather than issuing a single unexplained score.

What to change now

Three shifts are worth making without waiting for a budget cycle. Update awareness content so it no longer leans on spelling and obvious errors, and instead trains people to question unexpected requests and out-of-process demands. Make sure detection and reporting give weight to sender history, authentication results, link destinations and intent, not just message content, because those are the signals AI has not devalued. And treat any control whose core logic is "does this look suspicious to a human?" as one that attackers have been actively working to defeat.

The uncomfortable summary is that attackers adopted this technology before most defenders adapted to it. That is not a reason for alarm, and it is certainly not a reason to believe phishing is now unstoppable. It is a reason to stop defending against the phishing of five years ago. The messages have changed. The way we decide whether to trust them has to change with them.

Key takeaways

• Poor grammar was a proxy for an attacker's effort and origin, not for malicious intent. AI removed that proxy.
• Generative AI gives attackers faster research, cheaper personalisation, and spear-phishing quality at mass-phishing scale.
• The NCSC assesses that AI's clearest near-term uplift for attackers is in social engineering, and that it helps low-skill attackers the most.
• Awareness training built on obvious examples now teaches a skill attackers have specifically neutralised.
• Detection should lean on sender identity, authentication, link destinations, intent and behaviour, which fluent writing cannot disguise.

Frequently asked questions

Has AI made phishing emails harder to detect? It has made them harder to detect by reading alone. Generative AI removes the spelling and grammar errors that used to signal a fake, so users can no longer rely on how a message reads. Detection based on sender identity, authentication, link destinations and the nature of the request is less affected.

Did AI invent new phishing techniques? Not really. The core techniques, impersonation, urgency, credential theft and payment fraud, are long-standing. AI mainly lowers the cost and skill needed to execute them well, and lets attackers personalise at scale.

Is security awareness training now pointless? No, but the content has to change. Training people to spot bad grammar is training a skill attackers have defeated. Training people to question unexpected or out-of-process requests, regardless of how well written they are, remains valuable.

Can AI help defenders as well as attackers? Yes. AI can weigh a message's intent, compare it against a sender's normal behaviour and explain why something looks wrong. The important limitation is that it should support analysts with evidence, not replace judgement with an unexplained score.

Sources and further reading

• National Cyber Security Centre, The near-term impact of AI on the cyber threat.
• Verizon, 2024 Data Breach Investigations Report (social engineering and the human element).
• Centre for Emerging Technology and Security (CETaS), research on generative AI in cybersecurity.

See how Osiris weighs sender identity, intent and link destinations together, and explains why an email was flagged. Book a walkthrough.