Reported by Siddharth Venkataramakrishnan

Increasing sophistication

Identifying the scale of AI use by scammers is a difficult task, says Alex West, banking and payments fraud specialist at consultant PwC. He was one of the authors of a report into the impact of AI on fraud and scams last December in collaboration with cross-industry coalition Stop Scams UK. This identified the kind of “voice cloning” that targeted Fitzgerald as one of biggest ways in which criminals are expected to use AI.

“Scammers are already very successful, and it could be that they just don’t need to use this type of tech, or it could be that they are using AI and we just aren’t able to distinguish when it has been used,” he says. “[But] it’s clearly going to drive an increase in more sophisticated scam attempts.”

Steve Cornwell, head of fraud risk at high street lender TSB, says the rising sophistication of the technology was a major worry for banks.

“If you think of the way Generative AI is coming along, how long [is it] before that AI solution could have a real-time conversation with you [using] a synthetic voice?” he says.

Figures from banking industry trade body UK Finance show a welcome trend, with fraud losses falling by 8 per cent year on year in 2022. 

But one senior politician who did not wish to be named says that increased adoption of AI — OpenAI’s ChatGPT reached around 100mn monthly users in two months — could reverse that trend.

“Scammers are very well financed and entrepreneurial,” the person says. “That’s the thing I’m concerned about.”

Data from Cifas, a not-for-profit fraud prevention service in the UK, also gives cause for concern. While data from 2022 shows identity fraud rose by nearly a quarter, reports of AI tools being used to try and fool banks’ systems increased by 84 per cent.

“We’re seeing an increased use of deepfake images, videos and audio being used during application processes, along with synthetic identities being identified as a result of ‘liveness’ checks that are now being carried out at the application stage,” warns Stephen Dalton, director of intelligence at Cifas.

Speaking at Davos on Wednesday, Mary Callahan Erdoes, JPMorgan’s head of asset and wealth management, said the use of AI by cyber criminals was a big concern. The bank spent $15bn on technology annually in recent years and employed 62,000 technologists, with many focused solely on combating the rise in cyber crime.

“The fraudsters get smarter, savvier, quicker, more devious, more mischievous,” she added.

PwC and Stop Scams also identified artificially generated videos, better known as deepfakes, as a major risk. The technology, which only emerged in 2019, has rapidly advanced, says Henry Ajder, an expert on AI-generated media, who has advised companies including Meta, Adobe and EY.

How the scourge spreads

Financial institutions have long criticised social media platforms as vectors for fraud. Last summer, a deepfake of money saving expert Martin Lewis and X owner Elon Musk spread across social media, promoting a product it referred to as “Quantum AI”.

Lewis himself took to the X platform, formerly called Twitter, in July to warn about the scam. Some of the videos, aimed at a British audience, featured apparent BBC broadcasts, which were deepfakes of prime minister Rishi Sunak extolling the benefits of Quantum AI. 

While a number of the videos have been removed or are inactive, other accounts simply copy and paste the same material.

The AI is not perfect. In one video which has now been removed, the purported Sunak stumbles over the pronunciation of words like “provided”.

And despite the high-tech name, the operation is surprisingly manual. Links from the deepfakes lead people to hand over their telephone numbers. Call centre operatives then take over, persuading people to hand over money.

Nevertheless, West emphasises that for criminals, scams are a volume game, and AI can tip the balance in enough cases to make it believable.

“Making content more believable — and convincing just a small percentage more people — can have a big pay-off for the fraudster,” he says.

One such case was a former medical assistant in the US, who fell victim to an investment scammer on X who used AI to impersonate Elon Musk.

“This started back in March, and we only became aware in August, after she had already taken very large sums out of her retirement account to try to pay [the investment] account,” says one family member.

While the process began using direct messages, the criminal also used a filter to take on Musk’s appearance, video calling the victim to convince her to hand over almost $400,000 which she would supposedly invest in X.

Meanwhile, on Alphabet’s YouTube, a spate of fake bitcoin giveaways featuring an AI-generated Michael Saylor led the former MicroStrategy chief executive to release a warning on X. “Be careful out there, and remember there is no such thing as a free lunch.” The deepfakes, posted by a host of accounts which have since been banned, were labelled as “live” videos to make them more believable.

Ajder says platforms have taken steps to fight back against the increasing flow of AI-generated content. In September, TikTok announced users would be required to label AI-generated content, while Google’s DeepMind announced a watermark for AI images in August. 

But Ajder was also wary of the record of social media companies, which have often implemented apparently clear policies in a piecemeal fashion. A lack of resources leads to ineffective enforcement, he says. 

The UK government’s stance on AI and fraud has been mixed. In a speech in July, Financial Conduct Authority chief executive Nikhil Rathi mentioned the potential impact of AI on “cyber fraud, cyber attacks and identity fraud” in a speech on regulating new technologies. At a Treasury select committee hearing in December, Rathi also warned that criminals were “making unfettered use of AI” to manipulate markets.

The FCA says that “as AI is further adopted, investment in fraud prevention and operational and cyber resilience will have to speed up”. 

But the government did not explicitly mention the technology in its anti-fraud strategy last May or in a voluntary “online fraud charter” for Big Tech platforms revealed in November.

Read full report: https://www.ft.com/content/beea7f8a-2fa9-4b63-a542-88be231b0266