The fraudsters are accused of targeting hundreds of thousands of individuals with scams leveraging Gemini-branded websites.
Google frequently highlights how its generative AI products are being used to innovate, expand businesses, and benefit society—or so it claims. Naturally, AI is also being exploited for criminal purposes. Google has launched a new legal action against a Chinese group known as Outsider Enterprise, which is allegedly behind a large-scale AI-driven scam operation. Google states it is collaborating with law enforcement and mobile carriers to combat this.
Based on Google’s legal complaint, Outsider Enterprise operates via Telegram. The group provides phishing-as-a-service to individuals who may lack the technical expertise to create fraudulent websites and text campaigns independently. In its Telegram channels, Outsider Enterprise reportedly gave instructions on using Google’s Gemini AI to develop websites that mimic those of Google, YouTube, and government entities like New York’s E-ZPass. The group offered close to 300 scam templates.
Google reports that scams facilitated by Outsider Enterprise led to over 2.5 million text messages being sent to Android users. Approximately 55,000 of those messages occurred within a two-week span last month. Overall, Google has identified 9,000 fake websites and 1 million URLs linked to the scam network.
The text messages often referenced account issues or problems with a package delivery. When users clicked the links, they were directed to one of those fraudulent websites, crafted by Gemini to appear authentic. The cybercriminals used these sites to steal personal information and banking credentials. Google’s filing does not provide an estimate of the money stolen through Outsider Enterprise scams, but the blog post notes that hundreds of individuals have lost some amount of funds.
Google collaborated with AT&T, Verizon, and T-Mobile to block many of these malicious text messages, and Google notes that its on-device scam detection in Google Messages likely helped reduce the number of successful phishing attempts as well. This AI-powered feature reportedly blocks 10 billion scam texts each month, so it’s reasonable to assume it intercepted at least some Outsider Enterprise activity.
Legal measures for AI threats
Google has previously filed lawsuits against scammers, but this marks the first time it has directly targeted a group accused of using Gemini as part of its scams. Google discusses the security features integrated into Gemini whenever it announces a new model, but these can conflict with the broader need for chatbots to follow instructions and satisfy users. This results in thousands of scammers using Gemini to construct fake websites.
In addition to its civil lawsuit, Google is assisting the FBI’s cybercrime division with a parallel criminal investigation. However, the identities behind Outsider Enterprise remain unknown, and even if Google had names, there is limited recourse when the perpetrators are based in China. The company can target fraudulent domains and Telegram accounts in an effort to disrupt Outsider Enterprise’s operations, but the scams may simply evolve.
Google believes the AI era demands new law enforcement approaches, so it is using this opportunity to reiterate its public support for several pieces of legislation. The company has highlighted seven potential federal laws, including the National Strategy for Combating Scams Act, the Strategic Task Force on Scam Prevention Act, and the AI Plan Act.
Most of the legislation Google endorses calls for one or more federal law enforcement agencies to establish task forces aimed at countering the threat of AI-assisted scams and market manipulation. One law (the Artificial Intelligence Public Awareness and Education Campaign Act) focuses on enhancing the public’s ability to identify malicious uses of AI. However, the industry’s pursuit of human-like intelligence in AI systems will only make this content harder for people to detect, regardless of well-intentioned government legislation.