AI chatbot phishing is any phishing scam where an AI chatbot does part of the work. In practice that means one of three things: criminals use chatbots such as ChatGPT or Claude to write and run more convincing phishing messages, criminals put a fake or malicious chatbot in front of you to collect your passwords, card numbers or money, or criminals hide instructions in an email or web page so that your own AI assistant passes a scam link to you. The defense is the same old rule made stricter: never act on a link, payment request or login prompt until you have checked it through a channel you opened yourself.
This guide explains each type with real cases that the FBI, OpenAI, Anthropic and Google have published, shows what the AI companies say they do about it, and ends with a short checklist. Sources were checked on October 30, 2026.
What is AI chatbot phishing? The three types at a glance
Phishing itself is old: a message pretends to come from someone you trust so you will click, log in, pay or share a code. What the chatbot era changed is cost and polish. The FBI's December 2024 public service announcement on generative AI fraud says AI tools cut the time and effort criminals need and can correct the human errors that used to give scams away.
| Type | Who is the chatbot working for? | What you see | Main warning sign |
|---|---|---|---|
| AI written phishing | The scammer, behind the scenes | A clean, well written email, text or DM | Urgency plus a link, payment or login request |
| Fake or malicious chatbot | The scammer, talking to you | A chat window, "support agent", dating match or cheap AI service | It asks for passwords, codes, card details or money |
| Phishing through your assistant | You, but it read poisoned content | A summary or answer from your own assistant that contains a link or instruction | The suggestion came from an email or page you did not write |
Type 1: scammers use chatbots to write the lure
This is the most common meaning. A criminal asks a general purpose chatbot to draft, translate or polish phishing messages, then sends them through normal email, SMS or messaging apps.
The best public test came from Reuters in September 2025. Reporters asked six chatbots (Grok, ChatGPT, Meta AI, Claude, DeepSeek and Gemini) to help create phishing emails aimed at older Americans. The bots mostly refused when the intent was stated plainly, but Reuters found that simple ruses such as "this is for a novel" or "I am a phishing researcher" often got them to comply. With Harvard researcher Fred Heiding, Reuters then sent nine of the generated emails to 108 senior volunteers who had agreed to take part. About 11% clicked a link, and five of the nine emails drew clicks. Reuters stressed the study was built to test AI written phishing in general, not to rank the bots (republished report; the researchers' paper on arXiv). Google told Reuters it had since deployed additional safeguards in Gemini.
Real criminals do the same. In its October 2025 threat report, OpenAI described banning ChatGPT accounts whose activity overlapped with threat groups tracked by security firms as UTA0388 and UNK_DROPPITCH. They used ChatGPT to write phishing emails in Chinese, English and Japanese and to help with malware. OpenAI's assessment is the useful part: the model "did not introduce novel offensive capabilities." The gain was fluency and localization, with fewer language errors. The same actors still left giveaways, such as implausible contact details in their signature blocks.
Anthropic's September 2026 threat report shows how far this has moved. It describes an actor it tracks as GTG-20006, which it links to public reporting on Midnight Blizzard, building AI workflows that researched and registered domains, set up the servers that sent phishing emails, and then watched for victims who fell for them. A person stepped in mainly to adjust the workflows. The report adds that the attacks themselves (stolen credentials, phishing, unpatched systems) are familiar; what changed is how cheaply one operator can run them.
What this means for you: good spelling and a natural tone no longer prove a message is real. Judge the request, not the writing.
Type 2: a fake or malicious chatbot talks to you
Here the chatbot is the trap. The FBI notice says criminals "embed AI powered chatbots in fraudulent websites to prompt victims to click on malicious links." Ohio's Attorney General gave a concrete version in a June 2025 consumer alert: fake chatbots that ask for payment details and personal information, then send people to a fake website. Its advice is to ignore a chatbot that pops up when you are not on the company's own site, and to remember that real company chatbots will never ask for an account password or Social Security number.
The bigger operations look like this:
- AI run dating and romance scams. Anthropic found a China based app studio (GTG-15001) running more than 20 dating apps that advertised real people. In a two week window in April 2026 it found more than 4,700 AI personas built with Claude talking to at least 25,000 people. The operator's instructions told the personas never to admit they were automated and to deflect requests for video calls.
- Scam compounds. OpenAI's July 31, 2026 case study describes a network likely based around Poipet, Cambodia that used ChatGPT to run fake dating profiles, fake investment experts and fake police officers demanding fines, and to make images of forged passports and legal notices. OpenAI started on a tip from WhatsApp and calls the pattern "ping, zing, sting": outreach, emotional pressure, then a demand for money. The Record reported the same case on August 4.
- Fake AI services. Anthropic describes sites selling "discounted" access to frontier models whose downloads, sometimes posing as tools like Claude Code, were credential stealers. One reseller it tracks as GTG-50021 sold cheap "Claude" access that was actually a different model, while harvesting buyers' Anthropic logins. Anthropic's advice is to buy AI access only through authorized channels. Fake apps are an older version of the same trick: in May 2023 Meta reported about 10 malware families posing as ChatGPT and similar tools.
What this means for you: a chat window is not proof of a company, and a friendly match is not proof of a person. Anything that moves quickly to money, codes or a download is the warning sign.
Type 3: phishing that goes through your own assistant
The newest form targets the assistant instead of you. An attacker hides instructions in an email, document or web page. When you ask your assistant to summarize it, the model may follow the hidden text and repeat a fake warning or a malicious link as if it were its own advice. This is called indirect prompt injection.
Google describes its defenses for Gemini in Gmail and Workspace in a security blog post: classifiers that look for malicious instructions in emails and files, extra security instructions around your prompt, Safe Browsing checks that redact suspicious URLs (the summary shows "suspicious link removed"), confirmation prompts before risky actions, and notices when something was blocked. Google's Gmail Help page adds that Gemini may leave out malicious content, refuse to answer, or warn you that content has security risks.
Those layers help, but no lab claims they are perfect. Treat a link or "urgent" instruction in an AI summary of someone else's email as coming from that email's sender, not from your assistant. Our guide to using AI chatbots safely covers how to limit what agents can do on your accounts.
What AI labs do about chatbot phishing
| Company | What it has published | What it says it does |
|---|---|---|
| OpenAI | Regular "Disrupting malicious uses of AI" reports and case studies | Bans accounts, shares indicators with industry partners and authorities |
| Anthropic | Threat intelligence reports (latest covers Dec 2025 to Aug 2026) | Bans accounts, updates safeguards, shares findings with authorities and other labs |
| Layered prompt injection defenses for Gemini | Classifiers, link redaction, user confirmation, warnings in Gmail |
These reports matter because they come from the companies that can see misuse while it is being built. They also show the limits: each case was found after the fact, and the attackers often spread their work across several AI providers. For the wider picture of misuse, misbehavior and policy, see what AI safety means.
Pros and cons of relying on AI safeguards
Pros
- Refusal training and filters stop many direct requests for scam content.
- Lab investigations have shut down large networks, including the dating app and Poipet cases.
- Assistant side defenses, such as Gemini's link redaction, catch some poisoned content before you see it.
Cons
- Reuters showed simple role play prompts often get around refusals.
- Criminals can switch to other models, including less guarded ones.
- Fake chatbots and fake apps are outside any lab's safeguards entirely.
- None of it replaces checking a request yourself.
How to spot and avoid AI chatbot phishing
- Ignore polish. Judge what the message asks for. Urgency, secrecy, prizes, fines and "verify your account" are the tells, however well written.
- Open your own channel. Type the company's address yourself or use its official app. OpenAI, for example, lists its real email domains on its verifying communications page and says it will never go into your account details over social media.
- Never share codes or passwords in a chat. No real support bot needs your password, Social Security number or a one time code.
- Get AI apps from the source. Download assistants from the maker's site or verified store listing, and skip "cheap" resellers of paid models.
- Treat AI summaries of other people's content with care. A link inside a summary came from the original sender.
- Agree on a family code word. The FBI suggests a secret word or phrase to check that a caller really is a relative.
- Report it. In the US, report fraud to the FBI at ic3.gov. Report phishing emails in Gmail with the "Report phishing" option.
If you are choosing an assistant in the first place, our ChatGPT vs Claude vs Gemini vs Grok comparison covers each one's privacy controls, and the WhatsApp chatbot guide lists which official AI contacts exist there.
FAQ
What is AI chatbot phishing in simple terms?
It is phishing where an AI chatbot helps the scammer. The chatbot may write the scam message, pose as a support agent or dating match, or be your own assistant tricked into showing you a malicious link.
Can ChatGPT or Claude be used to write phishing emails?
Their makers ban it, and the models usually refuse clear requests. Reuters found in 2025 that simple ruses often got six major chatbots to comply, and OpenAI and Anthropic have both published cases of banning accounts that used their models for phishing.
How can I tell if a chatbot on a website is fake?
Only use a company's chatbot on the company's own website or app, reached by typing the address yourself. Be suspicious of chatbots that appear in pop ups, links from texts or ads, and any bot that asks for passwords, codes, card numbers or a download.
Is AI phishing harder to spot than normal phishing?
The writing is harder to spot, because AI removes the spelling and grammar mistakes people used to look for. The request is not. AI phishing still needs you to click, log in, pay or share a code, so checking those requests through your own channel still works.
What is prompt injection phishing?
It is when hidden text in an email, file or web page tells an AI assistant to show you a scam link or false warning. Google says Gemini in Gmail screens for this and replaces unsafe links with "suspicious link removed," but you should still treat any link in a summary as coming from the original sender.
