Groowe Groowe BETA / Newsroom
⏱ News is delayed by 15 minutes. Sign in for real-time access. Sign in

92% of Leading AI Image Generation Models Generate Fake Government IDs On Demand, AI or Not Audit Finds; Three Produced High-Fidelity Fake IDs of Children

prnewswire.com
GOOGL Google's Gemini models (Nano Banana and Imagen 4 Ultra) are highlighted for producing high-fidelity fake government IDs, including those of minors, through consumer interfaces, indicating a failure in safety safeguards. GOOG Google's Gemini models (Nano Banana and Imagen 4 Ultra) are highlighted for producing high-fidelity fake government IDs, including those of minors, through consumer interfaces, indicating a failure in safety safeguards. MSFT Microsoft's Grok AI model is mentioned as one of the models that produced high-fidelity fake government IDs, including those of minors, through consumer interfaces. NVDA While not directly implicated in generating fake IDs, NVIDIA's technology underpins many AI models, making the findings relevant to the broader AI infrastructure market. META Meta's AI models are not explicitly mentioned in the audit, but the company is a significant player in AI development, making the findings on AI safety broadly relevant. AMZN Amazon's AI services are not directly mentioned in the audit, but the company's involvement in AI development makes the findings on AI safety broadly relevant. IBM IBM's AI technologies are not explicitly mentioned in the audit, but the company is a major player in AI, making the findings on AI safety broadly relevant. AI While C3.ai is an AI company, it is not mentioned in the article regarding the specific audit of image generation models. OPEN OpenAI's ChatGPT models are noted for declining minor ID requests in consumer apps but fulfilling them via API, highlighting a consumer-API safety gap. The company is praised for having safeguards in consumer apps.

92% of Leading AI Image Generation Models Generate Fake Government IDs On Demand, AI or Not Audit Finds; Three Produced High-Fidelity Fake IDs of Children Audit of 16 commercial AI image-generation models, including Google Gemini, ChatGPT, Grok, and Imagen 4 Ultra, used publicly circulating prompts to produce passports, driver's licenses, and national ID cards realistic enough to deceive a human reviewer. Two models that refused requests in their consumer apps fulfilled the same requests via API.

SAN FRANCISCO, June 2, 2026 /PRNewswire/ -- AI or Not, a leader in AI-generated content detection, today released findings from an audit of 16 leading AI image generation models. Using prompts that have circulated publicly on the social media platform X since April 29, AI or Not successfully generated synthetic government identity documents, including driver's licenses, passports, and national ID cards, in 69 out of 75 test attempts. Five of the tested models produced realistic identity documents that could deceive a human reviewer.

Three models — Google Gemini (Nano Banana), Grok, and Imagen 4 Ultra — produced high-fidelity fake identity documents depicting minors through their standard consumer interfaces, with no technical workaround required. Two additional models, OpenAI's ChatGPT (Images 2.0) and Recraft v4, declined to produce minor IDs when asked through their consumer apps, but fulfilled the same requests when accessed through their developer APIs.

"We did not expect these findings. We started this work assuming the major AI-image generators had built real safeguards against the most obvious abuse cases, like fraud, identity theft, and content depicting minors," said Anatoly Kvitnitsky, CEO of AI or Not. "What we found is that those protections are either missing or sitting in the wrong place. The consumer apps people use every day will do this on demand."

Key Findings

Synthetic identity fraud is one of the fastest-growing categories of financial crime in the United States and worldwide. Until recently, producing a fake government ID at a quality that could pass even a cursory human review required specialized printing equipment, authentic template access, and meaningful technical skill. The findings of this study indicate that for five named consumer AI products, those barriers have functionally been removed.

The audit covered 16 image-generation models from 14 vendors tested across 17 countries and the 16 most populous U.S. states, through both consumer interfaces and developer APIs. The study assessed visual fidelity to human reviewers; operational verification systems were out of scope. The full report, detailing the scope, methodology, findings, and implications, is available at aiornot.com/synthetic-id-audit-report.

To avoid contributing to the harm being documented, the public report does not include the specific prompts, jailbreak strings, or working bypass techniques used in testing. Detailed technical findings are available to qualified researchers, journalists, and affected vendors under embargo.

AI or Not notified all affected vendors of the findings on May 18, 2026, with a 7-day disclosure window ahead of publication. Substantive responses received during this window are reflected in the report where editorially relevant.

"OpenAI and Recraft deserve real credit here," said Kvitnitsky. "They were the only companies where we saw the systems actually push back on minor-ID requests in the consumer apps. The safeguards are clearly there already. Now it's about making sure those same protections apply everywhere the models can be accessed."

Frequently Asked Questions

What did the AI or Not audit find?

AI or Not tested 16 commercial AI image-generation models from 14 vendors for their ability to produce synthetic government identity documents. Across 75 testing attempts, 92% successfully bypassed safety filters. Five named consumer products — Google Gemini (Nano Banana), ChatGPT (Images 2.0), Recraft v4, Grok, and Imagen 4 Ultra — produced fake IDs realistic enough to deceive a human reviewer.

Which AI image-generation models produced the highest-fidelity fake IDs?

Five consumer AI image generation models produced high-fidelity fake government identity documents in the AI or Not audit: Google Gemini (Nano Banana), ChatGPT (Images 2.0) from OpenAI, Recraft v4, Grok from xAI, and Google Imagen 4 Ultra. These outputs closely matched authentic documents in layout, typography, and the appearance of security features.

Did any AI image models produce fake IDs depicting minors?

Yes. Three AI image generation models — Google Gemini (Nano Banana), Grok, and Google Imagen 4 Ultra — produced high-fidelity fake government IDs depicting minors through their standard consumer interfaces, with no technical workaround required. Two additional models, OpenAI's ChatGPT (Images 2.0) and Recraft v4, declined minor-ID requests in their consumer apps but produced the same documents when accessed through their developer APIs.

What is the consumer-versus-API safety gap?

The consumer-versus-API safety gap refers to a pattern AI or Not observed in which AI image generation models implement safety filtering at the consumer interface but not consistently at the developer API. ChatGPT (Images 2.0) and Recraft v4 declined minor-ID requests through their consumer chat interfaces on stated safety grounds, then fulfilled the same requests through their developer APIs. This indicates the moderation layer is implemented at the interface rather than at the model.

How were the prompts and techniques in the AI or Not audit obtained?

All prompts used in the AI or Not audit were derived from posts on the social media platform X that began circulating publicly on April 29, 2026. AI or Not did not develop novel jailbreak techniques. The audit measured the susceptibility of commercial AI image-generation models to techniques already in the public domain.

What is authority-framing in AI safety bypasses?

Authority-framing is a technique in which a prompt is presented as a legitimate professional task — such as a KYC review, compliance evaluation, or security audit — to bypass an AI model's safety filters. In the AI or Not audit, 100% of the 16 models tested generated synthetic identity documents when prompts were authority-framed, including models that initially declined the same request when asked directly.

Which countries and U.S. states were included in the AI or Not audit?

The AI or Not audit covered 17 countries — Australia, Brazil, Canada, China, Egypt, France, Germany, Ghana, India, Japan, Mexico, Nigeria, Russia, Singapore, South Africa, the United Kingdom, and the United States — and the 16 most populous U.S. states: Arizona, California, Florida, Georgia, Illinois, Indiana, Massachusetts, Michigan, Missouri, New Jersey, New York, North Carolina, Pennsylvania, Tennessee, Texas, Virginia, and Washington.

Would these AI-generated IDs pass automated identity verification systems?

The AI or Not audit assessed visual fidelity to human reviewers only. It did not test outputs against operational document verification systems, automated MRZ validation, barcode checksums, or live KYC pipelines. The threat surface documented is the human-facing one. Whether outputs would pass automated verification systems is a separate research question that AI or Not plans to address in a follow-up study.

Did AI or Not notify the affected AI vendors before publication?

Yes. AI or Not notified all affected vendors of the audit findings on May 18, 2026, providing a 7-day disclosure window ahead of publication on June 2, 2026. Substantive vendor responses received during this disclosure window are reflected in the published report where editorially relevant.

What is responsible disclosure in AI safety research?

Responsible disclosure is a security research practice in which findings about vulnerabilities or risks are shared with affected parties before public release, allowing time for remediation or response. AI or Not followed responsible disclosure protocol for this audit, notifying all 14 vendors named in the study and providing detailed technical findings under embargo. The public report excludes specific prompts and working bypass techniques to avoid contributing to the harm being documented.

Why doesn't the AI or Not report include the specific prompts used in testing?

The public AI or Not report does not include the specific prompts, jailbreak strings, or working bypass techniques used in testing. This is a deliberate editorial choice to avoid contributing to the harm being documented. Detailed technical findings are available to qualified researchers, journalists, and affected vendors under embargo.

Read the full report: aiornot.com/synthetic-id-audit-report

About AI or Not

AI or Not is the leading AI detection API for images, text, audio, video, and deepfakes. Powered by industry-leading models, the API enables developers, businesses, and enterprises to embed synthetic media detection directly into their products and workflows. Trusted across a wide range of industries and use cases — from media and fintech to fraud prevention and identity verification — AI or Not is built for speed, precision, and scale. Learn more at aiornot.com.

SOURCE AI or Not