not.bot

Problems: Deepfakes and Inauthentic Content

This document opens the not.bot™ Problems series, which presents public evidence for the problems not.bot exists to solve. This one covers fabricated content: cloned voices, synthetic video, and impersonation at scale. Every figure below carries its source and date. The incidents are on the public record.

Making a fake costs almost nothing

Researchers at the Oxford Internet Institute analyzed 35,000 deepfake image generators available for public download. Building a new one requires as few as 20 photographs of the target, 24 GB of video memory, and about 15 minutes on a consumer computer ("Deepfakes on Demand," FAccT 2025). Microsoft Research demonstrated in January 2023 that its VALL-E system synthesizes personalized speech from a 3-second recording of a voice it has never heard. OpenAI built a tool that clones a voice from 15 seconds of audio, then declined to release it, judging the risk of misuse too high (March 2024). When the company that built the tool will not ship it, the threat assessment is settled.

The capability is cheap enough to demonstrate as theater. At Berkshire Hathaway's May 2026 shareholder event, Greg Abel played a video deepfake of Warren Buffett with Buffett sitting in the audience. "That was done with zero input from Warren," Abel told shareholders. "We were able to obtain that with information that's out there and replicate those actions and that voice" (Money, May 6, 2026).

Documented incidents

A US$25.6 million wire through a deepfaked video call. In January 2024, a finance employee at Arup, the British engineering firm, joined a video call with the company's CFO and several colleagues. Every other participant on the call was a deepfake recreation of a real coworker. The employee made 15 transfers totaling HK$200 million (US$25.6 million) to five bank accounts (CNN, February 4 and May 16, 2024; South China Morning Post, May 2024). The funds were not recovered.

Voice clones against bank call centers. In spring 2023, a software-generated copy of a Florida investor's voice called his Bank of America representative and tried to redirect a large transfer (The New York Times, August 30, 2023). The banker caught it. The damage arrived anyway: she stopped trusting calls and emails from the real customer, and it took ten days and an in-person visit to restore the relationship. A fraud that fails still breaks the trust the relationship ran on.

A coordinated campaign impersonating real doctors. The New York Times documented (September 5, 2025) a global operation that hijacked the likenesses of physicians at UCSF, Stanford, Harvard, and other institutions, using AI video and voice to sell unproven health products. One Stanford scientist's face fronted at least six YouTube channels carrying hundreds of AI-narrated videos. One physician's fake was convincing enough that her own mother believed it.

A single fake ad with 35 million views. CNN's chief medical correspondent Sanjay Gupta described fake videos of himself selling cures for Alzheimer's disease and diabetes (CNN Podcasts, September 23, 2025). One had been viewed 35 million times. His own former professor was fooled. Victims paid hundreds of dollars for products that never arrived, then wrote to the real Gupta in anger and shame.

An €830,000 romance scam. A French interior designer lost €830,000 over 18 months to scammers using AI-generated images and messages of Brad Pitt (TF1, January 2025; Euronews, January 15, 2025). When her story aired she faced a wave of public ridicule, and the broadcaster withdrew the interview to protect her.

Sexualized fakes of a head of government. AI-generated lingerie images of Italian Prime Minister Giorgia Meloni circulated as authentic in May 2026 and drew public criticism of her before they were exposed as fakes (Forbes, May 6, 2026). Meloni's response named the asymmetry: "Deepfakes are a dangerous tool, because they can deceive, manipulate and strike anyone. I can defend myself. Many others cannot."

The scale

  • Deloitte's Center for Financial Services projects that generative AI will drive US fraud losses from US$12.3 billion in 2023 to US$40 billion by 2027 under its aggressive adoption scenario, and cites a 700% increase in fintech deepfake incidents during 2023 (May 2024).
  • The UK Home Office states that an estimated 8 million deepfakes were shared in 2025, up from 500,000 in 2023 (February 2026).
  • Scams of all kinds cost Americans US$158 billion a year, a figure that prompted the first coordinated US national anti-scam strategy (Aspen Institute National Task Force on Fraud and Scam Prevention, September 30, 2025).
  • Of 2,000 deepfake generator models analyzed in the Oxford study, 96% target identifiable women without any suggestion of consent, and 99% of sexual deepfakes target women and girls (FAccT 2025, citing Internet Matters, 2024).
  • The World Economic Forum's Cybercrime Atlas surveyed 25 face-swap and camera-injection tools selling for US$10 to US$3,000 and documented the chain that defeats remote identity checks: AI-generated identity documents, face-swapped video matching those documents, and camera injection feeding the synthetic video into live biometric verification (January 2026). Its summary: "identity itself has become synthetic, scalable and weaponizable."

Detection loses the arms race

Generators train against detectors. Deloitte describes deepfake systems built to keep checking and updating their ability to fool computer-based detection, and notes that for audio, "the technology industry is behind in developing tools to identify fake content" (May 2024). The World Economic Forum reaches the same end point: deepfake fraud "may never be fully eliminated," only "contained, deterred and made economically unviable" (January 2026).

Takedown fares no better. In the doctor-impersonation campaign, a Stanford team spent hours reporting fake videos and posting warnings under them; the warnings were deleted within a minute. One impersonated oncologist, after her reports and a legal letter went unanswered, paid US$260 to a takedown service that failed. The platform said it was unaware of the fakes until a reporter called (The New York Times, September 5, 2025). Measure any takedown clock against a fake that reached 35 million viewers.

Detection asks whether content looks fake, and that question gets harder with every model generation. The question that stays answerable is whether content carries proof of authenticity.

Regulation arrives, and creates demand for verification

  • The US TAKE IT DOWN Act, signed May 19, 2025, requires platforms to remove non-consensual intimate imagery, including AI "digital forgeries," within 48 hours of a valid request. FTC enforcement began May 2026, with penalties up to US$53,088 per violation.
  • Italy's Law 132/2025, in force October 10, 2025, is the first comprehensive national AI law in the EU. It criminalizes disseminating, without consent, AI-altered images, video, or voice capable of misleading as to authenticity and causing unjust harm, with penalties of one to five years.
  • 46 US states have laws addressing non-consensual intimate deepfakes, and 30 regulate election deepfakes (Public Citizen trackers, early 2026). The limits are visible too: a federal court struck down Hawaii's election-deepfake law on First Amendment grounds (D. Haw., January 30, 2026).
  • EU AI Act Article 50 applies from August 2, 2026: deployers must disclose deepfakes, and providers must mark synthetic content in machine-readable form.
  • ETSI TS 119 461, the EU identity-proofing standard under eIDAS 2.0, names "deep fakes" as a technique imposters use and requires accredited-lab testing against injection and presentation attacks before the end of 2026 (February 2025).
  • Denmark has proposed copyright-style rights in a person's own face, voice, and body, the first proposal of its kind in Europe (June 2025; before parliament as of June 2026).

These laws assign duties: remove within 48 hours, disclose synthetic content, test identity checks against injection attacks. None of them can prove a given piece of content authentic. Regulation raises the cost of fakery and creates compliance demand for verification infrastructure. The authenticity question stays open.

Who bears the cost

Financial institutions. Voice clones target call centers and the fraud trajectory points at US$40 billion by 2027 (Deloitte, May 2024). The losses come with liability fights: "customer relationships may be tested when determining whether a fraud loss is to be borne by customers or their financial institutions" (Deloitte, May 2024). A Federal Reserve survey of more than 360 US institutions found scams the most common fraud type and concern about mule-account activity up 12 points in a year (2024).

Enterprises. The Arup pattern is repeatable: contact through a messaging app, an urgent video call, a deepfaked executive, a wire. The identity checks that gate remote onboarding are themselves a target; the WEF documents the bypass chain end to end (January 2026).

Public figures, executives, and clinicians. Anyone whose voice and face exist in abundance online is raw material. The cost lands as reputation damage, lost audience trust, and takedown labor that does not end.

Women. 96% of deepfake generator models target identifiable women (Oxford Internet Institute, FAccT 2025). Non-consensual intimate imagery is the largest single category of deepfake harm, and the reason most of the new criminal statutes exist.

Platforms. Statutory removal duties now carry per-violation penalties and active FTC enforcement, against a moderation problem that defeats both automated detection and manual review.

Consumers. Romance scams, fake cures, products that never arrive. Victims report shame more than anger, and shame suppresses reporting, so official figures undercount the harm.

The deepest cost: authentic content loses its authority

A fake does damage twice. The first harm lands on whoever it defrauds or defames. The second lands on everything authentic, because every real recording now competes with the possibility that it is fake. In the documented incidents above, the people fooled included a physician's mother and a neurosurgeon's former professor: people with personal knowledge of the target. Gupta's conclusion from inside the problem: "all content, even good content, ultimately gets tainted by this... Nobody believes anything. Everybody's suspicious of each other. And unless you can touch the person or talk to them directly, they are also suspect, because it could be an AI impersonator" (CNN Podcasts, September 23, 2025).

Meloni offered the public a rule: "verify before believing, and believe before sharing" (Forbes, May 6, 2026). The rule is correct. Today the ordinary reader has no way to follow it.

What an adequate solution requires

The evidence defines the requirement set:

  1. Deterministic proof. A verifier needs a yes-or-no answer about authenticity. Probabilistic detection decays as generators improve; a cryptographic check does not.
  2. Binding to a human. Proof must tie content to an accountable person. Devices, accounts, and communication channels are all spoofable, as the Arup call showed.
  3. Survival through distribution. Proof must survive screenshots, re-encoding, cropping, and forwarding, because content travels stripped of its metadata.
  4. Verification open to anyone. Checking must be free and require no enrollment. The people who most need to verify are the audience, and the audience cannot be asked to subscribe first.
  5. Speed at distribution scale. A fake reaches 35 million viewers faster than any takedown process. Authenticity must be checkable at the moment of viewing, by the viewer.
  6. No new surveillance. An authenticity layer that tracks who signs, who verifies, and what they look at would trade one harm for another. The proof must work without anyone watching.

Content Provenance and Digital Signatures (Doc #2) describes how not.bot meets these requirements with visible cryptographic signatures (patent pending) that travel with the content itself.

View this page as Markdown