Matt Burgess, Wired (Apple News+ link):
Apple and Google have been ordered to take down apps that can “nudify” or “undress” people and told that they must stop profiting from the harmful technology, according to cease-and-desist letters sent to the companies seen by WIRED.
On Thursday, San Francisco city attorney David Chiu sent legal notices to Apple and Google demanding that they remove from their app stores 13 face-swapping apps, which allow users to create AI-generated nonconsensual nude images. The letters say the Silicon Valley giants should stop “aiding and abetting” the sale of explicit deepfake images and “sever” business relationships with the app developers. […]
The city attorney, whose office previously took legal action against 16 popular deepfake websites, says Apple and Google have likely “made millions of dollars in fees” from apps that offer nudification, and they should improve their moderation processes to stop them appearing in their stores in the first place.
One might be tempted to ask why these apps were approved for sale in the first place. We can simply chalk it up to ineptitude, indifference, and income. The app review process is notoriously porous, most reviewers are treadmilling a daily quota, and none of Apple, Google, or developers want to give up the potentially enormous revenue.
I’m sure hardline “free speech absolutists” like Elon Musk will object, but I’d love to see this picked up by cities and states across the country.
(No word on whether the demand included Musk’s Grok, which elicited (unheeded) calls for a ban earlier this year for generating deepfakes.)
Apple’s and Google’s statements are models of deflection. First, let’s establish the baseline expectation for this content:
Both Apple and Google have developer policies that prohibit pornography, abuse, and harassment on their platforms. They have previously removed dozens of nudify and deepfake apps, after reports by researchers and journalists.
Here’s Google’s response:
Google spokesperson Dan Jackson tells WIRED that the company has deleted “hundreds” of apps with nudifying features for policy violations, including the five Android apps flagged by Chiu’s office, among other steps to restrict access to them.
“Google Play does not allow apps that contain sexual content, and we continually take proactive steps to detect and remove apps with harmful content,” Jackson says in a statement. “When violations are reported to us, we investigate and take swift action, which in the case of these apps has included suspending hundreds of violating apps and restricting related search terms like ‘nudify’ on our store."
Here’s Apple’s:
Apple tells WIRED that developers are ultimately responsible for the content of their apps, and those who fail to follow the rules will have their apps removed. The company also says users can report potentially problematic apps themselves. Adam Dema, an Apple spokesperson, tells WIRED in a statement that the company has removed three of the apps flagged by San Francisco city attorney and is “in the process of terminating their developer accounts from our program.” The other four developers must “address policy violations or risk being removed as well,” the company says.
“The App Store was designed to be a safe and trusted place for users, and we have always strictly prohibited apps designed to generate, distribute, or consume pornography," Dema says in the statement. “‘Nudification’ apps are against our App Review Guidelines and we have proactively rejected many of these apps and removed many others, including when users have flagged them via our reporting tools.”
Neither company takes responsibility for allowing “many” to “hundreds” of apps that violate their guidelines onto their stores. Instead of attributing the failures to lapses in their review processes, with promises to improve, they assert the problem is with developers themselves. It’s not our fault developers violated our very clear and precise rules!, I can hear them whine. Oooh, those wily developers!
I find Apple’s “developers are ultimately responsible” to be especially weak, running counter to its own argument that App Review’s vigilance is necessary to ensure a safe storefront. Instead, it places the onus to police content on developers (and customers), as though the review process Apple regularly touts for its fraud prevention is simply incapable of identifying and rejecting these apps.
Some customers want to violate women, and developers want to profit from it. Apple’s job is to short-circuit that by preventing such apps from ever getting past App Review—and immediately banning developers who ship innocuous-seeming apps that mutate after approval. As often as Apple might claim success at this, its failure rate remains too high.
(I fixate on Apple here because I’m most familiar with its App Store and App Review teams and policies, but you can apply this critique in equal measure to Google.)
Yes, it’s true that developers pull all sorts of shenanigans to get their apps through the system. And yes, the porous app review process is riddled with inconsistently applied rules and reviewers more interested in hitting their numbers than in getting it “right.”
Ultimately, this is an Apple (and Google) app review problem. They need to do better.




