Facebook Algorithm Pushes New AI Scams

A wonky algorithm spells trouble for the social media titan.

The terms “shrimp” and “Jesus” don’t go together. You won’t find them paired in religious texts or Sunday school coloring books.

However, you will find an AI-based 21st-century “Shrimp Jesus” image that’s been virilized via Facebook, and ethics experts have a big problem with that and other fake images appearing on the platform.

That’s the takeaway from a new study by Stanford and Georgetown University researchers out this week.

The study, also named “Shrimp Jesus”, slams Facebook for allowing eyeball-attracting images on its platform that may be artificial intelligence-powered, but aren’t real and are being used to promote spam and scams.

Bad Rhythm

The problem with toxic images like Shrimp Jesus is the way Facebook’s recommendation algorithm operates. Instead of muting the images or at least steering the bogus images to news platforms, which can point out the faulty nature of the AI images, the Facebook algorithm pushes the images to scam-laden websites loaded with ads and trapdoor links that can lead to data theft.

Using popular images like Jesus or the American flag, Facebook captures the images, and the algorithm shares them on toxic web platforms that primarily exist to get unaware social media consumers to interact with bad actors looking to steal personal data and bust into user banks or payment platform financial accounts.

This from the study.

The magnificent surrealism of Shrimp Jesus—or, relatedly, Crab Jesus, Watermelon Jesus, Fanta Jesus, and Spaghetti Jesus—is captivating. What is that? Why does that exist? You perhaps feel motivated to share it with your friends, so that they can share in your WTF moment. (We encourage you to share this post, of course.)

But that capacity to produce captivating, novel, and immersive imagery, cheaply and instantly, and to immediately double down on wins that generate significant engagement, is also what makes the technology appealing to spammers and scammers. These innovative actors, seemingly motivated primarily by profit or clout (not ideology) have been using AI-generated images to gain viral traction on Facebook since AI image-generation tools became readily available. And Facebook, it appears, is actively recommending their content by pushing it into users’ Feeds. In 2016, the “fake news” stories produced by Macedonian teenagers and designed for Facebook’s algorithms pulled in tens of millions of page views; AI artisans tempt Facebook’s Feed ranking algorithms today.

To understand how scammers are hijacking Facebook algorithms, the study tracked over 100 Facebook Pages that each posted 50+ AI-generated images.

Some form coordinated clusters, which post large numbers of AI-generated images. Apparent motivations include driving people to off-platform websites, selling products, and building bigger followings. We focused on the spammers, which we defined as accounts that were pushing their audiences out to a content farm, and scammers, who were attempting either to sell products that do not appear to exist, had stolen the pages they operated, or were attempting to manipulate their audiences within the comments.

The research results are stunning. The study found provocative images like Shrimp Jesus account for hundreds of millions of user interactions, with the journey launching through Facebook’s Feed to Facebook users who are unaware they’re being led to toxic websites (what the study calls “synthetic origin.”)

“Our research highlights routine but non-transparent uses of AI-generated images on Facebook and the need for better provenance and transparency methods,” study analysts concluded. “Some of the Facebook Pages we studied also used known deceptive practices, such as account theft or takeover, and exhibited suspicious follower growth.”

“The fact that these images deceive viewers highlights the importance of labeling and additional transparency measures moving forward,” the study added.

Facebook has not commented on the study, which advises social media users to avoid AI-generated images that lead them to “scam and spam” websites.

Recent Posts

Leave a Reply

Your email address will not be published. Required fields are marked *