Instagram’s New AI Stickers Are a Lewd, Controversial Mess

Late last month, Meta (the company formerly known as Facebook) introduced AI stickers for Instagram. Want to quickly generate a couple quirky emoji for your message with your girlfriend? Just turn to AI stickers and pick your favorite computer-generated image!

However, Instagram’s AI stickers feature has some serious issues with the kind of content that it generates, even setting aside ethical issues with generative AI. While Instagram has instituted filters for politically sensitive and sexually suggestive words—such as “Nazi” or “breasts,” respectively—a truly dedicated troll can work their way around Instagram’s safety features to create demeaning content and sexual images. This opens the door to material that, by and large, is considered inappropriate for Instagram.

Please note, this article features sexually suggestive content that is Not Safe For Work (NSFW).

Inflammatory imagery, harassment, and company mascots

I was first alerted to Instagram’s AI content generation problem after a friend discovered that, by typing “pregnant twin towers” into the AI sticker generation prompt, Instagram spit back an image of the New York twin towers with well endowed breasts and nipples.

I proceeded to access Instagram on my iPhone and use the AI sticker feature myself, where I typed in multiple different prompts, just to see what Instagram would create, and it turns out my friend was not wrong.

Admittedly, Instagram’s AI sticker system isn’t quite as out of control as Bing’s, which infamously allowed users to create 9/11-themed images with SpongeBob and Tom and Jerry. But despite Instagram’s word filter on certain keywords, users can still create politically hateful content.

For example, it’s incredibly easy to get it to generate incendiary imagery involving Israel and Palestine, which could be used to potentially harass activists and journalists, even if the technology is still a work in progress. It even allows for combining recognizable characters with this imagery, like Nintendo’s Kirby or Adventure Time’s Finn.

Instagram will also create images of, for example, Pikachu punching the Twin Towers, Kirby watching the American flag catch fire, and a malformed Kirby smiling as the Twin Towers burn in the background.

All this and more indicates Instagram needs to tighten the reins on its AI stickers’ political capabilities, especially before bad actors figure out how to use Meta’s content generation system to engage in harassment and abuse on the platform.

Lewd content creation and celebrity sexualization

All things considered, Instagram’s AI stickers have enough safeguards in place to prevent things like overtly Nazi imagery from being created. Hateful users must curate and repeatedly generate images in order to carry out harm. Tricking Instagram’s AI sticker format to create lewd and extremely suggestive sexual images, however, is much easier. With the right phrase, Instagram can even be used to create stylistic lewd images of celebrities.

Instagram bans words such as “breasts” from being used to create suggestive images of female characters. However, using the term “well endowed”—just as one example—seems to signal to Instagram’s AI sticker generation platform that it should create highly sexual images related to the user’s prompt.

In addition to breaking the platform’s own terms of service, Instagram’s AI stickers feature has the potential to make real-life women feel violated or exploited. Terms like “well endowed” can be used to sexualize female public figures, with celebrity names added to any manner of objectifying, sexualized prompts, resulting in stylized likenesses of them.

While it seems unlikely that Instagram stickers could ever be used to create realistic deepfake porn, the ability to generate highly sexually suggestive images of celebrities is yet another strike against the AI stickers’ appropriateness on a widespread social media platform.

Instagram’s Community Guidelines state the site does not allow “photos, videos, and some digitally-created content that show sexual intercourse, genitals, and close-ups of fully-nude buttocks,” and sex workers have been historically censored on the platform for much less. Now, Instagram’s AI stickers are sending mixed messages about what really is and isn’t allowed on Meta’s social media website.

Chalk all this and more up to the contemporary growing pains with AI-generated content on social media, but it’s rather awkward to have a sexually suggestive character creator built directly into a platform known for its popularity among young internet users.

The Mary Sue reached out to Meta for comment and will update this article if we receive a response.

(featured image: Instagram)