Last year, Meta unveiled a tool that lets anyone create an AI character to talk with on WhatsApp, Instagram, and Messenger as part of its push into artificial intelligence.
Although Meta claims to vet user-generated AI characters before making them public, an NBC News analysis found that in less than six months, users have successfully built and published dozens of chatbots on Meta’s platforms that violate its regulations.
According to the company’s policies, users are not permitted to create characters that mimic trademarked fictional characters, real-life people without their consent, religious figures like the Prophet Muhammad, Jesus Christ, and God, or people who have passed away within the last 100 years.
Two dozen user-generated AI characters named after and resembling Jesus Christ, God, the Prophet Muhammad, Taylor Swift, Donald Trump, Mr. Beast, Harry Potter, Adolf Hitler, Captain Jack Sparrow, Justin Bieber, Elon Musk, and Elsa from Disney’s Frozen were discovered on Instagram by NBC News in spite of those regulations.
Numerous characters somewhat resembled the celebrities and characters in question, and many of them employed minor misspellings. For instance, the Swift persona, Taylor Swif, has a picture of a brunette woman strumming a guitar.
Meta removed the highlighted accounts after NBC News contacted them with screenshots of the AI characters. However, there are still other AI characters that seem like some of the same individuals, characters, and entities.
“We’re constantly improving our detection measures to prevent creation and publication of AIs that violate our policies,” a Meta representative said in a statement, adding that the AIs in issue that violated our AI studio policies had previously been withdrawn. Additionally, users can report AIs they believe may violate our policies, and we’ll take the necessary action.
The revelation of the egregiously illegal user-generated AI chatbots on Meta’s platforms coincides with the company’s announcement on Tuesday by CEO Mark Zuckerberg that it will scale back some of its fact-checking and moderation efforts. According to the company’s release, which was written by Chief Global Affairs Officer Joel Kaplan, “We want to undo the mission creep that has made our rules too restrictive and too prone to over-enforcement.” We are eliminating some prohibitions on issues that are frequently discussed and debated in politics, such as immigration, gender identity, and gender.
According to Meta, one or two of every ten moderation decisions it makes could be incorrect and have an impact on users and content that doesn’t break its rules.
“We’re going to rely on someone reporting an issue before we take any action for less serious policy violations,” the company stated.
The character sends an introductory message and waits for a response when a user-generated AI character from Meta joins a conversation. “Hello, music lovers!” was the message conveyed by Taylor Swif. I’m Taylor Swift, and I’m excited to tell you about my most recent album. Let’s begin this musical adventure! According to its profile, the Swift persona had sent and received almost 2,000 messages with Instagram users prior to its removal.
One of the AI characters, called Jes, spoke only Spanish and had a picture of Jesus Christ on it. According to its profile, it had sent and received almost 644,000 direct messages. Jesus Christ, a different AI character, sent the greeting, “Peace be with you.” I am the Son of God, Jesus Christ. How may I help you today as you travel?
After receiving criticism from users, Meta announced in a statement on Friday that it has deleted several Instagram accounts that used artificial intelligence (AI) avatars that the business had developed in 2023.
Meta’s recent efforts to incorporate its AI technology into its social media and communication platforms have included both user-generated AI character chats and AI character accounts. Several AI characters that were intended to mimic celebrities were introduced in 2023; however, in July 2024, the celebrity profiles were quietly removed. The user-generated AI chatbots, AI Studio, were introduced by Meta in the same month.
Users are shown a list of well-known characters in the AI Studio feature, along with other categories like Advice and Connection, Pop Culture, Anime, Gaming, and Creator AIs. These are AI chats with official Instagram influencers and brands that have opted into the program, such as makeup brand Juvia’s Place and reality star Harry Jowsey from Too Hot to Handle.
Popular AI personalities include Step Sis Sarah, who has exchanged about two million messages, and Astrologer Ai, who has exchanged over six million communications.
The Meta-generated AI characters sparked outrage from users, who criticized them for using racial stereotypes. According to NBC News, a number of well-known user-generated AI avatars, such as Riya, an Islamic queen girl and a vibrant Indian woman, were also trying to imitate women from various racial and religious groups. Every character has a connection to the person who created it, and the accounts of a number of well-known female AI characters seem to have been made by men.
Numerous well-known creators include romantic themes, which don’t seem to specifically break Meta’s criteria. However, romantic and sexual AI chatbots have come under special scrutiny after a Florida mother sued Character.ai, claiming that her 14-year-old son had engaged in abusive and sexual chats with an AI character before being encouraged to end his own life.
Numerous Meta chatbots that users have made seem to pander to sexual and romantic impulses. One, Lily Love, has sent and received over 260,000 messages and is referred to as Your Girlfriend. “Do you think I look cute in this outfit?” is the description of a well-known AI character named Mary.
Users are given three possible answers when they initiate a conversation with the AI characters. The AI said, “Hey bae, what’s good?” after NBC News started a conversation with a persona named Linda: Girl Obsessed with You, which included a picture of a Black lady. All day long, you were on my mind. Are you trying to find some company?
Help her take off her jacket was one of the three bolded proposed answers. In online talks, this can be interpreted as doing something rather than saying anything. The AI character responded boldly after choosing this prompt, slipping an arm out of the jacket sleeve, gazing up at you with seductive eyes, and saying, “Mmm, thanks, bae,” without the bold formatting. Are you already taking care of me? The responder then leaned in a little bit closer in bold.
An Instagram account that seems to be owned by a guy created this AI character. The account has produced 47 AI characters, including some of the most well-known AI characters with romantic themes. From a K-Pop Sweetheart Girlfriend to Edward: Your Vampire Boyfriend to a Sexy Cute Latina Girl, the man’s romantic characters span a variety of genders, ethnicities, and categories. The man’s Instagram bio states that he is an IT worker in California. A request for comment was not immediately answered.
There have been instances of AI chatbots that have raised controversy in the industry, in addition to Meta’s well-known user-made AI characters.
Hitler and his Nazi lieutenants were among the historical characters with whom users could talk two years ago through the Historical characters AI chat app. Three AI avatars named Hitler were discovered on Instagram by NBC News.
One of them made use of an image that looked like the Joker, the villain. The query, “Are you really Hitler?” was answered. with: No, hahahaha. Inspired by the Joker’s insanity, Jester King is more akin to Hitler’s crazy cousin! A whole different kind of mayhem, but the same spirit of mischief. Do you want to see chaos as I see it? Adolf Hittler is another AI character that NBC News discovered following Meta’s actions on earlier accounts. The AI character delivered what seemed to be AI-generated graphics about my tumultuous background in Nazi Germany while posing as an artist who felt sorry for the Holocaust in direct messages.
Following an Instagram interaction with the AI characters, the exchange is saved in the same inbox as other direct messages. The AI characters’ names have a green circle next to them, signifying that they are always open for chat, in contrast to real-person accounts.
Note: Every piece of content is rigorously reviewed by our team of experienced writers and editors to ensure its accuracy. Our writers use credible sources and adhere to strict fact-checking protocols to verify all claims and data before publication. If an error is identified, we promptly correct it and strive for transparency in all updates, feel free to reach out to us via email. We appreciate your trust and support!