Table of Contents
OpenAI has discovered that during the early testing of ChatGPT’s Voice Mode, some users were forming personal connections with the AI
OpenAI has issued a warning about its recently released Voice Mode feature for ChatGPT, highlighting concerns that users might form social relationships with the AI model.
Join our Telegram Channel for Updates
This information was disclosed in the company’s System Card for GPT-4o, which provides a detailed analysis of potential risks and safeguards associated with the AI model. One of the key risks identified is the possibility of users anthropomorphizing the chatbot—attributing human characteristics to it—and developing emotional attachments.
The concern stems from the Voice Mode’s ability to modulate speech and express emotions in a way that closely resembles human interaction. During early testing, which included both red-teaming and internal user testing, OpenAI observed instances where users appeared to form social bonds with the AI.
In one notable case, a user expressed a sense of connection by saying, “This is our last day together” to the AI, prompting the company to consider the long-term implications of such attachments.
OpenAI is particularly concerned that this emotional reliance on AI could negatively impact human-to-human interactions. While the feature could provide companionship for lonely individuals, it also has the potential to undermine healthy relationships.
Additionally, prolonged AI-human interactions might alter social norms. For example, users can interrupt the AI at any time, a behavior that would be considered rude in human interactions.
Another issue raised by OpenAI is the potential for increased persuasiveness of the AI if users begin to trust it more deeply over time. While the current persuasion score of the models is not yet a significant concern, this could change as users develop stronger bonds with the AI.
At present, the AI company does not have a solution for these concerns but plans to continue monitoring the situation. The company stated, “We intend to further study the potential for emotional reliance, and ways in which deeper integration of our model’s and systems’ many features with the audio modality may drive behavior.”
