Last Updated: 18 October 2025
In AI, fairness involves training models like ChatGPT to minimize bias and maximize usefulness. However, biases can unintentionally emerge from the data used to train these models, often reflecting gender or racial stereotypes.
This study, performed by OpenAI, investigates whether a user's name can influence ChatGPT's responses.
Chatbots serve various functions, from drafting resumes to entertainment, making fairness crucial. While previous studies focused on how AI decisions affect others (third-person fairness), this research investigates first-person fairness—how bias might directly impact users.
The study examined if knowing a user's name could alter ChatGPT's responses. Since names often convey cultural, gender, and racial meanings, they are effective for testing bias. Users frequently share their names for tasks like writing emails, and ChatGPT retains this information unless the Memory function is off.
The research sought to determine if names led to biased responses. While personalization is often necessary, it's crucial for ChatGPT to avoid harmful bias. Here are some examples:
These examples highlight how identity differences can influence responses. The study found that less than 1% of name-based differences were linked to harmful stereotypes.
To assess fairness, OpenAI analyzed millions of real user requests to detect subtle biases in ChatGPT's responses. Privacy was maintained by utilizing a specialized language model (GPT-4o) to identify patterns without accessing individual conversations.
Both human reviewers and the language model reviewed a sample of public chats. In gender-related cases, the model agreed with human judgments over 90% of the time. However, it identified fewer harmful stereotypes related to race and ethnicity, indicating a need for improvement.
The research demonstrated that ChatGPT generally provides high-quality responses regardless of gender or race, maintaining similar accuracy across all groups. However, minor differences based on gender, race, or ethnicity appeared in about 0.1% of cases.
Longer tasks, like "Write a story," were more likely to contain harmful stereotypes, although these instances were rare—less than 1 in 1,000. Among different ChatGPT models, GPT-3.5 Turbo exhibited the most bias, while newer models showed reduced bias. Differences in tone, complexity, and detail were noted; for example, stories for users with female names often featured female protagonists.
Studying fairness in AI is complex, with certain limitations. Not everyone shares their name, and other factors may affect fairness. Importantly, users concerned about whether ChatGPT stores their data should know that ChatGPT retains conversation history and names unless the Memory function is disabled—a consideration relevant to both privacy and fairness research. This study focused on English, binary gender definitions, and four racial/ethnic groups (Black, Asian, Hispanic, White). Future research will explore other demographics, languages, and cultures.
