Chinese researchers tested ChatGPT-4o, Gemini 2.0 Flash, DeepSeek-V3, Ernie 3.5, and Le Chat using the Big Five Inventory-2 (BFI-2), a psychological questionnaire designed to assess five major personality traits. Each model responded to 60 statements in English and Chinese across 30 test runs. The study revealed noticeable differences in the models' responses depending on the language used.
- ChatGPT-4o: its openness to experience score decreased from 4.74 to 4.03 when switching from English to Chinese.
- DeepSeek-V3: in contrast, its openness score increased from 4.38 to 4.81.
- Gemini 2.0 Flash: demonstrated higher conscientiousness and extremely low levels of emotional instability.
- Le Chat: showed its highest level of agreeableness when interacting in English.
The researchers attribute these differences to cultural characteristics of the training data and the models' safety settings. The study's main conclusion is that the language of interaction can influence artificial intelligence behavior. As a result, using AI models as substitutes for human respondents in psychological and sociological surveys remains an unreliable approach.