arXiv:2606.20205: Psychological Profiles of LLMs Are Largely a Measurement Artifact, Not a Stable Personality
The paper arXiv:2606.20205 tested 56 instruction-tuned language models with standardized psychological and preference instruments. Using variance decomposition, the authors show that directional response bias explains 81 to 90 percent of differences among models, compared to only 9 to 16 percent in humans, concluding that psychological profiles of models are largely a measurement artifact, not a stable personality.
This article was generated using artificial intelligence from primary sources.
The paper arXiv:2606.20205 re-examines the increasingly popular practice of psychological profiling of language models — applying personality and preference tests, originally designed for humans, to large language models. Researchers tested 56 instruction-tuned models with standardized psychological and preference instruments.
What Was Discovered
Using variance decomposition, a statistical method that separates sources of variation, the authors found that directional response bias — a model’s tendency to select certain answers regardless of content — explains 81 to 90 percent of differences among models. In humans, that share is only 9 to 16 percent. The difference means that what appears as a model’s “personality” mostly comes from a measurement artifact, not a stable trait.
Why This Matters
According to the paper, profiles change depending on the questions used, so the results of the same tests are neither reliable nor comparable. The authors call for the development of purpose-built instruments for model assessment rather than borrowing human psychological scales. The finding is a warning about the increasingly common headlines claiming that a particular model has a certain “character” — such claims often rest on a measurement artifact.
Frequently Asked Questions
- How many models were tested?
- 56 instruction-tuned language models were tested with standardized psychological and preference instruments.
- What is the share of response bias?
- Directional response bias explains 81 to 90 percent of variance among models, while in humans that share is only 9 to 16 percent.
- What do the authors recommend?
- They recommend developing purpose-built instruments for model assessment, as profiles change depending on the questions used.