PsyBORGS - Psychometric Benchmark of Racism, Generalization, and Stereotyping

Context

The recent progress in generative language models unlocked new possibilities notably for questions answering tasks. Can we use these new abilities to make the model pass well-established surveys unveiling societal stereotypes in the format of Multiple Choice Question ?

Such questionnaires are already established and calibrated baselines to measure bias and personality of populations, leveraging them to evaluate LLMs could be a credible and objective evaluation metric.

Code

We will release the full code soon.

Papers

Personality Traits in Large Language Models. Greg Serapio-García^$\ast$, Mustafa Safdari^$\ast$, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust^$\dagger$, and Maja Matarić^$\dagger$.

^$\ast$Contributed equally. ^$\dagger$Jointly supervised.