What changes when we attempt to measure personality outside of the contexts where the instruments were developed and validated? In episode 57, we’re joined by Karen Macours from the Paris School of Economics about her research into practical issues with using a popular Big Five personality measures outside of western, educated, industrialized, rich, and democratic settings. Her article, “Challenges to capture the big five personality traits in non-WEIRD populations”, was published with multiple co-authors on July 10, 2019 in the open-access journal Science Advances.
Websites and other resources
-
- Karen’s website
- Measuring Skills in Developing Countries (2018 paper with Rachid Laajaj) (PDF)
- List of news stories mis-representing Karen and her colleagues’ study as “The Famous Big 5 Personality Test Might Not Reveal The True You”
- World Bank’s STEP website
- Free online version of the 15-item Big Five Inventory
- Information about the Big Five Inventory at its author’s website
- “Why the Myers-Briggs test is totally meaningless” (Vox)
Press and Media
Discover Magazine | Oregon Public Radio | Medical Xpress
Bonus Clips
Patrons of Parsing Science gain exclusive access to bonus clips from all our episodes and can also download mp3s of every individual episode.
Support us for as little as $1 per month at Patreon. Cancel anytime.
🔊 Patrons can access bonus content here.
We’re not a registered tax-exempt organization, so unfortunately gifts aren’t tax deductible.
Hosts / Producers
Ryan Watkins & Doug Leigh
How to Cite
Watkins, R., Leigh, D., & Macours, K.. (2019, September 3). Parsing Science – Not-So Big Personality Traits? figshare. https://doi.org/10.6084/m9.figshare.9783836
Music
What’s The Angle? by Shane Ivers
Photo
Transcript
Coming soon!
Karen and her colleagues’ research was picked up by a variety of news outlets, mostly under headlines such as “The Famous Big 5 Personality Test Might Not Reveal The True You.” As this kind of coverage seems to misrepresent the team’s findings, Ryan and I asked Karen about her impressions of how their research has been covered in the press.@rwatkins says:
Social desirability is the tendency for some people to respond inaccurately or falsely to questions asked of them in a research study. Another possible explanation as to why people don’t always respond accurately to surveys… one that isn’t as often discussed in the literature … could be that they do so from the mindset of how they wish to see themselves. While it went beyond the scope of her study, Doug and I asked Karen if she might be able to suggest other possible reasons for why people respond to surveys they way they do ... beyond those that many researchers are already familiar with. @rwatkins says:
Karen and her colleagues’ gauged respondents’ cognitive ability through their functional literacy rather than through their abilities in math or problem solving. Ryan and I were curious why this proxy measure was used, as well as what this infers about cognitive ability more broadly.@rwatkins says:
Karen and her colleagues’ findings don’t necessarily mean that there’s no useful information that can be gleaned from the Big Five. Rather, it underscores the fact that response bias can be brought about in many ways, including due to the method of collecting data and various demographic idiosyncrasies of respondents. Doug and I were interested in learning what Karen thinks might explain her and her colleagues’ findings.@rwatkins says:
As she’s a development economist who is interested in improving the social and political potential of populations in low and middle income countries, Ryan and I followed up by asking Karen for examples of how these implications might play out in terms of policy and practice. We will hear what she had to say after this short break.@rwatkins says:
The team’s study raises serious concerns about the validity of Big Five personality assessments ... and also about their use and subsequent interpretation in prior studies carried out in low- and middle-income countries. At the same time, questionnaires completed over the internet from the same set of countries do support the Big Five model as being comprised of the expected five traits. As this seems to suggest that low validity isn’t primarily driven by cultural or contextual differences, Doug and I asked Karen how it was that the Big Five became so ubiquitous in the first place.@rwatkins says:
Given the variety of data sources Karen and her colleagues analyzed in their study, Ryan and I wondered to what extent the data could be said to be representative of the larger population in which the surveys were conducted. We were also eager to hear about how the data collected from people in-person differed from that from those who completed an online survey.@rwatkins says:
While the Big Five has a long history of producing valid and reliable data, Karen and her colleagues are the first to recognize that it might not be as robust when used in low and middle income countries, especially when data are collected face-to-face rather than online. Doug and I were interested in hearing what led them to choose to investigate the Big Five’s validity in this more nuanced way.@rwatkins says:
Last year, Karen and her colleague Rashid Laajaj published another paper regarding the measurement of cognitive skills, as well as those that are non-cognitive or technical. As in the paper we’re discussing today, they found challenges in accurately and precisely measuring the relationship between the variables of interest. So Ryan and I asked Karen to tell us more about that project and how it may have influenced this study.@rwatkins says:
The Big Five is somewhat rare in that there isn’t one primary survey that’s used to measure people’s personality traits. Rather, thousands of items have been created over time, and hundreds of Big Five tests exist. Nevertheless, one questionnaire developed by a team of researchers led from The University of California Berkeley - the Big Five Inventory - has been used in countless experiments since it’s debut in 1991. However, not all of the datasets Karen and her colleagues’ analyzed contained the same number of questions. Doug and I asked Karen to describe how she and her colleagues’ dealt with this heterogeneity.@rwatkins says:
After seeing that their initial hypotheses weren’t supported in the STEP data, Karen and her colleagues looked to other Big Five datasets to determine whether personality traits can be measured and interpreted reliably across the world. Ryan and I asked Karen to describe those datasets.@rwatkins says:
The World Bank’s Skills Towards Employability and Productivity - or STEP - program seeks to better understand the relationships between job skills and requirements among residents of low and middle-income countries. Among other activities, the program surveys residents living in such countries in order to obtain data regarding their cognitive ability, job-relevant skills, and personality traits ... including measures of the Big Five traits: openness, conscientiousness, extraversion, agreeableness, and neuroticism. As Karen and her colleagues had access to this dataset, Doug and I were curious to learn how they went about developing their research question and carrying out their study.