Using Dynamic Item Response Theory and Machine Learning based on Natural Language Processing to improve the reliability of the Operant Motive Test (OMT)

The study of implicit motives is marked with a striking paradox: while tests for implicit motives like the Operant Motive Test (OMT) have proven their predictive power, they show only weak reliability scores using classical indices like Cronbach’s Alpha. The OMT is a picture-based procedure that asks respondents to generate imaginative verbal behavior that is later coded for the presence of affiliation, power, and achievement-related motive content by trained coders. The contribution of the article is twofold. First, the article builds on novel dynamic Thurstonian model to address the reliability issue, based on OMT data of 7.060 participants. Second, we developed a machine learning model based on Natural Language Processing (NLP) using the BERT model, with the aim of increasing the internal consistency of the coding compared to manual coding. The advantage of the BERT-based Machine Learning model is that it codes not only one motive for each picture, as Human experts do, but assigns four codings for each question of every picture. These 4 x 15 scores were analyzed for internal consistency (Cronbach’s Alpha). Both approaches achieve comparable reliability estimates. Squared correlation reliability using dynamic Thurstonian model were .75 for the Affiliation motive, .65 for the Power motive, and .61 for the Achievement motive, respectively. Internal consistencies (Cronbach’s Alpha) were .61 for the Affiliation motive, .71 for the Power motive, and .66 for the Achievement motive. Furthermore, the coding of the ML model was indistinguishable from that of a human expert and showed similar prognostic validity indices predicting managerial responsibility.

Motivation Science - In Press