An Example of the SCORS-G Scales for Developing AI-Based Application (1.5 CEs) - COPY 8

Abstract

Interest in the utility of Artificial Intelligence (AI) for a range of clinical and research applications is rising. Advances in natural language processing and increasing availability of AI models holds promise for those engaged in narrative-based research. Several narrative-based metrics for attitudinal and sentiment-based research currently exist; however, it is unclear if AI is capable of assisting in the types of complex narrative analysis conducted by psychologists. The four talks in this symposium focus on the utility of AI models for rating narratives using the Social Cognitions and Object Relations Scales - Global Rating Method (SCORS-G). The first paper describes the genesis of the project, reviews a prompt tuning process using SCORS-G resources, and describes agreement between expert raters and the AI models. The second paper builds on the first by employing these prompts to rate newly obtained narratives. It explores inter-rater reliability among AI models, associations between AI models and human raters, and associations between self-report measures and SCORS-G ratings made by human and AI raters. The third paper focuses on potential applications of an AI-informed application for making SCORS-G ratings, reviews potential uses in multiple settings, and emphasizes the need to make use of an iterative approach for developing such an application. The fourth paper highlights current progress, while being realistic about current limitations. It describes a multi-site, multi discipline research agenda for building on present results. While focused on the SCORS-G, the research agenda described could be used as a model for anyone seeking to develop an AI-informed application for assessment. Together, these papers suggest that AI holds some promise for those interested in narrative assessment, while ultimately arguing that a systematic process of research and refinement is necessary prior to broad deployment and adoption.

Chair

Barry Dauphin | University of Detroit Mercy

Discussant

Jenelle Slavin-Mulford | Augusta University

Goals & Objectives

1. Critique the advantages and disadvantages of using Artificial Intelligence (AI) for narrative rating systems.
2. Analyze the reliability of AI for the Social Cognition and Object Relations scales- Global Rating System (SCORS-G)
3. Utilize an iterative approach in the development of AI-augmented scoring for narrative data in personality research