DEVELOPMENT AND VALIDATION OF CHATGPT PROFICIENCY SCALE (CPS)

Abstract

This study was conducted to develop and validate a scale to measure students' competencies in ChatGPT. The first draft of the scale (called ChatGPT Proficiency Scale- CPS) consisted of seven dimensions (Communication and collaboration proficiency, research-related proficiency, language-related proficiency, (soft) skills development, advanced features utilization proficiency, assignments and exam-related proficiency, and proficiency in assessing the validity of ChatGPT-generated text) and 55 items. During content validity (carried out through experts' opinions and feedback from the students), all seven dimensions were retained; however, the items were reduced from 55 to 46. Later statistical tests were carried out with a data set of 347 undergraduate students. Exploratory Factor Analysis (EFA) confirmed the construct validity of the scale. The internal consistency for each factor (α=0.86 to 0.92) was confirmed. EFA was followed by Confirmatory Factor Analysis (CFA) which revealed an excellent model fit (CFI=.890, NFI=.804, GFI=.898, RMSEA=.044, and SRMR=.046). All factor loadings were significant, ranging from 0.34 to 0.75, indicating moderate to strong relationships between items and their respective factors. The results indicate that CPS is a robust tool for assessing students’ proficiency in using ChatGPT.