3.1. Participants

The participants in this study were 131 pre-service principals, including 59 females and 72 males. In terms of educational background, the majority held a master’s degree (\(n =115\)), followed by doctoral degrees (\(n =11\)) and bachelor’s degrees (\(n =5\)). The average age of the participants was 47 years and the average length of service was 21 years.

The training curriculum was structured around a series of developmental themes. The assessment covered four major areas: written reports, oral presentations, daily performance, and a final written examination. Although most counties consider the training program a form of pre-service development, where grades are used primarily for reference, some counties do factor these scores into the selection order for principal appointments. As a result, pre-service principals generally regard the assessment outcomes as significant.

Eighteen raters were involved in evaluating the school leadership projects, including 12 internal evaluators who served as mentor principals and six external evaluators comprising academic experts. The assessment adopted a balanced incomplete block (BIB) design to ensure a fair and efficient distribution of scoring tasks among the raters.

Each assignment was independently assessed by at least two mentoring principals and at least one external reviewer to ensure objectivity and scoring consistency. However, owing to the large number of pre-service principals, it was difficult for assessors to provide written feedback on each assignment. Consequently, most evaluators assigned scores without any accompanying comments or suggestions, which resulted in limited guidance for improvement.

3.2. Human Scoring Process

During the pre-training phase, an orientation session is conducted with the pre-service principals to introduce the course structure and assessment components. As the training progresses to school-based applications, each trainee is required to submit a School Improvement Plan based on his or her original school placement. Submissions are uploaded through an online principal assessment platform that includes built-in scoring functionality. This system allows the evaluators to read, score, and provide feedback on specific criteria. All assignment records are stored in a system for process-based reviews and auditing.

The assessment process consists of four phases:

1. Phases 1–2: Mentoring sessions during which pre-service principals begin drafting improvement plans under the guidance of their mentor principals.

2. Phase 3: On-site visit to the trainee’s original school for field observation and interaction with the school principal and staff.

3. Phase 4: Additional mentoring sessions to further develop the plan based on mentor feedback and school observations.

The assessment rubric includes seven dimensions (Table 1) that evaluate core leadership competencies such as systems thinking, vision formulation, strategic planning, innovative leadership, and self-awareness. Each dimension is scored on a five-point scale:

• Scores 4–5 indicate proficiency (advanced or innovative performance).

• Scores 2–3 indicate basic competence (generally aligned with expectations).

• Score 1 indicates needed improvement (lacking clarity or coherence).

The rubric was designed on the basis of criterion-referenced assessment principles, meaning that performance is evaluated against predetermined standards rather than relative to peer performance. The goal is to understand each trainee’s competency development across the targeted leadership domains.

3.3. ChatGPT Implementation Process

In this study, the ChatGPT API was integrated into the primary digital system developed for the project. The main purpose was to evaluate participants’ School Improvement Plans and compare AI-generated assessments with human ratings based on the same scoring rubric.

All past assignments were double-rated by experienced principals and expert scholars. The final score was determined when two researchers reached a consensus on the evaluation. For each scoring dimension, 200 previous reports were randomly selected as exemplars to train the ChatGPT model. To ensure independent judgment and prevent potential cross-article influence, each document was evaluated in a separate and newly initiated ChatGPT session. During the evaluation process, prompts were designed to include detailed scoring rubrics, replacing simple rating instructions with comprehensive descriptions of each performance level. ChatGPT was also required to provide justification for its ratings before assigning scores. This reasoning step has been shown to improve LLM performance when executing complex tasks ¹⁴.

3.4. AI-Based Scoring Procedure

ChatGPT was employed as an AI-assisted rater to evaluate School Improvement Plans submitted by pre-service principals. A structured prompt was used to fine-tune the model’s evaluative function, simulating the perspective of an experienced school principal with expertise in analyzing and assessing educational plans. The AI assessed each plan on the basis of seven predefined dimensions aligned with leadership and school management competencies.

For each dimension, the AI assigned a score ranging from 1 to 5, where 1 indicated poor performance and 5 indicated excellent performance. The scoring rubric emphasizes clarity, feasibility, specificity, and alignment with school development goals. In addition to the numeric scores, the AI-generated feedback included short rationale statements to justify the ratings based on the content of each submitted plan.

This AI-generated evaluation was then compared with scores assigned by human raters to examine inter-rater consistency, scoring discrepancies, and the potential of ChatGPT to serve as a reliable assessment assistant in professional development settings.

4.1. Descriptive Statistics

Table 2 presents descriptive statistics of the seven leadership performance constructs obtained by comparing ratings from human raters and an AI-based scoring system. Across all constructs, human raters consistently assigned higher mean scores than the AI. For instance, in the Basic (BA) dimension, the human mean was 4.02, whereas the AI mean was notably lower at 3.23. Similar gaps were observed in Strategy (ST; human \(=3.92\); AI \(=2.97\)) and Reflection (RE; human \(=4.03\); AI \(=3.50\)), suggesting that human evaluators tended to be more generous or lenient than the AI system.

Standard deviations indicate that human scores were generally more variable, with values ranging from 0.74 to 0.81, compared to AI standard deviations, which ranged from 0.56 to 0.70. This result implies that the AI ratings were more narrowly distributed and may reflect a more conservative or uniform scoring pattern.

Distribution characteristics further differentiated the two rater types. Human scores consistently exhibited negative skewness (e.g., Reflection \(=-0.71\); Basic \(=-0.65\)), indicating that most ratings clustered at the higher end of the scale. In contrast, the AI ratings displayed less skew or even positive skewness (e.g., Vision [VI] \(=0.73\)), suggesting a more symmetric or lower-centered distribution. Additionally, human ratings tended to be more leptokurtic, with higher kurtosis values (e.g., Basic \(=1.19\); Improvement [IM] \(=0.75\)), denoting peaked distributions with fewer extreme responses. AI scores, on the other hand, were generally platykurtic or near-normal, with lower kurtosis values (e.g., Strategy \(=-0.72\); Basic \(=-0.94\)).

Overall, these results revealed distinct patterns between the human and AI scores. Human raters provided higher and more dispersed ratings, which were potentially influenced by subjective perceptions or contextual empathy. In contrast, AI-generated scores were more conservative, consistent, and centered, reflecting algorithmic precision but potentially reduced sensitivity to nuanced performance. These differences underscore the importance of considering both reliability and interpretability when integrating AI systems into educational and leadership assessment contexts.

4.2. Inter-Construct Correlations: Human vs. AI Ratings

Table 3 presents the Pearson correlation coefficients for the seven leadership constructs separately for the human raters and AI-generated scores. All reported correlations were statistically significant (\(p<.01\)), as indicated by double asterisks (\(^{**}\)).

Among human raters, moderate to strong positive correlations were observed across all constructs. Notably, Plan (PL) showed the strongest correlations with Strategy (\(r =.60\)), Improvement (\(r =.59\)), and Vision (\(r =.55\)), suggesting that these constructs were perceived as tightly interrelated within human evaluations. Similarly, Improvement and Reflection exhibited a strong association (\(r =.65\)), indicating that human raters often view these two dimensions as co-occurring in effective leadership planning.

By contrast, the AI-generated scores displayed different correlation structures. Although some relationships were similarly strong, such as Basic and Vision (\(r =.72\)) or Basic and Plan (\(r =.70\)), other construct pairs had notably weaker associations. For instance, Strategy correlated only modestly with Plan (\(r =.24\)) and Curriculum (CU; \(r =.22\)), indicating that AI may treat these constructs as more distinct or independent in its assessment logic. Similarly, the correlation between Improvement and Reflection was lower (\(r =.48\)) compared to that observed for human scores.

Overall, these findings suggest that, although AI and human ratings were generally aligned in structure, the AI scoring model exhibited more differentiated construct relationships, potentially reflecting its rule-based or data-driven parsing of content features. Conversely, human raters may apply a more holistic or integrative judgment, resulting in higher inter-construct coherence. This discrepancy highlights the importance of reviewing not only the mean scores but also the internal consistency patterns when comparing human and AI evaluative systems.

4.3. Inter-Rater Reliability Analysis

To evaluate the consistency of ratings across evaluators, ICCs were computed using a two-way mixed-effects model with a consistency definition. This method assumes that the subjects are random and the raters are fixed, and evaluates the extent to which raters assign consistent ratings across multiple targets, excluding between-measure variance.

For human raters, the ICC for single measures was .55 (95% CI \(=[.51,\, .59]\)), indicating a moderate level of agreement among individual raters. The ICC for the average measures, which represents the reliability of the aggregated score across raters, was .896, indicating excellent consistency. The \(F\)-test for the null hypothesis of no reliability was statistically significant, \(F(452, 2712)=9.65\), \(p<.001\), suggesting that the observed agreement among raters was not due to chance.

The AI rater exhibited a single-measure ICC of .49 (95% CI \(=[.42,\, .57]\)), indicating moderate reliability at the individual level. The average-measures ICC was .87, showing strong interrater reliability when the ratings were averaged. The corresponding \(F\)-test result was also significant: \(F(130, 780)=7.93\), \(p<.001\) (Table 4).

These results confirm that, although human raters may show moderate variability in scoring, the overall scoring system, particularly when using aggregated ratings, demonstrated high reliability. Using multiple raters per case strengthens the dependability of the evaluation process and ensures robust assessment outcomes.