AI tools vs human in english narrative writing scoring: A comparative study of scoring agreement among human raters, DeepSeek and ChatGPT in exam context

Document Type

Article

Publication Date

4-1-2026

Abstract

A growing body of literature is examining the agreement between artificial intelligence (AI) and humans in scoring. However, empirical findings remain inconclusive, ranging from a strong correlation to a large difference in the scores assigned by the two media. In response, this mixed-methods study compared scoring agreement and evaluation rationales across three sources, including ChatGPT-4, DeepSeek-R1 and human ratings from a city-level standarised English narrative writing examination (n = 212). Quantitatively, the findings revealed strong relative agreement between the two AI models, while only moderate alignment was observed between each AI tool and the human ratings. Qualitative analyses of a focus group interview and scoring rationales indicated that these discrepancies reflect systematic differences in evaluation orientation. Human raters demonstrated more flexibility and sensitivity to contextual and presentation-related factors in evaluating writing quality, whereas AI tools prioritised linguistic accuracy and structural regularity and remained unaffected by surface features such as handwriting or erasures. Rather than suggesting simple scoring errors, the results reveal systematic differences between human and AI judgment, with implications for the principled integration of AI into high-stakes assessment.

Keywords

AI Scoring, AI Writing Evaluation, Comparative study, Human raters, Mixed Methods study, Narrative writing evaluation

Publication Title

Assessing Writing

ISSN

1075-2935

DOI

10.1016/j.asw.2026.101040

Volume

68

Publisher

Elsevier Ltd

Share

COinS