SPOT THE BOT: A comparative quality assessment of AI-generated written instructions for lung ultrasound training
Document Type
Article
Publication Date
1-1-2026
Abstract
Background: The rapid proliferation of artificial intelligence (AI) in medical education has outpaced the development of quality assurance methods for AI-generated content. This study provides the first systematic evaluation of AI-generated instructional materials for lung ultrasound (LUS) training. Methods: The ATLAS study employed a cross-sectional, multi-rater evaluation design comparing six instruction sources (five AI systems and human-generated content) across ten LUS content sessions. Expert evaluators (n=39) assessed materials using five standardized domains: Medical Accuracy, Evidence Completeness, Clarity, Practical Utility, and Pedagogical Quality. Statistical analysis included Kruskal-Wallis tests and pairwise comparisons with Bonferroni correction. Results: Significant differences existed between instruction sources (H=92.582, p < 0.001). Manus AI achieved the highest overall rating (4.55±0.83) and significantly outperformed human instructions in Medical Accuracy (p=0.0002) and Evidence Completeness (p < 0.001). Gemini AI (3.94±0.97) performed statistically equivalent to human instructions (4.23±1.00). ChatGPT (2.62±1.35) and Meta (1.53±1.02) performed significantly worse than human instructions (p < 0.001). Clarity emerged as the most discriminating criterion with the widest performance range (1.53-4.54). Conclusions: Certain AI systems can generate high-quality LUS instructional materials that match or exceed human-generated content. However, significant quality variations across AI systems emphasize the critical importance of systematic evaluation before implementation. These findings support cautious but optimistic integration of high-performing AI systems into medical education with appropriate quality assurance measures.
Keywords
artificial intelligence, generative AI, instructional design, lung ultrasound, medical education, point-of-care ultrasound, quality assessment
Publication Title
Ultrasound Journal
ISSN
2524-8987
DOI
10.5826/tuj.2026.18495
Recommended Citation
Wong, Adrian; Roslan, Nurul Liana; Ho, Serene; Tan, Rou An; Noor, Julina; Via, Gabriele; and Corradi, Francesco, "SPOT THE BOT: A comparative quality assessment of AI-generated written instructions for lung ultrasound training" (2026). Research Publications (2026 to 2030). 2078.
https://knova.um.edu.my/research_publications_2026_2030/2078
Volume
18
Issue
S1
Publisher
Mattioli 1885