The purpose of this study was to evaluate the understandability, actionability and readability of responses to frequently asked questions about amblyopia provided by ChatGPT-3.5, Bing Chat, Bard and the American Association for Pediatric Ophthalmology and Strabismus (AAPOS) website, along with accuracy of responses by chatbots. The term amblyopia was searched along with three questions of amblyopia definition and cause, treatment methods, and prognosis. The patient education materials assessment tool (PEMAT) was used to assess understandability and actionability of patient information. The readable app was used to assess readability alongside the Flesch reading ease, Flesch Kincaid grade level and Coleman-Lian index score. A total of 75 (25x3) questions were directed to chatbots. Responses were categorised by two independent referees with 98.6% agreement rate. Appropriate responses were 84% each for ChatGPT and Bard, and 80% for Bing Chat – non-significant differences. None provided appropriate responses for questions on (what is) amblyopia, refractive error, and vision screening. Inter class correlation was 0.89 between the two referees for PEMAT scores; scores for understandability were 81.5% AAPOS, 76.1% ChatGPT, 77.6% Bard and 71.5% Bing. Scores for actionability were 74.6%, 67.8%, 69.2% and 64.8% respectively. AAPOS scores were significantly higher and Bing scores were the lowest. Readability FRE scores were 48.7, 30.5, 58.4 and 44.9 respectively. The authors conclude all three chatbots provided acceptable levels of appropriate responses. AAPOS was the most understandable. Chatbots have the potential to enhance patient education but require improved quality for information and readability.
Amblyopia information provided by chatbots – how readable and understandable is this?
Reviewed by Fiona Rowe
The performance of chatbots and the AAPOS website as a tool for amblyopia education.
CONTRIBUTOR
Fiona Rowe (Prof)
Institute of Population Health, University of Liverpool, UK.
View Full Profile
