Postdoctoral Scholars Association

Background: The integration of artificial intelligence, specifically large language models, is increasingly prevalent in healthcare. This study examines an artificial intelligence chatbot (AIC) in orthopaedic patient education in a simulated setting. Methods: The fine-tuned AIC with custom prompts was used in two phases. First, the AIC engaged with 20 example patient personas (EP), offering personalized questions to prepare for appointments. The AIC sorted questions into those answerable with online resources (group 1) and those needing an orthopaedic surgeon’s expertise (group 2). In the second stage, the AIC provided answers for group 1 questions, citing medical journals or qualified patient education websites. Readability was assessed using the Simple Measure of Gobbledygook reading score (SMOG), and the citation accuracy was verified. Results: The AIC classified 42% of questions as group 1, with the remainder as group 2. Answers for 55% of EPs were at an “undergraduate level”, while the rest were at a “graduate level”, per SMOG scores. Citation accuracy was 100% for the AIC-generated responses. Conclusion: Although AIC prompts were not customized to readability, the readability level was higher, therefore less accessible, than that of typical materials. However, the pre-appointment questions were appropriate for orthopaedic visits, suggesting AICs could enhance patient engagement in clinical settings.