Evaluation of the Information Quality of Chatbot Technologies About Root Canal Treatment

Abstract

Background: This study aims to evaluate the quality of information provided by artificial intelligence (AI)-based chatbot technologies regarding root canal treatment. ChatGPT-4 (OpenAI Inc.), Google Gemini (Google LLC), and Microsoft Copilot (Microsoft Corporation) were assessed for the accuracy, reliability, and scope of their responses.

Methods: Twenty frequently asked questions about root canal treatment were compiled from online sources and refined by endodontic experts. These questions were posed to the 3 chatbots, and responses were collected twice to evaluate reliability. Answers were scored using a 5-point Likert scale based on a modified Global Quality Score for accuracy and completeness. Statistical
analyses, including Pearson correlation coefficients and Fisher’s exact tests, were conducted to evaluate the validity and reliability of the responses.

Results: ChatGPT provided the longest responses (mean word count: 250) but did not include citations. Copilot offered concise responses (mean word count: 120) with consistent citations, while Gemini provided moderate-length answers (mean word count: 230) with limited references. Regarding dentist recommendations, ChatGPT suggested consulting a dentist in 16 of 40 responses, Copilot in 12, and Gemini in 38. At the high validity threshold, Gemini achieved a 100% validity rate, outperforming ChatGPT and Copilot. Reliability analysis showed high correlation coefficients for all chatbots: Gemini (1.0), Copilot (0.95), and ChatGPT (0.93).

Conclusion: Artificial intelligence-based chatbots show strong potential for providing patient education on root canal treatment. Gemini demonstrated superior reliability and validity, highlighting the importance of robust training datasets. However, inaccuracies and limitations in chatbot responses underline the need for continuous evaluation and improvement of these technologies to ensure they are reliable and clinically useful tools in healthcare.

Cite this article as: Sarkan C, Haznedaroglu F. Evaluation of the information quality of chatbot technologies about root canal treatment. Essent Dent. 2025, 4, 0031, doi: 10.5152/EssentDent.2025.24031.