Grant: Project Grant

Background and Significance: Current access to rheumatology care is inadequate in Canada. With the increasing demand for rheumatologists and limited resources, particularly physician time, artificial intelligence tools such as large language models (LLMs) have been explored to free up physician capacity while maintaining, if not improving, access to rheumatology care for patients. However, we recently showed that currently available LLM chatbots may provide inaccurate answers to rheumatology questions. Fortunately, LLM chatbots can be trained on very specific information which should make their answers more accurate. Aim: Build a rheumatology-specific LLM chatbot ('ChatRheum') and evaluate its performance with respect to: A) accuracy and comprehensiveness B) empathy and readability Methods: Stage 1: Identification of rheumatology information sources. We will survey rheumatologists and rheumatology patients about where they get rheumatology information. Stage 2: Model development. We will develop an LLM chatbot (ChatRheum) trained on the rheumatology resources identified in Stage 1. Stage 3. Model evaluation. Using online rheumatology Q & A banks, we will compare ChatRheum answers against those of generic LLM chatbots (eg. ChatGPT) and rheumatologists. Expected Outcomes/Impacts: We expect that we will be able to create a source-verified LLM chatbot that will perform better than currently available generic LLM chatbots on accuracy and comprehensiveness (and on-par with rheumatologists), and better than rheumatologists on empathy and readability when it comes to answering rheumatology patient questions. This will allow rheumatologists to divert their time from triage to patient appointments, thus reducing wait times for rheumatology patients. Further, this method of developing a source-verified LLM chatbot could be used to develop similar tools for other areas of medicine, potentially saving the healthcare system money, while increasing much needed capacity.