We’re riding the waves of a technological revolution in healthcare with Generative AI bringing great opportunity – and significant challenges. In this post, Jonathon Carr Brown, MD at Healthily, suggests we embrace innovation and cutting-edge tech but always keep ethics and safety at the helm
In the era of rapid technological advancement, integrating Generative AI into healthcare systems shows immense promise for enhancing patient care, outcomes and health insurance workflows.
Optimism around AI
This optimism is shared. A recent Deloitte survey found that 53% of the US public felt Generative AI would improve healthcare.
At Healthily, our team of data scientists and doctors has delved into the complexities and hazards of this innovation.
A smart prototype
Our prototype, powered by a Retrieval-Augmented Generation (RAG) model, shows promise. It seamlessly enables a chatbot to mimic human responses, answer general health questions, conduct symptom checks, offer guidance on accessing health services, and provide location-specific details.
Using Healthily’s proprietary Smart Search, we scour over 2,500 medically approved articles, identifying the most relevant answers. These are then summarized by a Generative Large Language Model (LLM) into conversational responses.
Our ‘intent detection’ technology recognizes when users discuss symptoms, guiding them gently into our assessment environment driven by a blend of AI and Bayesian logic. We can then tailor safe recommendations for health services and products to suit their needs.
Why we take time to build our models
While this is incredibly exciting and promising, it's currently too experimental for public use. Hazard assessments conducted by our Clinical Safety Officers on the prototype highlight the need for scrutiny, especially because we operate in the healthcare space (we can’t move fast and break things – as ‘things’ are people).
We wrote this piece as a ‘cautionary tale’ for those enticed by the allure of Generative AI health chatbots.
5 key questions to ask about Generative AI technology
When evaluating such technologies, consider:
How do you ensure accuracy? Our doctors identified 15 instances where Generative AI produced incorrect results due to various factors including ambiguous input or data bias.
How do you prevent incomplete answers? Design limitations of Generative AI systems often lead to short or inadequate responses that fail to address user queries safely.
How do you address gibberish responses? Termed as "hallucinations," these responses, though usually harmless, can occasionally mislead users.
How do you prevent users from mistaking the bot for a human? Users may quickly perceive the bot as human-like, potentially lowering their guard.
How do you prevent the bot from offering inappropriate advice? Users may elicit personalized advice bypassing the safeguards provided by the in-built symptom checker – designed to automatically kick in when symptoms are present – potentially leading to inappropriate responses.
While these issues will likely be resolved over time, Healthily’s clinical safety team found at least 24 areas where action was needed to stop or prevent a hazard.
Looking forward: proceed with caution
In the future, a Generative AI Health Chatbot integrated with a symptom checker and defined medical content dataset will be a powerful tool indeed. However, we must navigate and address these hazards before its widespread adoption.
In a recent TechCrunch article, Andrew Borkowski, chief AI officer at the VA Sunshine Healthcare Network, the U.S. Department of Veterans Affairs’ largest health system, warned that Generative AI’s deployment could be premature due to its “significant” limitations — and the concerns around its efficacy.
Rigorous validation, continuous monitoring, and ethical oversight are essential to mitigate risks and maximize AI's benefits in healthcare delivery. Only through responsible implementation and collaboration can we fully leverage AI's potential to improve patient outcomes and advance medicine.
Comments