Ethical Prompt Design for Health Equity: Preventing Hallucination and Addressing Bias in AI Diagnoses

Abstract

The use of large language models (LLMs) in healthcare is transforming how clinicians access information, generate insights, and support patient care decisions. These AI systems hold tremendous promise offering the ability to summarize complex clinical notes, suggest differential diagnoses, and assist in managing vast amounts of medical data. However, alongside these benefits come serious ethical and practical concerns. If not carefully guided, LLMs can generate hallucinations confident sounding yet entirely fabricated information which can mislead clinical decision making. Moreover, these models often inherit and perpetuate biases from the data they are trained on, potentially exacerbating disparities in care for already marginalized or underrepresented patient populations. This paper explores how ethical prompt design the careful crafting of instructions and context given to LLMs can help address these risks. We focus on two key challenges: reducing hallucinated AI responses in medical contexts and minimizing bias that could negatively impact care quality for certain demographic groups. To tackle these issues, we propose a human in the loop framework, where clinicians and domain experts actively shape and evaluate prompts to ensure the output is safe, inclusive, and grounded in evidence based medicine