When AI Gets It Wrong: Racial Bias in Mental Health Tools
Today, I want to highlight a new study that should be on the radar of anyone involved in mental health care: "Racial bias in AI-mediated psychiatric diagnosis and treatment: a qualitative comparison of four large language models," This study raises essential questions about the ethics and equity of using AI, particularly large language models (LLMs), in mental health contexts—including schools.
The Method
To uncover whether AI might be unintentionally reinforcing racial disparities in mental health care, researchers conducted a focused investigation into four major large language models (LLMs):
Claude
ChatGPT
Gemini
NewMes-15 (a locally trained, medical-specific model based on LLaMA3)
The Setup
The researchers designed ten psychiatric patient vignettes representing five diagnoses with known historical biases:
Depression
Anxiety
Schizophrenia
Eating disorders
ADHD
Each vignette was presented in three formats:
Race-neutral: All racial indicators were removed.
Race-implied: The patient name suggested African American identity.
Race-explicit: The patient was explicitly described as African American, using the same name.
Two expert raters (a clinical psychologist and a social psychologist) scored 120 AI responses for racial bias:
0 = Minimal difference
1 = Minor variation, possibly benign
2 = Meaningful change, likely race-related
3 = Clear evidence of a racially biased response
The raters were blinded to the source of each response and instructed to focus only on differences across the three conditions.
What Did They Find? The Unsettling Results
Treatment Bias Is Real
Across the board, LLMs exhibited more bias in treatment recommendations than in diagnostic judgments. This is especially alarming in schizophrenia and anxiety cases:
Gemini emphasized alcohol use in African American anxiety cases, but not in race-neutral ones.
Claude recommended guardianship for African American depression cases, but not others.
ChatGPT and NewMes-15 omitted ADHD medication suggestions for African American patients.
Minimal Diagnostic Bias
Diagnoses were generally consistent across all conditions, suggesting bias was more likely to influence how models responded than how they reasoned.
Differences Among Models
NewMes-15 showed the highest levels of treatment bias, frequently scoring a 3.
Gemini displayed the least overall bias, but still showed concerning patterns.
Explicit > Implicit Bias
Bias was more severe when race was explicitly mentioned, but it still emerged with only implied cues like names.
Implications for School Psychologists and Mental Health Providers
This study delivers a strong warning: we must approach AI with a critical eye, especially in diverse school communities.
AI May Reinforce Disparities: Even advanced LLMs can replicate historical patterns of discrimination.
Diagnosis Is Not the Whole Story: The real danger lies in skewed treatment recommendations, even if diagnoses appear sound.
Evaluate Outputs Carefully: AI-generated treatment suggestions must be reviewed with cultural and ethical awareness.
Local Models Aren't a Fix: NewMes-15's poor performance suggests that even "custom" models can perpetuate bias if trained on problematic data.
Training Data Matters: The racial representation and fairness of training datasets remain an unsolved challenge.
A Call to Action
This research is a wake-up call for the entire healthcare AI ecosystem: developers, clinicians, and policymakers must collaborate to develop transparent bias detection tools and equity-centered guidelines for safe and ethical AI use.
Until then, caution is not just advisable—it's essential. Our responsibility is to question, critique, and advocate for AI systems that serve all communities with fairness and respect.
What are your thoughts on this study? How should school psychologists and mental health professionals prepare for these challenges? Let me know in the comments below.