Today I want to talk about the potential role of LLMs like ChatGPT in mental health triage. A recent study explored how well these models can assess psychiatric emergencies, and I think it has some interesting implications for us in the school setting.

The Study: How Was it Conducted?

The study, titled Potential of ChatGPT in Youth Mental Health Emergency Triage: Comparative Analysis with Clinicians, focused on comparing the triage decisions of GPT-4 models to those of clinicians (nurse practitioners) using clinical vignettes. Here's a breakdown of their methods:

• Vignettes: Researchers created 22 realistic scenarios depicting common psychiatric emergencies in young adults, including psychosis, suicidal ideation, substance abuse, anxiety, and violent behavior. These scenarios included details like age, gender, symptoms, and history.

• GPT Models: They tested three versions of GPT-4: GPT-4o, GPT-4o Mini, and GPT-4 Legacy.

• Triage Criteria: Both the GPT models and clinicians were asked to assess each vignette based on three criteria: risk level (low, moderate, high), need for admission (yes/no), and urgency of evaluation (low, moderate, high).

• Evaluation: Psychiatrists evaluated the responses from the GPT models and clinicians on triage accuracy, clarity, completeness, and overall score.

Key Results: What Did They Find?

The results of the study were pretty interesting:

• Agreement: There was substantial agreement between the clinicians and the GPT models in their triage decisions.

• GPT Model Differences: There were some differences among the GPT models. GPT-4o and GPT-4 Legacy performed better than GPT-4o Mini (which is designed for speed) in terms of triage accuracy, clarity, and completeness.

• Over-Admission: The GPT models tended to recommend admission more often than clinicians. This is likely due to their programming, which prioritizes safety and minimizing risk. However, the GPT-4o model did not over-admit any patients.

• No Missed Admissions: Importantly, the GPT models did not miss any cases where clinicians recommended admission.

Implications for School Psychologists: How Does This Affect Us?

While we aren't typically doing crisis triage in the same way as emergency departments, there are some key takeaways for our work in schools:

• AI as a Support Tool: This study suggests that LLMs, could be valuable as a support tool for assessing student mental health needs. In our roles as school psychologists, we could potentially use these tools to help us quickly evaluate the severity of a situation, especially during periods of high demand (like during a crisis response). They could support us in making well-informed decisions, as these models are capable of organizing and collecting high-quality information.

• Identifying Risk: The study shows that LLMs are capable of assessing risk in a way that is similar to that of clinicians. For school psychologists, assessing risk is an important responsibility, and AI tools might provide an additional layer of support, particularly when we're dealing with multiple students in distress.

• Limitations: It's important to remember that AI is not a replacement for human judgment and care. It cannot replicate the "human touch" and empathy we provide. We need to be wary of the tendency for LLMs to over-refer.

• Ethical Considerations: There are also important ethical considerations to keep in mind as we consider integrating AI into our work. We must be aware of how the data used by these models is collected, stored, and used. We must always prioritize students’ best interests.

• Further Research Needed: The authors noted that more research is needed using real-world data and more diverse scenarios. We should keep an eye out for additional studies as this technology evolves and matures.

Moving Forward

This study shows that LLMs have the potential to be a tool in mental health triage and could be beneficial in the school psychology setting as well. It's worth keeping an eye on how this technology develops and consider how it might help us in our mission to support students' mental health. This technology could help us better serve our students and reduce the strain on emergency departments.

I believe that AI tools will be developed for this purpose for school psychologists soon - if you're working on this, please let me know!