Want to stay informed about the rapidly evolving intersection of AI and school psychology practice? Sign up for my weekly newsletter where I explore these emerging issues and their implications for our field.

The rapid rise of artificial intelligence (AI) in healthcare has brought both excitement and skepticism to the forefront of clinical practice. As psychologists, we often wonder: Can AI truly enhance or even rival human diagnostic reasoning? A recent study published in JAMA Network Open in October 2024, titled "Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial," offers some intriguing insights into these questions.

The study explored the potential of AI, specifically a large language model (LLM), to improve physicians' ability to reason through challenging diagnoses. While the findings focus on medical diagnoses, they raise important implications for psychology as we consider how AI might transform our field.

The Study: An Overview

Researchers conducted a single-blind, randomized controlled trial involving 50 physicians from academic medical institutions. Participants ranged from residents to attending physicians in family medicine, internal medicine, and emergency medicine. Using stratified randomization, the physicians were divided into two groups:

1. Control Group: Had access to conventional diagnostic resources.

2. Intervention Group: Had access to the same resources plus a commercial LLM chatbot (ChatGPT Plus, powered by GPT-4).

Both groups tackled six complex clinical vignettes based on real patient cases, designed to mimic initial diagnostic evaluations. The tasks included:

• Identifying top differential diagnoses.

• Listing supporting and opposing factors.

• Selecting the most likely diagnosis.

• Outlining next steps for evaluation.

The results revealed something striking: while the LLM itself scored significantly higher than the human participants, physicians using the LLM as a tool did not show notable improvements in diagnostic reasoning.

What This Means for Psychologists

Although the study focuses on medical diagnoses, its findings prompt several questions about the potential role of AI in psychological diagnosis and assessment:

Could AI Improve Diagnostic Accuracy?

The fact that the LLM outperformed physicians independently suggests that AI can process and analyze complex diagnostic information with remarkable accuracy. For psychologists, this raises the possibility that AI might improve the accuracy of differential diagnoses in mental health, potentially leading to better treatment outcomes.

However, psychological diagnoses often involve more subjective factors, such as understanding a client’s lived experience or observing nuanced behavioral cues, which AI may struggle to interpret.

Impact on Clinical Judgment and Decision-Making

The study authors highlighted the importance of refining human-computer interactions to unlock AI’s full potential. For psychologists, this means considering how AI tools might complement—rather than replace—our clinical judgment. While AI might streamline tasks like synthesizing assessment data or identifying patterns, it cannot replicate the relational and empathic aspects of psychological practice.

The Need for Training and Ethical Guidelines

The study underscores the importance of prompt engineering—crafting specific instructions to optimize AI performance. Psychologists will need training not only in using AI tools effectively but also in addressing ethical concerns, including:

• Protecting patient privacy and data security.

• Mitigating bias in AI algorithms.

• Ensuring transparency in how AI-derived insights are used in clinical decisions.

As psychologists, we must also grapple with broader questions: How do we maintain accountability when relying on AI? How do we address disparities if AI tools are less accessible to certain populations?

Conclusion: Harnessing AI with Care

While AI holds incredible promise for improving diagnostic accuracy, the JAMA Network Open study reminds us that technology alone is not enough. To truly benefit from AI, psychologists must approach it with a combination of curiosity and caution. By integrating AI thoughtfully into our practice—and addressing its ethical, technical, and human challenges—we can ensure that this powerful tool serves both clinicians and clients.

As we navigate this evolving landscape, one question remains at the forefront: How can we, as psychologists, harness AI to enhance our work while staying rooted in the human connection that defines our profession?

Stay connected with the latest developments in AI and school psychology! Sign up for my weekly newsletter where I dive deep into these important issues, provide practical guidance, and explore how we can thoughtfully integrate AI while maintaining the highest standards of practice in school psychology. Join our community of forward-thinking school psychology professionals as we navigate this transformative period in our field.