A cross-sectional study on ChatGPT’s alignment with clinical practice guidelines in musculoskeletal rehabilitation

Abstract

Background

AI models like ChatGPT have the potential to support musculoskeletal rehabilitation by providing clinical insights. However, their alignment with evidence-based guidelines needs evaluation before integration into physiotherapy practice.

Objective

To evaluate the performance of ChatGPT (GPT-4 model) in generating responses to musculoskeletal rehabilitation queries by comparing its recommendations with evidence-based clinical practice guidelines (CPGs).

Design

This study was designed as a cross-sectional observational study.

Methods

Twenty questions covering disease information, assessment, and rehabilitation were developed by two experienced physiotherapists specializing in musculoskeletal disorders. The questions were distributed across three anatomical regions: upper extremity (7 questions), lower extremity (9 questions), and spine (4 questions). ChatGPT’s responses were obtained and evaluated independently by two raters using a 5-point Likert scale assessing relevance, accuracy, clarity, completeness, and consistency. Weighted kappa values were calculated to assess inter-rater agreement and consistency within each category.

Results

ChatGPT’s responses received the highest average score for clarity (4.85), followed by accuracy (4.62), relevance (4.50), and completeness (4.20). Consistency received the lowest score (3.85). The highest agreement (weighted kappa = 0.90) was observed in the disease information category, whereas rehabilitation displayed relatively lower agreement (weighted kappa = 0.56). Variability in consistency and moderate weighted kappa values in relevance and clarity highlighted areas requiring improvement.

Conclusions

This study demonstrates ChatGPT’s potential in providing guideline-aligned information in musculoskeletal rehabilitation. However, due to observed limitations in consistency, completeness, and the ability to replicate nuanced clinical reasoning, its use should remain supplementary rather than as a primary decision-making tool. While it performed better in disease information, as evidenced by higher inter-rater agreement and scores, its performance in the rehabilitation category was comparatively lower, highlighting challenges in addressing complex, nuanced therapeutic interventions. This variability in consistency and domain-specific reasoning underscores the need for further refinement to ensure reliability in complex clinical scenarios.

Clinical trial number

Not applicable.

Background

Artificial intelligence (AI) has become an integral component of modern healthcare, offering innovative tools to enhance diagnostics, clinical decision-making, and patient care efficiency [1, 2]. Among AI applications, natural language processing (NLP) models such as ChatGPT have garnered significant attention due to their ability to generate human-like responses to complex medical inquiries [3, 4]. These models are trained on vast datasets derived from diverse information sources, enabling them to produce contextually relevant and coherent outputs. However, ensuring the alignment of AI-generated content with evidence-based clinical practice guidelines (CPGs) is paramount to establishing their reliability in medical practice [5].

Recent studies have demonstrated the potential of AI tools in various healthcare domains, including disease diagnosis, treatment planning, and patient education. In musculoskeletal physical therapy, ChatGPT has shown notable promise, achieving up to 80% compliance with CPGs [6,7,8]. Additionally, it has been proposed as a means to optimize workflows in resource-limited settings where rapid access to accurate clinical information is crucial [9]. Despite these advantages, concerns persist regarding the consistency and reliability of AI-generated responses, particularly in complex clinical scenarios that demand nuanced clinical reasoning [10, 11]. While AI models like ChatGPT can generate well-structured and plausible content, they may also produce inaccurate or misleading information, underscoring the necessity for critical evaluation by healthcare professionals [12].

Musculoskeletal rehabilitation, a cornerstone of physical therapy, plays a crucial role in restoring function, alleviating pain, and enhancing the quality of life for individuals with musculoskeletal disorders [13]. For physiotherapists, adherence to evidence-based CPGs ensures the effectiveness and standardization of treatments, thereby optimizing patient outcomes and maintaining professional accountability [14, 15]. Integrating AI systems such as ChatGPT presents an opportunity to support physiotherapists by providing rapid, guideline-consistent recommendations, particularly in time-constrained or resource-limited settings [9]. Previous research has predominantly examined ChatGPT’s responses in diagnosing specific musculoskeletal pathologies or surgical conditions, such as lumbar radicular pain [11], degenerative spondylolisthesis [7], as well as surgical procedures like anterior cruciate ligament (ACL) reconstruction [10], and rotator cuff repairs [16]. Additionally, its application in clinical decision support has been explored [8]. However, physiotherapy practice encompasses not only diagnostics but also detailed patient assessment and evidence-based rehabilitation interventions, requiring comprehensive clinical reasoning beyond mere diagnostic capabilities. Yet, the current literature lacks an in-depth evaluation of ChatGPT’s capability to provide physiotherapy-specific recommendations, particularly regarding assessment strategies and rehabilitation interventions aligned with CPGs. Unlike previous studies that focused primarily on medical diagnoses or surgical scenarios, our study specifically investigates how ChatGPT performs in physiotherapy-specific domains such as assessment and rehabilitation—core components of physiotherapeutic management in musculoskeletal care. Therefore, the novelty of this study lies in its focused evaluation of ChatGPT’s performance in three physiotherapy-specific domains: disease information, patient assessment, and evidence-based rehabilitation practices—areas that have been less explored in previous research. By identifying areas where AI aligns with or diverges from established guidelines, this research seeks to inform the potential role of ChatGPT in clinical education and decision support within musculoskeletal rehabilitation.

Methods

Study design

This study employed a cross-sectional observational design to compare recommendations generated by evidence-based CPGs with those provided by ChatGPT’s GPT-4 model for musculoskeletal conditions. A total of twenty questions were systematically developed by two physiotherapists, each with over eight years of clinical experience in musculoskeletal rehabilitation. The development process began by identifying the most commonly encountered musculoskeletal conditions and clinical decision-making challenges in physiotherapy. Only conditions with established and accessible CPGs were included to ensure objective benchmarking. The questions were then categorized into three key domains: disease information, patient assessment, and rehabilitation. Additionally, efforts were made to ensure a balanced anatomical distribution—seven questions for the upper extremity, nine for the lower extremity, and four for the spine. A purposive sampling approach was adopted, and the number of questions was determined based on the need to cover a diverse range of physiotherapy-relevant scenarios while ensuring feasibility for expert rating and statistical analysis.

Each question was submitted in a new session to avoid memory retention or contextual influence between prompts. While ChatGPT does not learn from individual prompts in real time, periodic updates released by OpenAI may influence model performance; therefore, specifying the data collection period enhances reproducibility. The model was explicitly instructed to respond from the perspective of a physiotherapist to simulate clinical reasoning and decision-making comparable to that of an experienced practitioner. All responses were recorded verbatim to ensure accuracy and consistency during subsequent analysis.

Outcome measurement

The responses generated by ChatGPT were independently assessed by two musculoskeletal physiotherapists, each with over eight years of clinical experience. The two physiotherapists who evaluated the responses were also involved in developing the questions. To mitigate potential bias, each question was discussed and evaluated independently by both raters using a predefined Likert scale, and inter-rater agreement was statistically analyzed using weighted kappa coefficients.

A 5-point Likert scale (ranging from 1 = Strongly Disagree to 5 = Strongly Agree) was used to evaluate the following predefined response characteristics:

Relevance

Does the response directly address the question posed?

Accuracy

Is the information provided accurate and consistent with current clinical practice guidelines?

Clarity

Is the response well-organized and easy to comprehend?

Completeness

Does the response comprehensively address all aspects of the question?

Consistency

How consistent are the responses generated by ChatGPT when the same question is posed multiple times?

To facilitate comparative analysis, Table 1 presents the questions, CPG-derived answers, and ChatGPT’s verbatim responses, offering a comprehensive overview of the dataset.

Table 1 Questions with corresponding ChatGPT and guideline-based responses

Statistical analysis

The average scores from the two raters’ Likert scale evaluations were computed. To assess inter-rater agreement, weighted kappa coefficients were calculated for each evaluation criterion: relevance, accuracy, clarity, completeness, and consistency. Furthermore, the dataset was stratified into three primary categories—disease information, assessment, and rehabilitation—to examine inter-rater agreement within each category. Weighted kappa values were computed both for the overall dataset and within these specific domains.

All statistical analyses, including weighted kappa computations, were conducted using IBM SPSS Statistics version 25.

Results

A comparative analysis was conducted on the 20 predefined questions and their corresponding ChatGPT-generated responses, with the original questions, ChatGPT responses, and guideline-based answers presented in Table 1.

Section-wise average scores revealed that clarity received the highest mean score of 4.85, suggesting that ChatGPT’s responses were well-structured and easy to comprehend. In contrast, consistency obtained the lowest mean score of 3.85, indicating notable variability in responses when the same question was posed multiple times. Additional scores included accuracy at 4.62, relevance at 4.50, and completeness at 4.20.

Inter-rater agreement, evaluated using weighted kappa values, varied across the assessed criteria. Consistency exhibited the highest inter-rater agreement, with a weighted kappa of 0.88, followed by completeness (κ = 0.71) and accuracy (κ = 0.57). Moderate agreement was observed for relevance (κ = 0.45), whereas clarity exhibited the lowest agreement, with a weighted kappa of 0.27. The question-wise average scores and weighted kappa values for each criterion are detailed in Table 2.

Table 2 Question-wise average scores with weighted kappa values

Further subgroup analysis based on question categories (disease information, assessment, and rehabilitation) revealed notable differences. The disease information category exhibited the highest agreement, with a weighted kappa of 0.90, suggesting ChatGPT provided consistent and accurate responses in this domain. In contrast, the rehabilitation category showed lower agreement, with a weighted kappa of 0.56, indicating greater variability in ChatGPT’s performance when addressing rehabilitation-related queries. The average scores and weighted kappa values for each category are summarized in Table 3.

Table 3 Category-wise average scores and weighted kappa values

Discussion

The findings of this study provide valuable insights into both the potential and limitations of ChatGPT’s GPT-4 model in addressing queries related to musculoskeletal rehabilitation. By systematically comparing ChatGPT’s responses with evidence-based CPGs, this study highlights the model’s ability to generate clinically relevant and accurate information while also identifying areas requiring improvement. ChatGPT demonstrated notable strengths in clarity and accuracy, as reflected in its high ratings. The highest mean score for clarity underscores the AI model’s ability to present well-structured and easily comprehensible information. Another noteworthy strength was the model’s performance in relevance and accuracy (4.62), reflecting its ability to provide contextually appropriate and largely evidence-aligned responses. Furthermore, the high weighted kappa values for consistency and completeness in the disease information category suggest that ChatGPT’s responses are not only comprehensive but also exhibit minimal variability when addressing well-defined queries. Despite these strengths, the lowest scores for completeness and consistency highlight areas in need of improvement. Variability in responses to repeated queries raises concerns about reliability, particularly in scenarios requiring precise and consistent recommendations. The rehabilitation category exhibited the lowest weighted kappa value, suggesting challenges in addressing the nuanced and complex aspects of therapeutic interventions.

These results align with prior research emphasizing the strengths and challenges of AI-driven decision-support tools in healthcare [17, 18]. For instance, Bilika et al. explored ChatGPT’s application in physiotherapy decision-making, underscoring the importance of its cautious use and the need for informed judgment in clinical practice [19]. Such clarity enhances the usability of ChatGPT as an educational tool for physiotherapists and other healthcare professionals, particularly in environments where rapid access to clear and concise information is critical. These results are consistent with earlier studies demonstrating ChatGPT’s adherence to clinical guidelines in musculoskeletal care, with reported compliance rates reaching up to 80% in similar contexts [8]. This finding underscores ChatGPT’s potential as a decision-support tool, particularly for routine clinical queries. This may be attributed to limitations in ChatGPT’s training data, particularly concerning specialized and context-sensitive rehabilitation scenarios. For example, Gianola et al. found inconsistencies in ChatGPT’s recommendations for lumbosacral radicular pain compared to CPGs, raising concerns about accuracy and internal consistency [11]. Similarly, Sawamura et al. [12] concluded that although ChatGPT can generate accurate responses, its reference reliability and selection remain notable limitations. Consequently, they emphasized the necessity of cautious use, as ChatGPT is not entirely dependable for clinical decision-making. Additionally, the moderate inter-rater agreement for relevance and the low agreement for clarity emphasize the need for refining ChatGPT’s training algorithms to enhance alignment with evidence-based guidelines. Such variability in content quality has also been documented in studies examining AI-generated responses to complex medical queries [20].

Limitations

This study has several limitations that should be acknowledged. First, only two assessors were involved in the evaluation, which may limit the generalizability of inter-rater agreement results. Second, the assessors also developed the questions, introducing potential bias despite independent scoring and statistical analysis of agreement. Third, while the assessment criteria were informed by existing literature and expert consensus, the rating tool itself has not been psychometrically validated. Additionally, the study was limited to a fixed set of 20 questions, which, although diverse, may not fully represent the range of real-world clinical scenarios. Finally, the findings reflect ChatGPT’s performance during a specific time window and may not apply to future updates of the model.

Implications for clinical practice

The findings of this study carry significant implications for the integration of AI tools like ChatGPT into musculoskeletal rehabilitation. While the model demonstrates promise in delivering clear and accurate information, its limitations necessitate careful implementation. ChatGPT should be utilized as a supplementary resource rather than a primary decision-making tool, ensuring that AI-generated responses are cross-checked against established clinical guidelines to maintain accuracy. Furthermore, the results underscore ChatGPT’s potential role in clinical education. By providing readily accessible, guideline-consistent information, ChatGPT can serve as a valuable resource in the training of physiotherapists and other healthcare professionals. However, further refinement of the model is necessary to enhance its ability to address the complexities of rehabilitation interventions and ensure adherence to domain-specific practices. Particularly, improving the reliability of ChatGPT’s recommendations in rehabilitation contexts could significantly expand its clinical utility. At the same time, educators should be mindful that reliance on AI tools does not replace the need for students to engage critically with original sources, learn how to interpret clinical guidelines, and develop independent reasoning skills.

In summary, ChatGPT exhibits high potential in musculoskeletal rehabilitation education and information retrieval but requires cautious integration into practice due to its variability in complex clinical reasoning tasks. Ongoing evaluation and refinement of AI tools are essential to enhance their reliability, particularly in context-specific applications such as physiotherapy.

Future directions

To address the identified limitations, future research should focus on fine-tuning ChatGPT for specific healthcare domains, particularly musculoskeletal rehabilitation. Incorporating domain-specific datasets and enhancing the model’s ability to interpret complex clinical scenarios could substantially improve its practical applicability. Additionally, longitudinal studies assessing the long-term impact of ChatGPT’s integration into clinical practice are warranted.

Furthermore, the role of contextual and placebo effects in shaping user perceptions of AI-generated responses requires further exploration. As highlighted in this study, non-specific factors—such as response presentation and perceived authority—may influence both the acceptability and perceived reliability of AI tools. Investigating these factors could inform strategies to optimize the design and deployment of AI systems in healthcare, ensuring their effective and responsible use. Future research should include controlled trials assessing the impact of ChatGPT use in clinical education and patient care decision-making, particularly in real-time clinical scenarios.

Conclusion

This study underscores ChatGPT’s dual role as both a promising and a challenging tool in musculoskeletal rehabilitation. While it excels in clarity, relevance, and accuracy, its limitations in consistency and domain-specific reasoning necessitate careful oversight and ongoing refinement. Addressing these challenges could enable ChatGPT to evolve into a reliable decision-support tool, ultimately enhancing clinical practice, education, and patient care in musculoskeletal rehabilitation.

Key Points

Findings

• ChatGPT’s responses demonstrated high clarity (average score of 4.85) and relevance (4.50) when compared to clinical guidelines for musculoskeletal rehabilitation.

• The lowest score was observed in consistency (3.85), highlighting variability in repeated responses.

Implications

• These findings suggest that ChatGPT has potential as a supplementary tool for physiotherapists, offering guideline-aligned recommendations in musculoskeletal care.

• Enhancing the model’s consistency could improve its reliability for routine clinical use and decision support.

Caution

• The study relied on a predefined set of questions, which may not represent the full complexity of clinical practice.

• The evaluation was based on subjective scoring, which, despite using experienced raters, introduces potential variability. Furthermore, although the assessment criteria were informed by existing literature and expert consensus, the rating tool itself has not been formally validated.