Impact of academic collaboration and quality of clinical orthopaedic research conducted in low- and middle-income countries

Background: Little is known about the quality of orthopaedic investigations conducted in low- and middle-income countries (LMICs). Academic collaboration is one model to build research capacity and improve research quality. Our study aimed to determine (1) the quality of clinical orthopaedic research conducted in LMICs, (2) the World Bank Regions and LMICs that publish the highest quality studies, (3) the pattern of collaboration among investigators and (4) whether academic collaboration between LMIC and non-LMIC investigators is associated with studies that have higher levels of evidence. Methods: Orthopaedic studies from 2004 to 2014 conducted in LMICs were extracted from multiple electronic databases. The World Bank Region, level of evidence and author country-affiliation were recorded. Collaboration was defined as a study that included an LMIC with non-LMIC investigator. Results: There were 958 studies that met inclusion criteria of 22,714 searched. Ninety-seven (10.1%) of included studies achieved Level 1 or 2 evidence, but case series (52.3%) were the most common. Collaboration occurred in 14.4% of studies and the vast majority of these (88.4%) were among academic institutions. Collaborative studies were more likely to be Level 1 or 2 (20.3% vs. 8.4%, p < 0.01), prospective (34.8% vs. 22.9% p = 0.04) and controlled (29.7% vs. 14.4%, p < 0.01) compared to non-collaborative studies. Conclusions: Although orthopaedic studies in LMICs rarely reach Level 1 or 2 evidence, studies published through academic collaboration between LMIC and non-LMIC investigators are associated with higher levels of evidence and more prospective, controlled designs.


Introduction
There is increasing recognition of the importance of musculoskeletal disease as a global health issue, particularly as it relates to injury. An estimated 1.2 million deaths and 50 million nonfatal injuries occur each year due to road traffic accidents alone [1], with traumatic injuries contributing to more global disability than human immunodeficiency virus (HIV), tuberculosis and malaria combined [2,3]. The burden of injury is greatest in low-and middle-income countries (LMICs), where resource-limited orthopaedic surgeons face a daunting volume of musculoskeletal disease [2,4]. Efforts to address this disparity in care have manifested through surgical missions, donated implants and educational programmes [4,5]. Less, however, is known about the impact of clinical research [4,5]. Recent literature suggests that clinical orthopaedic research conducted in LMICs may help to answer relevant clinical questions and shape public health policy [4]. Although a growing body of research exists to inform treatment for orthopaedic disease in high-income countries (HICs), generalizing results to LMICs is difficult due to delays in treatment, differences in training and availability of equipment [6,7]. Thus, it would be ideal for orthopaedic surgeons practicing in LMICs to base their treatment on high-quality research conducted in a similar setting [4].
To date, the quality of clinical research conducted in LMICs has not been assessed. Orthopaedic surgeons weigh a study's level of evidence (LOE) before determining how the results apply clinical practice [8][9][10], as Level 1 or 2 studies can potentially reveal important therapeutic, diagnostic, prognostic and economic outcomes [11,12]. Given the importance of study quality in shaping treatment protocol, the state of orthopaedic research in developing countries requires comprehensive review. Some have hypothesized that collaboration between investigators from LMICs and non-LMICs (e.g. upper-middle and high-income countries) can improve study quality [4,13]. While there is growing evidence that investigators from LMICs strive to answer clinically relevant questions through research, they may lack resources, training and protected time away from clinical responsibilities [14]. However, collaboration with academically affiliated non-LMIC authors may help local investigators overcome these barriers through funding support, research education and division of labour [4]. Thus, investigations are needed to evaluate how international academic partnership can improve study quality.
Our study aimed to determine (1) the quality of clinical orthopaedic research conducted in LMICs, (2) the World Bank Regions and LMICs that publish the highest quality studies, (3) the pattern of collaboration among investigators and (4) whether academic collaboration between LMIC and non-LMIC investigators is associated with studies that have higher levels of evidence.

Material and methods
We conducted a scoping review using the Arksey and O'Malley framework with modifications from Levac et al. and Daudt et al. [15][16][17]. Five authors sought and assessed orthopaedic studies conducted in LMICs that were published between June 2004 and June 2014. The LMICs were defined as any country listed as a ''Low-income'' or ''Lower-middleincome'' economy according to the 2014 World Bank classification [18]. Eighty-two countries met our study's definition of LMIC (Table 1). The studies were identified by comprehensive textword and MeSH-based electronic searches of PubMed/ MEDLINE, EMBASE and Cochrane library that was developed with assistance from a research librarian. Our search strategy combined terms for orthopaedic surgery, LMICs, musculoskeletal injury, musculoskeletal anatomy and human studies to be as inclusive as possible (Appendix).
Full-text articles were assessed for eligibility in the order of title, abstract and manuscript. Included studies (1) deal primarily with a low-or lower-middle-income country (LMIC), (2) pertain to orthopaedic surgery, (3) enrol humans and (4) were original peer-reviewed publications. Studies that reported three cases or fewer, were non-English and pertained to a highincome country (HIC) at war, animals, biomechanics or laboratory values were excluded.
Each included study was read in its entirety and a REDCap survey for data extraction was created that identified the study location, author's academic affiliation and study quality. Study location and author affiliations were categorized into eight World Bank Regions: East Asia and Pacific, Europe and Central Asia, Latin America and the Caribbean, Middle East and North Africa, Latin America and the Caribbean, Middle East and North Africa, North America, South Asia (Table 2). Collaboration was determined by the investigator countryaffiliation and defined as LMIC-only, Multicentre (LMIC with LMIC investigator), Collaborative (LMIC with non-LMIC investigator) and non-LMIC (e.g. upper-middle or high-income country investigators) only. Academic partnerships were defined as a study where the authors from both LMICs and non-LMICs were affiliated with an academic institution.
Finally, the study quality was assessed by the levels of evidence (LOE), presence of control group, prospective or retrospective design, type of study and presence of randomization. Due to the heterogeneity of study design in the global orthopaedic literature, which includes qualitative and epidemiologic studies that cannot be categorized with the LOE scale, no other quality assessment tools were utilized. To determine the level of evidence of each study, we used the 2015 scale adopted by the Journal of Bone and Joint Surgery and derived from recommendations given by the Centre for Evidence-Based Medicine in Oxford, United Kingdom [19,20]. The LOE scale divides studies into four categories: Diagnostic, Prognostic, Therapeutic, and Economic and Decision Analyses. Each category can then be subdivided into Level 1-5 evidence, with each level having its own definition [19]. Level 1 and 2 studies were considered high levels of evidence, while those of Level 3, 4 and 5 were considered low levels of evidence. Studies that had no levels of evidence, such as epidemiologic or qualitative studies, were also noted. LOE was used as the primary indicator of study quality due to its ubiquitous use as a validated measure of study strength [9,11,[21][22][23]. In addition, studies that do include control groups, collect data prospectively and randomize interventions have been shown to produce higher quality evidence than studies that do not [8,10].

Data analysis
Descriptive statistics were used to summarize all data. Discrete variables were summarized as counts or proportions, and skewed continuous variables were reported as medians with interquartile ranges (IQRs). Chi-square analysis was used to determine the association between nominal variables. Significance was set at p < 0.05.

Results
Out of 22,714 unique articles assessed, 958 met all inclusion criteria ( Figure 1). Over the past decade, 265 (27.6%) of clinical orthopaedic research studies conducted in LMICs were epidemiologic or qualitative and had no LOE. Of the clinical studies that could be assessed with levels of evidence, only 97 (10.1%) were Level 1 or 2. Studies were most commonly designed as prospective or retrospective case series      partnerships. Collaborative studies were more likely to be Level 1 or 2 (20.3% vs. 8.4%, p < 0.01), prospective (34.8% vs. 22.9% p = 0.04) and controlled (29.7% vs. 14.4%, p < 0.01) compared to non-collaborative studies (Table 4). There was no difference in randomization between Collaborative and non-collaborative studies (p = 0.53).

Discussion
While the importance of evidence-based clinical practice is increasingly recognized in high-income countries, there is an enormous gap in the volume and quality of musculoskeletal research relevant to clinicians and patients in LMICs. Our study shows that the majority of clinical orthopaedic researches published in LMICs are designed as case series, and these studies rarely attain high levels of evidence, a phenomenon that has persisted over the past decade. Notably, there appears to be no significant difference between the proportion of Level 1 and 2 studies published in 2004 and the years leading up to 2014. Studies with low levels of evidence cannot accurately determine the treatment outcome and may be harmful if misinterpreted [11]. Our data shows that while the majority of therapeutic studies conducted in LMICs recommended the use of the tested intervention, they were mostly designed as case series. Given the importance of research in the effort to treat global orthopaedic disease, efforts to enable local researchers to produce high-quality studies are warranted.  One factor associated with higher quality research is academic collaboration. Our data suggest that while studies produced by collaboration between LMIC and non-LMIC investigators account for less than 15% of total research, they are associated with higher levels of evidence and more prospective, controlled designs.
Our finding that orthopaedic research conducted in LMICs was unlikely to attain high levels of evidence is similar to the literature published in major Western journals. For instance, Reich et al. evaluated seven years' worth of publications from The Journal of Bone and Joint Surgery (American) and found that only one of the Level 1 and 2 studies published was conducted in an LMIC [12]. In addition, the proportion of high level studies published in LMICs does not compare favorably with that of non-LMIC studies. An analysis of three American journals found that 23% of studies published in the last decade achieved Level 1 or 2 evidence [9], more than double the proportion of Level 1 and 2 LMIC-conducted studies during the same time frame. Notably, our data also suggests that the overall quality of LMIC publications has not improved over the past decade, which lies in stark contrast to studies of high impact journals that suggest the number of high-quality studies has increased significantly over time [9,12]. Future research into why a significant gap in research quality exists between high-resource and low-resource environments is warranted.
Even though the quality of research from LMICs is lower overall, studies conducted in South Asia and Sub-Saharan Africa were more likely to attain high levels of evidence. Notably, Pakistan, India, Nigeria and Nepal produced over twothirds of Level 1 and 2 studies in the past decade. In contrast, 59 of the world's 82 LMICs did not produce a single Level 1 or 2 study over the same time frame. This finding is similar to recent literature that indirectly suggests LMICs from South Asia produce the highest quality studies in Western journals compared to other LMIC regions [12]. However, it is still unknown why this disparity by region exists, which warrants further investigation. Collaboration between LMIC and non-LMIC investigators is associated with significantly higher quality of published clinical research with respect to levels of evidence, prospective design and presence of control groups. Notably, over 88% of these studies represented academic partnerships, in which both the LMIC and non-LMIC investigators were affiliated with an academic institution. Although, it is unclear why Collaborative studies are associated with higher research quality, recent literature has hypothesized that investigators from high-income countries can enable their counterparts in LMICs to conduct research through longitudinal partnership [4,13,24]. For example, Morshed et al. outline a potential model of partnership in which investigators from an academic HIC institution helped LMIC-based surgeons develop a prospective cohort study through assistance in protocol development, research training, funding and resource procurement [4]. Academic affiliation of LMIC partners may also be an important ingredient for successful partnerships, since academic advancement and research requirement for trainees are often cited by LMIC authors as incentives for research participation [4,14].
Our study had several limitations. First, given the heterogeneity of included articles, we were unable to apply additional standardized quality assessment tools, which may have affected our ability to gauge the study quality. However, LOE has been shown to be a validated measure of study quality and have adequate interobserver reliability [9,21]. Second, our review only included studies since June 2004 and could have been strengthened by including more publication years. However, our study provides a snapshot of study quality over the last decade and is by far the most comprehensive existing review of the global orthopaedics literature. Third, as a literature review, our study design cannot prove causal inference exists between academic collaboration and study quality. Our data does, however, show that non-LMIC and LMIC collaboration is associated with higher study quality, which warrants further investigation. Finally, our study did not include primarily non-English publications, which may have excluded highquality studies published in other languages. However, recent literature suggests that language restrictions of reviews may not produce meaningful bias [25], although future studies with broader language requirements are needed.

Conclusion
Recent literature has shown that many investigators who practice in LMICs have a strong interest in participating in research studies [14]. However, surgeons in LMICs contend with a lack of formalized research training, resources and protected time as significant barriers to improving study quality [4,14]. These barriers may explain why clinical orthopaedic research conducted in LMICs rarely attains high levels of evidence compared to those conducted in non-LMICs, a phenomenon that has not improved in the past decade. Thus, more research is needed to determine how LMIC investigators can best overcome impediments to high-quality research in resource-poor environments. Academic collaboration is one of the most promising solutions, as it is associated with higher quality studies published over the last decade. Thus, greater investment of resources into developing and investigating the impact of academic collaboration is warranted.

Funding
No funding to disclose.