Ciccotti MG

Anterior cruciate ligament (ACL) injuries affect >175,000 people each year,¹ with >100,000 Americans undergoing ACL reconstruction annually.² Due to the high impact this injury has on the general population, and especially on athletes, it is important to determine the factors that influence a patient’s selection of a particular graft type. With increasing access to information and other outside influences, surgeons should attempt to provide as much objective information as possible in order to allow patients to make appropriate informed decisions regarding their graft choice for ACL surgery.

While autografts are used in >60% of primary ACL reconstructions, allografts are used in >80% of revision procedures.³ Both autografts and allografts offer advantages and disadvantages, and the advantages of each may depend on patient age, activity level, and occupation.⁴ For example, graft rerupture rates have been shown to be higher in patients with ACL allografts⁴, while kneeling pain has been shown to be worse in patients with bone-patellar tendon-bone (BPTB) autografts compared to hamstring autografts⁵ as well as BPTB allografts.⁴

Patient satisfaction rates are high for ACL autografts and allografts. Boonriong and Kietsiriroje⁶ have shown visual analog scale (VAS) patient satisfaction score averages to be 88 out of 100 for BPTB autografts and 93 out of 100 for hamstring tendon autografts. Fox and colleagues⁷ showed that 87% of patients were completely or mostly satisfied following revision ACL reconstruction with patellar tendon allograft. Cohen and colleagues⁸ evaluated 240 patients undergoing primary ACL reconstruction; 63.3% underwent ACL reconstruction with an allograft and 35.4% with an autograft. Of all patients enrolled in the study, 93% were satisfied with their graft choice, with 12.7% of patients opting to choose another graft if in the same situation again. Of those patients, 63.3% would have switched from an autograft to allograft. Although these numbers represent high patient satisfaction following a variety of ACL graft types, it is important to continue to identify graft selection factors in order to maximize patient outcomes.

The purposes of this prospective study were to assess patients’ knowledge of their graft type used for ACL reconstruction, to determine the most influential factors involved in graft selection, and to determine the level of satisfaction with the graft of choice at a minimum of 1-year follow-up. Based on a previous retrospective study,8 we hypothesized that physician recommendation would be the most influential factor in ACL graft selection. We also hypothesized that patients receiving an autograft would be more accurate in stating their graft harvest location compared to allograft patients.

Materials and Methods

We prospectively enrolled 304 patients who underwent primary ACL reconstruction from January 2008 to September 2013. Surgery was performed by 9 different surgeons within the same practice. All patients undergoing primary ACL reconstruction were eligible for the study.

All surgeons explained to each patient the pros and cons of each graft choice based upon peer-reviewed literature. Each patient was allowed to choose autograft or allograft, although most of the surgeons strongly encourage patients under age 25 years to choose autograft. One of the surgeons specifically encourages a patellar tendon autograft in patients under age 30 to 35 years, except for those patients with a narrow patellar tendon on magnetic resonance imaging, in which case he recommends a hamstring autograft. Another surgeon also specifically encourages patellar tendon autograft in patients under 35 years, except in skeletally immature patients, for whom he encourages hamstring autograft. However, none of the surgeons prohibited patients from choosing autograft or allograft, regardless of age.

The Institutional Review Board at our institution provided approval for this study. At the first postoperative follow-up appointment, each patient completed a questionnaire asking to select from a list the type (“your own” or “a cadaver”) and harvest site of the graft that was used for the surgery. Patients were also asked how they decided upon that graft type by ranking a list of 4 factors from 1 to 4. These included (1) physician recommendation, (2) family/friend’s recommendation, (3) coach’s recommendation, and (4) the media. Patients had the option of ranking more than one factor as most important in their decision. In addition, patients were asked to list any other factors that influenced their decision regarding graft type.

At a minimum of 1 year following surgery, patients completed the same questionnaire described above. In addition, patients were asked if they were satisfied with their graft and whether they would choose the same graft type if undergoing ACL reconstruction again. Patients who would have chosen a different graft were asked which graft they would have chosen and why. Any patient who experienced graft rupture prior to follow-up was included in the analysis.

Statistical Analysis

Chi square tests were used to compare dichotomous outcomes. A type I error of less than 5% (P < .05) was considered statistically significant.

Results

At least 1 year following ACL reconstruction, 213 of 304 patients (70%) successfully completed the same questionnaire as they did at their first postoperative follow-up appointment. The mean age of these patients at the time of surgery was 31.9 ± 11.0 years (range, 13.9 to 58.0 years). The mean follow-up time was 1.4 ± 0.4 years (range, 1.0 to 2.6 years), and 59% of these patients were male.

Autografts were used for 139 patients (139/304, 46%), allografts for 156 patients (156/304, 51%), and hybrid grafts for 9 patients (9/304, 3%). Overall, 77% of patients were accurate in stating the type of graft used for their ACL reconstruction, including 88% of autograft patients, 71% of allograft patients, and 11% of hybrid graft patients (Table 1). Patients who underwent reconstruction with an autograft were significantly more accurate in stating their graft type compared to patients with an allograft (P < .001). Graft type by surgeon is shown in Table 2. A statistically significant difference was found in the proportion of patients choosing autograft versus allograft based on surgeon (P < .0001).

When asked which type of graft was used for their surgery, 12 of 304 patients (4%) did not know their graft type or harvest location. Twenty-nine patients stated that their graft was an allograft but did not know the harvest location. Five patients stated that their graft was an autograft but did not know the harvest location. The 34 patients who classified their choice of graft but did not know the harvest site (11%) stated their surgeon never told them where their graft was from or they did not remember. A complete list of graft type responses is shown in Table 3.

Of the 29 patients who stated that their graft was an allograft but did not know the harvest location, 19 (66%) had a tibialis anterior allograft, 7 (24%) had a BPTB allograft, 2 (7%) had an Achilles tendon allograft, and 1 (3%) had a tibialis anterior autograft.

Physician recommendation was the most important decision-making factor listed for 82% of patients at their first postoperative appointment (Table 4). In addition to the 4 factors listed on our survey, patients were allowed to write in other factors involved in their decision. The most popular answers included recovery time, personal research on graft types, and prior personal experience with ACL reconstruction on the contralateral knee.

At the time of 1-year follow-up, 205 of 213 patients (96%) said they were satisfied with their graft choice (Table 5). All 4 unsatisfied autograft patients received a hamstring autograft, 3 of which were performed by the same surgeon. No significant difference was found in satisfaction rates between patients with autograft vs allograft (P = .87). There was a higher satisfaction rate among patients with a BPTB autograft compared to those with a hamstring autograft (P = .043). Of the unsatisfied patients, 3 patients stated that their graft had failed in the time prior to follow-up and 2 patients stated that they were having donor site pain following surgery with hamstring autograft and would consider an allograft if the reconstruction were repeated (Table 6). Two patients stated that they were unsatisfied with their graft but would need to do more research before deciding on a different graft type.

As shown in Tables 5 and 6, there is a discrepancy between the number of patients who were unsatisfied with their graft and the number of patients who stated that they would switch to a different graft type if they were to have ACL reconstruction again. A number of patients stated that they were satisfied with their graft, yet they would switch to a different graft. The main reasons for this related to issues from a hamstring autograft harvest site. One patient noted that although she was satisfied with her graft, she would switch after doing further research.

Discussion

Determining the decision-making factors for patients choosing between graft types for ACL reconstruction is important to ensure that patients can make a decision based on objective information. Several previous studies have evaluated patient selection of ACL grafts.^8-10 All 3 of these studies showed that surgeon recommendation is the primary factor in a patient’s decision. Similar to previous studies, we also found that physician recommendation is the most influential factor involved in this decision.

At an average follow-up of 41 months, Cohen and colleagues⁸ found that 1.3% of patients did not know whether they received an autograft or allograft for their ACL reconstruction. Furthermore, 50.7% of patients stating they received an allograft in Cohen’s study⁸ were unsure of the harvest location. In our study, 4% of patients at their first postoperative visit did not know whether they had received an autograft or allograft and 10% of patients stating they received an allograft selected an unknown harvest site. In contrast, only 2% of autograft patients in our study were unsure of the harvest location at their first postoperative appointment. It is likely that, over time, patients with an allograft forget the harvest location, whereas autograft patients are more likely to remember the location of harvest. This is especially true in patients with anterior knee pain or hamstring pain following ACL reconstruction with a BPTB or hamstring tendon autograft, respectively.

In terms of patients’ knowledge of their graft type, we found an overall accuracy of 77%, with 88% of autograft patients, 71% of allograft patients, and 11% of hybrid graft patients remembering their graft type and harvest location. Although we do not believe it to be critical for patients to remember these details, we do believe that patients who do not know their graft type likely relied on the recommendation of their physician.

We found a significant difference in the proportion of patients choosing autograft vs allograft based on surgeon, despite these surgeons citing available data in the literature to each patient and ultimately allowing each patient to make his or her own decision. This is partly due to the low sample size of most of the surgeons involved. However, the main reason for this distortion is likely that different surgeons may highlight different aspects of the literature to “spin” patients towards one graft or another in certain cases.

Currently, there remains a lack of clarity in the literature on appropriate ACL graft choices for patients. With constant new findings being published on different aspects of various grafts, it is important for surgeons to remain up to date with the literature. Nevertheless, we believe that certain biases are inevitable among surgeons due to unique training experiences as well as experience with their own patients.

Cohen and colleagues8 found that only 7% of patients reported that their own personal research influenced their decision, and only 6.4% of patients reported the media as their primary decision-making factor. Cheung and colleagues⁹ conducted a retrospective study and found that more than half of patients did significant personal research prior to making a decision regarding their graft type. Most of this research was done using medical websites and literature. Koh and colleagues¹⁰ noted that >80% of patients consulted the internet for graft information before making a decision. Koh’s study¹⁰ was performed in Korea and therefore the high prevalence of internet use may be culturally-related.

Overall, quality of information for patients undergoing ACL reconstruction is mixed across the internet, with only 22.5% of top websites being affiliated with an academic institution and 35.5% of websites authored by private physicians or physician groups.¹¹ Although a majority of internet websites offer discussion into the condition and surgical procedure of ACL reconstruction, less than half of these websites share the equally important information on the eligibility for surgery and concomitant complications following surgery.¹¹In our study, only 39 patients (13%) listed the media as either the first (13, 4%) or second (26, 9%) most important factor in their graft decision. Clearly there is some discrepancy between studies regarding the influence of personal research and media. There are a few potential reasons for this. First, we did not explicitly ask patients if their own personal research had any influence on their graft decision. Rather, we asked patients to rank their decision-making factors, and few patients ranked the media as their first or second greatest influence. Second, the word “media” was used in our questionnaire rather than “online research” or “internet.” It may seem somewhat vague to patients what the word “media” really means in terms of their own research, whereas listing “online research” or “internet” as selection options may have influenced patient responses.

In our study, we asked patients for any additional factors that influenced their graft choice. Thirteen patients (4%) noted that “personal research” through internet, orthopaedic literature, and the media influenced their graft decision. This corroborates the idea that “media” may have seemed vague to some patients. Of these patients, 9 chose an autograft and 4 chose an allograft. The relative ease in accessing information regarding graft choice in ACL reconstruction should be noted. Numerous websites offer advice, graft options, and commentary from group practices and orthopaedic surgeons. Whether or not these sources provide reasonable support for one graft vs another graft remains to be answered. The physician should be responsible for providing the patient with this collected objective information.

In our study, 205 patients (96%) were satisfied with their graft choice at the time of follow-up, with 15 patients (7%) stating that they would have chosen a different graft type if they could redo the operation. Cheung and colleagues⁹ found a satisfaction rate of 87.4% at an average follow-up time of 19 months, with 4.6% stating they would have chosen a different graft type. Many factors can contribute to patient satisfaction after ACL reconstruction. Looking at patient variables such as age, demographics, occupation, activity level, surgical technique including tunnel placement and fixation, postoperative rehabilitation, and graft type may influence the success of the patient after ACL reconstruction.

The strengths of this study include the patient population size with 1-year follow-up as well as the prospective study design. In comparison to a previous retrospective study in 2009 by Cohen and colleagues⁸with a sample size of 240 patients, our study collected 213 patients with 70% follow-up at minimum 1 year. Collecting data prospectively ensures accurate representation of the factors influencing each patient’s graft selection, while follow-up data was useful for patient satisfaction.

The limitations of this study include the percentage of patients lost from follow-up as well as any bias generated from the organization of the questionnaire. Unfortunately, with a younger, transient population of patients undergoing ACL reconstruction in a major metropolitan area, a percentage of patients are lost to follow-up. Many attempts were made to locate these patients. Another potential limitation was the order of decision factors listed on the questionnaire. These factors were not ordered randomly on each survey, but were listed in the following order: (1) physician recommendation (2) family/friend’s recommendation (3) coach’s recommendation and (4) the media. This may have influenced patient responses. The organization of these factors in the questionnaire started with physician recommendation, which may have influenced the patient’s initial thought process of which factor had the greatest influence in their graft decision. In addition, for the surveys completed at least 1 year following surgery, some patients were contacted via e-mail and others via telephone. Thus, some patients may have changed their answers if they were able to see the questions rather than hearing the questions. We believe this is particularly true of the question regarding graft harvest site.

Our study indicates that the majority of patients undergoing ACL reconstruction are primarily influenced by the physician’s recommendation.

Anterior cruciate ligament (ACL) injuries affect >175,000 people each year,¹ with >100,000 Americans undergoing ACL reconstruction annually.² Due to the high impact this injury has on the general population, and especially on athletes, it is important to determine the factors that influence a patient’s selection of a particular graft type. With increasing access to information and other outside influences, surgeons should attempt to provide as much objective information as possible in order to allow patients to make appropriate informed decisions regarding their graft choice for ACL surgery.

While autografts are used in >60% of primary ACL reconstructions, allografts are used in >80% of revision procedures.³ Both autografts and allografts offer advantages and disadvantages, and the advantages of each may depend on patient age, activity level, and occupation.⁴ For example, graft rerupture rates have been shown to be higher in patients with ACL allografts⁴, while kneeling pain has been shown to be worse in patients with bone-patellar tendon-bone (BPTB) autografts compared to hamstring autografts⁵ as well as BPTB allografts.⁴

Patient satisfaction rates are high for ACL autografts and allografts. Boonriong and Kietsiriroje⁶ have shown visual analog scale (VAS) patient satisfaction score averages to be 88 out of 100 for BPTB autografts and 93 out of 100 for hamstring tendon autografts. Fox and colleagues⁷ showed that 87% of patients were completely or mostly satisfied following revision ACL reconstruction with patellar tendon allograft. Cohen and colleagues⁸ evaluated 240 patients undergoing primary ACL reconstruction; 63.3% underwent ACL reconstruction with an allograft and 35.4% with an autograft. Of all patients enrolled in the study, 93% were satisfied with their graft choice, with 12.7% of patients opting to choose another graft if in the same situation again. Of those patients, 63.3% would have switched from an autograft to allograft. Although these numbers represent high patient satisfaction following a variety of ACL graft types, it is important to continue to identify graft selection factors in order to maximize patient outcomes.

The purposes of this prospective study were to assess patients’ knowledge of their graft type used for ACL reconstruction, to determine the most influential factors involved in graft selection, and to determine the level of satisfaction with the graft of choice at a minimum of 1-year follow-up. Based on a previous retrospective study,8 we hypothesized that physician recommendation would be the most influential factor in ACL graft selection. We also hypothesized that patients receiving an autograft would be more accurate in stating their graft harvest location compared to allograft patients.

Materials and Methods

We prospectively enrolled 304 patients who underwent primary ACL reconstruction from January 2008 to September 2013. Surgery was performed by 9 different surgeons within the same practice. All patients undergoing primary ACL reconstruction were eligible for the study.

All surgeons explained to each patient the pros and cons of each graft choice based upon peer-reviewed literature. Each patient was allowed to choose autograft or allograft, although most of the surgeons strongly encourage patients under age 25 years to choose autograft. One of the surgeons specifically encourages a patellar tendon autograft in patients under age 30 to 35 years, except for those patients with a narrow patellar tendon on magnetic resonance imaging, in which case he recommends a hamstring autograft. Another surgeon also specifically encourages patellar tendon autograft in patients under 35 years, except in skeletally immature patients, for whom he encourages hamstring autograft. However, none of the surgeons prohibited patients from choosing autograft or allograft, regardless of age.

The Institutional Review Board at our institution provided approval for this study. At the first postoperative follow-up appointment, each patient completed a questionnaire asking to select from a list the type (“your own” or “a cadaver”) and harvest site of the graft that was used for the surgery. Patients were also asked how they decided upon that graft type by ranking a list of 4 factors from 1 to 4. These included (1) physician recommendation, (2) family/friend’s recommendation, (3) coach’s recommendation, and (4) the media. Patients had the option of ranking more than one factor as most important in their decision. In addition, patients were asked to list any other factors that influenced their decision regarding graft type.

At a minimum of 1 year following surgery, patients completed the same questionnaire described above. In addition, patients were asked if they were satisfied with their graft and whether they would choose the same graft type if undergoing ACL reconstruction again. Patients who would have chosen a different graft were asked which graft they would have chosen and why. Any patient who experienced graft rupture prior to follow-up was included in the analysis.

Statistical Analysis

Chi square tests were used to compare dichotomous outcomes. A type I error of less than 5% (P < .05) was considered statistically significant.

Results

At least 1 year following ACL reconstruction, 213 of 304 patients (70%) successfully completed the same questionnaire as they did at their first postoperative follow-up appointment. The mean age of these patients at the time of surgery was 31.9 ± 11.0 years (range, 13.9 to 58.0 years). The mean follow-up time was 1.4 ± 0.4 years (range, 1.0 to 2.6 years), and 59% of these patients were male.

Autografts were used for 139 patients (139/304, 46%), allografts for 156 patients (156/304, 51%), and hybrid grafts for 9 patients (9/304, 3%). Overall, 77% of patients were accurate in stating the type of graft used for their ACL reconstruction, including 88% of autograft patients, 71% of allograft patients, and 11% of hybrid graft patients (Table 1). Patients who underwent reconstruction with an autograft were significantly more accurate in stating their graft type compared to patients with an allograft (P < .001). Graft type by surgeon is shown in Table 2. A statistically significant difference was found in the proportion of patients choosing autograft versus allograft based on surgeon (P < .0001).

When asked which type of graft was used for their surgery, 12 of 304 patients (4%) did not know their graft type or harvest location. Twenty-nine patients stated that their graft was an allograft but did not know the harvest location. Five patients stated that their graft was an autograft but did not know the harvest location. The 34 patients who classified their choice of graft but did not know the harvest site (11%) stated their surgeon never told them where their graft was from or they did not remember. A complete list of graft type responses is shown in Table 3.

Of the 29 patients who stated that their graft was an allograft but did not know the harvest location, 19 (66%) had a tibialis anterior allograft, 7 (24%) had a BPTB allograft, 2 (7%) had an Achilles tendon allograft, and 1 (3%) had a tibialis anterior autograft.

Physician recommendation was the most important decision-making factor listed for 82% of patients at their first postoperative appointment (Table 4). In addition to the 4 factors listed on our survey, patients were allowed to write in other factors involved in their decision. The most popular answers included recovery time, personal research on graft types, and prior personal experience with ACL reconstruction on the contralateral knee.

At the time of 1-year follow-up, 205 of 213 patients (96%) said they were satisfied with their graft choice (Table 5). All 4 unsatisfied autograft patients received a hamstring autograft, 3 of which were performed by the same surgeon. No significant difference was found in satisfaction rates between patients with autograft vs allograft (P = .87). There was a higher satisfaction rate among patients with a BPTB autograft compared to those with a hamstring autograft (P = .043). Of the unsatisfied patients, 3 patients stated that their graft had failed in the time prior to follow-up and 2 patients stated that they were having donor site pain following surgery with hamstring autograft and would consider an allograft if the reconstruction were repeated (Table 6). Two patients stated that they were unsatisfied with their graft but would need to do more research before deciding on a different graft type.

As shown in Tables 5 and 6, there is a discrepancy between the number of patients who were unsatisfied with their graft and the number of patients who stated that they would switch to a different graft type if they were to have ACL reconstruction again. A number of patients stated that they were satisfied with their graft, yet they would switch to a different graft. The main reasons for this related to issues from a hamstring autograft harvest site. One patient noted that although she was satisfied with her graft, she would switch after doing further research.

Discussion

Determining the decision-making factors for patients choosing between graft types for ACL reconstruction is important to ensure that patients can make a decision based on objective information. Several previous studies have evaluated patient selection of ACL grafts.^8-10 All 3 of these studies showed that surgeon recommendation is the primary factor in a patient’s decision. Similar to previous studies, we also found that physician recommendation is the most influential factor involved in this decision.

At an average follow-up of 41 months, Cohen and colleagues⁸ found that 1.3% of patients did not know whether they received an autograft or allograft for their ACL reconstruction. Furthermore, 50.7% of patients stating they received an allograft in Cohen’s study⁸ were unsure of the harvest location. In our study, 4% of patients at their first postoperative visit did not know whether they had received an autograft or allograft and 10% of patients stating they received an allograft selected an unknown harvest site. In contrast, only 2% of autograft patients in our study were unsure of the harvest location at their first postoperative appointment. It is likely that, over time, patients with an allograft forget the harvest location, whereas autograft patients are more likely to remember the location of harvest. This is especially true in patients with anterior knee pain or hamstring pain following ACL reconstruction with a BPTB or hamstring tendon autograft, respectively.

In terms of patients’ knowledge of their graft type, we found an overall accuracy of 77%, with 88% of autograft patients, 71% of allograft patients, and 11% of hybrid graft patients remembering their graft type and harvest location. Although we do not believe it to be critical for patients to remember these details, we do believe that patients who do not know their graft type likely relied on the recommendation of their physician.

We found a significant difference in the proportion of patients choosing autograft vs allograft based on surgeon, despite these surgeons citing available data in the literature to each patient and ultimately allowing each patient to make his or her own decision. This is partly due to the low sample size of most of the surgeons involved. However, the main reason for this distortion is likely that different surgeons may highlight different aspects of the literature to “spin” patients towards one graft or another in certain cases.

Currently, there remains a lack of clarity in the literature on appropriate ACL graft choices for patients. With constant new findings being published on different aspects of various grafts, it is important for surgeons to remain up to date with the literature. Nevertheless, we believe that certain biases are inevitable among surgeons due to unique training experiences as well as experience with their own patients.

Cohen and colleagues8 found that only 7% of patients reported that their own personal research influenced their decision, and only 6.4% of patients reported the media as their primary decision-making factor. Cheung and colleagues⁹ conducted a retrospective study and found that more than half of patients did significant personal research prior to making a decision regarding their graft type. Most of this research was done using medical websites and literature. Koh and colleagues¹⁰ noted that >80% of patients consulted the internet for graft information before making a decision. Koh’s study¹⁰ was performed in Korea and therefore the high prevalence of internet use may be culturally-related.

Overall, quality of information for patients undergoing ACL reconstruction is mixed across the internet, with only 22.5% of top websites being affiliated with an academic institution and 35.5% of websites authored by private physicians or physician groups.¹¹ Although a majority of internet websites offer discussion into the condition and surgical procedure of ACL reconstruction, less than half of these websites share the equally important information on the eligibility for surgery and concomitant complications following surgery.¹¹In our study, only 39 patients (13%) listed the media as either the first (13, 4%) or second (26, 9%) most important factor in their graft decision. Clearly there is some discrepancy between studies regarding the influence of personal research and media. There are a few potential reasons for this. First, we did not explicitly ask patients if their own personal research had any influence on their graft decision. Rather, we asked patients to rank their decision-making factors, and few patients ranked the media as their first or second greatest influence. Second, the word “media” was used in our questionnaire rather than “online research” or “internet.” It may seem somewhat vague to patients what the word “media” really means in terms of their own research, whereas listing “online research” or “internet” as selection options may have influenced patient responses.

In our study, we asked patients for any additional factors that influenced their graft choice. Thirteen patients (4%) noted that “personal research” through internet, orthopaedic literature, and the media influenced their graft decision. This corroborates the idea that “media” may have seemed vague to some patients. Of these patients, 9 chose an autograft and 4 chose an allograft. The relative ease in accessing information regarding graft choice in ACL reconstruction should be noted. Numerous websites offer advice, graft options, and commentary from group practices and orthopaedic surgeons. Whether or not these sources provide reasonable support for one graft vs another graft remains to be answered. The physician should be responsible for providing the patient with this collected objective information.

In our study, 205 patients (96%) were satisfied with their graft choice at the time of follow-up, with 15 patients (7%) stating that they would have chosen a different graft type if they could redo the operation. Cheung and colleagues⁹ found a satisfaction rate of 87.4% at an average follow-up time of 19 months, with 4.6% stating they would have chosen a different graft type. Many factors can contribute to patient satisfaction after ACL reconstruction. Looking at patient variables such as age, demographics, occupation, activity level, surgical technique including tunnel placement and fixation, postoperative rehabilitation, and graft type may influence the success of the patient after ACL reconstruction.

The strengths of this study include the patient population size with 1-year follow-up as well as the prospective study design. In comparison to a previous retrospective study in 2009 by Cohen and colleagues⁸with a sample size of 240 patients, our study collected 213 patients with 70% follow-up at minimum 1 year. Collecting data prospectively ensures accurate representation of the factors influencing each patient’s graft selection, while follow-up data was useful for patient satisfaction.

The limitations of this study include the percentage of patients lost from follow-up as well as any bias generated from the organization of the questionnaire. Unfortunately, with a younger, transient population of patients undergoing ACL reconstruction in a major metropolitan area, a percentage of patients are lost to follow-up. Many attempts were made to locate these patients. Another potential limitation was the order of decision factors listed on the questionnaire. These factors were not ordered randomly on each survey, but were listed in the following order: (1) physician recommendation (2) family/friend’s recommendation (3) coach’s recommendation and (4) the media. This may have influenced patient responses. The organization of these factors in the questionnaire started with physician recommendation, which may have influenced the patient’s initial thought process of which factor had the greatest influence in their graft decision. In addition, for the surveys completed at least 1 year following surgery, some patients were contacted via e-mail and others via telephone. Thus, some patients may have changed their answers if they were able to see the questions rather than hearing the questions. We believe this is particularly true of the question regarding graft harvest site.

Our study indicates that the majority of patients undergoing ACL reconstruction are primarily influenced by the physician’s recommendation.

Anterior cruciate ligament (ACL) injuries affect >175,000 people each year,¹ with >100,000 Americans undergoing ACL reconstruction annually.² Due to the high impact this injury has on the general population, and especially on athletes, it is important to determine the factors that influence a patient’s selection of a particular graft type. With increasing access to information and other outside influences, surgeons should attempt to provide as much objective information as possible in order to allow patients to make appropriate informed decisions regarding their graft choice for ACL surgery.

While autografts are used in >60% of primary ACL reconstructions, allografts are used in >80% of revision procedures.³ Both autografts and allografts offer advantages and disadvantages, and the advantages of each may depend on patient age, activity level, and occupation.⁴ For example, graft rerupture rates have been shown to be higher in patients with ACL allografts⁴, while kneeling pain has been shown to be worse in patients with bone-patellar tendon-bone (BPTB) autografts compared to hamstring autografts⁵ as well as BPTB allografts.⁴

Patient satisfaction rates are high for ACL autografts and allografts. Boonriong and Kietsiriroje⁶ have shown visual analog scale (VAS) patient satisfaction score averages to be 88 out of 100 for BPTB autografts and 93 out of 100 for hamstring tendon autografts. Fox and colleagues⁷ showed that 87% of patients were completely or mostly satisfied following revision ACL reconstruction with patellar tendon allograft. Cohen and colleagues⁸ evaluated 240 patients undergoing primary ACL reconstruction; 63.3% underwent ACL reconstruction with an allograft and 35.4% with an autograft. Of all patients enrolled in the study, 93% were satisfied with their graft choice, with 12.7% of patients opting to choose another graft if in the same situation again. Of those patients, 63.3% would have switched from an autograft to allograft. Although these numbers represent high patient satisfaction following a variety of ACL graft types, it is important to continue to identify graft selection factors in order to maximize patient outcomes.

The purposes of this prospective study were to assess patients’ knowledge of their graft type used for ACL reconstruction, to determine the most influential factors involved in graft selection, and to determine the level of satisfaction with the graft of choice at a minimum of 1-year follow-up. Based on a previous retrospective study,8 we hypothesized that physician recommendation would be the most influential factor in ACL graft selection. We also hypothesized that patients receiving an autograft would be more accurate in stating their graft harvest location compared to allograft patients.

Materials and Methods

We prospectively enrolled 304 patients who underwent primary ACL reconstruction from January 2008 to September 2013. Surgery was performed by 9 different surgeons within the same practice. All patients undergoing primary ACL reconstruction were eligible for the study.

All surgeons explained to each patient the pros and cons of each graft choice based upon peer-reviewed literature. Each patient was allowed to choose autograft or allograft, although most of the surgeons strongly encourage patients under age 25 years to choose autograft. One of the surgeons specifically encourages a patellar tendon autograft in patients under age 30 to 35 years, except for those patients with a narrow patellar tendon on magnetic resonance imaging, in which case he recommends a hamstring autograft. Another surgeon also specifically encourages patellar tendon autograft in patients under 35 years, except in skeletally immature patients, for whom he encourages hamstring autograft. However, none of the surgeons prohibited patients from choosing autograft or allograft, regardless of age.

The Institutional Review Board at our institution provided approval for this study. At the first postoperative follow-up appointment, each patient completed a questionnaire asking to select from a list the type (“your own” or “a cadaver”) and harvest site of the graft that was used for the surgery. Patients were also asked how they decided upon that graft type by ranking a list of 4 factors from 1 to 4. These included (1) physician recommendation, (2) family/friend’s recommendation, (3) coach’s recommendation, and (4) the media. Patients had the option of ranking more than one factor as most important in their decision. In addition, patients were asked to list any other factors that influenced their decision regarding graft type.

At a minimum of 1 year following surgery, patients completed the same questionnaire described above. In addition, patients were asked if they were satisfied with their graft and whether they would choose the same graft type if undergoing ACL reconstruction again. Patients who would have chosen a different graft were asked which graft they would have chosen and why. Any patient who experienced graft rupture prior to follow-up was included in the analysis.

Statistical Analysis

Chi square tests were used to compare dichotomous outcomes. A type I error of less than 5% (P < .05) was considered statistically significant.

Results

At least 1 year following ACL reconstruction, 213 of 304 patients (70%) successfully completed the same questionnaire as they did at their first postoperative follow-up appointment. The mean age of these patients at the time of surgery was 31.9 ± 11.0 years (range, 13.9 to 58.0 years). The mean follow-up time was 1.4 ± 0.4 years (range, 1.0 to 2.6 years), and 59% of these patients were male.

Autografts were used for 139 patients (139/304, 46%), allografts for 156 patients (156/304, 51%), and hybrid grafts for 9 patients (9/304, 3%). Overall, 77% of patients were accurate in stating the type of graft used for their ACL reconstruction, including 88% of autograft patients, 71% of allograft patients, and 11% of hybrid graft patients (Table 1). Patients who underwent reconstruction with an autograft were significantly more accurate in stating their graft type compared to patients with an allograft (P < .001). Graft type by surgeon is shown in Table 2. A statistically significant difference was found in the proportion of patients choosing autograft versus allograft based on surgeon (P < .0001).

When asked which type of graft was used for their surgery, 12 of 304 patients (4%) did not know their graft type or harvest location. Twenty-nine patients stated that their graft was an allograft but did not know the harvest location. Five patients stated that their graft was an autograft but did not know the harvest location. The 34 patients who classified their choice of graft but did not know the harvest site (11%) stated their surgeon never told them where their graft was from or they did not remember. A complete list of graft type responses is shown in Table 3.

Of the 29 patients who stated that their graft was an allograft but did not know the harvest location, 19 (66%) had a tibialis anterior allograft, 7 (24%) had a BPTB allograft, 2 (7%) had an Achilles tendon allograft, and 1 (3%) had a tibialis anterior autograft.

Physician recommendation was the most important decision-making factor listed for 82% of patients at their first postoperative appointment (Table 4). In addition to the 4 factors listed on our survey, patients were allowed to write in other factors involved in their decision. The most popular answers included recovery time, personal research on graft types, and prior personal experience with ACL reconstruction on the contralateral knee.

At the time of 1-year follow-up, 205 of 213 patients (96%) said they were satisfied with their graft choice (Table 5). All 4 unsatisfied autograft patients received a hamstring autograft, 3 of which were performed by the same surgeon. No significant difference was found in satisfaction rates between patients with autograft vs allograft (P = .87). There was a higher satisfaction rate among patients with a BPTB autograft compared to those with a hamstring autograft (P = .043). Of the unsatisfied patients, 3 patients stated that their graft had failed in the time prior to follow-up and 2 patients stated that they were having donor site pain following surgery with hamstring autograft and would consider an allograft if the reconstruction were repeated (Table 6). Two patients stated that they were unsatisfied with their graft but would need to do more research before deciding on a different graft type.

As shown in Tables 5 and 6, there is a discrepancy between the number of patients who were unsatisfied with their graft and the number of patients who stated that they would switch to a different graft type if they were to have ACL reconstruction again. A number of patients stated that they were satisfied with their graft, yet they would switch to a different graft. The main reasons for this related to issues from a hamstring autograft harvest site. One patient noted that although she was satisfied with her graft, she would switch after doing further research.

Discussion

Determining the decision-making factors for patients choosing between graft types for ACL reconstruction is important to ensure that patients can make a decision based on objective information. Several previous studies have evaluated patient selection of ACL grafts.^8-10 All 3 of these studies showed that surgeon recommendation is the primary factor in a patient’s decision. Similar to previous studies, we also found that physician recommendation is the most influential factor involved in this decision.

At an average follow-up of 41 months, Cohen and colleagues⁸ found that 1.3% of patients did not know whether they received an autograft or allograft for their ACL reconstruction. Furthermore, 50.7% of patients stating they received an allograft in Cohen’s study⁸ were unsure of the harvest location. In our study, 4% of patients at their first postoperative visit did not know whether they had received an autograft or allograft and 10% of patients stating they received an allograft selected an unknown harvest site. In contrast, only 2% of autograft patients in our study were unsure of the harvest location at their first postoperative appointment. It is likely that, over time, patients with an allograft forget the harvest location, whereas autograft patients are more likely to remember the location of harvest. This is especially true in patients with anterior knee pain or hamstring pain following ACL reconstruction with a BPTB or hamstring tendon autograft, respectively.

In terms of patients’ knowledge of their graft type, we found an overall accuracy of 77%, with 88% of autograft patients, 71% of allograft patients, and 11% of hybrid graft patients remembering their graft type and harvest location. Although we do not believe it to be critical for patients to remember these details, we do believe that patients who do not know their graft type likely relied on the recommendation of their physician.

We found a significant difference in the proportion of patients choosing autograft vs allograft based on surgeon, despite these surgeons citing available data in the literature to each patient and ultimately allowing each patient to make his or her own decision. This is partly due to the low sample size of most of the surgeons involved. However, the main reason for this distortion is likely that different surgeons may highlight different aspects of the literature to “spin” patients towards one graft or another in certain cases.

Currently, there remains a lack of clarity in the literature on appropriate ACL graft choices for patients. With constant new findings being published on different aspects of various grafts, it is important for surgeons to remain up to date with the literature. Nevertheless, we believe that certain biases are inevitable among surgeons due to unique training experiences as well as experience with their own patients.

Cohen and colleagues8 found that only 7% of patients reported that their own personal research influenced their decision, and only 6.4% of patients reported the media as their primary decision-making factor. Cheung and colleagues⁹ conducted a retrospective study and found that more than half of patients did significant personal research prior to making a decision regarding their graft type. Most of this research was done using medical websites and literature. Koh and colleagues¹⁰ noted that >80% of patients consulted the internet for graft information before making a decision. Koh’s study¹⁰ was performed in Korea and therefore the high prevalence of internet use may be culturally-related.

Overall, quality of information for patients undergoing ACL reconstruction is mixed across the internet, with only 22.5% of top websites being affiliated with an academic institution and 35.5% of websites authored by private physicians or physician groups.¹¹ Although a majority of internet websites offer discussion into the condition and surgical procedure of ACL reconstruction, less than half of these websites share the equally important information on the eligibility for surgery and concomitant complications following surgery.¹¹In our study, only 39 patients (13%) listed the media as either the first (13, 4%) or second (26, 9%) most important factor in their graft decision. Clearly there is some discrepancy between studies regarding the influence of personal research and media. There are a few potential reasons for this. First, we did not explicitly ask patients if their own personal research had any influence on their graft decision. Rather, we asked patients to rank their decision-making factors, and few patients ranked the media as their first or second greatest influence. Second, the word “media” was used in our questionnaire rather than “online research” or “internet.” It may seem somewhat vague to patients what the word “media” really means in terms of their own research, whereas listing “online research” or “internet” as selection options may have influenced patient responses.

In our study, we asked patients for any additional factors that influenced their graft choice. Thirteen patients (4%) noted that “personal research” through internet, orthopaedic literature, and the media influenced their graft decision. This corroborates the idea that “media” may have seemed vague to some patients. Of these patients, 9 chose an autograft and 4 chose an allograft. The relative ease in accessing information regarding graft choice in ACL reconstruction should be noted. Numerous websites offer advice, graft options, and commentary from group practices and orthopaedic surgeons. Whether or not these sources provide reasonable support for one graft vs another graft remains to be answered. The physician should be responsible for providing the patient with this collected objective information.

In our study, 205 patients (96%) were satisfied with their graft choice at the time of follow-up, with 15 patients (7%) stating that they would have chosen a different graft type if they could redo the operation. Cheung and colleagues⁹ found a satisfaction rate of 87.4% at an average follow-up time of 19 months, with 4.6% stating they would have chosen a different graft type. Many factors can contribute to patient satisfaction after ACL reconstruction. Looking at patient variables such as age, demographics, occupation, activity level, surgical technique including tunnel placement and fixation, postoperative rehabilitation, and graft type may influence the success of the patient after ACL reconstruction.

The strengths of this study include the patient population size with 1-year follow-up as well as the prospective study design. In comparison to a previous retrospective study in 2009 by Cohen and colleagues⁸with a sample size of 240 patients, our study collected 213 patients with 70% follow-up at minimum 1 year. Collecting data prospectively ensures accurate representation of the factors influencing each patient’s graft selection, while follow-up data was useful for patient satisfaction.

The limitations of this study include the percentage of patients lost from follow-up as well as any bias generated from the organization of the questionnaire. Unfortunately, with a younger, transient population of patients undergoing ACL reconstruction in a major metropolitan area, a percentage of patients are lost to follow-up. Many attempts were made to locate these patients. Another potential limitation was the order of decision factors listed on the questionnaire. These factors were not ordered randomly on each survey, but were listed in the following order: (1) physician recommendation (2) family/friend’s recommendation (3) coach’s recommendation and (4) the media. This may have influenced patient responses. The organization of these factors in the questionnaire started with physician recommendation, which may have influenced the patient’s initial thought process of which factor had the greatest influence in their graft decision. In addition, for the surveys completed at least 1 year following surgery, some patients were contacted via e-mail and others via telephone. Thus, some patients may have changed their answers if they were able to see the questions rather than hearing the questions. We believe this is particularly true of the question regarding graft harvest site.

Our study indicates that the majority of patients undergoing ACL reconstruction are primarily influenced by the physician’s recommendation.

Patients are learning about health and disease more independently than before, but such self-education may pose a unique challenge for practicing physicians. Although educated patients can assist in the critical appraisal of treatment options,¹ misinformed patients may have preconceived treatment biases and unrealistic expectations. More than 66 million Americans use the Internet daily, and recent surveys have shown 86% have used the Internet for health-related information.^2,3 With Internet use increasing, the number of patients turning to the web for medical information is increasing as well.⁴ For many patients, this information can be useful in making decisions about their health and health care.⁵

Although accessing medical information from the Internet has grown exponentially, analysis of information quality has grown considerably slower.⁶ With no regulatory body monitoring content, there is easy circumvention of the peer review process, an essential feature of academic publishing.⁷ With no external regulation, the information retrieved may be incorrect, outdated, or misleading. Many orthopedic studies have analyzed Internet content about numerous diagnoses.^3-6,8-18 Most of these studies have found this information highly variable and of poor quality.

We conducted a study to evaluate and analyze rotator cuff repair information available to the general public through the Internet; to assess changes in the quality of information over time; to determine if sites sponsored by academic institutions offered higher-quality information; and to assess whether the readability of the material varied according to DISCERN scores.

Rotator cuff repairs are among the most common surgeries performed by orthopedic surgeons. To our knowledge, this is the first study to assess the quality of web information about rotator cuff repairs. We hypothesized that the quality of information would positively correlate with the reading level of the material presented, that academic institutions would present the highest-quality information, and that the type of information presented would change over time.

Materials and Methods

We used the search phrase rotator cuff repair on the 3 most popular search engines: Google, Yahoo!, and Bing. Google is the dominant engine, taking 83.06% of total market share, followed by Yahoo! (6.86%) and Bing (4.27%).5 The first 50 websites identified by each search engine were selected for evaluation, excluding duplicates or overlapping websites. Similarly, advertisements and strictly video results lacking text were excluded. After each engine was queried, a master list of 150 websites was created for individual evaluation and assessment. To assess changes in results over time, we performed 2 searches, on November 16, 2011, and May 18, 2014.

The content of each website was analyzed for authorship, ability to contact the author, discussion of disorder, surgical treatment, complications, surgical eligibility, rehabilitation, other treatment options, and use of peer-reviewed sources. Authorship was placed in 1 of 6 categories:

1. Academic—university-affiliated physician or research group.

2. Private—physician or group without stated affiliation to an academic organization.

3. Industry—manufacturing or marketing company advertising a product or service for profit.

4. News source—bulletin or article without affiliation to a hospital or an academic institution.

5. Public education—individual or organization with noncommercial website providing third-party information (eg, Wikipedia, About.com).

6. Blog—website publishing an individual’s personal experiences in diary or journal form.

Websites were also assessed for accuracy and validity based on presence or absence of Health On the Net code (HONcode) certification and DISCERN score. Designed by the Health On the Net Foundation in 1996, HONcode provides a framework for disseminating high-quality medical information over the web.¹⁹ Website owners can request that their sites be evaluated for HONcode certification; a site that qualifies can display the HONcode seal.²⁰ The DISCERN project, initially funded by the National Health Service in the United Kingdom, judges the quality of written information available on health-related websites.²¹ It determines the quality of a publication on the basis of 16 questions: The first 8 address the publication’s reliability, the next 7 involve specific details of treatment choices, and the last is an overall rating of the website.

Website readability was assessed with the Flesch-Kincaid test. This test, designed under contract with the US Navy in 1975, has been used in other orthopedic studies.¹⁹ Regression analysis was performed to check for correlation between website readability and DISCERN score. Analysis of variance was used to analyze differences between scores.

Results

We performed a comprehensive analysis of the top 50 websites from each of the 3 search engines (N = 150 websites) (Figures 1–5, Table). Regarding authorship, our 2 searches demonstrated similar values (Figure 1). In 2011, 21% of websites were associated with an academic institution, 38% were authored by private physicians or hospital or physician groups not associated with an academic institution, 11.5% were industry-sponsored, 5% were news bulletins or media reports, 21.5% were public education websites, and 3% were personal blogs. Our 2014 search found a similar distribution of contributors. Between 2011 and 2014, the largest change was in academic authors, which decreased by 7%, from 21% to 14%. Percentage of websites authored by private physicians remained constant from the first to the second search: 38%.

When the 2011 and 2014 website content was compared, several changes were noted. Percentage of websites providing an author contact method increased from 21% to 50% (Figure 2), percentage detailing rotator cuff repairs increased from 82% to 91%, and percentage introducing treatment options in addition to surgical management increased from 11.5% to 61%. Percentage discussing surgical eligibility, however, decreased from 43% to 18%. Percentage citing peer-reviewed sources remained relatively constant (28%, 26%), as did percentage discussing surgical technique for rotator cuff repair (55%, 59%) (Figure 3). A major decrease was found in percentage of websites discussing surgical complications, 42% in 2011 down to 25% in 2014, whereas a major increase was found in percentage discussing rehabilitation, from 39% in 2011 up to 73% in 2014. In 2014, no websites discussed double- versus single-row surgery—compared with 6% in 2011. False claims remained low between 2011 and 2014. In both searches, no website guaranteed a return to sport, and few made claims of painless or bloodless surgery.

DISCERN scores for websites found during the 2014 search were averaged for each of the 6 authorship groups (Figure 4). The highest DISCERN scores were given to academic institution websites (51.6) and public education websites (49). For the academic websites, this difference was significant relative to news, blog, and private physician websites (Ps = .012, .001, .001) The lowest DISCERN scores were given to news organization websites and personal blogs. DISCERN scores were 43.8 for industry sources and 40.7 for private physician groups; the difference was not significant (P = .229). Overall mean DISCERN score for all websites was 44. Eleven percent of websites were HONcode-certified.

No correlation was found between website readability and DISCERN score; correlation coefficient r was .01 (Figure 5). For the websites in 2014, mean Flesch-Kincaid readability score was 50.17, and mean grade level was 10.98; coefficient of determination r² was 0.00012.

The Table compares our data with data from other orthopedic studies that have analyzed the quality of Internet information about various orthopedic injuries, diseases, and procedures.^{3-6,8,9,11-18} With its mean DISCERN score of 44, the present rotator cuff tear study was ranked third of 6 studies that have used this scoring system to analyze website content. Of these 6 studies, those reviewing osteosarcoma and juvenile idiopathic arthritis were ranked highest (mean scores, 49.8 and 48.9, respectively), and the study reviewing scoliosis surgery was ranked lowest (38.9). Bruce-Brand and colleagues9 recently found a mean DISCERN score of 41 for anterior cruciate ligament (ACL) reconstruction. When considering HONcode-certified websites, our Internet search for rotator cuff tears found the third lowest percentage, 10.5%, compared with the other studies (Table); the highest percentage, 30%, was found for websites discussing concussions in athletes. When considering authorship, our rotator cuff study found the third highest percentage, 76%, authored by academic centers, physicians, and public education websites; the highest percentage was found in websites discussing ACL reconstruction. Websites discussing ACL reconstruction also had the highest percentage of websites authored by industry.⁹

Discussion

To our knowledge, this is the first study specifically analyzing the quality of Internet information about rotator cuff repairs. A similar study, conducted by Starman and colleagues¹⁵ in 2010, addressed the quality of web information about 10 common sports medicine diagnoses, one of which was rotator cuff tears. In that study, only 16 of the websites included discussed rotator cuff tears. In addition, the authors used a customized, HONcode-based grading system to analyze each website, making their data difficult to compare across studies.

Ideally, a high-quality medical website should be written by a credible source and should cover a disorder, treatment options, eligibility, rehabilitation, and complications. As there is no standard grading system for analyzing web content about rotator cuff repairs, we analyzed the websites for specific information we thought should be included in a high-quality website (Figures 2, 3). When considering authorship, we found academic centers, private physicians, and educational sources comprised 76% of the sources; industry sources made up only 12%. Similar findings were noted by investigators analyzing Internet information about other orthopedic topics, including ACL reconstruction, lumbar arthroplasty, osteosarcoma, and cervical spine surgery.^5,11,22 Studies analyzing websites for information on ACL tears and distal radius fractures found have a higher percentage of industry-sponsored websites.^9,10

DISCERN showed that the highest-quality information came from websites with academic affiliations, consistent with previous studies,^3,9,17 and its mean score (51.6) was significantly higher than the scores for private physician websites, news sites, and blogs (Ps = .001, .016, .001); the least reliable information was from personal blogs and news outlets. Of note, mean DISCERN score was higher for the industry websites we found than for private physician websites (43.8 vs 40.7), though the difference was not significant (P = .229). Previous investigators considered number of industry-sponsored websites as a marker of poor quality of information relating to a given topic; however, given the DISCERN scores in our study, this might not necessarily be true.⁶ Based on the present study’s data, websites affiliated with academic institutions would be recommended for patients searching for high-quality information about rotator cuff tears.

Given DISCERN scores across studies, information about rotator cuff tears ranked below information about osteosarcoma and juvenile idiopathic arthritis but above information about scoliosis, cervical spine surgery, and ACL reconstruction (Table). DISCERN scores must be compared across studies, as there are no definitions for good and poor DISCERN scores.

Of the 4 studies that analyzed percentage of websites citing peer-reviewed sources, only our study and the study of cervical spine surgery¹⁸ analyzed that percentage as well as DISCERN score. Percentage citing peer-reviewed sources was 26% for rotator cuff tears and 24% for cervical spine surgery; the respective DISCERN scores were 44 and 43.6. As only these 2 studies could be compared, no real correlation between percentage of websites citing peer-reviewed sources and the quality of the content on a given topic can be assessed. More research into this relationship is needed. One already delineated association is the correlation between HONcode-certified sites and high DISCERN scores.²¹ For high-quality medical information, physicians can direct their patients both to academic institution websites and to HONcode-certified websites.

When we compared the present study with previous investigations, we found a large difference between search results for a given topic. In 2013, Duncan and colleagues⁶ and Bruce-Brand and colleagues⁹ used similar study designs (eg, search terms, search engines) for their investigations of quality of web information. Their results, however, were widely different. For example, percentages of industry authorship were 4.5% (Duncan and colleagues⁶) and 64% (Bruce-Brand and colleagues⁹). This inconsistency between studies conducted during similar periods might be related to what appears at the top of the results queue for a search. Duncan and colleagues⁶ analyzed 200 websites, Bruce-Brand and colleagues⁹ only 45. Industries may have made financial arrangements and used search engine optimization techniques to have their websites listed first in search results.

In our study, we also analyzed how web information has changed over time. On the Internet, information changes daily, and we hypothesized that the content found during our 2 searches (2011, 2014) would yield different results. Surprisingly, the data were similar, particularly concerning authorship (Figures 1, 2). In both searches, the largest authorship source was private physician or physician groups (38% in 2011 and 2014). Other authorship sources showed little change in percentage between searches. As for content, we found both increases and decreases in specific web information. Ability to contact authors increased from 21% (2011) to 50% (2014). We think it is important that websites offer a communication channel to people who read the medical information the sites provide. Percentage of websites discussing nonoperative treatment options increased from 11.5% to 61%. Therefore, patients in 2014 were being introduced to more options (in addition to surgery) for managing shoulder pain—an improvement in quality of information between the searches. Percentage of websites discussing surgical eligibility, however, decreased from 43% to 18%—a negative development in information quality. Another decrease, from 42% to 25%, was found for websites discussing surgical complications. Given the data as a whole, and our finding both negative and positive changes, it appears the quality of web content has not improved significantly. Interestingly, no websites discussed double- versus single-row surgery in 2014, but 6% did so in 2011.

Lost in the discussion of quality and reliability of information is whether patients comprehend what they are reading.²³ Yi and colleagues¹⁹ recentlyassessed the readability level of arthroscopy information in articles published online by the American Academy of Orthopaedic Surgeons (AAOS) and the Arthroscopy Association of North America (AANA). The investigators used the Flesch-Kincaid readability test to determine readability level in terms of grade level. They found that the majority of the patient education articles on the AAOS and AANA sites had a readability level far above the national average; only 4 articles were written at or below the eighth-grade level, the current average reading level in the United States.²⁴ Information that is not comprehensible is of no use to patients, and information that physicians and researchers consider high-quality might not be what patients consider high-quality. As we pursue higher-quality web content, we need to consider that its audience includes nonmedical readers, our patients. In the present study, we found that the readability of a website had no correlation with the site’s DISCERN score (Figure 5). Therefore, for information about rotator cuff repairs, higher-quality websites are no harder than lower-quality sites for patients to comprehend. The Flesch-Kincaid readability test is flawed in that it considers only total number of syllables per word and words per sentence, not nontextual elements of patient education materials, such as illustrations on a website. The 10.98 mean grade level found in our study is higher than the levels found for most studies reviewed by Yi and colleagues.¹⁹

This study had several limitations. During an Internet search, the number of websites a user visits drops precipitously after the first page of results. Studies have shown the top 20 sites in a given search receive 97% of the views, and the top 3 receive 58.4%. Whether patients visit websites far down in the list of 150 we found in our given search is unknown. Last, the Flesch-Kincaid readability test is flawed in several ways but nevertheless is used extensively in research. Grading is based on number of words and syllables used in a given sentence; it does not take into account the complexity or common usage of a given word or definition. Therefore, websites may receive low Flesch-Kincaid scores—indicating ease of reading—despite their use of complex medical terminology and jargon that complicate patients’ comprehension of the material.

Conclusion

Numerous authors have evaluated orthopedic patients’ accessing of medical information from the Internet. Although the Internet makes access easier, unreliable content can lead patients to develop certain notions about the direction of their care and certain expectations regarding their clinical outcomes. With there being no regulatory body monitoring content, the peer review process, an essential feature of academic publishing, can be easily circumvented.²⁵

In this study, the highest-quality websites had academic affiliations. Quality of information about rotator cuff repairs was similar to what was found for other orthopedic topics in comparable studies. Surprisingly, there was little change in authorship and content of web information between our 2 search periods (2011, 2014). Although there has been a rapid increase in the number of medical websites, quality of content seems not to have changed significantly. Patients look to physicians for guidance but increasingly are accessing the Internet for additional information. It is essential that physicians understand the quality of information available on the Internet when counseling patients regarding surgery.

Patients are learning about health and disease more independently than before, but such self-education may pose a unique challenge for practicing physicians. Although educated patients can assist in the critical appraisal of treatment options,¹ misinformed patients may have preconceived treatment biases and unrealistic expectations. More than 66 million Americans use the Internet daily, and recent surveys have shown 86% have used the Internet for health-related information.^2,3 With Internet use increasing, the number of patients turning to the web for medical information is increasing as well.⁴ For many patients, this information can be useful in making decisions about their health and health care.⁵

Although accessing medical information from the Internet has grown exponentially, analysis of information quality has grown considerably slower.⁶ With no regulatory body monitoring content, there is easy circumvention of the peer review process, an essential feature of academic publishing.⁷ With no external regulation, the information retrieved may be incorrect, outdated, or misleading. Many orthopedic studies have analyzed Internet content about numerous diagnoses.^3-6,8-18 Most of these studies have found this information highly variable and of poor quality.

We conducted a study to evaluate and analyze rotator cuff repair information available to the general public through the Internet; to assess changes in the quality of information over time; to determine if sites sponsored by academic institutions offered higher-quality information; and to assess whether the readability of the material varied according to DISCERN scores.

Rotator cuff repairs are among the most common surgeries performed by orthopedic surgeons. To our knowledge, this is the first study to assess the quality of web information about rotator cuff repairs. We hypothesized that the quality of information would positively correlate with the reading level of the material presented, that academic institutions would present the highest-quality information, and that the type of information presented would change over time.

Materials and Methods

We used the search phrase rotator cuff repair on the 3 most popular search engines: Google, Yahoo!, and Bing. Google is the dominant engine, taking 83.06% of total market share, followed by Yahoo! (6.86%) and Bing (4.27%).5 The first 50 websites identified by each search engine were selected for evaluation, excluding duplicates or overlapping websites. Similarly, advertisements and strictly video results lacking text were excluded. After each engine was queried, a master list of 150 websites was created for individual evaluation and assessment. To assess changes in results over time, we performed 2 searches, on November 16, 2011, and May 18, 2014.

The content of each website was analyzed for authorship, ability to contact the author, discussion of disorder, surgical treatment, complications, surgical eligibility, rehabilitation, other treatment options, and use of peer-reviewed sources. Authorship was placed in 1 of 6 categories:

1. Academic—university-affiliated physician or research group.

2. Private—physician or group without stated affiliation to an academic organization.

3. Industry—manufacturing or marketing company advertising a product or service for profit.

4. News source—bulletin or article without affiliation to a hospital or an academic institution.

5. Public education—individual or organization with noncommercial website providing third-party information (eg, Wikipedia, About.com).

6. Blog—website publishing an individual’s personal experiences in diary or journal form.

Websites were also assessed for accuracy and validity based on presence or absence of Health On the Net code (HONcode) certification and DISCERN score. Designed by the Health On the Net Foundation in 1996, HONcode provides a framework for disseminating high-quality medical information over the web.¹⁹ Website owners can request that their sites be evaluated for HONcode certification; a site that qualifies can display the HONcode seal.²⁰ The DISCERN project, initially funded by the National Health Service in the United Kingdom, judges the quality of written information available on health-related websites.²¹ It determines the quality of a publication on the basis of 16 questions: The first 8 address the publication’s reliability, the next 7 involve specific details of treatment choices, and the last is an overall rating of the website.

Website readability was assessed with the Flesch-Kincaid test. This test, designed under contract with the US Navy in 1975, has been used in other orthopedic studies.¹⁹ Regression analysis was performed to check for correlation between website readability and DISCERN score. Analysis of variance was used to analyze differences between scores.

Results

We performed a comprehensive analysis of the top 50 websites from each of the 3 search engines (N = 150 websites) (Figures 1–5, Table). Regarding authorship, our 2 searches demonstrated similar values (Figure 1). In 2011, 21% of websites were associated with an academic institution, 38% were authored by private physicians or hospital or physician groups not associated with an academic institution, 11.5% were industry-sponsored, 5% were news bulletins or media reports, 21.5% were public education websites, and 3% were personal blogs. Our 2014 search found a similar distribution of contributors. Between 2011 and 2014, the largest change was in academic authors, which decreased by 7%, from 21% to 14%. Percentage of websites authored by private physicians remained constant from the first to the second search: 38%.

When the 2011 and 2014 website content was compared, several changes were noted. Percentage of websites providing an author contact method increased from 21% to 50% (Figure 2), percentage detailing rotator cuff repairs increased from 82% to 91%, and percentage introducing treatment options in addition to surgical management increased from 11.5% to 61%. Percentage discussing surgical eligibility, however, decreased from 43% to 18%. Percentage citing peer-reviewed sources remained relatively constant (28%, 26%), as did percentage discussing surgical technique for rotator cuff repair (55%, 59%) (Figure 3). A major decrease was found in percentage of websites discussing surgical complications, 42% in 2011 down to 25% in 2014, whereas a major increase was found in percentage discussing rehabilitation, from 39% in 2011 up to 73% in 2014. In 2014, no websites discussed double- versus single-row surgery—compared with 6% in 2011. False claims remained low between 2011 and 2014. In both searches, no website guaranteed a return to sport, and few made claims of painless or bloodless surgery.

DISCERN scores for websites found during the 2014 search were averaged for each of the 6 authorship groups (Figure 4). The highest DISCERN scores were given to academic institution websites (51.6) and public education websites (49). For the academic websites, this difference was significant relative to news, blog, and private physician websites (Ps = .012, .001, .001) The lowest DISCERN scores were given to news organization websites and personal blogs. DISCERN scores were 43.8 for industry sources and 40.7 for private physician groups; the difference was not significant (P = .229). Overall mean DISCERN score for all websites was 44. Eleven percent of websites were HONcode-certified.

No correlation was found between website readability and DISCERN score; correlation coefficient r was .01 (Figure 5). For the websites in 2014, mean Flesch-Kincaid readability score was 50.17, and mean grade level was 10.98; coefficient of determination r² was 0.00012.

The Table compares our data with data from other orthopedic studies that have analyzed the quality of Internet information about various orthopedic injuries, diseases, and procedures.^{3-6,8,9,11-18} With its mean DISCERN score of 44, the present rotator cuff tear study was ranked third of 6 studies that have used this scoring system to analyze website content. Of these 6 studies, those reviewing osteosarcoma and juvenile idiopathic arthritis were ranked highest (mean scores, 49.8 and 48.9, respectively), and the study reviewing scoliosis surgery was ranked lowest (38.9). Bruce-Brand and colleagues9 recently found a mean DISCERN score of 41 for anterior cruciate ligament (ACL) reconstruction. When considering HONcode-certified websites, our Internet search for rotator cuff tears found the third lowest percentage, 10.5%, compared with the other studies (Table); the highest percentage, 30%, was found for websites discussing concussions in athletes. When considering authorship, our rotator cuff study found the third highest percentage, 76%, authored by academic centers, physicians, and public education websites; the highest percentage was found in websites discussing ACL reconstruction. Websites discussing ACL reconstruction also had the highest percentage of websites authored by industry.⁹

Discussion

To our knowledge, this is the first study specifically analyzing the quality of Internet information about rotator cuff repairs. A similar study, conducted by Starman and colleagues¹⁵ in 2010, addressed the quality of web information about 10 common sports medicine diagnoses, one of which was rotator cuff tears. In that study, only 16 of the websites included discussed rotator cuff tears. In addition, the authors used a customized, HONcode-based grading system to analyze each website, making their data difficult to compare across studies.

Ideally, a high-quality medical website should be written by a credible source and should cover a disorder, treatment options, eligibility, rehabilitation, and complications. As there is no standard grading system for analyzing web content about rotator cuff repairs, we analyzed the websites for specific information we thought should be included in a high-quality website (Figures 2, 3). When considering authorship, we found academic centers, private physicians, and educational sources comprised 76% of the sources; industry sources made up only 12%. Similar findings were noted by investigators analyzing Internet information about other orthopedic topics, including ACL reconstruction, lumbar arthroplasty, osteosarcoma, and cervical spine surgery.^5,11,22 Studies analyzing websites for information on ACL tears and distal radius fractures found have a higher percentage of industry-sponsored websites.^9,10

DISCERN showed that the highest-quality information came from websites with academic affiliations, consistent with previous studies,^3,9,17 and its mean score (51.6) was significantly higher than the scores for private physician websites, news sites, and blogs (Ps = .001, .016, .001); the least reliable information was from personal blogs and news outlets. Of note, mean DISCERN score was higher for the industry websites we found than for private physician websites (43.8 vs 40.7), though the difference was not significant (P = .229). Previous investigators considered number of industry-sponsored websites as a marker of poor quality of information relating to a given topic; however, given the DISCERN scores in our study, this might not necessarily be true.⁶ Based on the present study’s data, websites affiliated with academic institutions would be recommended for patients searching for high-quality information about rotator cuff tears.

Given DISCERN scores across studies, information about rotator cuff tears ranked below information about osteosarcoma and juvenile idiopathic arthritis but above information about scoliosis, cervical spine surgery, and ACL reconstruction (Table). DISCERN scores must be compared across studies, as there are no definitions for good and poor DISCERN scores.

Of the 4 studies that analyzed percentage of websites citing peer-reviewed sources, only our study and the study of cervical spine surgery¹⁸ analyzed that percentage as well as DISCERN score. Percentage citing peer-reviewed sources was 26% for rotator cuff tears and 24% for cervical spine surgery; the respective DISCERN scores were 44 and 43.6. As only these 2 studies could be compared, no real correlation between percentage of websites citing peer-reviewed sources and the quality of the content on a given topic can be assessed. More research into this relationship is needed. One already delineated association is the correlation between HONcode-certified sites and high DISCERN scores.²¹ For high-quality medical information, physicians can direct their patients both to academic institution websites and to HONcode-certified websites.

When we compared the present study with previous investigations, we found a large difference between search results for a given topic. In 2013, Duncan and colleagues⁶ and Bruce-Brand and colleagues⁹ used similar study designs (eg, search terms, search engines) for their investigations of quality of web information. Their results, however, were widely different. For example, percentages of industry authorship were 4.5% (Duncan and colleagues⁶) and 64% (Bruce-Brand and colleagues⁹). This inconsistency between studies conducted during similar periods might be related to what appears at the top of the results queue for a search. Duncan and colleagues⁶ analyzed 200 websites, Bruce-Brand and colleagues⁹ only 45. Industries may have made financial arrangements and used search engine optimization techniques to have their websites listed first in search results.

In our study, we also analyzed how web information has changed over time. On the Internet, information changes daily, and we hypothesized that the content found during our 2 searches (2011, 2014) would yield different results. Surprisingly, the data were similar, particularly concerning authorship (Figures 1, 2). In both searches, the largest authorship source was private physician or physician groups (38% in 2011 and 2014). Other authorship sources showed little change in percentage between searches. As for content, we found both increases and decreases in specific web information. Ability to contact authors increased from 21% (2011) to 50% (2014). We think it is important that websites offer a communication channel to people who read the medical information the sites provide. Percentage of websites discussing nonoperative treatment options increased from 11.5% to 61%. Therefore, patients in 2014 were being introduced to more options (in addition to surgery) for managing shoulder pain—an improvement in quality of information between the searches. Percentage of websites discussing surgical eligibility, however, decreased from 43% to 18%—a negative development in information quality. Another decrease, from 42% to 25%, was found for websites discussing surgical complications. Given the data as a whole, and our finding both negative and positive changes, it appears the quality of web content has not improved significantly. Interestingly, no websites discussed double- versus single-row surgery in 2014, but 6% did so in 2011.

Lost in the discussion of quality and reliability of information is whether patients comprehend what they are reading.²³ Yi and colleagues¹⁹ recentlyassessed the readability level of arthroscopy information in articles published online by the American Academy of Orthopaedic Surgeons (AAOS) and the Arthroscopy Association of North America (AANA). The investigators used the Flesch-Kincaid readability test to determine readability level in terms of grade level. They found that the majority of the patient education articles on the AAOS and AANA sites had a readability level far above the national average; only 4 articles were written at or below the eighth-grade level, the current average reading level in the United States.²⁴ Information that is not comprehensible is of no use to patients, and information that physicians and researchers consider high-quality might not be what patients consider high-quality. As we pursue higher-quality web content, we need to consider that its audience includes nonmedical readers, our patients. In the present study, we found that the readability of a website had no correlation with the site’s DISCERN score (Figure 5). Therefore, for information about rotator cuff repairs, higher-quality websites are no harder than lower-quality sites for patients to comprehend. The Flesch-Kincaid readability test is flawed in that it considers only total number of syllables per word and words per sentence, not nontextual elements of patient education materials, such as illustrations on a website. The 10.98 mean grade level found in our study is higher than the levels found for most studies reviewed by Yi and colleagues.¹⁹

This study had several limitations. During an Internet search, the number of websites a user visits drops precipitously after the first page of results. Studies have shown the top 20 sites in a given search receive 97% of the views, and the top 3 receive 58.4%. Whether patients visit websites far down in the list of 150 we found in our given search is unknown. Last, the Flesch-Kincaid readability test is flawed in several ways but nevertheless is used extensively in research. Grading is based on number of words and syllables used in a given sentence; it does not take into account the complexity or common usage of a given word or definition. Therefore, websites may receive low Flesch-Kincaid scores—indicating ease of reading—despite their use of complex medical terminology and jargon that complicate patients’ comprehension of the material.

Conclusion

Numerous authors have evaluated orthopedic patients’ accessing of medical information from the Internet. Although the Internet makes access easier, unreliable content can lead patients to develop certain notions about the direction of their care and certain expectations regarding their clinical outcomes. With there being no regulatory body monitoring content, the peer review process, an essential feature of academic publishing, can be easily circumvented.²⁵

In this study, the highest-quality websites had academic affiliations. Quality of information about rotator cuff repairs was similar to what was found for other orthopedic topics in comparable studies. Surprisingly, there was little change in authorship and content of web information between our 2 search periods (2011, 2014). Although there has been a rapid increase in the number of medical websites, quality of content seems not to have changed significantly. Patients look to physicians for guidance but increasingly are accessing the Internet for additional information. It is essential that physicians understand the quality of information available on the Internet when counseling patients regarding surgery.

Patients are learning about health and disease more independently than before, but such self-education may pose a unique challenge for practicing physicians. Although educated patients can assist in the critical appraisal of treatment options,¹ misinformed patients may have preconceived treatment biases and unrealistic expectations. More than 66 million Americans use the Internet daily, and recent surveys have shown 86% have used the Internet for health-related information.^2,3 With Internet use increasing, the number of patients turning to the web for medical information is increasing as well.⁴ For many patients, this information can be useful in making decisions about their health and health care.⁵

Although accessing medical information from the Internet has grown exponentially, analysis of information quality has grown considerably slower.⁶ With no regulatory body monitoring content, there is easy circumvention of the peer review process, an essential feature of academic publishing.⁷ With no external regulation, the information retrieved may be incorrect, outdated, or misleading. Many orthopedic studies have analyzed Internet content about numerous diagnoses.^3-6,8-18 Most of these studies have found this information highly variable and of poor quality.

We conducted a study to evaluate and analyze rotator cuff repair information available to the general public through the Internet; to assess changes in the quality of information over time; to determine if sites sponsored by academic institutions offered higher-quality information; and to assess whether the readability of the material varied according to DISCERN scores.

Rotator cuff repairs are among the most common surgeries performed by orthopedic surgeons. To our knowledge, this is the first study to assess the quality of web information about rotator cuff repairs. We hypothesized that the quality of information would positively correlate with the reading level of the material presented, that academic institutions would present the highest-quality information, and that the type of information presented would change over time.

Materials and Methods

We used the search phrase rotator cuff repair on the 3 most popular search engines: Google, Yahoo!, and Bing. Google is the dominant engine, taking 83.06% of total market share, followed by Yahoo! (6.86%) and Bing (4.27%).5 The first 50 websites identified by each search engine were selected for evaluation, excluding duplicates or overlapping websites. Similarly, advertisements and strictly video results lacking text were excluded. After each engine was queried, a master list of 150 websites was created for individual evaluation and assessment. To assess changes in results over time, we performed 2 searches, on November 16, 2011, and May 18, 2014.

The content of each website was analyzed for authorship, ability to contact the author, discussion of disorder, surgical treatment, complications, surgical eligibility, rehabilitation, other treatment options, and use of peer-reviewed sources. Authorship was placed in 1 of 6 categories:

1. Academic—university-affiliated physician or research group.

2. Private—physician or group without stated affiliation to an academic organization.

3. Industry—manufacturing or marketing company advertising a product or service for profit.

4. News source—bulletin or article without affiliation to a hospital or an academic institution.

5. Public education—individual or organization with noncommercial website providing third-party information (eg, Wikipedia, About.com).

6. Blog—website publishing an individual’s personal experiences in diary or journal form.

Websites were also assessed for accuracy and validity based on presence or absence of Health On the Net code (HONcode) certification and DISCERN score. Designed by the Health On the Net Foundation in 1996, HONcode provides a framework for disseminating high-quality medical information over the web.¹⁹ Website owners can request that their sites be evaluated for HONcode certification; a site that qualifies can display the HONcode seal.²⁰ The DISCERN project, initially funded by the National Health Service in the United Kingdom, judges the quality of written information available on health-related websites.²¹ It determines the quality of a publication on the basis of 16 questions: The first 8 address the publication’s reliability, the next 7 involve specific details of treatment choices, and the last is an overall rating of the website.

Website readability was assessed with the Flesch-Kincaid test. This test, designed under contract with the US Navy in 1975, has been used in other orthopedic studies.¹⁹ Regression analysis was performed to check for correlation between website readability and DISCERN score. Analysis of variance was used to analyze differences between scores.

Results

We performed a comprehensive analysis of the top 50 websites from each of the 3 search engines (N = 150 websites) (Figures 1–5, Table). Regarding authorship, our 2 searches demonstrated similar values (Figure 1). In 2011, 21% of websites were associated with an academic institution, 38% were authored by private physicians or hospital or physician groups not associated with an academic institution, 11.5% were industry-sponsored, 5% were news bulletins or media reports, 21.5% were public education websites, and 3% were personal blogs. Our 2014 search found a similar distribution of contributors. Between 2011 and 2014, the largest change was in academic authors, which decreased by 7%, from 21% to 14%. Percentage of websites authored by private physicians remained constant from the first to the second search: 38%.

When the 2011 and 2014 website content was compared, several changes were noted. Percentage of websites providing an author contact method increased from 21% to 50% (Figure 2), percentage detailing rotator cuff repairs increased from 82% to 91%, and percentage introducing treatment options in addition to surgical management increased from 11.5% to 61%. Percentage discussing surgical eligibility, however, decreased from 43% to 18%. Percentage citing peer-reviewed sources remained relatively constant (28%, 26%), as did percentage discussing surgical technique for rotator cuff repair (55%, 59%) (Figure 3). A major decrease was found in percentage of websites discussing surgical complications, 42% in 2011 down to 25% in 2014, whereas a major increase was found in percentage discussing rehabilitation, from 39% in 2011 up to 73% in 2014. In 2014, no websites discussed double- versus single-row surgery—compared with 6% in 2011. False claims remained low between 2011 and 2014. In both searches, no website guaranteed a return to sport, and few made claims of painless or bloodless surgery.

DISCERN scores for websites found during the 2014 search were averaged for each of the 6 authorship groups (Figure 4). The highest DISCERN scores were given to academic institution websites (51.6) and public education websites (49). For the academic websites, this difference was significant relative to news, blog, and private physician websites (Ps = .012, .001, .001) The lowest DISCERN scores were given to news organization websites and personal blogs. DISCERN scores were 43.8 for industry sources and 40.7 for private physician groups; the difference was not significant (P = .229). Overall mean DISCERN score for all websites was 44. Eleven percent of websites were HONcode-certified.

No correlation was found between website readability and DISCERN score; correlation coefficient r was .01 (Figure 5). For the websites in 2014, mean Flesch-Kincaid readability score was 50.17, and mean grade level was 10.98; coefficient of determination r² was 0.00012.

The Table compares our data with data from other orthopedic studies that have analyzed the quality of Internet information about various orthopedic injuries, diseases, and procedures.^{3-6,8,9,11-18} With its mean DISCERN score of 44, the present rotator cuff tear study was ranked third of 6 studies that have used this scoring system to analyze website content. Of these 6 studies, those reviewing osteosarcoma and juvenile idiopathic arthritis were ranked highest (mean scores, 49.8 and 48.9, respectively), and the study reviewing scoliosis surgery was ranked lowest (38.9). Bruce-Brand and colleagues9 recently found a mean DISCERN score of 41 for anterior cruciate ligament (ACL) reconstruction. When considering HONcode-certified websites, our Internet search for rotator cuff tears found the third lowest percentage, 10.5%, compared with the other studies (Table); the highest percentage, 30%, was found for websites discussing concussions in athletes. When considering authorship, our rotator cuff study found the third highest percentage, 76%, authored by academic centers, physicians, and public education websites; the highest percentage was found in websites discussing ACL reconstruction. Websites discussing ACL reconstruction also had the highest percentage of websites authored by industry.⁹

Discussion

To our knowledge, this is the first study specifically analyzing the quality of Internet information about rotator cuff repairs. A similar study, conducted by Starman and colleagues¹⁵ in 2010, addressed the quality of web information about 10 common sports medicine diagnoses, one of which was rotator cuff tears. In that study, only 16 of the websites included discussed rotator cuff tears. In addition, the authors used a customized, HONcode-based grading system to analyze each website, making their data difficult to compare across studies.

Ideally, a high-quality medical website should be written by a credible source and should cover a disorder, treatment options, eligibility, rehabilitation, and complications. As there is no standard grading system for analyzing web content about rotator cuff repairs, we analyzed the websites for specific information we thought should be included in a high-quality website (Figures 2, 3). When considering authorship, we found academic centers, private physicians, and educational sources comprised 76% of the sources; industry sources made up only 12%. Similar findings were noted by investigators analyzing Internet information about other orthopedic topics, including ACL reconstruction, lumbar arthroplasty, osteosarcoma, and cervical spine surgery.^5,11,22 Studies analyzing websites for information on ACL tears and distal radius fractures found have a higher percentage of industry-sponsored websites.^9,10

DISCERN showed that the highest-quality information came from websites with academic affiliations, consistent with previous studies,^3,9,17 and its mean score (51.6) was significantly higher than the scores for private physician websites, news sites, and blogs (Ps = .001, .016, .001); the least reliable information was from personal blogs and news outlets. Of note, mean DISCERN score was higher for the industry websites we found than for private physician websites (43.8 vs 40.7), though the difference was not significant (P = .229). Previous investigators considered number of industry-sponsored websites as a marker of poor quality of information relating to a given topic; however, given the DISCERN scores in our study, this might not necessarily be true.⁶ Based on the present study’s data, websites affiliated with academic institutions would be recommended for patients searching for high-quality information about rotator cuff tears.

Given DISCERN scores across studies, information about rotator cuff tears ranked below information about osteosarcoma and juvenile idiopathic arthritis but above information about scoliosis, cervical spine surgery, and ACL reconstruction (Table). DISCERN scores must be compared across studies, as there are no definitions for good and poor DISCERN scores.

Of the 4 studies that analyzed percentage of websites citing peer-reviewed sources, only our study and the study of cervical spine surgery¹⁸ analyzed that percentage as well as DISCERN score. Percentage citing peer-reviewed sources was 26% for rotator cuff tears and 24% for cervical spine surgery; the respective DISCERN scores were 44 and 43.6. As only these 2 studies could be compared, no real correlation between percentage of websites citing peer-reviewed sources and the quality of the content on a given topic can be assessed. More research into this relationship is needed. One already delineated association is the correlation between HONcode-certified sites and high DISCERN scores.²¹ For high-quality medical information, physicians can direct their patients both to academic institution websites and to HONcode-certified websites.

When we compared the present study with previous investigations, we found a large difference between search results for a given topic. In 2013, Duncan and colleagues⁶ and Bruce-Brand and colleagues⁹ used similar study designs (eg, search terms, search engines) for their investigations of quality of web information. Their results, however, were widely different. For example, percentages of industry authorship were 4.5% (Duncan and colleagues⁶) and 64% (Bruce-Brand and colleagues⁹). This inconsistency between studies conducted during similar periods might be related to what appears at the top of the results queue for a search. Duncan and colleagues⁶ analyzed 200 websites, Bruce-Brand and colleagues⁹ only 45. Industries may have made financial arrangements and used search engine optimization techniques to have their websites listed first in search results.

In our study, we also analyzed how web information has changed over time. On the Internet, information changes daily, and we hypothesized that the content found during our 2 searches (2011, 2014) would yield different results. Surprisingly, the data were similar, particularly concerning authorship (Figures 1, 2). In both searches, the largest authorship source was private physician or physician groups (38% in 2011 and 2014). Other authorship sources showed little change in percentage between searches. As for content, we found both increases and decreases in specific web information. Ability to contact authors increased from 21% (2011) to 50% (2014). We think it is important that websites offer a communication channel to people who read the medical information the sites provide. Percentage of websites discussing nonoperative treatment options increased from 11.5% to 61%. Therefore, patients in 2014 were being introduced to more options (in addition to surgery) for managing shoulder pain—an improvement in quality of information between the searches. Percentage of websites discussing surgical eligibility, however, decreased from 43% to 18%—a negative development in information quality. Another decrease, from 42% to 25%, was found for websites discussing surgical complications. Given the data as a whole, and our finding both negative and positive changes, it appears the quality of web content has not improved significantly. Interestingly, no websites discussed double- versus single-row surgery in 2014, but 6% did so in 2011.

Lost in the discussion of quality and reliability of information is whether patients comprehend what they are reading.²³ Yi and colleagues¹⁹ recentlyassessed the readability level of arthroscopy information in articles published online by the American Academy of Orthopaedic Surgeons (AAOS) and the Arthroscopy Association of North America (AANA). The investigators used the Flesch-Kincaid readability test to determine readability level in terms of grade level. They found that the majority of the patient education articles on the AAOS and AANA sites had a readability level far above the national average; only 4 articles were written at or below the eighth-grade level, the current average reading level in the United States.²⁴ Information that is not comprehensible is of no use to patients, and information that physicians and researchers consider high-quality might not be what patients consider high-quality. As we pursue higher-quality web content, we need to consider that its audience includes nonmedical readers, our patients. In the present study, we found that the readability of a website had no correlation with the site’s DISCERN score (Figure 5). Therefore, for information about rotator cuff repairs, higher-quality websites are no harder than lower-quality sites for patients to comprehend. The Flesch-Kincaid readability test is flawed in that it considers only total number of syllables per word and words per sentence, not nontextual elements of patient education materials, such as illustrations on a website. The 10.98 mean grade level found in our study is higher than the levels found for most studies reviewed by Yi and colleagues.¹⁹

This study had several limitations. During an Internet search, the number of websites a user visits drops precipitously after the first page of results. Studies have shown the top 20 sites in a given search receive 97% of the views, and the top 3 receive 58.4%. Whether patients visit websites far down in the list of 150 we found in our given search is unknown. Last, the Flesch-Kincaid readability test is flawed in several ways but nevertheless is used extensively in research. Grading is based on number of words and syllables used in a given sentence; it does not take into account the complexity or common usage of a given word or definition. Therefore, websites may receive low Flesch-Kincaid scores—indicating ease of reading—despite their use of complex medical terminology and jargon that complicate patients’ comprehension of the material.

Conclusion

Numerous authors have evaluated orthopedic patients’ accessing of medical information from the Internet. Although the Internet makes access easier, unreliable content can lead patients to develop certain notions about the direction of their care and certain expectations regarding their clinical outcomes. With there being no regulatory body monitoring content, the peer review process, an essential feature of academic publishing, can be easily circumvented.²⁵

In this study, the highest-quality websites had academic affiliations. Quality of information about rotator cuff repairs was similar to what was found for other orthopedic topics in comparable studies. Surprisingly, there was little change in authorship and content of web information between our 2 search periods (2011, 2014). Although there has been a rapid increase in the number of medical websites, quality of content seems not to have changed significantly. Patients look to physicians for guidance but increasingly are accessing the Internet for additional information. It is essential that physicians understand the quality of information available on the Internet when counseling patients regarding surgery.

As defined by the American Academy of Orthopaedic Surgeons (AAOS) in 1996, conflict of interest (COI) is the “circumstance that exists when, because of personal financial gain, an individual has the potential to be less than objective when called on to reach a judgment or interpret a result.”¹ In medical research, COIs often occur in relationships between physician-researchers and pharmaceutical, medical device, and biotechnology companies. These relationships usually take the form of research grants but can also arise when the researcher has a financial interest in the product being tested or in the company that manufactures the product.

 Although constructive collaboration between academic medicine and industry has worked to improve health care and ultimately benefit patients, potential drawbacks of such relationships include sequestration and suppression of results that may be disadvantageous to the industry sponsor,² increased likelihood of reporting positive results (pro-industry),^3-7 and biased study designs.⁸ The nature of such relationships may threaten the integrity of scientific studies, the objectivity of medical education, the quality of patient care, and the public’s trust in medicine.⁹

Financial relationships and affiliations are increasing as we seek to answer a growing number of clinical questions—with funding often being a limiting factor. At national scientific meetings, the number of presentations reporting COIs reflects this trend. Paper and poster presentations accepted for annual meetings of the Orthopaedic Trauma Association (OTA) and reporting a COI increased from 7.6% in 1993 to 12.6% in 2002 (P = .0129).²

Medical subspecialties outside of orthopedics are experiencing similar trends. Most notable is the American Psychiatric Association (APA). After the APA published a mandatory financial COI disclosure policy in the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), the percentage of task force members reporting industry relationships increased by 12%.¹⁰ Analysis of the DSM-5 panels demonstrated that the panels with the largest percentage of reported COIs are those for which pharmacological treatment is the first-line intervention, including the panels for mood disorders (67%), psychotic disorders (83%) and sleep/wake disorders (100%).¹⁰ Moreover, the industry ties reported are to the pharmaceutical companies that manufacture the medications used to treat these disorders or to companies that service the pharmaceutical industry.¹⁰

The degree to which financial COIs affect the interpretation of the orthopedic literature has never been quantified. Although it is clear that COIs can confound the results and reporting of data, how the medical community uses disclosures when interpreting the literature and when formulating opinions that may or may not affect their practice patterns is largely unknown.

We conducted a study to evaluate how a hypothetical financial COI disclosure would influence the interpretation of data by orthopedic clinicians. We also wanted to determine the reliability of the data as perceived in association with different study designs, levels of evidence, research institutional settings, and reporting of positive or negative results.

Methods

We asked members of the Arthroscopy Association of North America (AANA) and the American Orthopaedic Society for Sports Medicine (AOSSM) to complete a multiple-choice situational questionnaire (Table). The questionnaire assesses the degree to which respondents use COI disclosures when interpreting the literature. It further explores the perceived clinical value of a study with a given reported COI, assuming variations in study design, research institutional setting, and significance of results. The fictional research team disclosed the project was funded by a pharmaceutical company and all team members received consulting compensation. The survey and study were reviewed and approved by our institutional review board. The survey consisted of 14 multiple-choice questions that allowed for only 1 answer selection per person and allowed survey takers to skip questions they did not wish to answer. The survey questions and associated response options appear in edited form in the Table. A link to the questionnaire (https://www.surveymonkey.com/s/MPCCLCX) was sent with a message explaining the study. The responses to the questionnaire constituted the data.

Results

We sent a request to participate in the survey to 750 physicians and received 522 responses (overall response rate, 70%). The response rate for each question equaled or exceeded 98%.

The majority of respondents (95.6%) were male. Ninety-nine percent of respondents were orthopedic surgeons. The Northeast (US) was the most common geographical practice location of respondents (32%), followed by the Midwest (19.1%) and the Southeast (16.6%). Most respondents (40%) had been in practice for more than 20 years; 67% had been in practice a minimum of 10 years. The majority (68.8%) were employed by private practice groups, either single specialty (57.8%) or multispecialty (11%).

Eighty percent of respondents strongly agreed that COI disclosure is important when interpreting study results, 62% reported always reading the disclosure slide during academy or other meeting presentations, and 41% reported always using this information when deciding how to interpret scientific data.

Seventy-five percent of respondents thought the study—an academic-center case series with significant results in favor of the pharmaceutical company funding the study—was biased (42% indicated biased with merit, 33% biased without merit). Twenty-three percent thought the study was possibly biased, but likely trustworthy given the academic institutional affiliation. When the study setting was changed to community hospital, 95% thought the study was biased (51% biased with merit, 44% biased without merit). With the same study performed at an academic center, and no statistically significant results (not in favor of the pharmaceutical company funding the study), 88% thought the study had merit (46% biased with merit, 42% unbiased with merit).

When the study design was changed to a randomized controlled trial (level I evidence) conducted at an academic center with negative results, an overwhelming 95% of respondents thought the study had merit (33% biased with merit, 62% unbiased with merit). Given the same study design at an academic center, with positive results, 78% still thought the study had merit (39% biased with merit, 39% unbiased with merit). An additional 18% thought the study was biased, but still likely trustworthy given the academic institutional affiliation. Finally, given a randomized controlled trial and positive results, but with the research setting a small community practice, 90% thought the study had merit (51% biased with merit, 39% unbiased with merit). The percentage of respondents who found the study biased and likely without merit increased from 3.7% to 9.5% when the institutional affiliation changed from academic to community.

Discussion

As governmental funding sources become increasingly limited, the role of industry sponsorship of orthopedic research has grown. Potential drawbacks and biases of such research support have been well described—most notably, increased positive result reporting, suppression of results that may be disadvantageous to the industry sponsor, and biased study designs.^2-8 However, the extent to which financial COIs affect the orthopedic medical community’s interpretation of the literature has never been quantified. To our knowledge, the present study is the first to quantify the impact of reported COI on study interpretation.

Our goal was to examine how reported financial COIs influence the interpretation of the literature by the orthopedic medical community. Moreover, we wanted to determine the perceived reliability of the data when variables (study design, institutional affiliation, positive vs negative results) were changed. The results of our survey indicate that, when a financial COI is reported, study reliability is perceived as highest when negative results were found.

Our survey noted a discrepancy between the documented importance of the hypothetical research team’s COI disclosure and the use of such disclosures when interpreting study results. Eighty percent of respondents agreed that COI disclosure is important when interpreting study results, but only 62% reported always reading disclosures, and even fewer (41%) reported always using the information when interpreting results. It is unclear exactly why this trend exists, as one would expect the percentages to be more similar. These particular survey questions were formed around using COI disclosures when interpreting study results during academic presentations at national meetings and not during the review of published literature. It is possible that positioning the COI disclosure at the beginning of a presentation has an effect, but only 3.7% of respondents indicated they seldom remembered the disclosure by the end of the presentation. The results of our survey may have varied if the questions had targeted reading and interpreting the literature.

Interestingly, the results of these survey questions tended to be more consistent with rates of reported financial COI by presenters at national orthopedic meetings. A study published in the New England Journal of Medicine found that less than 80% of orthopedic surgeons reported their disclosures at a large annual meeting (AAOS), even when the disclosure involved payments pertinent to the research they were presenting.⁵ When the payments were indirectly related to the research, the percentage of surgeons reporting disclosures was 50%, almost the same as the disclosure rate for unrelated payments.⁵

When the study was changed to a level I randomized controlled trial, more survey respondents found it to be less biased and have more merit. Although it would seem intuitive for a study with a higher level of evidence to carry more clinical value during interpretation, this may not hold true in the setting of industry-sponsored clinical trials. Several studies have documented a significant association between the reporting of positive results and industry-sponsored randomized clinical trials. In 2008, Khan and colleagues³ examined 100 orthopedic randomized clinical trials reported in 5 major orthopedic subspecialty journals over a 2-year period. The association between industry funding and favorable outcome in all original randomized clinical trials was strong and significant (P < .001). This is not surprising, given the amount of time and money required for a well-designed clinical study. Commercial products with preclinical promise are pushed to testing in a clinical trial, whereas resources would not be wasted on products lacking preclinical merit.

The most important variable affecting interpretation of study merit by survey respondents was the reporting of negative results. As more researchers are developing COIs, many studies are discovering a relationship between COIs and outcomes of research studies. Reviewing the adult total joint literature, Ezzet⁸ found an industry funding rate of 50%. Positive results were reported in 93% of cases in commercially funded studies versus 37% of cases in independently funded studies. Furthermore, no negative results were reported by investigators who were receiving royalties from the respective companies.

Studies across the medical literature have also found this association between industry sponsorship and reporting of positive results. One such study, reported by Valachis and colleagues⁷ in the Journal of Clinical Oncology, examined more than 80 economic analyses of targeted oncologic therapies and found the studies funded by pharmaceutical companies were more likely to report favorable qualitative cost estimates. In addition, when studies with a COI disclosure were examined, those reporting any financial relationship with a manufacturer (eg, author affiliation, funding) were more likely than those without such a relationship to report favorable results.

Our study had several limitations. First, as most of the survey respondents were orthopedic surgeons, extrapolating their data to the medical community at large may not be appropriate, as each specialty may view industry affiliations differently. In addition, respondents were asked to base their interpretations of a study on conclusions we predetermined—no direct visualization of the data set or statistical testing methods. It is possible that these responses may have been different had the respondents had the opportunity to further evaluate the study in question. In a recent study, Altwairgi and colleagues¹¹ found that 10% of randomized clinical trials involving lung cancer treatment were reported with different conclusions in their full manuscripts relative to their abstracts. We think our survey design perhaps best mimics an annual meeting environment in which participants have very limited ability to interpret studies and may rely more heavily on the factors we investigated—study design, significance of findings, and setting, all similar to information presented in an abstract—when making informed decisions. Although our response rate was only 70%, this is comparable to or better than the rates in similar survey studies that used email-based questionnaires.^12,13

Another limitation was that our survey may have forced respondents into answers they did not entirely agree with, given the limited options of the multiple-choice response format and the specific wording of the questions. Our conclusions may have been more dramatic when we were evaluating whether the study was deemed meritorious or not. However, there is no adopted standard for evaluating the extent of bias perceived by a clinician. We thought it was important to include answer options indicating a study had merit despite obvious bias in design and execution. That a study had merit can mean different things. It may change clinical practice, may require further study and reproducibility, or may not be significant enough to matter. Asking follow-up questions to evaluate this perception among the respondents could have provided validity to the term merit. Further studies in this field are needed to determine how studies are interpreted and translated into clinical practice by various clinicians.

Conclusion

Although the present study is not a quantitative analysis of the determination of bias in the orthopedic community, it is the first to evaluate orthopedic surgeons’ perceptions on the basis of key fundamentals of orthopedic research relative to COI. It is clear from our study results that introducing levels of evidence to the orthopedic milieu has had a significant impact both on the quality of research and on the foundational use of deductive reasoning when interpreting the literature. Reporting negative outcomes is perhaps the most important factor in eliminating the perception of bias among orthopedic surgeons. To what extent a perceived COI plays into medical decision-making and the ultimate treatment of patients is still relatively unknown.

As defined by the American Academy of Orthopaedic Surgeons (AAOS) in 1996, conflict of interest (COI) is the “circumstance that exists when, because of personal financial gain, an individual has the potential to be less than objective when called on to reach a judgment or interpret a result.”¹ In medical research, COIs often occur in relationships between physician-researchers and pharmaceutical, medical device, and biotechnology companies. These relationships usually take the form of research grants but can also arise when the researcher has a financial interest in the product being tested or in the company that manufactures the product.

 Although constructive collaboration between academic medicine and industry has worked to improve health care and ultimately benefit patients, potential drawbacks of such relationships include sequestration and suppression of results that may be disadvantageous to the industry sponsor,² increased likelihood of reporting positive results (pro-industry),^3-7 and biased study designs.⁸ The nature of such relationships may threaten the integrity of scientific studies, the objectivity of medical education, the quality of patient care, and the public’s trust in medicine.⁹

Financial relationships and affiliations are increasing as we seek to answer a growing number of clinical questions—with funding often being a limiting factor. At national scientific meetings, the number of presentations reporting COIs reflects this trend. Paper and poster presentations accepted for annual meetings of the Orthopaedic Trauma Association (OTA) and reporting a COI increased from 7.6% in 1993 to 12.6% in 2002 (P = .0129).²

Medical subspecialties outside of orthopedics are experiencing similar trends. Most notable is the American Psychiatric Association (APA). After the APA published a mandatory financial COI disclosure policy in the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), the percentage of task force members reporting industry relationships increased by 12%.¹⁰ Analysis of the DSM-5 panels demonstrated that the panels with the largest percentage of reported COIs are those for which pharmacological treatment is the first-line intervention, including the panels for mood disorders (67%), psychotic disorders (83%) and sleep/wake disorders (100%).¹⁰ Moreover, the industry ties reported are to the pharmaceutical companies that manufacture the medications used to treat these disorders or to companies that service the pharmaceutical industry.¹⁰

The degree to which financial COIs affect the interpretation of the orthopedic literature has never been quantified. Although it is clear that COIs can confound the results and reporting of data, how the medical community uses disclosures when interpreting the literature and when formulating opinions that may or may not affect their practice patterns is largely unknown.

We conducted a study to evaluate how a hypothetical financial COI disclosure would influence the interpretation of data by orthopedic clinicians. We also wanted to determine the reliability of the data as perceived in association with different study designs, levels of evidence, research institutional settings, and reporting of positive or negative results.

Methods

We asked members of the Arthroscopy Association of North America (AANA) and the American Orthopaedic Society for Sports Medicine (AOSSM) to complete a multiple-choice situational questionnaire (Table). The questionnaire assesses the degree to which respondents use COI disclosures when interpreting the literature. It further explores the perceived clinical value of a study with a given reported COI, assuming variations in study design, research institutional setting, and significance of results. The fictional research team disclosed the project was funded by a pharmaceutical company and all team members received consulting compensation. The survey and study were reviewed and approved by our institutional review board. The survey consisted of 14 multiple-choice questions that allowed for only 1 answer selection per person and allowed survey takers to skip questions they did not wish to answer. The survey questions and associated response options appear in edited form in the Table. A link to the questionnaire (https://www.surveymonkey.com/s/MPCCLCX) was sent with a message explaining the study. The responses to the questionnaire constituted the data.

Results

We sent a request to participate in the survey to 750 physicians and received 522 responses (overall response rate, 70%). The response rate for each question equaled or exceeded 98%.

The majority of respondents (95.6%) were male. Ninety-nine percent of respondents were orthopedic surgeons. The Northeast (US) was the most common geographical practice location of respondents (32%), followed by the Midwest (19.1%) and the Southeast (16.6%). Most respondents (40%) had been in practice for more than 20 years; 67% had been in practice a minimum of 10 years. The majority (68.8%) were employed by private practice groups, either single specialty (57.8%) or multispecialty (11%).

Eighty percent of respondents strongly agreed that COI disclosure is important when interpreting study results, 62% reported always reading the disclosure slide during academy or other meeting presentations, and 41% reported always using this information when deciding how to interpret scientific data.

Seventy-five percent of respondents thought the study—an academic-center case series with significant results in favor of the pharmaceutical company funding the study—was biased (42% indicated biased with merit, 33% biased without merit). Twenty-three percent thought the study was possibly biased, but likely trustworthy given the academic institutional affiliation. When the study setting was changed to community hospital, 95% thought the study was biased (51% biased with merit, 44% biased without merit). With the same study performed at an academic center, and no statistically significant results (not in favor of the pharmaceutical company funding the study), 88% thought the study had merit (46% biased with merit, 42% unbiased with merit).

When the study design was changed to a randomized controlled trial (level I evidence) conducted at an academic center with negative results, an overwhelming 95% of respondents thought the study had merit (33% biased with merit, 62% unbiased with merit). Given the same study design at an academic center, with positive results, 78% still thought the study had merit (39% biased with merit, 39% unbiased with merit). An additional 18% thought the study was biased, but still likely trustworthy given the academic institutional affiliation. Finally, given a randomized controlled trial and positive results, but with the research setting a small community practice, 90% thought the study had merit (51% biased with merit, 39% unbiased with merit). The percentage of respondents who found the study biased and likely without merit increased from 3.7% to 9.5% when the institutional affiliation changed from academic to community.

Discussion

As governmental funding sources become increasingly limited, the role of industry sponsorship of orthopedic research has grown. Potential drawbacks and biases of such research support have been well described—most notably, increased positive result reporting, suppression of results that may be disadvantageous to the industry sponsor, and biased study designs.^2-8 However, the extent to which financial COIs affect the orthopedic medical community’s interpretation of the literature has never been quantified. To our knowledge, the present study is the first to quantify the impact of reported COI on study interpretation.

Our goal was to examine how reported financial COIs influence the interpretation of the literature by the orthopedic medical community. Moreover, we wanted to determine the perceived reliability of the data when variables (study design, institutional affiliation, positive vs negative results) were changed. The results of our survey indicate that, when a financial COI is reported, study reliability is perceived as highest when negative results were found.

Our survey noted a discrepancy between the documented importance of the hypothetical research team’s COI disclosure and the use of such disclosures when interpreting study results. Eighty percent of respondents agreed that COI disclosure is important when interpreting study results, but only 62% reported always reading disclosures, and even fewer (41%) reported always using the information when interpreting results. It is unclear exactly why this trend exists, as one would expect the percentages to be more similar. These particular survey questions were formed around using COI disclosures when interpreting study results during academic presentations at national meetings and not during the review of published literature. It is possible that positioning the COI disclosure at the beginning of a presentation has an effect, but only 3.7% of respondents indicated they seldom remembered the disclosure by the end of the presentation. The results of our survey may have varied if the questions had targeted reading and interpreting the literature.

Interestingly, the results of these survey questions tended to be more consistent with rates of reported financial COI by presenters at national orthopedic meetings. A study published in the New England Journal of Medicine found that less than 80% of orthopedic surgeons reported their disclosures at a large annual meeting (AAOS), even when the disclosure involved payments pertinent to the research they were presenting.⁵ When the payments were indirectly related to the research, the percentage of surgeons reporting disclosures was 50%, almost the same as the disclosure rate for unrelated payments.⁵

When the study was changed to a level I randomized controlled trial, more survey respondents found it to be less biased and have more merit. Although it would seem intuitive for a study with a higher level of evidence to carry more clinical value during interpretation, this may not hold true in the setting of industry-sponsored clinical trials. Several studies have documented a significant association between the reporting of positive results and industry-sponsored randomized clinical trials. In 2008, Khan and colleagues³ examined 100 orthopedic randomized clinical trials reported in 5 major orthopedic subspecialty journals over a 2-year period. The association between industry funding and favorable outcome in all original randomized clinical trials was strong and significant (P < .001). This is not surprising, given the amount of time and money required for a well-designed clinical study. Commercial products with preclinical promise are pushed to testing in a clinical trial, whereas resources would not be wasted on products lacking preclinical merit.

The most important variable affecting interpretation of study merit by survey respondents was the reporting of negative results. As more researchers are developing COIs, many studies are discovering a relationship between COIs and outcomes of research studies. Reviewing the adult total joint literature, Ezzet⁸ found an industry funding rate of 50%. Positive results were reported in 93% of cases in commercially funded studies versus 37% of cases in independently funded studies. Furthermore, no negative results were reported by investigators who were receiving royalties from the respective companies.

Studies across the medical literature have also found this association between industry sponsorship and reporting of positive results. One such study, reported by Valachis and colleagues⁷ in the Journal of Clinical Oncology, examined more than 80 economic analyses of targeted oncologic therapies and found the studies funded by pharmaceutical companies were more likely to report favorable qualitative cost estimates. In addition, when studies with a COI disclosure were examined, those reporting any financial relationship with a manufacturer (eg, author affiliation, funding) were more likely than those without such a relationship to report favorable results.

Our study had several limitations. First, as most of the survey respondents were orthopedic surgeons, extrapolating their data to the medical community at large may not be appropriate, as each specialty may view industry affiliations differently. In addition, respondents were asked to base their interpretations of a study on conclusions we predetermined—no direct visualization of the data set or statistical testing methods. It is possible that these responses may have been different had the respondents had the opportunity to further evaluate the study in question. In a recent study, Altwairgi and colleagues¹¹ found that 10% of randomized clinical trials involving lung cancer treatment were reported with different conclusions in their full manuscripts relative to their abstracts. We think our survey design perhaps best mimics an annual meeting environment in which participants have very limited ability to interpret studies and may rely more heavily on the factors we investigated—study design, significance of findings, and setting, all similar to information presented in an abstract—when making informed decisions. Although our response rate was only 70%, this is comparable to or better than the rates in similar survey studies that used email-based questionnaires.^12,13

Another limitation was that our survey may have forced respondents into answers they did not entirely agree with, given the limited options of the multiple-choice response format and the specific wording of the questions. Our conclusions may have been more dramatic when we were evaluating whether the study was deemed meritorious or not. However, there is no adopted standard for evaluating the extent of bias perceived by a clinician. We thought it was important to include answer options indicating a study had merit despite obvious bias in design and execution. That a study had merit can mean different things. It may change clinical practice, may require further study and reproducibility, or may not be significant enough to matter. Asking follow-up questions to evaluate this perception among the respondents could have provided validity to the term merit. Further studies in this field are needed to determine how studies are interpreted and translated into clinical practice by various clinicians.

Conclusion

Although the present study is not a quantitative analysis of the determination of bias in the orthopedic community, it is the first to evaluate orthopedic surgeons’ perceptions on the basis of key fundamentals of orthopedic research relative to COI. It is clear from our study results that introducing levels of evidence to the orthopedic milieu has had a significant impact both on the quality of research and on the foundational use of deductive reasoning when interpreting the literature. Reporting negative outcomes is perhaps the most important factor in eliminating the perception of bias among orthopedic surgeons. To what extent a perceived COI plays into medical decision-making and the ultimate treatment of patients is still relatively unknown.

As defined by the American Academy of Orthopaedic Surgeons (AAOS) in 1996, conflict of interest (COI) is the “circumstance that exists when, because of personal financial gain, an individual has the potential to be less than objective when called on to reach a judgment or interpret a result.”¹ In medical research, COIs often occur in relationships between physician-researchers and pharmaceutical, medical device, and biotechnology companies. These relationships usually take the form of research grants but can also arise when the researcher has a financial interest in the product being tested or in the company that manufactures the product.

 Although constructive collaboration between academic medicine and industry has worked to improve health care and ultimately benefit patients, potential drawbacks of such relationships include sequestration and suppression of results that may be disadvantageous to the industry sponsor,² increased likelihood of reporting positive results (pro-industry),^3-7 and biased study designs.⁸ The nature of such relationships may threaten the integrity of scientific studies, the objectivity of medical education, the quality of patient care, and the public’s trust in medicine.⁹

Financial relationships and affiliations are increasing as we seek to answer a growing number of clinical questions—with funding often being a limiting factor. At national scientific meetings, the number of presentations reporting COIs reflects this trend. Paper and poster presentations accepted for annual meetings of the Orthopaedic Trauma Association (OTA) and reporting a COI increased from 7.6% in 1993 to 12.6% in 2002 (P = .0129).²

Medical subspecialties outside of orthopedics are experiencing similar trends. Most notable is the American Psychiatric Association (APA). After the APA published a mandatory financial COI disclosure policy in the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), the percentage of task force members reporting industry relationships increased by 12%.¹⁰ Analysis of the DSM-5 panels demonstrated that the panels with the largest percentage of reported COIs are those for which pharmacological treatment is the first-line intervention, including the panels for mood disorders (67%), psychotic disorders (83%) and sleep/wake disorders (100%).¹⁰ Moreover, the industry ties reported are to the pharmaceutical companies that manufacture the medications used to treat these disorders or to companies that service the pharmaceutical industry.¹⁰

The degree to which financial COIs affect the interpretation of the orthopedic literature has never been quantified. Although it is clear that COIs can confound the results and reporting of data, how the medical community uses disclosures when interpreting the literature and when formulating opinions that may or may not affect their practice patterns is largely unknown.

We conducted a study to evaluate how a hypothetical financial COI disclosure would influence the interpretation of data by orthopedic clinicians. We also wanted to determine the reliability of the data as perceived in association with different study designs, levels of evidence, research institutional settings, and reporting of positive or negative results.

Methods

We asked members of the Arthroscopy Association of North America (AANA) and the American Orthopaedic Society for Sports Medicine (AOSSM) to complete a multiple-choice situational questionnaire (Table). The questionnaire assesses the degree to which respondents use COI disclosures when interpreting the literature. It further explores the perceived clinical value of a study with a given reported COI, assuming variations in study design, research institutional setting, and significance of results. The fictional research team disclosed the project was funded by a pharmaceutical company and all team members received consulting compensation. The survey and study were reviewed and approved by our institutional review board. The survey consisted of 14 multiple-choice questions that allowed for only 1 answer selection per person and allowed survey takers to skip questions they did not wish to answer. The survey questions and associated response options appear in edited form in the Table. A link to the questionnaire (https://www.surveymonkey.com/s/MPCCLCX) was sent with a message explaining the study. The responses to the questionnaire constituted the data.

Results

We sent a request to participate in the survey to 750 physicians and received 522 responses (overall response rate, 70%). The response rate for each question equaled or exceeded 98%.

The majority of respondents (95.6%) were male. Ninety-nine percent of respondents were orthopedic surgeons. The Northeast (US) was the most common geographical practice location of respondents (32%), followed by the Midwest (19.1%) and the Southeast (16.6%). Most respondents (40%) had been in practice for more than 20 years; 67% had been in practice a minimum of 10 years. The majority (68.8%) were employed by private practice groups, either single specialty (57.8%) or multispecialty (11%).

Eighty percent of respondents strongly agreed that COI disclosure is important when interpreting study results, 62% reported always reading the disclosure slide during academy or other meeting presentations, and 41% reported always using this information when deciding how to interpret scientific data.

Seventy-five percent of respondents thought the study—an academic-center case series with significant results in favor of the pharmaceutical company funding the study—was biased (42% indicated biased with merit, 33% biased without merit). Twenty-three percent thought the study was possibly biased, but likely trustworthy given the academic institutional affiliation. When the study setting was changed to community hospital, 95% thought the study was biased (51% biased with merit, 44% biased without merit). With the same study performed at an academic center, and no statistically significant results (not in favor of the pharmaceutical company funding the study), 88% thought the study had merit (46% biased with merit, 42% unbiased with merit).

When the study design was changed to a randomized controlled trial (level I evidence) conducted at an academic center with negative results, an overwhelming 95% of respondents thought the study had merit (33% biased with merit, 62% unbiased with merit). Given the same study design at an academic center, with positive results, 78% still thought the study had merit (39% biased with merit, 39% unbiased with merit). An additional 18% thought the study was biased, but still likely trustworthy given the academic institutional affiliation. Finally, given a randomized controlled trial and positive results, but with the research setting a small community practice, 90% thought the study had merit (51% biased with merit, 39% unbiased with merit). The percentage of respondents who found the study biased and likely without merit increased from 3.7% to 9.5% when the institutional affiliation changed from academic to community.

Discussion

As governmental funding sources become increasingly limited, the role of industry sponsorship of orthopedic research has grown. Potential drawbacks and biases of such research support have been well described—most notably, increased positive result reporting, suppression of results that may be disadvantageous to the industry sponsor, and biased study designs.^2-8 However, the extent to which financial COIs affect the orthopedic medical community’s interpretation of the literature has never been quantified. To our knowledge, the present study is the first to quantify the impact of reported COI on study interpretation.

Our goal was to examine how reported financial COIs influence the interpretation of the literature by the orthopedic medical community. Moreover, we wanted to determine the perceived reliability of the data when variables (study design, institutional affiliation, positive vs negative results) were changed. The results of our survey indicate that, when a financial COI is reported, study reliability is perceived as highest when negative results were found.

Our survey noted a discrepancy between the documented importance of the hypothetical research team’s COI disclosure and the use of such disclosures when interpreting study results. Eighty percent of respondents agreed that COI disclosure is important when interpreting study results, but only 62% reported always reading disclosures, and even fewer (41%) reported always using the information when interpreting results. It is unclear exactly why this trend exists, as one would expect the percentages to be more similar. These particular survey questions were formed around using COI disclosures when interpreting study results during academic presentations at national meetings and not during the review of published literature. It is possible that positioning the COI disclosure at the beginning of a presentation has an effect, but only 3.7% of respondents indicated they seldom remembered the disclosure by the end of the presentation. The results of our survey may have varied if the questions had targeted reading and interpreting the literature.

Interestingly, the results of these survey questions tended to be more consistent with rates of reported financial COI by presenters at national orthopedic meetings. A study published in the New England Journal of Medicine found that less than 80% of orthopedic surgeons reported their disclosures at a large annual meeting (AAOS), even when the disclosure involved payments pertinent to the research they were presenting.⁵ When the payments were indirectly related to the research, the percentage of surgeons reporting disclosures was 50%, almost the same as the disclosure rate for unrelated payments.⁵

When the study was changed to a level I randomized controlled trial, more survey respondents found it to be less biased and have more merit. Although it would seem intuitive for a study with a higher level of evidence to carry more clinical value during interpretation, this may not hold true in the setting of industry-sponsored clinical trials. Several studies have documented a significant association between the reporting of positive results and industry-sponsored randomized clinical trials. In 2008, Khan and colleagues³ examined 100 orthopedic randomized clinical trials reported in 5 major orthopedic subspecialty journals over a 2-year period. The association between industry funding and favorable outcome in all original randomized clinical trials was strong and significant (P < .001). This is not surprising, given the amount of time and money required for a well-designed clinical study. Commercial products with preclinical promise are pushed to testing in a clinical trial, whereas resources would not be wasted on products lacking preclinical merit.

The most important variable affecting interpretation of study merit by survey respondents was the reporting of negative results. As more researchers are developing COIs, many studies are discovering a relationship between COIs and outcomes of research studies. Reviewing the adult total joint literature, Ezzet⁸ found an industry funding rate of 50%. Positive results were reported in 93% of cases in commercially funded studies versus 37% of cases in independently funded studies. Furthermore, no negative results were reported by investigators who were receiving royalties from the respective companies.

Studies across the medical literature have also found this association between industry sponsorship and reporting of positive results. One such study, reported by Valachis and colleagues⁷ in the Journal of Clinical Oncology, examined more than 80 economic analyses of targeted oncologic therapies and found the studies funded by pharmaceutical companies were more likely to report favorable qualitative cost estimates. In addition, when studies with a COI disclosure were examined, those reporting any financial relationship with a manufacturer (eg, author affiliation, funding) were more likely than those without such a relationship to report favorable results.

Our study had several limitations. First, as most of the survey respondents were orthopedic surgeons, extrapolating their data to the medical community at large may not be appropriate, as each specialty may view industry affiliations differently. In addition, respondents were asked to base their interpretations of a study on conclusions we predetermined—no direct visualization of the data set or statistical testing methods. It is possible that these responses may have been different had the respondents had the opportunity to further evaluate the study in question. In a recent study, Altwairgi and colleagues¹¹ found that 10% of randomized clinical trials involving lung cancer treatment were reported with different conclusions in their full manuscripts relative to their abstracts. We think our survey design perhaps best mimics an annual meeting environment in which participants have very limited ability to interpret studies and may rely more heavily on the factors we investigated—study design, significance of findings, and setting, all similar to information presented in an abstract—when making informed decisions. Although our response rate was only 70%, this is comparable to or better than the rates in similar survey studies that used email-based questionnaires.^12,13

Another limitation was that our survey may have forced respondents into answers they did not entirely agree with, given the limited options of the multiple-choice response format and the specific wording of the questions. Our conclusions may have been more dramatic when we were evaluating whether the study was deemed meritorious or not. However, there is no adopted standard for evaluating the extent of bias perceived by a clinician. We thought it was important to include answer options indicating a study had merit despite obvious bias in design and execution. That a study had merit can mean different things. It may change clinical practice, may require further study and reproducibility, or may not be significant enough to matter. Asking follow-up questions to evaluate this perception among the respondents could have provided validity to the term merit. Further studies in this field are needed to determine how studies are interpreted and translated into clinical practice by various clinicians.

Conclusion

Although the present study is not a quantitative analysis of the determination of bias in the orthopedic community, it is the first to evaluate orthopedic surgeons’ perceptions on the basis of key fundamentals of orthopedic research relative to COI. It is clear from our study results that introducing levels of evidence to the orthopedic milieu has had a significant impact both on the quality of research and on the foundational use of deductive reasoning when interpreting the literature. Reporting negative outcomes is perhaps the most important factor in eliminating the perception of bias among orthopedic surgeons. To what extent a perceived COI plays into medical decision-making and the ultimate treatment of patients is still relatively unknown.