Kane PW

Patients are learning about health and disease more independently than before, but such self-education may pose a unique challenge for practicing physicians. Although educated patients can assist in the critical appraisal of treatment options,¹ misinformed patients may have preconceived treatment biases and unrealistic expectations. More than 66 million Americans use the Internet daily, and recent surveys have shown 86% have used the Internet for health-related information.^2,3 With Internet use increasing, the number of patients turning to the web for medical information is increasing as well.⁴ For many patients, this information can be useful in making decisions about their health and health care.⁵

Although accessing medical information from the Internet has grown exponentially, analysis of information quality has grown considerably slower.⁶ With no regulatory body monitoring content, there is easy circumvention of the peer review process, an essential feature of academic publishing.⁷ With no external regulation, the information retrieved may be incorrect, outdated, or misleading. Many orthopedic studies have analyzed Internet content about numerous diagnoses.^3-6,8-18 Most of these studies have found this information highly variable and of poor quality.

We conducted a study to evaluate and analyze rotator cuff repair information available to the general public through the Internet; to assess changes in the quality of information over time; to determine if sites sponsored by academic institutions offered higher-quality information; and to assess whether the readability of the material varied according to DISCERN scores.

Rotator cuff repairs are among the most common surgeries performed by orthopedic surgeons. To our knowledge, this is the first study to assess the quality of web information about rotator cuff repairs. We hypothesized that the quality of information would positively correlate with the reading level of the material presented, that academic institutions would present the highest-quality information, and that the type of information presented would change over time.

Materials and Methods

We used the search phrase rotator cuff repair on the 3 most popular search engines: Google, Yahoo!, and Bing. Google is the dominant engine, taking 83.06% of total market share, followed by Yahoo! (6.86%) and Bing (4.27%).5 The first 50 websites identified by each search engine were selected for evaluation, excluding duplicates or overlapping websites. Similarly, advertisements and strictly video results lacking text were excluded. After each engine was queried, a master list of 150 websites was created for individual evaluation and assessment. To assess changes in results over time, we performed 2 searches, on November 16, 2011, and May 18, 2014.

The content of each website was analyzed for authorship, ability to contact the author, discussion of disorder, surgical treatment, complications, surgical eligibility, rehabilitation, other treatment options, and use of peer-reviewed sources. Authorship was placed in 1 of 6 categories:

1. Academic—university-affiliated physician or research group.

2. Private—physician or group without stated affiliation to an academic organization.

3. Industry—manufacturing or marketing company advertising a product or service for profit.

4. News source—bulletin or article without affiliation to a hospital or an academic institution.

5. Public education—individual or organization with noncommercial website providing third-party information (eg, Wikipedia, About.com).

6. Blog—website publishing an individual’s personal experiences in diary or journal form.

Websites were also assessed for accuracy and validity based on presence or absence of Health On the Net code (HONcode) certification and DISCERN score. Designed by the Health On the Net Foundation in 1996, HONcode provides a framework for disseminating high-quality medical information over the web.¹⁹ Website owners can request that their sites be evaluated for HONcode certification; a site that qualifies can display the HONcode seal.²⁰ The DISCERN project, initially funded by the National Health Service in the United Kingdom, judges the quality of written information available on health-related websites.²¹ It determines the quality of a publication on the basis of 16 questions: The first 8 address the publication’s reliability, the next 7 involve specific details of treatment choices, and the last is an overall rating of the website.

Website readability was assessed with the Flesch-Kincaid test. This test, designed under contract with the US Navy in 1975, has been used in other orthopedic studies.¹⁹ Regression analysis was performed to check for correlation between website readability and DISCERN score. Analysis of variance was used to analyze differences between scores.

Results

We performed a comprehensive analysis of the top 50 websites from each of the 3 search engines (N = 150 websites) (Figures 1–5, Table). Regarding authorship, our 2 searches demonstrated similar values (Figure 1). In 2011, 21% of websites were associated with an academic institution, 38% were authored by private physicians or hospital or physician groups not associated with an academic institution, 11.5% were industry-sponsored, 5% were news bulletins or media reports, 21.5% were public education websites, and 3% were personal blogs. Our 2014 search found a similar distribution of contributors. Between 2011 and 2014, the largest change was in academic authors, which decreased by 7%, from 21% to 14%. Percentage of websites authored by private physicians remained constant from the first to the second search: 38%.

When the 2011 and 2014 website content was compared, several changes were noted. Percentage of websites providing an author contact method increased from 21% to 50% (Figure 2), percentage detailing rotator cuff repairs increased from 82% to 91%, and percentage introducing treatment options in addition to surgical management increased from 11.5% to 61%. Percentage discussing surgical eligibility, however, decreased from 43% to 18%. Percentage citing peer-reviewed sources remained relatively constant (28%, 26%), as did percentage discussing surgical technique for rotator cuff repair (55%, 59%) (Figure 3). A major decrease was found in percentage of websites discussing surgical complications, 42% in 2011 down to 25% in 2014, whereas a major increase was found in percentage discussing rehabilitation, from 39% in 2011 up to 73% in 2014. In 2014, no websites discussed double- versus single-row surgery—compared with 6% in 2011. False claims remained low between 2011 and 2014. In both searches, no website guaranteed a return to sport, and few made claims of painless or bloodless surgery.

DISCERN scores for websites found during the 2014 search were averaged for each of the 6 authorship groups (Figure 4). The highest DISCERN scores were given to academic institution websites (51.6) and public education websites (49). For the academic websites, this difference was significant relative to news, blog, and private physician websites (Ps = .012, .001, .001) The lowest DISCERN scores were given to news organization websites and personal blogs. DISCERN scores were 43.8 for industry sources and 40.7 for private physician groups; the difference was not significant (P = .229). Overall mean DISCERN score for all websites was 44. Eleven percent of websites were HONcode-certified.

No correlation was found between website readability and DISCERN score; correlation coefficient r was .01 (Figure 5). For the websites in 2014, mean Flesch-Kincaid readability score was 50.17, and mean grade level was 10.98; coefficient of determination r² was 0.00012.

The Table compares our data with data from other orthopedic studies that have analyzed the quality of Internet information about various orthopedic injuries, diseases, and procedures.^{3-6,8,9,11-18} With its mean DISCERN score of 44, the present rotator cuff tear study was ranked third of 6 studies that have used this scoring system to analyze website content. Of these 6 studies, those reviewing osteosarcoma and juvenile idiopathic arthritis were ranked highest (mean scores, 49.8 and 48.9, respectively), and the study reviewing scoliosis surgery was ranked lowest (38.9). Bruce-Brand and colleagues9 recently found a mean DISCERN score of 41 for anterior cruciate ligament (ACL) reconstruction. When considering HONcode-certified websites, our Internet search for rotator cuff tears found the third lowest percentage, 10.5%, compared with the other studies (Table); the highest percentage, 30%, was found for websites discussing concussions in athletes. When considering authorship, our rotator cuff study found the third highest percentage, 76%, authored by academic centers, physicians, and public education websites; the highest percentage was found in websites discussing ACL reconstruction. Websites discussing ACL reconstruction also had the highest percentage of websites authored by industry.⁹

Discussion

To our knowledge, this is the first study specifically analyzing the quality of Internet information about rotator cuff repairs. A similar study, conducted by Starman and colleagues¹⁵ in 2010, addressed the quality of web information about 10 common sports medicine diagnoses, one of which was rotator cuff tears. In that study, only 16 of the websites included discussed rotator cuff tears. In addition, the authors used a customized, HONcode-based grading system to analyze each website, making their data difficult to compare across studies.

Ideally, a high-quality medical website should be written by a credible source and should cover a disorder, treatment options, eligibility, rehabilitation, and complications. As there is no standard grading system for analyzing web content about rotator cuff repairs, we analyzed the websites for specific information we thought should be included in a high-quality website (Figures 2, 3). When considering authorship, we found academic centers, private physicians, and educational sources comprised 76% of the sources; industry sources made up only 12%. Similar findings were noted by investigators analyzing Internet information about other orthopedic topics, including ACL reconstruction, lumbar arthroplasty, osteosarcoma, and cervical spine surgery.^5,11,22 Studies analyzing websites for information on ACL tears and distal radius fractures found have a higher percentage of industry-sponsored websites.^9,10

DISCERN showed that the highest-quality information came from websites with academic affiliations, consistent with previous studies,^3,9,17 and its mean score (51.6) was significantly higher than the scores for private physician websites, news sites, and blogs (Ps = .001, .016, .001); the least reliable information was from personal blogs and news outlets. Of note, mean DISCERN score was higher for the industry websites we found than for private physician websites (43.8 vs 40.7), though the difference was not significant (P = .229). Previous investigators considered number of industry-sponsored websites as a marker of poor quality of information relating to a given topic; however, given the DISCERN scores in our study, this might not necessarily be true.⁶ Based on the present study’s data, websites affiliated with academic institutions would be recommended for patients searching for high-quality information about rotator cuff tears.

Given DISCERN scores across studies, information about rotator cuff tears ranked below information about osteosarcoma and juvenile idiopathic arthritis but above information about scoliosis, cervical spine surgery, and ACL reconstruction (Table). DISCERN scores must be compared across studies, as there are no definitions for good and poor DISCERN scores.

Of the 4 studies that analyzed percentage of websites citing peer-reviewed sources, only our study and the study of cervical spine surgery¹⁸ analyzed that percentage as well as DISCERN score. Percentage citing peer-reviewed sources was 26% for rotator cuff tears and 24% for cervical spine surgery; the respective DISCERN scores were 44 and 43.6. As only these 2 studies could be compared, no real correlation between percentage of websites citing peer-reviewed sources and the quality of the content on a given topic can be assessed. More research into this relationship is needed. One already delineated association is the correlation between HONcode-certified sites and high DISCERN scores.²¹ For high-quality medical information, physicians can direct their patients both to academic institution websites and to HONcode-certified websites.

When we compared the present study with previous investigations, we found a large difference between search results for a given topic. In 2013, Duncan and colleagues⁶ and Bruce-Brand and colleagues⁹ used similar study designs (eg, search terms, search engines) for their investigations of quality of web information. Their results, however, were widely different. For example, percentages of industry authorship were 4.5% (Duncan and colleagues⁶) and 64% (Bruce-Brand and colleagues⁹). This inconsistency between studies conducted during similar periods might be related to what appears at the top of the results queue for a search. Duncan and colleagues⁶ analyzed 200 websites, Bruce-Brand and colleagues⁹ only 45. Industries may have made financial arrangements and used search engine optimization techniques to have their websites listed first in search results.

In our study, we also analyzed how web information has changed over time. On the Internet, information changes daily, and we hypothesized that the content found during our 2 searches (2011, 2014) would yield different results. Surprisingly, the data were similar, particularly concerning authorship (Figures 1, 2). In both searches, the largest authorship source was private physician or physician groups (38% in 2011 and 2014). Other authorship sources showed little change in percentage between searches. As for content, we found both increases and decreases in specific web information. Ability to contact authors increased from 21% (2011) to 50% (2014). We think it is important that websites offer a communication channel to people who read the medical information the sites provide. Percentage of websites discussing nonoperative treatment options increased from 11.5% to 61%. Therefore, patients in 2014 were being introduced to more options (in addition to surgery) for managing shoulder pain—an improvement in quality of information between the searches. Percentage of websites discussing surgical eligibility, however, decreased from 43% to 18%—a negative development in information quality. Another decrease, from 42% to 25%, was found for websites discussing surgical complications. Given the data as a whole, and our finding both negative and positive changes, it appears the quality of web content has not improved significantly. Interestingly, no websites discussed double- versus single-row surgery in 2014, but 6% did so in 2011.

Lost in the discussion of quality and reliability of information is whether patients comprehend what they are reading.²³ Yi and colleagues¹⁹ recentlyassessed the readability level of arthroscopy information in articles published online by the American Academy of Orthopaedic Surgeons (AAOS) and the Arthroscopy Association of North America (AANA). The investigators used the Flesch-Kincaid readability test to determine readability level in terms of grade level. They found that the majority of the patient education articles on the AAOS and AANA sites had a readability level far above the national average; only 4 articles were written at or below the eighth-grade level, the current average reading level in the United States.²⁴ Information that is not comprehensible is of no use to patients, and information that physicians and researchers consider high-quality might not be what patients consider high-quality. As we pursue higher-quality web content, we need to consider that its audience includes nonmedical readers, our patients. In the present study, we found that the readability of a website had no correlation with the site’s DISCERN score (Figure 5). Therefore, for information about rotator cuff repairs, higher-quality websites are no harder than lower-quality sites for patients to comprehend. The Flesch-Kincaid readability test is flawed in that it considers only total number of syllables per word and words per sentence, not nontextual elements of patient education materials, such as illustrations on a website. The 10.98 mean grade level found in our study is higher than the levels found for most studies reviewed by Yi and colleagues.¹⁹

This study had several limitations. During an Internet search, the number of websites a user visits drops precipitously after the first page of results. Studies have shown the top 20 sites in a given search receive 97% of the views, and the top 3 receive 58.4%. Whether patients visit websites far down in the list of 150 we found in our given search is unknown. Last, the Flesch-Kincaid readability test is flawed in several ways but nevertheless is used extensively in research. Grading is based on number of words and syllables used in a given sentence; it does not take into account the complexity or common usage of a given word or definition. Therefore, websites may receive low Flesch-Kincaid scores—indicating ease of reading—despite their use of complex medical terminology and jargon that complicate patients’ comprehension of the material.

Conclusion

Numerous authors have evaluated orthopedic patients’ accessing of medical information from the Internet. Although the Internet makes access easier, unreliable content can lead patients to develop certain notions about the direction of their care and certain expectations regarding their clinical outcomes. With there being no regulatory body monitoring content, the peer review process, an essential feature of academic publishing, can be easily circumvented.²⁵

In this study, the highest-quality websites had academic affiliations. Quality of information about rotator cuff repairs was similar to what was found for other orthopedic topics in comparable studies. Surprisingly, there was little change in authorship and content of web information between our 2 search periods (2011, 2014). Although there has been a rapid increase in the number of medical websites, quality of content seems not to have changed significantly. Patients look to physicians for guidance but increasingly are accessing the Internet for additional information. It is essential that physicians understand the quality of information available on the Internet when counseling patients regarding surgery.

Patients are learning about health and disease more independently than before, but such self-education may pose a unique challenge for practicing physicians. Although educated patients can assist in the critical appraisal of treatment options,¹ misinformed patients may have preconceived treatment biases and unrealistic expectations. More than 66 million Americans use the Internet daily, and recent surveys have shown 86% have used the Internet for health-related information.^2,3 With Internet use increasing, the number of patients turning to the web for medical information is increasing as well.⁴ For many patients, this information can be useful in making decisions about their health and health care.⁵

Although accessing medical information from the Internet has grown exponentially, analysis of information quality has grown considerably slower.⁶ With no regulatory body monitoring content, there is easy circumvention of the peer review process, an essential feature of academic publishing.⁷ With no external regulation, the information retrieved may be incorrect, outdated, or misleading. Many orthopedic studies have analyzed Internet content about numerous diagnoses.^3-6,8-18 Most of these studies have found this information highly variable and of poor quality.

We conducted a study to evaluate and analyze rotator cuff repair information available to the general public through the Internet; to assess changes in the quality of information over time; to determine if sites sponsored by academic institutions offered higher-quality information; and to assess whether the readability of the material varied according to DISCERN scores.

Rotator cuff repairs are among the most common surgeries performed by orthopedic surgeons. To our knowledge, this is the first study to assess the quality of web information about rotator cuff repairs. We hypothesized that the quality of information would positively correlate with the reading level of the material presented, that academic institutions would present the highest-quality information, and that the type of information presented would change over time.

Materials and Methods

We used the search phrase rotator cuff repair on the 3 most popular search engines: Google, Yahoo!, and Bing. Google is the dominant engine, taking 83.06% of total market share, followed by Yahoo! (6.86%) and Bing (4.27%).5 The first 50 websites identified by each search engine were selected for evaluation, excluding duplicates or overlapping websites. Similarly, advertisements and strictly video results lacking text were excluded. After each engine was queried, a master list of 150 websites was created for individual evaluation and assessment. To assess changes in results over time, we performed 2 searches, on November 16, 2011, and May 18, 2014.

The content of each website was analyzed for authorship, ability to contact the author, discussion of disorder, surgical treatment, complications, surgical eligibility, rehabilitation, other treatment options, and use of peer-reviewed sources. Authorship was placed in 1 of 6 categories:

1. Academic—university-affiliated physician or research group.

2. Private—physician or group without stated affiliation to an academic organization.

3. Industry—manufacturing or marketing company advertising a product or service for profit.

4. News source—bulletin or article without affiliation to a hospital or an academic institution.

5. Public education—individual or organization with noncommercial website providing third-party information (eg, Wikipedia, About.com).

6. Blog—website publishing an individual’s personal experiences in diary or journal form.

Websites were also assessed for accuracy and validity based on presence or absence of Health On the Net code (HONcode) certification and DISCERN score. Designed by the Health On the Net Foundation in 1996, HONcode provides a framework for disseminating high-quality medical information over the web.¹⁹ Website owners can request that their sites be evaluated for HONcode certification; a site that qualifies can display the HONcode seal.²⁰ The DISCERN project, initially funded by the National Health Service in the United Kingdom, judges the quality of written information available on health-related websites.²¹ It determines the quality of a publication on the basis of 16 questions: The first 8 address the publication’s reliability, the next 7 involve specific details of treatment choices, and the last is an overall rating of the website.

Website readability was assessed with the Flesch-Kincaid test. This test, designed under contract with the US Navy in 1975, has been used in other orthopedic studies.¹⁹ Regression analysis was performed to check for correlation between website readability and DISCERN score. Analysis of variance was used to analyze differences between scores.

Results

We performed a comprehensive analysis of the top 50 websites from each of the 3 search engines (N = 150 websites) (Figures 1–5, Table). Regarding authorship, our 2 searches demonstrated similar values (Figure 1). In 2011, 21% of websites were associated with an academic institution, 38% were authored by private physicians or hospital or physician groups not associated with an academic institution, 11.5% were industry-sponsored, 5% were news bulletins or media reports, 21.5% were public education websites, and 3% were personal blogs. Our 2014 search found a similar distribution of contributors. Between 2011 and 2014, the largest change was in academic authors, which decreased by 7%, from 21% to 14%. Percentage of websites authored by private physicians remained constant from the first to the second search: 38%.

When the 2011 and 2014 website content was compared, several changes were noted. Percentage of websites providing an author contact method increased from 21% to 50% (Figure 2), percentage detailing rotator cuff repairs increased from 82% to 91%, and percentage introducing treatment options in addition to surgical management increased from 11.5% to 61%. Percentage discussing surgical eligibility, however, decreased from 43% to 18%. Percentage citing peer-reviewed sources remained relatively constant (28%, 26%), as did percentage discussing surgical technique for rotator cuff repair (55%, 59%) (Figure 3). A major decrease was found in percentage of websites discussing surgical complications, 42% in 2011 down to 25% in 2014, whereas a major increase was found in percentage discussing rehabilitation, from 39% in 2011 up to 73% in 2014. In 2014, no websites discussed double- versus single-row surgery—compared with 6% in 2011. False claims remained low between 2011 and 2014. In both searches, no website guaranteed a return to sport, and few made claims of painless or bloodless surgery.

DISCERN scores for websites found during the 2014 search were averaged for each of the 6 authorship groups (Figure 4). The highest DISCERN scores were given to academic institution websites (51.6) and public education websites (49). For the academic websites, this difference was significant relative to news, blog, and private physician websites (Ps = .012, .001, .001) The lowest DISCERN scores were given to news organization websites and personal blogs. DISCERN scores were 43.8 for industry sources and 40.7 for private physician groups; the difference was not significant (P = .229). Overall mean DISCERN score for all websites was 44. Eleven percent of websites were HONcode-certified.

No correlation was found between website readability and DISCERN score; correlation coefficient r was .01 (Figure 5). For the websites in 2014, mean Flesch-Kincaid readability score was 50.17, and mean grade level was 10.98; coefficient of determination r² was 0.00012.

The Table compares our data with data from other orthopedic studies that have analyzed the quality of Internet information about various orthopedic injuries, diseases, and procedures.^{3-6,8,9,11-18} With its mean DISCERN score of 44, the present rotator cuff tear study was ranked third of 6 studies that have used this scoring system to analyze website content. Of these 6 studies, those reviewing osteosarcoma and juvenile idiopathic arthritis were ranked highest (mean scores, 49.8 and 48.9, respectively), and the study reviewing scoliosis surgery was ranked lowest (38.9). Bruce-Brand and colleagues9 recently found a mean DISCERN score of 41 for anterior cruciate ligament (ACL) reconstruction. When considering HONcode-certified websites, our Internet search for rotator cuff tears found the third lowest percentage, 10.5%, compared with the other studies (Table); the highest percentage, 30%, was found for websites discussing concussions in athletes. When considering authorship, our rotator cuff study found the third highest percentage, 76%, authored by academic centers, physicians, and public education websites; the highest percentage was found in websites discussing ACL reconstruction. Websites discussing ACL reconstruction also had the highest percentage of websites authored by industry.⁹

Discussion

To our knowledge, this is the first study specifically analyzing the quality of Internet information about rotator cuff repairs. A similar study, conducted by Starman and colleagues¹⁵ in 2010, addressed the quality of web information about 10 common sports medicine diagnoses, one of which was rotator cuff tears. In that study, only 16 of the websites included discussed rotator cuff tears. In addition, the authors used a customized, HONcode-based grading system to analyze each website, making their data difficult to compare across studies.

Ideally, a high-quality medical website should be written by a credible source and should cover a disorder, treatment options, eligibility, rehabilitation, and complications. As there is no standard grading system for analyzing web content about rotator cuff repairs, we analyzed the websites for specific information we thought should be included in a high-quality website (Figures 2, 3). When considering authorship, we found academic centers, private physicians, and educational sources comprised 76% of the sources; industry sources made up only 12%. Similar findings were noted by investigators analyzing Internet information about other orthopedic topics, including ACL reconstruction, lumbar arthroplasty, osteosarcoma, and cervical spine surgery.^5,11,22 Studies analyzing websites for information on ACL tears and distal radius fractures found have a higher percentage of industry-sponsored websites.^9,10

DISCERN showed that the highest-quality information came from websites with academic affiliations, consistent with previous studies,^3,9,17 and its mean score (51.6) was significantly higher than the scores for private physician websites, news sites, and blogs (Ps = .001, .016, .001); the least reliable information was from personal blogs and news outlets. Of note, mean DISCERN score was higher for the industry websites we found than for private physician websites (43.8 vs 40.7), though the difference was not significant (P = .229). Previous investigators considered number of industry-sponsored websites as a marker of poor quality of information relating to a given topic; however, given the DISCERN scores in our study, this might not necessarily be true.⁶ Based on the present study’s data, websites affiliated with academic institutions would be recommended for patients searching for high-quality information about rotator cuff tears.

Given DISCERN scores across studies, information about rotator cuff tears ranked below information about osteosarcoma and juvenile idiopathic arthritis but above information about scoliosis, cervical spine surgery, and ACL reconstruction (Table). DISCERN scores must be compared across studies, as there are no definitions for good and poor DISCERN scores.

Of the 4 studies that analyzed percentage of websites citing peer-reviewed sources, only our study and the study of cervical spine surgery¹⁸ analyzed that percentage as well as DISCERN score. Percentage citing peer-reviewed sources was 26% for rotator cuff tears and 24% for cervical spine surgery; the respective DISCERN scores were 44 and 43.6. As only these 2 studies could be compared, no real correlation between percentage of websites citing peer-reviewed sources and the quality of the content on a given topic can be assessed. More research into this relationship is needed. One already delineated association is the correlation between HONcode-certified sites and high DISCERN scores.²¹ For high-quality medical information, physicians can direct their patients both to academic institution websites and to HONcode-certified websites.

When we compared the present study with previous investigations, we found a large difference between search results for a given topic. In 2013, Duncan and colleagues⁶ and Bruce-Brand and colleagues⁹ used similar study designs (eg, search terms, search engines) for their investigations of quality of web information. Their results, however, were widely different. For example, percentages of industry authorship were 4.5% (Duncan and colleagues⁶) and 64% (Bruce-Brand and colleagues⁹). This inconsistency between studies conducted during similar periods might be related to what appears at the top of the results queue for a search. Duncan and colleagues⁶ analyzed 200 websites, Bruce-Brand and colleagues⁹ only 45. Industries may have made financial arrangements and used search engine optimization techniques to have their websites listed first in search results.

In our study, we also analyzed how web information has changed over time. On the Internet, information changes daily, and we hypothesized that the content found during our 2 searches (2011, 2014) would yield different results. Surprisingly, the data were similar, particularly concerning authorship (Figures 1, 2). In both searches, the largest authorship source was private physician or physician groups (38% in 2011 and 2014). Other authorship sources showed little change in percentage between searches. As for content, we found both increases and decreases in specific web information. Ability to contact authors increased from 21% (2011) to 50% (2014). We think it is important that websites offer a communication channel to people who read the medical information the sites provide. Percentage of websites discussing nonoperative treatment options increased from 11.5% to 61%. Therefore, patients in 2014 were being introduced to more options (in addition to surgery) for managing shoulder pain—an improvement in quality of information between the searches. Percentage of websites discussing surgical eligibility, however, decreased from 43% to 18%—a negative development in information quality. Another decrease, from 42% to 25%, was found for websites discussing surgical complications. Given the data as a whole, and our finding both negative and positive changes, it appears the quality of web content has not improved significantly. Interestingly, no websites discussed double- versus single-row surgery in 2014, but 6% did so in 2011.

Lost in the discussion of quality and reliability of information is whether patients comprehend what they are reading.²³ Yi and colleagues¹⁹ recentlyassessed the readability level of arthroscopy information in articles published online by the American Academy of Orthopaedic Surgeons (AAOS) and the Arthroscopy Association of North America (AANA). The investigators used the Flesch-Kincaid readability test to determine readability level in terms of grade level. They found that the majority of the patient education articles on the AAOS and AANA sites had a readability level far above the national average; only 4 articles were written at or below the eighth-grade level, the current average reading level in the United States.²⁴ Information that is not comprehensible is of no use to patients, and information that physicians and researchers consider high-quality might not be what patients consider high-quality. As we pursue higher-quality web content, we need to consider that its audience includes nonmedical readers, our patients. In the present study, we found that the readability of a website had no correlation with the site’s DISCERN score (Figure 5). Therefore, for information about rotator cuff repairs, higher-quality websites are no harder than lower-quality sites for patients to comprehend. The Flesch-Kincaid readability test is flawed in that it considers only total number of syllables per word and words per sentence, not nontextual elements of patient education materials, such as illustrations on a website. The 10.98 mean grade level found in our study is higher than the levels found for most studies reviewed by Yi and colleagues.¹⁹

This study had several limitations. During an Internet search, the number of websites a user visits drops precipitously after the first page of results. Studies have shown the top 20 sites in a given search receive 97% of the views, and the top 3 receive 58.4%. Whether patients visit websites far down in the list of 150 we found in our given search is unknown. Last, the Flesch-Kincaid readability test is flawed in several ways but nevertheless is used extensively in research. Grading is based on number of words and syllables used in a given sentence; it does not take into account the complexity or common usage of a given word or definition. Therefore, websites may receive low Flesch-Kincaid scores—indicating ease of reading—despite their use of complex medical terminology and jargon that complicate patients’ comprehension of the material.

Conclusion

Numerous authors have evaluated orthopedic patients’ accessing of medical information from the Internet. Although the Internet makes access easier, unreliable content can lead patients to develop certain notions about the direction of their care and certain expectations regarding their clinical outcomes. With there being no regulatory body monitoring content, the peer review process, an essential feature of academic publishing, can be easily circumvented.²⁵

In this study, the highest-quality websites had academic affiliations. Quality of information about rotator cuff repairs was similar to what was found for other orthopedic topics in comparable studies. Surprisingly, there was little change in authorship and content of web information between our 2 search periods (2011, 2014). Although there has been a rapid increase in the number of medical websites, quality of content seems not to have changed significantly. Patients look to physicians for guidance but increasingly are accessing the Internet for additional information. It is essential that physicians understand the quality of information available on the Internet when counseling patients regarding surgery.

Patients are learning about health and disease more independently than before, but such self-education may pose a unique challenge for practicing physicians. Although educated patients can assist in the critical appraisal of treatment options,¹ misinformed patients may have preconceived treatment biases and unrealistic expectations. More than 66 million Americans use the Internet daily, and recent surveys have shown 86% have used the Internet for health-related information.^2,3 With Internet use increasing, the number of patients turning to the web for medical information is increasing as well.⁴ For many patients, this information can be useful in making decisions about their health and health care.⁵

Although accessing medical information from the Internet has grown exponentially, analysis of information quality has grown considerably slower.⁶ With no regulatory body monitoring content, there is easy circumvention of the peer review process, an essential feature of academic publishing.⁷ With no external regulation, the information retrieved may be incorrect, outdated, or misleading. Many orthopedic studies have analyzed Internet content about numerous diagnoses.^3-6,8-18 Most of these studies have found this information highly variable and of poor quality.

We conducted a study to evaluate and analyze rotator cuff repair information available to the general public through the Internet; to assess changes in the quality of information over time; to determine if sites sponsored by academic institutions offered higher-quality information; and to assess whether the readability of the material varied according to DISCERN scores.

Rotator cuff repairs are among the most common surgeries performed by orthopedic surgeons. To our knowledge, this is the first study to assess the quality of web information about rotator cuff repairs. We hypothesized that the quality of information would positively correlate with the reading level of the material presented, that academic institutions would present the highest-quality information, and that the type of information presented would change over time.

Materials and Methods

We used the search phrase rotator cuff repair on the 3 most popular search engines: Google, Yahoo!, and Bing. Google is the dominant engine, taking 83.06% of total market share, followed by Yahoo! (6.86%) and Bing (4.27%).5 The first 50 websites identified by each search engine were selected for evaluation, excluding duplicates or overlapping websites. Similarly, advertisements and strictly video results lacking text were excluded. After each engine was queried, a master list of 150 websites was created for individual evaluation and assessment. To assess changes in results over time, we performed 2 searches, on November 16, 2011, and May 18, 2014.

The content of each website was analyzed for authorship, ability to contact the author, discussion of disorder, surgical treatment, complications, surgical eligibility, rehabilitation, other treatment options, and use of peer-reviewed sources. Authorship was placed in 1 of 6 categories:

1. Academic—university-affiliated physician or research group.

2. Private—physician or group without stated affiliation to an academic organization.

3. Industry—manufacturing or marketing company advertising a product or service for profit.

4. News source—bulletin or article without affiliation to a hospital or an academic institution.

5. Public education—individual or organization with noncommercial website providing third-party information (eg, Wikipedia, About.com).

6. Blog—website publishing an individual’s personal experiences in diary or journal form.

Websites were also assessed for accuracy and validity based on presence or absence of Health On the Net code (HONcode) certification and DISCERN score. Designed by the Health On the Net Foundation in 1996, HONcode provides a framework for disseminating high-quality medical information over the web.¹⁹ Website owners can request that their sites be evaluated for HONcode certification; a site that qualifies can display the HONcode seal.²⁰ The DISCERN project, initially funded by the National Health Service in the United Kingdom, judges the quality of written information available on health-related websites.²¹ It determines the quality of a publication on the basis of 16 questions: The first 8 address the publication’s reliability, the next 7 involve specific details of treatment choices, and the last is an overall rating of the website.

Website readability was assessed with the Flesch-Kincaid test. This test, designed under contract with the US Navy in 1975, has been used in other orthopedic studies.¹⁹ Regression analysis was performed to check for correlation between website readability and DISCERN score. Analysis of variance was used to analyze differences between scores.

Results

We performed a comprehensive analysis of the top 50 websites from each of the 3 search engines (N = 150 websites) (Figures 1–5, Table). Regarding authorship, our 2 searches demonstrated similar values (Figure 1). In 2011, 21% of websites were associated with an academic institution, 38% were authored by private physicians or hospital or physician groups not associated with an academic institution, 11.5% were industry-sponsored, 5% were news bulletins or media reports, 21.5% were public education websites, and 3% were personal blogs. Our 2014 search found a similar distribution of contributors. Between 2011 and 2014, the largest change was in academic authors, which decreased by 7%, from 21% to 14%. Percentage of websites authored by private physicians remained constant from the first to the second search: 38%.

When the 2011 and 2014 website content was compared, several changes were noted. Percentage of websites providing an author contact method increased from 21% to 50% (Figure 2), percentage detailing rotator cuff repairs increased from 82% to 91%, and percentage introducing treatment options in addition to surgical management increased from 11.5% to 61%. Percentage discussing surgical eligibility, however, decreased from 43% to 18%. Percentage citing peer-reviewed sources remained relatively constant (28%, 26%), as did percentage discussing surgical technique for rotator cuff repair (55%, 59%) (Figure 3). A major decrease was found in percentage of websites discussing surgical complications, 42% in 2011 down to 25% in 2014, whereas a major increase was found in percentage discussing rehabilitation, from 39% in 2011 up to 73% in 2014. In 2014, no websites discussed double- versus single-row surgery—compared with 6% in 2011. False claims remained low between 2011 and 2014. In both searches, no website guaranteed a return to sport, and few made claims of painless or bloodless surgery.

DISCERN scores for websites found during the 2014 search were averaged for each of the 6 authorship groups (Figure 4). The highest DISCERN scores were given to academic institution websites (51.6) and public education websites (49). For the academic websites, this difference was significant relative to news, blog, and private physician websites (Ps = .012, .001, .001) The lowest DISCERN scores were given to news organization websites and personal blogs. DISCERN scores were 43.8 for industry sources and 40.7 for private physician groups; the difference was not significant (P = .229). Overall mean DISCERN score for all websites was 44. Eleven percent of websites were HONcode-certified.

No correlation was found between website readability and DISCERN score; correlation coefficient r was .01 (Figure 5). For the websites in 2014, mean Flesch-Kincaid readability score was 50.17, and mean grade level was 10.98; coefficient of determination r² was 0.00012.

The Table compares our data with data from other orthopedic studies that have analyzed the quality of Internet information about various orthopedic injuries, diseases, and procedures.^{3-6,8,9,11-18} With its mean DISCERN score of 44, the present rotator cuff tear study was ranked third of 6 studies that have used this scoring system to analyze website content. Of these 6 studies, those reviewing osteosarcoma and juvenile idiopathic arthritis were ranked highest (mean scores, 49.8 and 48.9, respectively), and the study reviewing scoliosis surgery was ranked lowest (38.9). Bruce-Brand and colleagues9 recently found a mean DISCERN score of 41 for anterior cruciate ligament (ACL) reconstruction. When considering HONcode-certified websites, our Internet search for rotator cuff tears found the third lowest percentage, 10.5%, compared with the other studies (Table); the highest percentage, 30%, was found for websites discussing concussions in athletes. When considering authorship, our rotator cuff study found the third highest percentage, 76%, authored by academic centers, physicians, and public education websites; the highest percentage was found in websites discussing ACL reconstruction. Websites discussing ACL reconstruction also had the highest percentage of websites authored by industry.⁹

Discussion

To our knowledge, this is the first study specifically analyzing the quality of Internet information about rotator cuff repairs. A similar study, conducted by Starman and colleagues¹⁵ in 2010, addressed the quality of web information about 10 common sports medicine diagnoses, one of which was rotator cuff tears. In that study, only 16 of the websites included discussed rotator cuff tears. In addition, the authors used a customized, HONcode-based grading system to analyze each website, making their data difficult to compare across studies.

Ideally, a high-quality medical website should be written by a credible source and should cover a disorder, treatment options, eligibility, rehabilitation, and complications. As there is no standard grading system for analyzing web content about rotator cuff repairs, we analyzed the websites for specific information we thought should be included in a high-quality website (Figures 2, 3). When considering authorship, we found academic centers, private physicians, and educational sources comprised 76% of the sources; industry sources made up only 12%. Similar findings were noted by investigators analyzing Internet information about other orthopedic topics, including ACL reconstruction, lumbar arthroplasty, osteosarcoma, and cervical spine surgery.^5,11,22 Studies analyzing websites for information on ACL tears and distal radius fractures found have a higher percentage of industry-sponsored websites.^9,10

DISCERN showed that the highest-quality information came from websites with academic affiliations, consistent with previous studies,^3,9,17 and its mean score (51.6) was significantly higher than the scores for private physician websites, news sites, and blogs (Ps = .001, .016, .001); the least reliable information was from personal blogs and news outlets. Of note, mean DISCERN score was higher for the industry websites we found than for private physician websites (43.8 vs 40.7), though the difference was not significant (P = .229). Previous investigators considered number of industry-sponsored websites as a marker of poor quality of information relating to a given topic; however, given the DISCERN scores in our study, this might not necessarily be true.⁶ Based on the present study’s data, websites affiliated with academic institutions would be recommended for patients searching for high-quality information about rotator cuff tears.

Given DISCERN scores across studies, information about rotator cuff tears ranked below information about osteosarcoma and juvenile idiopathic arthritis but above information about scoliosis, cervical spine surgery, and ACL reconstruction (Table). DISCERN scores must be compared across studies, as there are no definitions for good and poor DISCERN scores.

Of the 4 studies that analyzed percentage of websites citing peer-reviewed sources, only our study and the study of cervical spine surgery¹⁸ analyzed that percentage as well as DISCERN score. Percentage citing peer-reviewed sources was 26% for rotator cuff tears and 24% for cervical spine surgery; the respective DISCERN scores were 44 and 43.6. As only these 2 studies could be compared, no real correlation between percentage of websites citing peer-reviewed sources and the quality of the content on a given topic can be assessed. More research into this relationship is needed. One already delineated association is the correlation between HONcode-certified sites and high DISCERN scores.²¹ For high-quality medical information, physicians can direct their patients both to academic institution websites and to HONcode-certified websites.

When we compared the present study with previous investigations, we found a large difference between search results for a given topic. In 2013, Duncan and colleagues⁶ and Bruce-Brand and colleagues⁹ used similar study designs (eg, search terms, search engines) for their investigations of quality of web information. Their results, however, were widely different. For example, percentages of industry authorship were 4.5% (Duncan and colleagues⁶) and 64% (Bruce-Brand and colleagues⁹). This inconsistency between studies conducted during similar periods might be related to what appears at the top of the results queue for a search. Duncan and colleagues⁶ analyzed 200 websites, Bruce-Brand and colleagues⁹ only 45. Industries may have made financial arrangements and used search engine optimization techniques to have their websites listed first in search results.

In our study, we also analyzed how web information has changed over time. On the Internet, information changes daily, and we hypothesized that the content found during our 2 searches (2011, 2014) would yield different results. Surprisingly, the data were similar, particularly concerning authorship (Figures 1, 2). In both searches, the largest authorship source was private physician or physician groups (38% in 2011 and 2014). Other authorship sources showed little change in percentage between searches. As for content, we found both increases and decreases in specific web information. Ability to contact authors increased from 21% (2011) to 50% (2014). We think it is important that websites offer a communication channel to people who read the medical information the sites provide. Percentage of websites discussing nonoperative treatment options increased from 11.5% to 61%. Therefore, patients in 2014 were being introduced to more options (in addition to surgery) for managing shoulder pain—an improvement in quality of information between the searches. Percentage of websites discussing surgical eligibility, however, decreased from 43% to 18%—a negative development in information quality. Another decrease, from 42% to 25%, was found for websites discussing surgical complications. Given the data as a whole, and our finding both negative and positive changes, it appears the quality of web content has not improved significantly. Interestingly, no websites discussed double- versus single-row surgery in 2014, but 6% did so in 2011.

Lost in the discussion of quality and reliability of information is whether patients comprehend what they are reading.²³ Yi and colleagues¹⁹ recentlyassessed the readability level of arthroscopy information in articles published online by the American Academy of Orthopaedic Surgeons (AAOS) and the Arthroscopy Association of North America (AANA). The investigators used the Flesch-Kincaid readability test to determine readability level in terms of grade level. They found that the majority of the patient education articles on the AAOS and AANA sites had a readability level far above the national average; only 4 articles were written at or below the eighth-grade level, the current average reading level in the United States.²⁴ Information that is not comprehensible is of no use to patients, and information that physicians and researchers consider high-quality might not be what patients consider high-quality. As we pursue higher-quality web content, we need to consider that its audience includes nonmedical readers, our patients. In the present study, we found that the readability of a website had no correlation with the site’s DISCERN score (Figure 5). Therefore, for information about rotator cuff repairs, higher-quality websites are no harder than lower-quality sites for patients to comprehend. The Flesch-Kincaid readability test is flawed in that it considers only total number of syllables per word and words per sentence, not nontextual elements of patient education materials, such as illustrations on a website. The 10.98 mean grade level found in our study is higher than the levels found for most studies reviewed by Yi and colleagues.¹⁹

This study had several limitations. During an Internet search, the number of websites a user visits drops precipitously after the first page of results. Studies have shown the top 20 sites in a given search receive 97% of the views, and the top 3 receive 58.4%. Whether patients visit websites far down in the list of 150 we found in our given search is unknown. Last, the Flesch-Kincaid readability test is flawed in several ways but nevertheless is used extensively in research. Grading is based on number of words and syllables used in a given sentence; it does not take into account the complexity or common usage of a given word or definition. Therefore, websites may receive low Flesch-Kincaid scores—indicating ease of reading—despite their use of complex medical terminology and jargon that complicate patients’ comprehension of the material.

Conclusion

Numerous authors have evaluated orthopedic patients’ accessing of medical information from the Internet. Although the Internet makes access easier, unreliable content can lead patients to develop certain notions about the direction of their care and certain expectations regarding their clinical outcomes. With there being no regulatory body monitoring content, the peer review process, an essential feature of academic publishing, can be easily circumvented.²⁵

In this study, the highest-quality websites had academic affiliations. Quality of information about rotator cuff repairs was similar to what was found for other orthopedic topics in comparable studies. Surprisingly, there was little change in authorship and content of web information between our 2 search periods (2011, 2014). Although there has been a rapid increase in the number of medical websites, quality of content seems not to have changed significantly. Patients look to physicians for guidance but increasingly are accessing the Internet for additional information. It is essential that physicians understand the quality of information available on the Internet when counseling patients regarding surgery.