Assessing the Reading Level of Online Sarcoma Patient Education Materials

Text from these PEMs were copied and pasted into separate Microsoft Word documents (Microsoft, Redmond, Washington). Advertisements, pictures, picture text, hyperlinks, copyright notices, page navigation links, paragraphs with no text, and any text that was not related to the given condition were deleted from the document to format the text for the readability software. Then, each Microsoft Word document was uploaded into the software package Readability Studio Professional (RSP) Edition Version 2012.1 for Windows (Oleander Software, Vandalia, Ohio). The 10 distinct readability instruments that were used to gauge the readability of each document were the Flesch Reading Ease score (FRE), the New Fog Count, the New Automated Readability Index, the Coleman-Liau Index (CLI), the Fry readability graph, the New Dale-Chall formula (NDC), the Gunning Frequency of Gobbledygook (Gunning FOG), the Powers-Sumner-Kearl formula, the Simple Measure of Gobbledygook (SMOG), and the Raygor Estimate Graph.

The FRE’s formula takes the average number of words per sentence and average number of syllables per word to compute a score ranging from 0 to 100 with 0 being the hardest to read.³⁶ The New Fog Count tallies the number of sentences, easy words, and hard words (polysyllables) to calculate the grade level of the document.³⁷ The New Automated Readability Index takes the average characters per word and average words per sentence to calculate a grade level for the document.³⁷ The CLI randomly samples a few hundred words from the document, averages the number of letters and sentences per sample, and calculates an estimated grade level.³⁸ The Fry readability graph selects samples of 100 words from the document, averages the number of syllables and sentences per 100 words, plots these data points on a graph, with the intersection determining the reading level.³⁹ The NDC uses a list of 3000 familiar words that most fourth-grade students know.⁴⁰ The percentage of difficult words, which are not on the list of familiar words, and the average sentence length in words are used to calculate the reading grade level of the document. The Gunning FOG uses the average sentence length in words and the percentage of hard words from a sample of at least 100 words to determine the reading grade level of the document.⁴¹ The Powers-Sumner-Kearl formula uses the average sentence length and percentage of monosyllables from a 100-word sample passage to calculate the reading grade level.⁴² The SMOG formula counts the number of polysyllabic words from 30 sentences and calculates the reading grade level of the document.⁴³ In contrast to other formulas that test for 50% to 75% comprehension, the SMOG formula tests for 100% comprehension. As a result, the SMOG formula generally assigns a reading level 2 grades higher than the Dale-Chall level. The Raygor Estimate Graph selects a 100-word passage, counts the number of sentences and number of words with 6 or more letters, and plots the 2 variables on a graph to determine the reading grade level.⁴⁴ The software package calculated the results from each reading instrument and reported the mean grade level score
for each document.

Results

We identified a total of 72 websites with relevant PEMs and included them in this study. Of these 72 websites, 36 websites were academic training centers, 10 were Google search hits, and 21 were from the Sarcoma Alliance list of sarcoma specialists. The remaining 5 websites were AAOS, Bonetumor.org, Sarcoma Alliance, Sarcoma Foundation of America, and Medscape. A list of conditions and treatments that were considered relevant PEMs is found in Appendix 1. A total of 774 articles were obtained from the 72 websites.

None of the websites had a mean readability score of 7 (seventh grade) or lower (Figures 1A, 1B). Mid-America Sarcoma Institute’s PEMs had the lowest mean readability score, 8.9. The lowest readability score was 5.3, which the New Fog Count readability instrument calculated for Vanderbilt University Medical Center’s (VUMC’s) PEMs (Appendix 2). The mean readability score of all websites was 11.4 (range, 8.9-15.5) (Appendix 2).

Seventy of 72 websites (97%) had PEMs that were fairly difficult or difficult, according to the FRE analysis (Figure 2). The American Cancer Society and Mid-America Sarcoma Institute had PEMs that were written in plain English. Sixty-nine of 72 websites (96%) had PEMs with a readability score of 10 or higher, according to the Raygor readability estimate (Figure 3). Using this instrument, the scores of the American Cancer Society and the University of Pennsylvania–Joan Karnell Cancer Center were 9; Mid-America Sarcoma Institute’s score was 8.

Discussion

Many cancer patients have turned to websites and online PEMs to gather health information about their condition.^10-17 Basch and colleagues¹⁰ reported almost a decade ago that 44% of cancer patients, as well as 60% of their companions, used the Internet to find cancer-related information.¹⁰ When LaCoursiere and colleagues³⁵ surveyed cancer patients, they found that patients handled their condition better and had less anxiety and uncertainty after using the Internet to find health information and support.³⁵ In addition, many orthopedic patients, specifically 46% of orthopedic community outpatients,⁴⁵ consult the Internet for information about their condition and future surgical procedures.^46,47