Abstract
The Internet is becoming an increasingly integral part in the delivery of health information in the 21st century. Approximately three of four Americans have Internet access at home, and this number is increasing. 1 A 2006 survey by the Pew Internet & American Life Project found that eight out of 10 American Internet users search for online health information (OHI). Two thirds of those searching for OHI start at a generalized search engine, whereas 27 percent initiate their search at a health-related website. The survey also demonstrated that most people seeking OHI are pleased about what they find, but some are frustrated or confused. 2 Given the high percentage of the public using the Internet to obtain medical information, it is important for patient-oriented health information to be easily understood.
Health literacy refers to the “degree to which individuals have the capacity to obtain, process and understand health information and services needed to make appropriate health decisions.” 3 In the 2003 National Assessment of Adult Literacy (NAAL), which examined the relationships between demographics and literacy, the majority of adults (53%) showed an intermediate level of health literacy. Twenty percent of adults with below basic health literacy got information about health issues from the Internet. 4 A general trend toward higher health literacy exists with each higher level of educational attainment. As previously described, educational level does not correlate with reading-grade level. With this in mind, it is important to develop educational materials appropriate for most patients.
Readability is defined as the “comprehension level a person must have to understand written material.” 5 Many methods have been developed to calculate readability. 6 The Flesch-Kincaid (FK) formula has been validated, is readily available, and is easy to apply. 7 The FK grade level formula is as follows: (0.39 × average number of words per sentence) + (11.8 × average number of syllables per word) − 15.59. Calculation of this formula for a discrete sample of text gives a U.S. grade level corresponding to that which an individual with that level of education should be able to read and understand. FK formula results above 12 refer to post-secondary school reading levels.
The average reading comprehension in the United States is at the eighth grade level. 8 To increase the percentage of individuals that can understand textual health, information experts have proposed it be presented at a sixth grade reading level. 9 - 11 Notably, the completed academic year often does not correlate with reading level. In fact, the reading level of patients tested at a public hospital averaged four grades below their reported completed grade level. 12
The patient section of the AAO–HNS website (www.entnet.org/healthinformation) provides information for patients to learn about “the health of the ear, nose, and throat.” Based on several studies, 13 - 15 the OHI provided by the academic societies of a number of medical specialties is written at a level higher than that recommended. In our review of the literature, there is no evaluation of the readability of the patient-oriented OHI presented on the AAO–HNS website. This study evaluates the readability of this material. We also demonstrate the reproducibility and ease of use of the FK formula to assess readability of written material.
Methods
This study was exempted from institutional review board review because the educational material on the website is publicly accessible and does not involve patient records. The patient information section of the AAO–HNS website was accessed on January 19, 2009, using a personal computer with commercially available Internet access.
The patient information section has six links with the following headings: “Ears,” “Throat,” “Nose and Mouth,” “Head and Neck,” “Cancer,” and “Pediatric.” Each section contains written information on various topics pertaining to each heading. Only entries related to patient education about diagnosis and treatment of diseases and disorders of the ears, nose, throat, and head and neck were included. A total of 108 unique entries were identified. Four articles were excluded from analysis: two with information in list form, one with an interactive diagram, and one written in Spanish. This left 104 entries for readability analysis.
The text content of each article meeting inclusion criteria was copied and pasted into a Microsoft Word (Microsoft Corp, Redmond, WA) document. Any text containing links to other websites, copyright information, pictures, navigation tools, or citations were omitted. The FK grade level for each article was determined using the readability calculator available within Microsoft Office Word 2003. To obtain the FK grade level from a Microsoft Office Word document, this function needs to be enabled by sequentially selecting “Tools,” “Options,” “Spelling & Grammar,” choosing the option “Show readability statistics,” then clicking “OK.” The result for the FK grade level is displayed by selecting the “Spelling & Grammar” tool or pressing “F7.”
The FK grade level was calculated using the above protocol for all articles meeting inclusion criteria by one member of the study team. The average grade level for each section was also calculated. The senior author (M.S.B.) also calculated the FK grade level for 52 entries to assess reproducibility of the analysis. The interobserver reliability for the FK grade was determined by calculating the intraclass correlation coefficient (ICC) using the VassarStats Statistical Computation website (http://faculty.vassar.edu/lowry/VassarStats.html). An intraclass correlation coefficient of 0 to 0.24 reflects poor correlation; 0.25 to 0.49, low; 0.50 to 0.69, fair; 0.7 to 0.89, good; and 0.9 to 1.0, excellent. 16
Results
A total of 104 articles were evaluated with the FK grade level formula. The average FK grade reading level of the articles was 10.8 (range 6.3-16.7; 95% CI, 10.4-11.2). Only two articles (1.9%) were written at the recommended sixth grade level. Eighty-one percent of the articles were written at a ninth grade level or higher (Fig 1). Each section of the website had an FK grade level above the tenth grade, with the highest level being the “Pediatric” section, at 11.6. The lowest grade level was the section on “Ears,” at 10.3. The “Head and Neck” section was 10.6, the “Throat” section was 11.1, and both “Nose and Mouth” and “Cancer” were 11.1 (Table 1). The intraclass correlation was good (r = 0.83) for the 52 articles that were independently reviewed.
Flesch-Kincaid grade level for each section of the American Academy of Otolaryngology–Head and Neck Surgery website
Distribution of FK grade levels for 104 articles on the AAO–HNS website.
Discussion
Many health-related websites are written at a high school or higher reading level. 13 , 14 , 17 , 18 In our study, the average grade reading level of patient education articles on the AAO–HNS website as calculated by the FK formula was 10.8. This was over four grade levels higher than the recommended sixth grade reading level. Only two articles were written at the recommended sixth grade level. Eighty-one percent of the articles were written above the eighth grade level.
To ensure that our analysis of the written material was reliable, two authors independently analyzed a set of the website articles using the aforementioned technique. In our study, there was good correlation (r = 0.83) between the authors' analyses as determined by the intraclass coefficient. Our protocol is similar to previously published studies; however, in these studies, only a sample of the article was used to determine the grade level. In our study, we utilized the entire passage. One limitation to this technique is that not all articles were written in the same format. For example, some articles contained some information in list form or with sentence fragments, which was not edited from the pasted text. This would artificially lower the readability grade level because there would be shorter sentences.
Readability analyses of oncology and orthopedic surgery OHI have shown similar results. 13 , 14 , 19 , 20 In a study by Sabharwal et al, only two percent of the articles on the American Academy of Orthopedic Surgeons patient education website had the recommended readability level of sixth grade or lower. Moreover, they investigated whether future revisions of the website demonstrated a change in the readability of the articles. They found that over time, with further revisions in the website material, there was no correlation between the FK grade level and when the article was written (r = 0.0003). 14
Numerous formulae exist for the calculation of text readability. The Flesch Reading Ease Formula, Simple Measure of Gobbledygook (SMOG) index, and FORCAST formula (J. Patrick
Readability formulae such as the FK readability formula have several limitations. First, these formulae assume a linear relationship between increased word and sentence length and increased reading ability requirements. 3 , 7 This may not be an accurate assessment of a passage because many words in otolaryngology would be considered difficult but consist of only a few syllables. Shorter sentences are considered easier to read, but they may not be grammatically correct, or they may contain difficult vocabulary. Second, readability formulae do not take into account the previous knowledge a person may have about a particular topic. For example, the word “cochlea” consists of only three syllables and as such may be considered an easy word, yet to someone not familiar with medical terminology, the ability to comprehend an article utilizing this word may be diminished. Last, readability formulae cannot measure text structure and do not account for word order, both of which factor into patient understanding. In the case of the computer-calculated FK formula, colons and semicolons are counted as the end of a sentence. 3 Thus, a passage with several compound sentences containing these punctuation marks may be evaluated as easier to read than, for example, a passage with short sentences and no colons or semicolons.
This analysis has shown that the average reading level for each article was higher than the recommended sixth grade reading level. Changes to the AAO–HNS website that may improve patient education include links to related topics, fact sheets summarizing important points, and use of illustrations on introductory web pages. Additionally, the patient section is written by “member physicians” of the AAO–HNS and is updated every three years. This suggests, but does not guarantee, current and relevant information from qualified individuals.
Although the AAO–HNS website is written at a higher level than that suggested for the general public, it is important to realize that readability is just one consideration in the evaluation of OHI comprehension. In designing future versions of the patient education section of the AAO–HNS website, including diagrams and pictures within the substance of each entry would likely improve comprehension. Physicians need to be cognizant of their patients' ability to read and comprehend written information and tailor their educational material appropriately.
Author Contributions
Disclosures