User login
- Several taxonomies exist for rating individual studies and the strength of recommendations, making the analysis of evidence confusing for practitioners.
- A new grading scale—the Strength of Recommendation Taxonomy (SORT)—will be used by several family medicine and primary care journals (required or optional), allowing readers to learn 1 consistently applied taxonomy of evidence.
- SORT is built around the information mastery framework, which emphasizes the use of patient-oriented outcomes that measure changes in morbidity or mortality. Levels of evidence from 1 to 3 for individual studies also are defined.
- An A-level recommendation is based on consistent and good-quality patient-oriented evidence; a B-level recommendation is based on inconsistent or limited-quality patient-oriented evidence; and a C-level recommendation is based on consensus, usual practice, opinion, disease-oriented evidence, or case series for studies of diagnosis, treatment, prevention, or screening.
Review articles (or overviews) are highly valued by physicians as a way to keep up-to-date with the medical literature. Sometimes though, these articles are based more on the authors’ personal experience, or anecdotes, or incomplete surveys of the literature than on a comprehensive collection of the best available evidence. To improve the quality of review articles, there is an ongoing effort in the medical publishing field to use more explicit grading of the strength of evidence on which recommendations are based.1-4
Making evidence easier to understand
Several journals, including American Family Physician and Journal of Family Practice, have adopted evidence-grading scales that are used in particular articles. Other organizations and publications have also developed evidence-grading scales. The diversity of these scales can be confusing for readers. More than 100 grading scales are in use by various medical publications.5 A level B recommendation in 1 journal may not mean the same thing in another. Even within 1 issue of a journal, evidence-grading scales often vary among the articles. Journal readers do not have the time, energy, or interest to interpret multiple grading scales, and more complex scales are difficult to integrate into daily practice.
Therefore the editors of the US family medicine and primary care journals (ie, American Family Physician, Family Medicine, Journal of Family Practice, Journal of the American Board of Family Practice, and BMJ-USA) and the Family Practice Inquiries Network (FPIN) came together to develop a unified taxonomy for the strength of recommendations based on a body of evidence. The new taxonomy should fulfill several objectives:
- Be uniform in most family medicine journals and electronic databases
- Allow authors to evaluate the strength of recommendation of a body of evidence
- Allow authors to rate the level of evidence for an individual study
- Be comprehensive and allow authors to evaluate studies of screening, diagnosis, therapy, prevention, and prognosis
- Be easy to use and not too time-consuming for authors, reviewers, and editors who may be content experts but not experts in critical appraisal or clinical epidemiology
- Be straightforward enough that primary care physicians can readily integrate the recommendations into daily practice.
Defining terms of evidence
A number of relevant terms must be defined for clarification.
Disease-oriented outcomes. These outcomes include intermediate, histopathologic, physiologic, or surrogate results (eg, blood sugar, blood pressure, flow rate, coronary plaque thickness) that may or may not reflect improvements in patient outcomes.
Patient-oriented outcomes. These are outcomes that matter to patients and help them live longer or better lives, including reduced morbidity, mortality, or symptoms, improved quality of life, or lower cost.
Level of evidence. The validity of an individual study is based on an assessment of its study design. According to some methodologies,6 levels of evidence can refer not only to individual studies but also to the quality of evidence from multiple studies about a specific question or the quality of evidence supporting a clinical intervention. For simplicity and consistency in this proposal, we use the term level of evidence to refer to individual studies.
Strength of recommendation. The strength (or grade) of a recommendation for clinical practice is based on a body of evidence (typically more than 1 study). This approach takes into account the level of evidence of individual studies, the type of outcomes measured by these studies (patient-oriented or disease-oriented), the number, consistency, and coherence of the evidence as a whole, and the relationship between benefits, harms, and costs.
Practice guideline (evidence-based). These guidelines are recommendations for practice that involve a comprehensive search of the literature, an evaluation of the quality of individual studies, and recommendation grades that reflect the quality of the supporting evidence. All search, critical appraisal, and grading methods should be described explicitly and be replicable by similarly skilled authors.
Practice guideline (consensus). Consensus guidelines are recommendations for practice based on expert opinions that typically do not include a systematic search, an assessment of the quality of individual studies, or a system to label the strength of recommendations explicitly.
Research evidence. This evidence is presented in publications of original research, involving collection of original data or the systematic review of other original research publications. It does not include editorials, opinion pieces, or review articles (other than systematic reviews or meta-analyses).
Review article. A nonsystematic overview of a topic is a review article. In most cases, it is not based on an exhaustive, structured review of the literature and does not evaluate the quality of included studies systematically.
Systematic reviews and meta-analyses. A systematic review is a critical assessment of existing evidence that addresses a focused clinical question, includes a comprehensive literature search, appraises the quality of studies, and reports results in a systematic manner. If the studies report comparable quantitative data and have a low degree of variation in their findings, a meta-analysis can be performed to derive a summary estimate of effect.
Most strength-of-evidence scales lack key elements
In March 2002, the Agency for Healthcare Research and Quality (AHRQ) published a report that summarized the state-of-the-art in methods of rating the strength of evidence.5 The report identified a large number of systems for rating the quality of individual studies: 20 for systematic reviews, 49 for randomized controlled trials, 19 for observational studies, and 18 for diagnostic test studies. It also identified 40 scales that graded the strength of a body of evidence consisting of 1 or more studies.
The authors of the AHRQ report proposed that any system for grading the strength of evidence should consider 3 key elements: quality, quantity, and consistency. Quality is the extent to which the identified studies minimize the opportunity for bias and is synonymous with the concept of validity. Quantity is the number of studies and subjects included in those studies. Consistency is the extent to which findings are similar between different studies on the same topic. Only 7 of the 40 systems identified and addressed all 3 elements.6-11
Strength of Recommendation Taxonomy (SORT) contains the key elements
The authors of this article represent the major family medicine journals in the United States and a large family practice academic consortium. Our process began with a series of electronic mail exchanges, was developed during a meeting of the editors, and continued through another series of electronic mail exchanges.
We decided our taxonomy for rating the strength of a recommendation should address the 3 key elements identified in the AHRQ report: quality, quantity, and consistency of evidence. We also were committed to creating a grading scale that could be applied by authors with varying degrees of expertise in evidence-based medicine and clinical epidemiology, and interpreted by physicians with little or no formal training in these areas. We believed that the taxonomy should address the issue of patientoriented evidence versus disease-oriented evidence explicitly and be consistent with the information mastery framework proposed by Slawson and Shaughnessy.2
After considering these criteria and reviewing the existing taxonomies for grading the strength of a recommendation, we decided that a new taxonomy was needed to reflect the needs of our specialty. Existing grading scales were focused on a particular kind of study (ie, prevention or treatment), were too complex, or did not take into account the type of outcome.
Our proposed taxonomy is called the Strength of Recommendations Taxonomy (SORT), and it is shown in Table 1. The taxonomy includes ratings of A, B, or C for the strength of recommendation for a body of evidence. The taxonomy also explains whether a body of evidence represents good-quality or limited-quality evidence, and whether evidence is consistent or inconsistent. The quality of individual studies is rated 1, 2, or 3; numbers are used to distinguish ratings of individual studies from the letters A, B, and C used to evaluate the strength of a recommendation based on a body of evidence. Figure 1 provides information about how to determine the strength of recommendation for management recommendations, and Figure 2 explains how to determine the level of evidence for an individual study. These 2 algorithms should be helpful to authors preparing papers for submission to family medicine journals. The algorithms are to be considered general guidelines, and special circumstances may dictate assignment of a different strength of recommendation (eg, a single, large, well-designed study in a diverse population may warrant an A-level recommendation).
Recommendations based only on improvements in surrogate or disease-oriented outcomes are always categorized as level C, because improvements in disease-oriented outcomes are not always associated with improve-ments in patient-oriented outcomes, as exemplified by several well-known findings from the medical literature. For example, doxazosin lowers blood pressure in African American patients—a seemingly beneficial outcome—but it also increases mortality.12 Similarly, encainide and flecainide reduce the incidence of arrhythmias after acute myocardial infarction, but they also increase mortality.13 Finasteride improves urinary flow rates, but it does not significantly improve urinary tract symptoms in patients with benign prostatic hypertrophy,14 while arthroscopic surgery for osteoarthritis of the knee improves the appearance of cartilage but does not reduce pain or improve joint function.15 Additional examples of clinical situations where disease-oriented evidence disagrees with patient—oriented evidence are shown in Table 2.12-24 Examples of how to apply the taxonomy are given in Table 3.
TABLE 1
How recommendations are graded for strength, and underlying individual studies are rated for quality
| In general, only key recommendations for readers require a grade of the “Strength of Recommendation.” Recommendations should be based on the highest quality evidence available. For example, vitamin E was found in some cohort studies (level 2 study quality) to have a benefit for cardiovascular protection, but good-quality randomized trials (level 1) have not confirmed this effect. Therefore, it is preferable to base clinical recommendations in a manuscript on the level 1 studies. | |||
| Strength of recommendation | Definition | ||
| A | Recommendation based on consistent and good-quality patient-oriented evidence.* | ||
| B | Recommendation based on inconsistent or limited-quality patient-oriented evidence.* | ||
| C | Recommendation based on consensus, usual practice, opinion, disease-oriented evidence,* or case series for studies of diagnosis, treatment, prevention, or screening | ||
| Use the following scheme to determine whether a study measuring patient-oriented outcomes is of good or limited quality, and whether the results are consistent or inconsistent between studies. | |||
| Study quality | Type of Study | ||
| Diagnosis | Treatment/prevention/screening | Prognosis | |
| Level 1—good-quality patient-oriented evidence | Validated clinical decision rule | SR/meta-analysis of RCTs with consistent findings | SR/meta-analysis of good-quality cohort studies |
| SR/meta-analysis of high-quality studies | High-quality individual RCT‡ All-or-none study§ | Prospective cohort study with good follow-up | |
| High-quality diagnostic cohort study† | |||
| Level 2—limited-quality patient-oriented evidence | Unvalidated clinical decision rule | SR/meta-analysis lower-quality clinical trials or of studies with inconsistent findings | SR/meta-analysis of lower-quality cohort studies or with inconsistent results |
| SR/meta-analysis of lower-quality studies or studies with inconsistent findings | Lower-quality clinical trial‡ or prospective cohort study Cohort study | Retrospective cohort study with poor follow-up | |
| Lower-quality diagnostic cohort study or diagnostic case-control study§ | Case-control study | Case-control study Case series | |
| Level 3—other evidence | Consensus guidelines, extrapolations from bench research, usual practice, opinion, other evidence disease-oriented evidence (intermediate or physiologic outcomes only), or case series for studies of diagnosis, treatment, prevention, or screening | ||
| Consistency across studies | |||
| Consistent | Most studies found similar or at least coherent conclusions (coherence means that differences are explainable); or If high-quality and up-to-date systematic reviews or meta-analyses exist, they support the recommendation | ||
| Inconsistent | Considerable variation among study findings and lack of coherence; or If high-quality and up-to-date systematic reviews or meta-analyses exist, they do not find consistent evidence in favor of the recommendation | ||
| *Patient-oriented evidence measures outcomes that matter to patients: morbidity, mortality, symptom improvement, cost reduction, and quality of life. Disease-oriented evidence measures intermediate, physiologic, or surrogate end points that may or may not reflect improvements in patient outcomes (ie, blood pressure, blood chemistry, physiologic function, and pathologic findings). | |||
| † High-quality diagnostic cohort study: cohort design, adequate size, adequate spectrum of patients, blinding, and a consistent, well-defined reference standard. | |||
| ‡ High-quality RCT: allocation concealed, blinding if possible, intention-to-treat analysis, adequate statistical power, adequate follow-up (greater than 80 percent). | |||
| § In an all-or-none study, the treatment causes a dramatic change in outcomes, such as antibiotics for meningitis or surgery for appendicitis, which precludes study in a controlled trial. | |||
| SR, systematic review; RCT, randomized controlled trial | |||
TABLE 2
Examples of inconsistency between disease-oriented and patient-oriented outcomes
| Therapy | Disease-oriented outcome | Patient-oriented outcome |
|---|---|---|
| Doxazosin for blood pressure12 | Reduces blood pressure | Increases morality in African Americans |
| Lidocaine for arrhythmia following acute myocardial infarction13 | Suppresses arrhythmias | Increases mortality |
| Finasteride for benign prostatic hypertrophy14 | Improves urinary flow rate | No clinically important change in symptom scores |
| Sleeping infants on their stomach or side16 | Knowledge of anatomy and physiology suggests that this will decrease the risk of aspiration | Increases risk of sudden infant death syndrome |
| Vitamin E for heart disease17 | Reduces levels of free radicals | No change in mortality |
| Histamine antagonists and proton pump inhibitors for nonulcer dyspepsia18 | Significantly reduces gastric pH levels | Little or no improvement in symptoms in patients with non-gastroesophageal reflux disease, nonulcer dyspepsia |
| Arthroscopic surgery for osteoarthritis of the knee15 | Improves appearance of cartilage after debridement | No change in function or symptoms at 1 year |
| Hormone therapy19 | Reduces low-density lipoprotein cholesterol, increases high-density lipoprotein cholesterol | No decrease in cardiovascular or all-cause mortality; an increase in cardiovascular events in all-cause mortality; an increase in cardiovascular events in women older than 60 years (Women’s Health Initiative) with combined hormone therapy |
| Insulin therapy in type 2 diabetes mellitus20 | Keeps blood sugar below 120 mg/dL (6.7 mmol/l) | Does not reduce overall mortality |
| Sodium fluoride for fracture prevention21 | Increases bone density | Does not reduce fracture rate |
| Lidocaine prophylaxis following acute myocardial infarction22 | Suppresses arrhythmias | Increases mortality |
| Clofibrate for hyperlipidemia23 | Reduces lipids | Does not reduce mortality |
| Beta-blockers for heart failure24 | Reduces cardiac output | Reduces mortality in moderate to severe disease |
TABLE 3
Examples of how to apply the SORT in practice
| Example 1: While a number of observational studies (level of evidence—2) suggested a cardiovascular benefit from vitamin E, a large, well-designed, randomized trial with a diverse patient population (level of evidence—1) showed the opposite. The strength of recommendation against routine, long-term use of vitamin E to prevent heart disease, based on the best available evidence, should be A. |
| Example 2: A Cochrane review finds 7 clinical trials that are consistent in their support of a mechanical intervention for low back pain, but the trials were poorly designed (ie, unblinded, nonrandomized, or with allocation to groups unconcealed). In this case, the strength of recommendation in favor of these mechanical interventions is B (consistent but lower-quality clinical trials). |
| Example 3: A meta-analysis finds 9 high-quality clinical trials of the use of a new drug in the treatment of pulmonary fibrosis. Two of the studies find harm, 2 find no benefit, and 5 show some benefit. The strength of recommendation in favor of this drug would be B (inconsistent results of good-quality, randomized controlled trials). |
| Example 4: A new drug increases the forced expiratory volume in 1 second (FEV1) and peak flow rate in patients with an acute asthma exacerbation. Data on symptom improvement is lacking. The strength of recommendation in favor of using this drug is C (disease-oriented evidence only). |
FIGURE 1
Determining the strength of a recommendation based on a body of evidence
FIGURE 2
Determining the level of evidence for an individual study
The advantages of SORT
We believe there are several advantages to our proposed taxonomy. It is straightforward and comprehensive, is easily applied by authors and physicians, and explicitly addresses the issue of patient-oriented versus disease-oriented evidence. The latter attribute distinguishes SORT from most other evidence grading scales. These strengths also create some limitations. Some clinicians may be concerned that the taxonomy is not as detailed in its assessment of study designs as others, such as that of the Centre for Evidence-Based Medicine (CEBM).25 However, the primary difference between the 2 taxonomies is that the CEBM version distinguishes between good and poor observational studies while the SORT version does not. We concluded that the advantages of a system that provides the physician with a clear recommendation that is strong (A), moderate (B), or weak (C) in its support of a particular intervention outweighs the theoretic benefit of distinguishing between lower quality and higher quality observational studies, particularly because there is no objective evidence that the latter distinction carries important differences in clinical recommendations.
Any publication applying SORT (or any other evidence-based taxonomy) should describe carefully the search process that preceded the assignment of a SORT rating. For example, authors could perform a comprehensive search of MEDLINE and the gray literature, a comprehensive search of MEDLINE alone, or a more focused search of MEDLINE plus secondary evidence-based sources of information.
Walkovers: Creating linkages with SORT
Some organizations, such as the CEBM,25 the Cochrane Collaboration,7 and the US Preventive Services Task Force (USPSTF),6 have developed their own grading scales for the strength of recommendations based on a body of evidence and are unlikely to abandon them. Other organizations, such as FPIN,26 publish their work in a variety of settings and must be able to move between taxonomies. We have developed a set of optional walkovers that suggest how authors, editors, and readers might move from 1 taxonomy to another. Walkovers for the CEBM and USPSTF taxonomies are shown in Table 4.
Many authors and experts in evidence-based medicine use the “Level of Evidence” taxonomy from the CEBM to rate the quality of individual studies.25 A walkover from the 5-level CEBM scale to the simpler 3-level SORT scale for individual studies is shown in Table 5.
TABLE 4
Suggested walkovers between taxonomies for assessing the strength of a recommendation based on a body of evidence
| SORT | CEBM | BMJ’s Clinical Evidence |
|---|---|---|
| A. Recommendation based on consistent and good-quality patient-oriented evidence | A. Consistent level 1 studies | Beneficial |
| B. Recommendation based on inconsistent or limited-quality patient-oriented evidence | B. Consistent level 2 or 3 studies or extrapolations from level 1 studies | Likely to be beneficial Likely to be ineffective or harmful (recommendation against) |
| C. Level 4 studies or extrapola-tions from level 2 or 3 studies | Unlikely to be beneficial (recommendation against) | |
| C. Recommendation based on consensus, usual practice, disease-oriented evidence, case series for studies of treatment or screening, and/on opinion | D. Level 5 evidence or troublingly inconsistent inconclusive studies of of any level | Unknown effectiveness |
| SORT, Strength of Evidence Taxonomy; CEBM, Centre for Evidence-Based Medicine; BMJ, BMJ Publishing Group. | ||
TABLE 5
Suggested walkover between CEBM and SORT for assessing the level of evidence of an individual study
| SORT | CEBM | |
|---|---|---|
| Treatment/screening | Other categories | |
| Level 1 | Levels 1a to 1c | Levels 1a to 1c |
| Level 2 | Level 2 or 3 | Levels 2 to 4 |
| Level 3 | Level 4 or 5 and any study that measures measures intermediate or surrogate outcomes | Level 5 andany study that intermediate or surrogate outcomes |
| CEBM, Centre for Evidence-Based Medicine; | ||
| SORT, Strength of Recommendation Taxonomy | ||
SORT can improve patient care
The SORT is a comprehensive taxonomy for evaluating the strength of a recommendation based on a body of evidence and the quality of an individual study. If applied consistently by authors and editors in the family medicine literature, it has the potential to make it easier for physicians to apply the results of research in their practice through the information mastery approach and to incorporate evidence-based medicine into their patient care.
Like any such grading scale, it is a work in progress. As we learn more about biases in study design, and as the authors and readers who use the taxonomy become more sophisticated about principles of information mastery, evidence-based medicine, and critical appraisal, it is likely to evolve. We remain open to suggestions from the primary care community for refining and improving SORT.
Acknowledgments
The authors thank Lee Green, MD, MPH, John Epling, MD, Kurt Stange, MD, PhD, and Margaret Gourlay, MD, for helpful comments on the manuscript. The authors indicate that they do not have any conflicts of interest. Sources of funding: none reported. This article has been simultaneously published in print and online by American Family Physician, Journal of Family Practice, Journal of the American Board of Family Practice, and online by Family Practice Inquiries Network. Copyright © 2004 American Family Physician, a publication of the American Academy of Family Physicians. All rights reserved.
1. Evidence-based medicine . A new approach to teaching the practice of medicine. JAMA 1992;268:2420-2425.
2. Slawson DC, Shaughnessy AF, Bennett JH. Becoming a medical information master: feeling good about not knowing everything. J Fam Pract 1994;38:505-513.
3. Shaughnessy AF, Slawson DC, Bennett JH. Becoming an information master: a guidebook to the medical information jungle. J Fam Pract 1994;39:489-499.
4. Siwek J, Gourlay ML, Slawson DC, Shaughnessy AF. How to write an evidence-based clinical review article. Am Fam Physician 2002;65:251-258.
5. Systems to rate the strength of scientific evidence. Summary, evidence report/technology assessment: number 47. AHRQ pub. no. 02-E015, March 2002. Agency for Healthcare Research and Quality, Rockville, Md. Available at: www.ahrq.gov/clinic/epcsums/strengthsum.htm. Accessed on November 13, 2003.
6. Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, et al. Current methods of the U.S. Preventive Services Task Force: a review of the process. Am J Prev Med 2001;20(3 suppl):21-35.
7. Clarke M, Oxman AD. Cochrane reviewer’s handbook 4.0. The Cochrane Collaboration, 2003. Available at: www.cochrane.org/resources/handbook/handbook.pdf. Accessed on November 13, 2003.
8. Gyorkos TW, Tannenbaum TN, Abrahamowicz M, Oxman AD, Scott EA, Millson ME, et al. An approach to the development of practice guidelines for community health interventions. Can J Public Health 1994;85(suppl 1):S8-S13.
9. Briss PA, Zaza S, Pappaioanou M, et al. Developing an evidence-based guide to community preventive services—methods. Am J Prev Med 2000;18(1 suppl):35-43.
10. Greer N, Mosser G, Logan G, Halaas GW. A practical approach to evidence grading. Jt Comm J Qual Improv 2000;26:700-712.
11. Guyatt GH, Haynes RB, Jaeschke RZ, et al. Users’ guides to the medical literature: XXV. Evidence-based medicine: principles for applying the users’ guides to patient care. JAMA 2000;284:1290-1296.
12. Major cardiovascular events in hypertensive patients randomized to doxazosin vs chlorthalidone: the antihypertensive and lipid-lowering treatment to prevent heart attack trial (ALLHAT) JAMA 2000;283:1967-1975.
13. Echt DS, Liebson PR, Mitchell LB, et al. Mortality and morbidity in patients receiving encainide, flecainide, or placebo. N Engl J Med 1991;324:781-788.
14. Lepor H, Williford WO, Barry MJ, et al. The efficacy of terazosin, finasteride, or both in benign prostatic hyperplasia. N Engl J Med 1996;335:533-539.
15. Moseley JB, O’Malley K, Petersen NJ, et al. A controlled trial of arthroscopic surgery for osteoarthritis of the knee. N Engl J Med 2002;347:81-88.
16. Dwyer T, Ponsonby AL. Sudden infant death syndrome: after the “back to sleep” campaign. BMJ 1996;313:180-181.
17. Yusuf S, Dagenais G, Pogue J, Bosch J, Sleight P. Vitamin E supplementation and cardiovascular events in high-risk patients. N Engl J Med 2000;342:154-160.
18. Moayyedi P, Soo S, Deeks J, Delaney B, Innes M, Forman D. Pharmacological interventions for non-ulcer dyspepsia. Cochrane Database Syst Rev 2003;(1):CD001960.-
19. Rossouw JE, Anderson GL, Prentice RL, et al. Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women’s Health Initiative randomized controlled trial. JAMA 2002;288:321-333.
20. Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). Lancet 1998;352:837-853.
21. Meunier PJ, Sebert JL, Reginster JY, et al. Fluoride salts are no better at preventing new vertebral fractures than calcium-vitamin D in postmenopausal osteoporosis: the FAVO Study. Osteoporos Int 1998;8:4-12.
22. MacMahon S, Collins R, Peto R, Koster RW, Yusuf S. Effects of prophylactic lidocaine in suspected acute myocardial infarction. An overview of results from the randomized, controlled trials. JAMA 1988;260:1910-1916.
23. Grumbach K. How effective is drug treatment of hypercholesterolemia? A guided tour of the major clinical trials for the primary care physician. J Am Board Fam Pract 1991;4:437-445.
24. Heidenreich PA, Lee TT, Massie BM. Effect of beta-blockade on mortality in patients with heart failure: a metaanalysis of randomized clinical trials. J Am Coll Cardiol 1997;30:27-34.
25. Centre for Evidence-Based Medicine. Levels of evidence and grades of recommendation. Available at: www.cebm.net/levels_of_evidence.asp. Accessed on November 13, 2003.
26. Family Practice Inquiries Network. (FPIN). Available at: www.fpin.org. Accessed on November 13, 2003.
- Several taxonomies exist for rating individual studies and the strength of recommendations, making the analysis of evidence confusing for practitioners.
- A new grading scale—the Strength of Recommendation Taxonomy (SORT)—will be used by several family medicine and primary care journals (required or optional), allowing readers to learn 1 consistently applied taxonomy of evidence.
- SORT is built around the information mastery framework, which emphasizes the use of patient-oriented outcomes that measure changes in morbidity or mortality. Levels of evidence from 1 to 3 for individual studies also are defined.
- An A-level recommendation is based on consistent and good-quality patient-oriented evidence; a B-level recommendation is based on inconsistent or limited-quality patient-oriented evidence; and a C-level recommendation is based on consensus, usual practice, opinion, disease-oriented evidence, or case series for studies of diagnosis, treatment, prevention, or screening.
Review articles (or overviews) are highly valued by physicians as a way to keep up-to-date with the medical literature. Sometimes though, these articles are based more on the authors’ personal experience, or anecdotes, or incomplete surveys of the literature than on a comprehensive collection of the best available evidence. To improve the quality of review articles, there is an ongoing effort in the medical publishing field to use more explicit grading of the strength of evidence on which recommendations are based.1-4
Making evidence easier to understand
Several journals, including American Family Physician and Journal of Family Practice, have adopted evidence-grading scales that are used in particular articles. Other organizations and publications have also developed evidence-grading scales. The diversity of these scales can be confusing for readers. More than 100 grading scales are in use by various medical publications.5 A level B recommendation in 1 journal may not mean the same thing in another. Even within 1 issue of a journal, evidence-grading scales often vary among the articles. Journal readers do not have the time, energy, or interest to interpret multiple grading scales, and more complex scales are difficult to integrate into daily practice.
Therefore the editors of the US family medicine and primary care journals (ie, American Family Physician, Family Medicine, Journal of Family Practice, Journal of the American Board of Family Practice, and BMJ-USA) and the Family Practice Inquiries Network (FPIN) came together to develop a unified taxonomy for the strength of recommendations based on a body of evidence. The new taxonomy should fulfill several objectives:
- Be uniform in most family medicine journals and electronic databases
- Allow authors to evaluate the strength of recommendation of a body of evidence
- Allow authors to rate the level of evidence for an individual study
- Be comprehensive and allow authors to evaluate studies of screening, diagnosis, therapy, prevention, and prognosis
- Be easy to use and not too time-consuming for authors, reviewers, and editors who may be content experts but not experts in critical appraisal or clinical epidemiology
- Be straightforward enough that primary care physicians can readily integrate the recommendations into daily practice.
Defining terms of evidence
A number of relevant terms must be defined for clarification.
Disease-oriented outcomes. These outcomes include intermediate, histopathologic, physiologic, or surrogate results (eg, blood sugar, blood pressure, flow rate, coronary plaque thickness) that may or may not reflect improvements in patient outcomes.
Patient-oriented outcomes. These are outcomes that matter to patients and help them live longer or better lives, including reduced morbidity, mortality, or symptoms, improved quality of life, or lower cost.
Level of evidence. The validity of an individual study is based on an assessment of its study design. According to some methodologies,6 levels of evidence can refer not only to individual studies but also to the quality of evidence from multiple studies about a specific question or the quality of evidence supporting a clinical intervention. For simplicity and consistency in this proposal, we use the term level of evidence to refer to individual studies.
Strength of recommendation. The strength (or grade) of a recommendation for clinical practice is based on a body of evidence (typically more than 1 study). This approach takes into account the level of evidence of individual studies, the type of outcomes measured by these studies (patient-oriented or disease-oriented), the number, consistency, and coherence of the evidence as a whole, and the relationship between benefits, harms, and costs.
Practice guideline (evidence-based). These guidelines are recommendations for practice that involve a comprehensive search of the literature, an evaluation of the quality of individual studies, and recommendation grades that reflect the quality of the supporting evidence. All search, critical appraisal, and grading methods should be described explicitly and be replicable by similarly skilled authors.
Practice guideline (consensus). Consensus guidelines are recommendations for practice based on expert opinions that typically do not include a systematic search, an assessment of the quality of individual studies, or a system to label the strength of recommendations explicitly.
Research evidence. This evidence is presented in publications of original research, involving collection of original data or the systematic review of other original research publications. It does not include editorials, opinion pieces, or review articles (other than systematic reviews or meta-analyses).
Review article. A nonsystematic overview of a topic is a review article. In most cases, it is not based on an exhaustive, structured review of the literature and does not evaluate the quality of included studies systematically.
Systematic reviews and meta-analyses. A systematic review is a critical assessment of existing evidence that addresses a focused clinical question, includes a comprehensive literature search, appraises the quality of studies, and reports results in a systematic manner. If the studies report comparable quantitative data and have a low degree of variation in their findings, a meta-analysis can be performed to derive a summary estimate of effect.
Most strength-of-evidence scales lack key elements
In March 2002, the Agency for Healthcare Research and Quality (AHRQ) published a report that summarized the state-of-the-art in methods of rating the strength of evidence.5 The report identified a large number of systems for rating the quality of individual studies: 20 for systematic reviews, 49 for randomized controlled trials, 19 for observational studies, and 18 for diagnostic test studies. It also identified 40 scales that graded the strength of a body of evidence consisting of 1 or more studies.
The authors of the AHRQ report proposed that any system for grading the strength of evidence should consider 3 key elements: quality, quantity, and consistency. Quality is the extent to which the identified studies minimize the opportunity for bias and is synonymous with the concept of validity. Quantity is the number of studies and subjects included in those studies. Consistency is the extent to which findings are similar between different studies on the same topic. Only 7 of the 40 systems identified and addressed all 3 elements.6-11
Strength of Recommendation Taxonomy (SORT) contains the key elements
The authors of this article represent the major family medicine journals in the United States and a large family practice academic consortium. Our process began with a series of electronic mail exchanges, was developed during a meeting of the editors, and continued through another series of electronic mail exchanges.
We decided our taxonomy for rating the strength of a recommendation should address the 3 key elements identified in the AHRQ report: quality, quantity, and consistency of evidence. We also were committed to creating a grading scale that could be applied by authors with varying degrees of expertise in evidence-based medicine and clinical epidemiology, and interpreted by physicians with little or no formal training in these areas. We believed that the taxonomy should address the issue of patientoriented evidence versus disease-oriented evidence explicitly and be consistent with the information mastery framework proposed by Slawson and Shaughnessy.2
After considering these criteria and reviewing the existing taxonomies for grading the strength of a recommendation, we decided that a new taxonomy was needed to reflect the needs of our specialty. Existing grading scales were focused on a particular kind of study (ie, prevention or treatment), were too complex, or did not take into account the type of outcome.
Our proposed taxonomy is called the Strength of Recommendations Taxonomy (SORT), and it is shown in Table 1. The taxonomy includes ratings of A, B, or C for the strength of recommendation for a body of evidence. The taxonomy also explains whether a body of evidence represents good-quality or limited-quality evidence, and whether evidence is consistent or inconsistent. The quality of individual studies is rated 1, 2, or 3; numbers are used to distinguish ratings of individual studies from the letters A, B, and C used to evaluate the strength of a recommendation based on a body of evidence. Figure 1 provides information about how to determine the strength of recommendation for management recommendations, and Figure 2 explains how to determine the level of evidence for an individual study. These 2 algorithms should be helpful to authors preparing papers for submission to family medicine journals. The algorithms are to be considered general guidelines, and special circumstances may dictate assignment of a different strength of recommendation (eg, a single, large, well-designed study in a diverse population may warrant an A-level recommendation).
Recommendations based only on improvements in surrogate or disease-oriented outcomes are always categorized as level C, because improvements in disease-oriented outcomes are not always associated with improve-ments in patient-oriented outcomes, as exemplified by several well-known findings from the medical literature. For example, doxazosin lowers blood pressure in African American patients—a seemingly beneficial outcome—but it also increases mortality.12 Similarly, encainide and flecainide reduce the incidence of arrhythmias after acute myocardial infarction, but they also increase mortality.13 Finasteride improves urinary flow rates, but it does not significantly improve urinary tract symptoms in patients with benign prostatic hypertrophy,14 while arthroscopic surgery for osteoarthritis of the knee improves the appearance of cartilage but does not reduce pain or improve joint function.15 Additional examples of clinical situations where disease-oriented evidence disagrees with patient—oriented evidence are shown in Table 2.12-24 Examples of how to apply the taxonomy are given in Table 3.
TABLE 1
How recommendations are graded for strength, and underlying individual studies are rated for quality
| In general, only key recommendations for readers require a grade of the “Strength of Recommendation.” Recommendations should be based on the highest quality evidence available. For example, vitamin E was found in some cohort studies (level 2 study quality) to have a benefit for cardiovascular protection, but good-quality randomized trials (level 1) have not confirmed this effect. Therefore, it is preferable to base clinical recommendations in a manuscript on the level 1 studies. | |||
| Strength of recommendation | Definition | ||
| A | Recommendation based on consistent and good-quality patient-oriented evidence.* | ||
| B | Recommendation based on inconsistent or limited-quality patient-oriented evidence.* | ||
| C | Recommendation based on consensus, usual practice, opinion, disease-oriented evidence,* or case series for studies of diagnosis, treatment, prevention, or screening | ||
| Use the following scheme to determine whether a study measuring patient-oriented outcomes is of good or limited quality, and whether the results are consistent or inconsistent between studies. | |||
| Study quality | Type of Study | ||
| Diagnosis | Treatment/prevention/screening | Prognosis | |
| Level 1—good-quality patient-oriented evidence | Validated clinical decision rule | SR/meta-analysis of RCTs with consistent findings | SR/meta-analysis of good-quality cohort studies |
| SR/meta-analysis of high-quality studies | High-quality individual RCT‡ All-or-none study§ | Prospective cohort study with good follow-up | |
| High-quality diagnostic cohort study† | |||
| Level 2—limited-quality patient-oriented evidence | Unvalidated clinical decision rule | SR/meta-analysis lower-quality clinical trials or of studies with inconsistent findings | SR/meta-analysis of lower-quality cohort studies or with inconsistent results |
| SR/meta-analysis of lower-quality studies or studies with inconsistent findings | Lower-quality clinical trial‡ or prospective cohort study Cohort study | Retrospective cohort study with poor follow-up | |
| Lower-quality diagnostic cohort study or diagnostic case-control study§ | Case-control study | Case-control study Case series | |
| Level 3—other evidence | Consensus guidelines, extrapolations from bench research, usual practice, opinion, other evidence disease-oriented evidence (intermediate or physiologic outcomes only), or case series for studies of diagnosis, treatment, prevention, or screening | ||
| Consistency across studies | |||
| Consistent | Most studies found similar or at least coherent conclusions (coherence means that differences are explainable); or If high-quality and up-to-date systematic reviews or meta-analyses exist, they support the recommendation | ||
| Inconsistent | Considerable variation among study findings and lack of coherence; or If high-quality and up-to-date systematic reviews or meta-analyses exist, they do not find consistent evidence in favor of the recommendation | ||
| *Patient-oriented evidence measures outcomes that matter to patients: morbidity, mortality, symptom improvement, cost reduction, and quality of life. Disease-oriented evidence measures intermediate, physiologic, or surrogate end points that may or may not reflect improvements in patient outcomes (ie, blood pressure, blood chemistry, physiologic function, and pathologic findings). | |||
| † High-quality diagnostic cohort study: cohort design, adequate size, adequate spectrum of patients, blinding, and a consistent, well-defined reference standard. | |||
| ‡ High-quality RCT: allocation concealed, blinding if possible, intention-to-treat analysis, adequate statistical power, adequate follow-up (greater than 80 percent). | |||
| § In an all-or-none study, the treatment causes a dramatic change in outcomes, such as antibiotics for meningitis or surgery for appendicitis, which precludes study in a controlled trial. | |||
| SR, systematic review; RCT, randomized controlled trial | |||
TABLE 2
Examples of inconsistency between disease-oriented and patient-oriented outcomes
| Therapy | Disease-oriented outcome | Patient-oriented outcome |
|---|---|---|
| Doxazosin for blood pressure12 | Reduces blood pressure | Increases morality in African Americans |
| Lidocaine for arrhythmia following acute myocardial infarction13 | Suppresses arrhythmias | Increases mortality |
| Finasteride for benign prostatic hypertrophy14 | Improves urinary flow rate | No clinically important change in symptom scores |
| Sleeping infants on their stomach or side16 | Knowledge of anatomy and physiology suggests that this will decrease the risk of aspiration | Increases risk of sudden infant death syndrome |
| Vitamin E for heart disease17 | Reduces levels of free radicals | No change in mortality |
| Histamine antagonists and proton pump inhibitors for nonulcer dyspepsia18 | Significantly reduces gastric pH levels | Little or no improvement in symptoms in patients with non-gastroesophageal reflux disease, nonulcer dyspepsia |
| Arthroscopic surgery for osteoarthritis of the knee15 | Improves appearance of cartilage after debridement | No change in function or symptoms at 1 year |
| Hormone therapy19 | Reduces low-density lipoprotein cholesterol, increases high-density lipoprotein cholesterol | No decrease in cardiovascular or all-cause mortality; an increase in cardiovascular events in all-cause mortality; an increase in cardiovascular events in women older than 60 years (Women’s Health Initiative) with combined hormone therapy |
| Insulin therapy in type 2 diabetes mellitus20 | Keeps blood sugar below 120 mg/dL (6.7 mmol/l) | Does not reduce overall mortality |
| Sodium fluoride for fracture prevention21 | Increases bone density | Does not reduce fracture rate |
| Lidocaine prophylaxis following acute myocardial infarction22 | Suppresses arrhythmias | Increases mortality |
| Clofibrate for hyperlipidemia23 | Reduces lipids | Does not reduce mortality |
| Beta-blockers for heart failure24 | Reduces cardiac output | Reduces mortality in moderate to severe disease |
TABLE 3
Examples of how to apply the SORT in practice
| Example 1: While a number of observational studies (level of evidence—2) suggested a cardiovascular benefit from vitamin E, a large, well-designed, randomized trial with a diverse patient population (level of evidence—1) showed the opposite. The strength of recommendation against routine, long-term use of vitamin E to prevent heart disease, based on the best available evidence, should be A. |
| Example 2: A Cochrane review finds 7 clinical trials that are consistent in their support of a mechanical intervention for low back pain, but the trials were poorly designed (ie, unblinded, nonrandomized, or with allocation to groups unconcealed). In this case, the strength of recommendation in favor of these mechanical interventions is B (consistent but lower-quality clinical trials). |
| Example 3: A meta-analysis finds 9 high-quality clinical trials of the use of a new drug in the treatment of pulmonary fibrosis. Two of the studies find harm, 2 find no benefit, and 5 show some benefit. The strength of recommendation in favor of this drug would be B (inconsistent results of good-quality, randomized controlled trials). |
| Example 4: A new drug increases the forced expiratory volume in 1 second (FEV1) and peak flow rate in patients with an acute asthma exacerbation. Data on symptom improvement is lacking. The strength of recommendation in favor of using this drug is C (disease-oriented evidence only). |
FIGURE 1
Determining the strength of a recommendation based on a body of evidence
FIGURE 2
Determining the level of evidence for an individual study
The advantages of SORT
We believe there are several advantages to our proposed taxonomy. It is straightforward and comprehensive, is easily applied by authors and physicians, and explicitly addresses the issue of patient-oriented versus disease-oriented evidence. The latter attribute distinguishes SORT from most other evidence grading scales. These strengths also create some limitations. Some clinicians may be concerned that the taxonomy is not as detailed in its assessment of study designs as others, such as that of the Centre for Evidence-Based Medicine (CEBM).25 However, the primary difference between the 2 taxonomies is that the CEBM version distinguishes between good and poor observational studies while the SORT version does not. We concluded that the advantages of a system that provides the physician with a clear recommendation that is strong (A), moderate (B), or weak (C) in its support of a particular intervention outweighs the theoretic benefit of distinguishing between lower quality and higher quality observational studies, particularly because there is no objective evidence that the latter distinction carries important differences in clinical recommendations.
Any publication applying SORT (or any other evidence-based taxonomy) should describe carefully the search process that preceded the assignment of a SORT rating. For example, authors could perform a comprehensive search of MEDLINE and the gray literature, a comprehensive search of MEDLINE alone, or a more focused search of MEDLINE plus secondary evidence-based sources of information.
Walkovers: Creating linkages with SORT
Some organizations, such as the CEBM,25 the Cochrane Collaboration,7 and the US Preventive Services Task Force (USPSTF),6 have developed their own grading scales for the strength of recommendations based on a body of evidence and are unlikely to abandon them. Other organizations, such as FPIN,26 publish their work in a variety of settings and must be able to move between taxonomies. We have developed a set of optional walkovers that suggest how authors, editors, and readers might move from 1 taxonomy to another. Walkovers for the CEBM and USPSTF taxonomies are shown in Table 4.
Many authors and experts in evidence-based medicine use the “Level of Evidence” taxonomy from the CEBM to rate the quality of individual studies.25 A walkover from the 5-level CEBM scale to the simpler 3-level SORT scale for individual studies is shown in Table 5.
TABLE 4
Suggested walkovers between taxonomies for assessing the strength of a recommendation based on a body of evidence
| SORT | CEBM | BMJ’s Clinical Evidence |
|---|---|---|
| A. Recommendation based on consistent and good-quality patient-oriented evidence | A. Consistent level 1 studies | Beneficial |
| B. Recommendation based on inconsistent or limited-quality patient-oriented evidence | B. Consistent level 2 or 3 studies or extrapolations from level 1 studies | Likely to be beneficial Likely to be ineffective or harmful (recommendation against) |
| C. Level 4 studies or extrapola-tions from level 2 or 3 studies | Unlikely to be beneficial (recommendation against) | |
| C. Recommendation based on consensus, usual practice, disease-oriented evidence, case series for studies of treatment or screening, and/on opinion | D. Level 5 evidence or troublingly inconsistent inconclusive studies of of any level | Unknown effectiveness |
| SORT, Strength of Evidence Taxonomy; CEBM, Centre for Evidence-Based Medicine; BMJ, BMJ Publishing Group. | ||
TABLE 5
Suggested walkover between CEBM and SORT for assessing the level of evidence of an individual study
| SORT | CEBM | |
|---|---|---|
| Treatment/screening | Other categories | |
| Level 1 | Levels 1a to 1c | Levels 1a to 1c |
| Level 2 | Level 2 or 3 | Levels 2 to 4 |
| Level 3 | Level 4 or 5 and any study that measures measures intermediate or surrogate outcomes | Level 5 andany study that intermediate or surrogate outcomes |
| CEBM, Centre for Evidence-Based Medicine; | ||
| SORT, Strength of Recommendation Taxonomy | ||
SORT can improve patient care
The SORT is a comprehensive taxonomy for evaluating the strength of a recommendation based on a body of evidence and the quality of an individual study. If applied consistently by authors and editors in the family medicine literature, it has the potential to make it easier for physicians to apply the results of research in their practice through the information mastery approach and to incorporate evidence-based medicine into their patient care.
Like any such grading scale, it is a work in progress. As we learn more about biases in study design, and as the authors and readers who use the taxonomy become more sophisticated about principles of information mastery, evidence-based medicine, and critical appraisal, it is likely to evolve. We remain open to suggestions from the primary care community for refining and improving SORT.
Acknowledgments
The authors thank Lee Green, MD, MPH, John Epling, MD, Kurt Stange, MD, PhD, and Margaret Gourlay, MD, for helpful comments on the manuscript. The authors indicate that they do not have any conflicts of interest. Sources of funding: none reported. This article has been simultaneously published in print and online by American Family Physician, Journal of Family Practice, Journal of the American Board of Family Practice, and online by Family Practice Inquiries Network. Copyright © 2004 American Family Physician, a publication of the American Academy of Family Physicians. All rights reserved.
- Several taxonomies exist for rating individual studies and the strength of recommendations, making the analysis of evidence confusing for practitioners.
- A new grading scale—the Strength of Recommendation Taxonomy (SORT)—will be used by several family medicine and primary care journals (required or optional), allowing readers to learn 1 consistently applied taxonomy of evidence.
- SORT is built around the information mastery framework, which emphasizes the use of patient-oriented outcomes that measure changes in morbidity or mortality. Levels of evidence from 1 to 3 for individual studies also are defined.
- An A-level recommendation is based on consistent and good-quality patient-oriented evidence; a B-level recommendation is based on inconsistent or limited-quality patient-oriented evidence; and a C-level recommendation is based on consensus, usual practice, opinion, disease-oriented evidence, or case series for studies of diagnosis, treatment, prevention, or screening.
Review articles (or overviews) are highly valued by physicians as a way to keep up-to-date with the medical literature. Sometimes though, these articles are based more on the authors’ personal experience, or anecdotes, or incomplete surveys of the literature than on a comprehensive collection of the best available evidence. To improve the quality of review articles, there is an ongoing effort in the medical publishing field to use more explicit grading of the strength of evidence on which recommendations are based.1-4
Making evidence easier to understand
Several journals, including American Family Physician and Journal of Family Practice, have adopted evidence-grading scales that are used in particular articles. Other organizations and publications have also developed evidence-grading scales. The diversity of these scales can be confusing for readers. More than 100 grading scales are in use by various medical publications.5 A level B recommendation in 1 journal may not mean the same thing in another. Even within 1 issue of a journal, evidence-grading scales often vary among the articles. Journal readers do not have the time, energy, or interest to interpret multiple grading scales, and more complex scales are difficult to integrate into daily practice.
Therefore the editors of the US family medicine and primary care journals (ie, American Family Physician, Family Medicine, Journal of Family Practice, Journal of the American Board of Family Practice, and BMJ-USA) and the Family Practice Inquiries Network (FPIN) came together to develop a unified taxonomy for the strength of recommendations based on a body of evidence. The new taxonomy should fulfill several objectives:
- Be uniform in most family medicine journals and electronic databases
- Allow authors to evaluate the strength of recommendation of a body of evidence
- Allow authors to rate the level of evidence for an individual study
- Be comprehensive and allow authors to evaluate studies of screening, diagnosis, therapy, prevention, and prognosis
- Be easy to use and not too time-consuming for authors, reviewers, and editors who may be content experts but not experts in critical appraisal or clinical epidemiology
- Be straightforward enough that primary care physicians can readily integrate the recommendations into daily practice.
Defining terms of evidence
A number of relevant terms must be defined for clarification.
Disease-oriented outcomes. These outcomes include intermediate, histopathologic, physiologic, or surrogate results (eg, blood sugar, blood pressure, flow rate, coronary plaque thickness) that may or may not reflect improvements in patient outcomes.
Patient-oriented outcomes. These are outcomes that matter to patients and help them live longer or better lives, including reduced morbidity, mortality, or symptoms, improved quality of life, or lower cost.
Level of evidence. The validity of an individual study is based on an assessment of its study design. According to some methodologies,6 levels of evidence can refer not only to individual studies but also to the quality of evidence from multiple studies about a specific question or the quality of evidence supporting a clinical intervention. For simplicity and consistency in this proposal, we use the term level of evidence to refer to individual studies.
Strength of recommendation. The strength (or grade) of a recommendation for clinical practice is based on a body of evidence (typically more than 1 study). This approach takes into account the level of evidence of individual studies, the type of outcomes measured by these studies (patient-oriented or disease-oriented), the number, consistency, and coherence of the evidence as a whole, and the relationship between benefits, harms, and costs.
Practice guideline (evidence-based). These guidelines are recommendations for practice that involve a comprehensive search of the literature, an evaluation of the quality of individual studies, and recommendation grades that reflect the quality of the supporting evidence. All search, critical appraisal, and grading methods should be described explicitly and be replicable by similarly skilled authors.
Practice guideline (consensus). Consensus guidelines are recommendations for practice based on expert opinions that typically do not include a systematic search, an assessment of the quality of individual studies, or a system to label the strength of recommendations explicitly.
Research evidence. This evidence is presented in publications of original research, involving collection of original data or the systematic review of other original research publications. It does not include editorials, opinion pieces, or review articles (other than systematic reviews or meta-analyses).
Review article. A nonsystematic overview of a topic is a review article. In most cases, it is not based on an exhaustive, structured review of the literature and does not evaluate the quality of included studies systematically.
Systematic reviews and meta-analyses. A systematic review is a critical assessment of existing evidence that addresses a focused clinical question, includes a comprehensive literature search, appraises the quality of studies, and reports results in a systematic manner. If the studies report comparable quantitative data and have a low degree of variation in their findings, a meta-analysis can be performed to derive a summary estimate of effect.
Most strength-of-evidence scales lack key elements
In March 2002, the Agency for Healthcare Research and Quality (AHRQ) published a report that summarized the state-of-the-art in methods of rating the strength of evidence.5 The report identified a large number of systems for rating the quality of individual studies: 20 for systematic reviews, 49 for randomized controlled trials, 19 for observational studies, and 18 for diagnostic test studies. It also identified 40 scales that graded the strength of a body of evidence consisting of 1 or more studies.
The authors of the AHRQ report proposed that any system for grading the strength of evidence should consider 3 key elements: quality, quantity, and consistency. Quality is the extent to which the identified studies minimize the opportunity for bias and is synonymous with the concept of validity. Quantity is the number of studies and subjects included in those studies. Consistency is the extent to which findings are similar between different studies on the same topic. Only 7 of the 40 systems identified and addressed all 3 elements.6-11
Strength of Recommendation Taxonomy (SORT) contains the key elements
The authors of this article represent the major family medicine journals in the United States and a large family practice academic consortium. Our process began with a series of electronic mail exchanges, was developed during a meeting of the editors, and continued through another series of electronic mail exchanges.
We decided our taxonomy for rating the strength of a recommendation should address the 3 key elements identified in the AHRQ report: quality, quantity, and consistency of evidence. We also were committed to creating a grading scale that could be applied by authors with varying degrees of expertise in evidence-based medicine and clinical epidemiology, and interpreted by physicians with little or no formal training in these areas. We believed that the taxonomy should address the issue of patientoriented evidence versus disease-oriented evidence explicitly and be consistent with the information mastery framework proposed by Slawson and Shaughnessy.2
After considering these criteria and reviewing the existing taxonomies for grading the strength of a recommendation, we decided that a new taxonomy was needed to reflect the needs of our specialty. Existing grading scales were focused on a particular kind of study (ie, prevention or treatment), were too complex, or did not take into account the type of outcome.
Our proposed taxonomy is called the Strength of Recommendations Taxonomy (SORT), and it is shown in Table 1. The taxonomy includes ratings of A, B, or C for the strength of recommendation for a body of evidence. The taxonomy also explains whether a body of evidence represents good-quality or limited-quality evidence, and whether evidence is consistent or inconsistent. The quality of individual studies is rated 1, 2, or 3; numbers are used to distinguish ratings of individual studies from the letters A, B, and C used to evaluate the strength of a recommendation based on a body of evidence. Figure 1 provides information about how to determine the strength of recommendation for management recommendations, and Figure 2 explains how to determine the level of evidence for an individual study. These 2 algorithms should be helpful to authors preparing papers for submission to family medicine journals. The algorithms are to be considered general guidelines, and special circumstances may dictate assignment of a different strength of recommendation (eg, a single, large, well-designed study in a diverse population may warrant an A-level recommendation).
Recommendations based only on improvements in surrogate or disease-oriented outcomes are always categorized as level C, because improvements in disease-oriented outcomes are not always associated with improve-ments in patient-oriented outcomes, as exemplified by several well-known findings from the medical literature. For example, doxazosin lowers blood pressure in African American patients—a seemingly beneficial outcome—but it also increases mortality.12 Similarly, encainide and flecainide reduce the incidence of arrhythmias after acute myocardial infarction, but they also increase mortality.13 Finasteride improves urinary flow rates, but it does not significantly improve urinary tract symptoms in patients with benign prostatic hypertrophy,14 while arthroscopic surgery for osteoarthritis of the knee improves the appearance of cartilage but does not reduce pain or improve joint function.15 Additional examples of clinical situations where disease-oriented evidence disagrees with patient—oriented evidence are shown in Table 2.12-24 Examples of how to apply the taxonomy are given in Table 3.
TABLE 1
How recommendations are graded for strength, and underlying individual studies are rated for quality
| In general, only key recommendations for readers require a grade of the “Strength of Recommendation.” Recommendations should be based on the highest quality evidence available. For example, vitamin E was found in some cohort studies (level 2 study quality) to have a benefit for cardiovascular protection, but good-quality randomized trials (level 1) have not confirmed this effect. Therefore, it is preferable to base clinical recommendations in a manuscript on the level 1 studies. | |||
| Strength of recommendation | Definition | ||
| A | Recommendation based on consistent and good-quality patient-oriented evidence.* | ||
| B | Recommendation based on inconsistent or limited-quality patient-oriented evidence.* | ||
| C | Recommendation based on consensus, usual practice, opinion, disease-oriented evidence,* or case series for studies of diagnosis, treatment, prevention, or screening | ||
| Use the following scheme to determine whether a study measuring patient-oriented outcomes is of good or limited quality, and whether the results are consistent or inconsistent between studies. | |||
| Study quality | Type of Study | ||
| Diagnosis | Treatment/prevention/screening | Prognosis | |
| Level 1—good-quality patient-oriented evidence | Validated clinical decision rule | SR/meta-analysis of RCTs with consistent findings | SR/meta-analysis of good-quality cohort studies |
| SR/meta-analysis of high-quality studies | High-quality individual RCT‡ All-or-none study§ | Prospective cohort study with good follow-up | |
| High-quality diagnostic cohort study† | |||
| Level 2—limited-quality patient-oriented evidence | Unvalidated clinical decision rule | SR/meta-analysis lower-quality clinical trials or of studies with inconsistent findings | SR/meta-analysis of lower-quality cohort studies or with inconsistent results |
| SR/meta-analysis of lower-quality studies or studies with inconsistent findings | Lower-quality clinical trial‡ or prospective cohort study Cohort study | Retrospective cohort study with poor follow-up | |
| Lower-quality diagnostic cohort study or diagnostic case-control study§ | Case-control study | Case-control study Case series | |
| Level 3—other evidence | Consensus guidelines, extrapolations from bench research, usual practice, opinion, other evidence disease-oriented evidence (intermediate or physiologic outcomes only), or case series for studies of diagnosis, treatment, prevention, or screening | ||
| Consistency across studies | |||
| Consistent | Most studies found similar or at least coherent conclusions (coherence means that differences are explainable); or If high-quality and up-to-date systematic reviews or meta-analyses exist, they support the recommendation | ||
| Inconsistent | Considerable variation among study findings and lack of coherence; or If high-quality and up-to-date systematic reviews or meta-analyses exist, they do not find consistent evidence in favor of the recommendation | ||
| *Patient-oriented evidence measures outcomes that matter to patients: morbidity, mortality, symptom improvement, cost reduction, and quality of life. Disease-oriented evidence measures intermediate, physiologic, or surrogate end points that may or may not reflect improvements in patient outcomes (ie, blood pressure, blood chemistry, physiologic function, and pathologic findings). | |||
| † High-quality diagnostic cohort study: cohort design, adequate size, adequate spectrum of patients, blinding, and a consistent, well-defined reference standard. | |||
| ‡ High-quality RCT: allocation concealed, blinding if possible, intention-to-treat analysis, adequate statistical power, adequate follow-up (greater than 80 percent). | |||
| § In an all-or-none study, the treatment causes a dramatic change in outcomes, such as antibiotics for meningitis or surgery for appendicitis, which precludes study in a controlled trial. | |||
| SR, systematic review; RCT, randomized controlled trial | |||
TABLE 2
Examples of inconsistency between disease-oriented and patient-oriented outcomes
| Therapy | Disease-oriented outcome | Patient-oriented outcome |
|---|---|---|
| Doxazosin for blood pressure12 | Reduces blood pressure | Increases morality in African Americans |
| Lidocaine for arrhythmia following acute myocardial infarction13 | Suppresses arrhythmias | Increases mortality |
| Finasteride for benign prostatic hypertrophy14 | Improves urinary flow rate | No clinically important change in symptom scores |
| Sleeping infants on their stomach or side16 | Knowledge of anatomy and physiology suggests that this will decrease the risk of aspiration | Increases risk of sudden infant death syndrome |
| Vitamin E for heart disease17 | Reduces levels of free radicals | No change in mortality |
| Histamine antagonists and proton pump inhibitors for nonulcer dyspepsia18 | Significantly reduces gastric pH levels | Little or no improvement in symptoms in patients with non-gastroesophageal reflux disease, nonulcer dyspepsia |
| Arthroscopic surgery for osteoarthritis of the knee15 | Improves appearance of cartilage after debridement | No change in function or symptoms at 1 year |
| Hormone therapy19 | Reduces low-density lipoprotein cholesterol, increases high-density lipoprotein cholesterol | No decrease in cardiovascular or all-cause mortality; an increase in cardiovascular events in all-cause mortality; an increase in cardiovascular events in women older than 60 years (Women’s Health Initiative) with combined hormone therapy |
| Insulin therapy in type 2 diabetes mellitus20 | Keeps blood sugar below 120 mg/dL (6.7 mmol/l) | Does not reduce overall mortality |
| Sodium fluoride for fracture prevention21 | Increases bone density | Does not reduce fracture rate |
| Lidocaine prophylaxis following acute myocardial infarction22 | Suppresses arrhythmias | Increases mortality |
| Clofibrate for hyperlipidemia23 | Reduces lipids | Does not reduce mortality |
| Beta-blockers for heart failure24 | Reduces cardiac output | Reduces mortality in moderate to severe disease |
TABLE 3
Examples of how to apply the SORT in practice
| Example 1: While a number of observational studies (level of evidence—2) suggested a cardiovascular benefit from vitamin E, a large, well-designed, randomized trial with a diverse patient population (level of evidence—1) showed the opposite. The strength of recommendation against routine, long-term use of vitamin E to prevent heart disease, based on the best available evidence, should be A. |
| Example 2: A Cochrane review finds 7 clinical trials that are consistent in their support of a mechanical intervention for low back pain, but the trials were poorly designed (ie, unblinded, nonrandomized, or with allocation to groups unconcealed). In this case, the strength of recommendation in favor of these mechanical interventions is B (consistent but lower-quality clinical trials). |
| Example 3: A meta-analysis finds 9 high-quality clinical trials of the use of a new drug in the treatment of pulmonary fibrosis. Two of the studies find harm, 2 find no benefit, and 5 show some benefit. The strength of recommendation in favor of this drug would be B (inconsistent results of good-quality, randomized controlled trials). |
| Example 4: A new drug increases the forced expiratory volume in 1 second (FEV1) and peak flow rate in patients with an acute asthma exacerbation. Data on symptom improvement is lacking. The strength of recommendation in favor of using this drug is C (disease-oriented evidence only). |
FIGURE 1
Determining the strength of a recommendation based on a body of evidence
FIGURE 2
Determining the level of evidence for an individual study
The advantages of SORT
We believe there are several advantages to our proposed taxonomy. It is straightforward and comprehensive, is easily applied by authors and physicians, and explicitly addresses the issue of patient-oriented versus disease-oriented evidence. The latter attribute distinguishes SORT from most other evidence grading scales. These strengths also create some limitations. Some clinicians may be concerned that the taxonomy is not as detailed in its assessment of study designs as others, such as that of the Centre for Evidence-Based Medicine (CEBM).25 However, the primary difference between the 2 taxonomies is that the CEBM version distinguishes between good and poor observational studies while the SORT version does not. We concluded that the advantages of a system that provides the physician with a clear recommendation that is strong (A), moderate (B), or weak (C) in its support of a particular intervention outweighs the theoretic benefit of distinguishing between lower quality and higher quality observational studies, particularly because there is no objective evidence that the latter distinction carries important differences in clinical recommendations.
Any publication applying SORT (or any other evidence-based taxonomy) should describe carefully the search process that preceded the assignment of a SORT rating. For example, authors could perform a comprehensive search of MEDLINE and the gray literature, a comprehensive search of MEDLINE alone, or a more focused search of MEDLINE plus secondary evidence-based sources of information.
Walkovers: Creating linkages with SORT
Some organizations, such as the CEBM,25 the Cochrane Collaboration,7 and the US Preventive Services Task Force (USPSTF),6 have developed their own grading scales for the strength of recommendations based on a body of evidence and are unlikely to abandon them. Other organizations, such as FPIN,26 publish their work in a variety of settings and must be able to move between taxonomies. We have developed a set of optional walkovers that suggest how authors, editors, and readers might move from 1 taxonomy to another. Walkovers for the CEBM and USPSTF taxonomies are shown in Table 4.
Many authors and experts in evidence-based medicine use the “Level of Evidence” taxonomy from the CEBM to rate the quality of individual studies.25 A walkover from the 5-level CEBM scale to the simpler 3-level SORT scale for individual studies is shown in Table 5.
TABLE 4
Suggested walkovers between taxonomies for assessing the strength of a recommendation based on a body of evidence
| SORT | CEBM | BMJ’s Clinical Evidence |
|---|---|---|
| A. Recommendation based on consistent and good-quality patient-oriented evidence | A. Consistent level 1 studies | Beneficial |
| B. Recommendation based on inconsistent or limited-quality patient-oriented evidence | B. Consistent level 2 or 3 studies or extrapolations from level 1 studies | Likely to be beneficial Likely to be ineffective or harmful (recommendation against) |
| C. Level 4 studies or extrapola-tions from level 2 or 3 studies | Unlikely to be beneficial (recommendation against) | |
| C. Recommendation based on consensus, usual practice, disease-oriented evidence, case series for studies of treatment or screening, and/on opinion | D. Level 5 evidence or troublingly inconsistent inconclusive studies of of any level | Unknown effectiveness |
| SORT, Strength of Evidence Taxonomy; CEBM, Centre for Evidence-Based Medicine; BMJ, BMJ Publishing Group. | ||
TABLE 5
Suggested walkover between CEBM and SORT for assessing the level of evidence of an individual study
| SORT | CEBM | |
|---|---|---|
| Treatment/screening | Other categories | |
| Level 1 | Levels 1a to 1c | Levels 1a to 1c |
| Level 2 | Level 2 or 3 | Levels 2 to 4 |
| Level 3 | Level 4 or 5 and any study that measures measures intermediate or surrogate outcomes | Level 5 andany study that intermediate or surrogate outcomes |
| CEBM, Centre for Evidence-Based Medicine; | ||
| SORT, Strength of Recommendation Taxonomy | ||
SORT can improve patient care
The SORT is a comprehensive taxonomy for evaluating the strength of a recommendation based on a body of evidence and the quality of an individual study. If applied consistently by authors and editors in the family medicine literature, it has the potential to make it easier for physicians to apply the results of research in their practice through the information mastery approach and to incorporate evidence-based medicine into their patient care.
Like any such grading scale, it is a work in progress. As we learn more about biases in study design, and as the authors and readers who use the taxonomy become more sophisticated about principles of information mastery, evidence-based medicine, and critical appraisal, it is likely to evolve. We remain open to suggestions from the primary care community for refining and improving SORT.
Acknowledgments
The authors thank Lee Green, MD, MPH, John Epling, MD, Kurt Stange, MD, PhD, and Margaret Gourlay, MD, for helpful comments on the manuscript. The authors indicate that they do not have any conflicts of interest. Sources of funding: none reported. This article has been simultaneously published in print and online by American Family Physician, Journal of Family Practice, Journal of the American Board of Family Practice, and online by Family Practice Inquiries Network. Copyright © 2004 American Family Physician, a publication of the American Academy of Family Physicians. All rights reserved.
1. Evidence-based medicine . A new approach to teaching the practice of medicine. JAMA 1992;268:2420-2425.
2. Slawson DC, Shaughnessy AF, Bennett JH. Becoming a medical information master: feeling good about not knowing everything. J Fam Pract 1994;38:505-513.
3. Shaughnessy AF, Slawson DC, Bennett JH. Becoming an information master: a guidebook to the medical information jungle. J Fam Pract 1994;39:489-499.
4. Siwek J, Gourlay ML, Slawson DC, Shaughnessy AF. How to write an evidence-based clinical review article. Am Fam Physician 2002;65:251-258.
5. Systems to rate the strength of scientific evidence. Summary, evidence report/technology assessment: number 47. AHRQ pub. no. 02-E015, March 2002. Agency for Healthcare Research and Quality, Rockville, Md. Available at: www.ahrq.gov/clinic/epcsums/strengthsum.htm. Accessed on November 13, 2003.
6. Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, et al. Current methods of the U.S. Preventive Services Task Force: a review of the process. Am J Prev Med 2001;20(3 suppl):21-35.
7. Clarke M, Oxman AD. Cochrane reviewer’s handbook 4.0. The Cochrane Collaboration, 2003. Available at: www.cochrane.org/resources/handbook/handbook.pdf. Accessed on November 13, 2003.
8. Gyorkos TW, Tannenbaum TN, Abrahamowicz M, Oxman AD, Scott EA, Millson ME, et al. An approach to the development of practice guidelines for community health interventions. Can J Public Health 1994;85(suppl 1):S8-S13.
9. Briss PA, Zaza S, Pappaioanou M, et al. Developing an evidence-based guide to community preventive services—methods. Am J Prev Med 2000;18(1 suppl):35-43.
10. Greer N, Mosser G, Logan G, Halaas GW. A practical approach to evidence grading. Jt Comm J Qual Improv 2000;26:700-712.
11. Guyatt GH, Haynes RB, Jaeschke RZ, et al. Users’ guides to the medical literature: XXV. Evidence-based medicine: principles for applying the users’ guides to patient care. JAMA 2000;284:1290-1296.
12. Major cardiovascular events in hypertensive patients randomized to doxazosin vs chlorthalidone: the antihypertensive and lipid-lowering treatment to prevent heart attack trial (ALLHAT) JAMA 2000;283:1967-1975.
13. Echt DS, Liebson PR, Mitchell LB, et al. Mortality and morbidity in patients receiving encainide, flecainide, or placebo. N Engl J Med 1991;324:781-788.
14. Lepor H, Williford WO, Barry MJ, et al. The efficacy of terazosin, finasteride, or both in benign prostatic hyperplasia. N Engl J Med 1996;335:533-539.
15. Moseley JB, O’Malley K, Petersen NJ, et al. A controlled trial of arthroscopic surgery for osteoarthritis of the knee. N Engl J Med 2002;347:81-88.
16. Dwyer T, Ponsonby AL. Sudden infant death syndrome: after the “back to sleep” campaign. BMJ 1996;313:180-181.
17. Yusuf S, Dagenais G, Pogue J, Bosch J, Sleight P. Vitamin E supplementation and cardiovascular events in high-risk patients. N Engl J Med 2000;342:154-160.
18. Moayyedi P, Soo S, Deeks J, Delaney B, Innes M, Forman D. Pharmacological interventions for non-ulcer dyspepsia. Cochrane Database Syst Rev 2003;(1):CD001960.-
19. Rossouw JE, Anderson GL, Prentice RL, et al. Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women’s Health Initiative randomized controlled trial. JAMA 2002;288:321-333.
20. Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). Lancet 1998;352:837-853.
21. Meunier PJ, Sebert JL, Reginster JY, et al. Fluoride salts are no better at preventing new vertebral fractures than calcium-vitamin D in postmenopausal osteoporosis: the FAVO Study. Osteoporos Int 1998;8:4-12.
22. MacMahon S, Collins R, Peto R, Koster RW, Yusuf S. Effects of prophylactic lidocaine in suspected acute myocardial infarction. An overview of results from the randomized, controlled trials. JAMA 1988;260:1910-1916.
23. Grumbach K. How effective is drug treatment of hypercholesterolemia? A guided tour of the major clinical trials for the primary care physician. J Am Board Fam Pract 1991;4:437-445.
24. Heidenreich PA, Lee TT, Massie BM. Effect of beta-blockade on mortality in patients with heart failure: a metaanalysis of randomized clinical trials. J Am Coll Cardiol 1997;30:27-34.
25. Centre for Evidence-Based Medicine. Levels of evidence and grades of recommendation. Available at: www.cebm.net/levels_of_evidence.asp. Accessed on November 13, 2003.
26. Family Practice Inquiries Network. (FPIN). Available at: www.fpin.org. Accessed on November 13, 2003.
1. Evidence-based medicine . A new approach to teaching the practice of medicine. JAMA 1992;268:2420-2425.
2. Slawson DC, Shaughnessy AF, Bennett JH. Becoming a medical information master: feeling good about not knowing everything. J Fam Pract 1994;38:505-513.
3. Shaughnessy AF, Slawson DC, Bennett JH. Becoming an information master: a guidebook to the medical information jungle. J Fam Pract 1994;39:489-499.
4. Siwek J, Gourlay ML, Slawson DC, Shaughnessy AF. How to write an evidence-based clinical review article. Am Fam Physician 2002;65:251-258.
5. Systems to rate the strength of scientific evidence. Summary, evidence report/technology assessment: number 47. AHRQ pub. no. 02-E015, March 2002. Agency for Healthcare Research and Quality, Rockville, Md. Available at: www.ahrq.gov/clinic/epcsums/strengthsum.htm. Accessed on November 13, 2003.
6. Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, et al. Current methods of the U.S. Preventive Services Task Force: a review of the process. Am J Prev Med 2001;20(3 suppl):21-35.
7. Clarke M, Oxman AD. Cochrane reviewer’s handbook 4.0. The Cochrane Collaboration, 2003. Available at: www.cochrane.org/resources/handbook/handbook.pdf. Accessed on November 13, 2003.
8. Gyorkos TW, Tannenbaum TN, Abrahamowicz M, Oxman AD, Scott EA, Millson ME, et al. An approach to the development of practice guidelines for community health interventions. Can J Public Health 1994;85(suppl 1):S8-S13.
9. Briss PA, Zaza S, Pappaioanou M, et al. Developing an evidence-based guide to community preventive services—methods. Am J Prev Med 2000;18(1 suppl):35-43.
10. Greer N, Mosser G, Logan G, Halaas GW. A practical approach to evidence grading. Jt Comm J Qual Improv 2000;26:700-712.
11. Guyatt GH, Haynes RB, Jaeschke RZ, et al. Users’ guides to the medical literature: XXV. Evidence-based medicine: principles for applying the users’ guides to patient care. JAMA 2000;284:1290-1296.
12. Major cardiovascular events in hypertensive patients randomized to doxazosin vs chlorthalidone: the antihypertensive and lipid-lowering treatment to prevent heart attack trial (ALLHAT) JAMA 2000;283:1967-1975.
13. Echt DS, Liebson PR, Mitchell LB, et al. Mortality and morbidity in patients receiving encainide, flecainide, or placebo. N Engl J Med 1991;324:781-788.
14. Lepor H, Williford WO, Barry MJ, et al. The efficacy of terazosin, finasteride, or both in benign prostatic hyperplasia. N Engl J Med 1996;335:533-539.
15. Moseley JB, O’Malley K, Petersen NJ, et al. A controlled trial of arthroscopic surgery for osteoarthritis of the knee. N Engl J Med 2002;347:81-88.
16. Dwyer T, Ponsonby AL. Sudden infant death syndrome: after the “back to sleep” campaign. BMJ 1996;313:180-181.
17. Yusuf S, Dagenais G, Pogue J, Bosch J, Sleight P. Vitamin E supplementation and cardiovascular events in high-risk patients. N Engl J Med 2000;342:154-160.
18. Moayyedi P, Soo S, Deeks J, Delaney B, Innes M, Forman D. Pharmacological interventions for non-ulcer dyspepsia. Cochrane Database Syst Rev 2003;(1):CD001960.-
19. Rossouw JE, Anderson GL, Prentice RL, et al. Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women’s Health Initiative randomized controlled trial. JAMA 2002;288:321-333.
20. Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes (UKPDS 33). Lancet 1998;352:837-853.
21. Meunier PJ, Sebert JL, Reginster JY, et al. Fluoride salts are no better at preventing new vertebral fractures than calcium-vitamin D in postmenopausal osteoporosis: the FAVO Study. Osteoporos Int 1998;8:4-12.
22. MacMahon S, Collins R, Peto R, Koster RW, Yusuf S. Effects of prophylactic lidocaine in suspected acute myocardial infarction. An overview of results from the randomized, controlled trials. JAMA 1988;260:1910-1916.
23. Grumbach K. How effective is drug treatment of hypercholesterolemia? A guided tour of the major clinical trials for the primary care physician. J Am Board Fam Pract 1991;4:437-445.
24. Heidenreich PA, Lee TT, Massie BM. Effect of beta-blockade on mortality in patients with heart failure: a metaanalysis of randomized clinical trials. J Am Coll Cardiol 1997;30:27-34.
25. Centre for Evidence-Based Medicine. Levels of evidence and grades of recommendation. Available at: www.cebm.net/levels_of_evidence.asp. Accessed on November 13, 2003.
26. Family Practice Inquiries Network. (FPIN). Available at: www.fpin.org. Accessed on November 13, 2003.