Michael C. Monuteaux, ScD

Non‐polio enteroviruses are the most common cause of aseptic meningitis in children.1 While bacterial meningitis requires parenteral antibiotics, aseptic meningitis requires only supportive care.1 Enteroviral reverse transcription polymerase chain reaction (EVPCR) testing of the cerebrospinal fluid (CSF) allows the virus to be detected with high sensitivity and specificity.2 Because children with a positive EVPCR test are at low risk of bacterial meningitis,3 access to rapid EVPCR results has the potential to impact the clinical management of children with meningitis.4, 5 We studied the impact of implementing an in‐hospital EVPCR testing protocol on the clinical management of children with meningitis in a single‐center retrospective cohort.

MATERIALS AND METHODS

RESULTS

Of the 593 children with meningitis, 152 (26%) were excluded for the reasons noted above. The 441 patients included in our analyses had the following final diagnoses: bacterial meningitis (1 patient with Streptococcus pneumoniae, 0.2%), pretreated culture‐negative meningitis (42 patients, 10%), and aseptic meningitis (398 patients, 90%).

We compared patient populations and EVPCR testing characteristics between the pre‐ and post‐study periods (Table 1). While CSF glucose differed between study periods, the difference was not felt to be clinically significant. However, during the post‐period, more children presented during enteroviral season. Clinicians were more likely to order an EVPCR test for children with aseptic, than bacterial, meningitis (213/370 [58%] vs 0/1 [0%]).

We evaluated the impact of the in‐hospital EVPCR test on the length of stay and duration of parenteral antibiotics for the 6 predefined patient groups (Table 2). Length of stay could be determined for 432 (98%) of study patients, and duration of parenteral antibiotics for 365 (83%). We found a clinically important decrease in both length of stay and duration of parenteral antibiotics for children with a positive EVPCR test in the post‐period. For every hour earlier the EVPCR results returned, length of stay was reduced by 0.3 hours ( = 0.3, 95% confidence interval [CI] 0.10.5), and parenteral antibiotics were reduced by 0.3 hours ( = 0.3, 95% CI 0.10.5). However, even in the post‐period, the median length of time from a positive EVPCR test result to hospital discharge was 14 hours (interquartile range, 533 hours).

We observed no trend in length of stay in either testing period ( = 0.17, 95% CI 3.9 to 3.6 pre vs = 1.64, 95% CI 6.3 to 3.0 post), with no change following the introduction of the faster EVPCR protocol (P = 0.52). We observed an increase in duration of parenteral antibiotics in the pre‐period ( = 5.4, 95% CI 0.3 to 10.6), with no trend in the post‐period ( = 1.7, 95% CI 5.2 to 1.8), but the difference was not significant (P = 0.08).

DISCUSSION

The in‐hospital EVPCR testing protocol reduced test turnaround time and increased testing. Children with a positive test had a shorter length of stay and duration of parenteral antibiotics. Decreasing the test turnaround time for EVPCR improved the care of children with enteroviral meningitis by reducing the length of unnecessary hospitalizations and parenteral antibiotics, with the potential for reducing the costs associated with these admissions.

Accurate identification of children with enteroviral meningitis, an often self‐limited infection requiring supportive care, can reduce hospitalization and unnecessary antibiotics. Previously, a positive EVPCR test result has been associated with a reduction in length of stay and of parenteral antibiotics,4, 5, 1012 with a direct correlation between test turnaround time and length of stay.12, 13 Moreover, positive EVPCR test results that were available prior to hospital discharge resulted in shorter length of hospital stay and duration of parenteral antibiotics.10

Our study is the largest to investigate the impact of implementing an in‐hospital EVPCR testing protocol, with the goal of making results available in a clinically useful time frame for all patients. Older EVPCR tests were typically performed in batches, or at centralized laboratories.4, 5, 1013 The in‐hospital EVPCR test utilized is a simple testing platform, which can be run multiple times daily. While there were higher charges associated with increased testing in the post‐period, these were more than offset by a reduced length of stay. Using study institution patient charges, we estimate that overall patient charges decreased approximately $80,000 in the post‐period, compared to the pre‐period (an average reduction of $375 per patient).

Many children were not discharged when a positive EVPCR test result became available. Some children with enteroviral meningitis will have persistent symptoms that require inpatient management. In addition, results that returned in the evening or nighttime were less likely to result in immediate hospital discharge. However, children with a positive EVPCR test are at very low risk for bacterial meningitis.3 As clinicians' knowledge of, and comfort with, the EVPCR test increase, this technology has the potential to further decrease the costs of caring for children with enteroviral meningitis.14

Our study had several limitations. First, it was retrospective; however, primary outcomes were objective measures accurately recorded in the medical record for most patients. Second, our study was single‐center, and findings may not be generalizable to other settings. Third, the management of children with meningitis may have been changing over the study period, independent of the in‐hospital EVPCR test. However, among children with a negative test, we observed no change in either of our primary outcomes. Fourth, given the large number of physicians involved with testing and treatment decisions, we could not adjust for clustering at the physician level. Fifth, we corrected CSF WBC in the case of a traumatic lumbar puncture (LP). Although use of this correction might underestimate the true CSF WBC count,6 the percentage of children with traumatic lumbar punctures was the same in both testing periods. Lastly, we evaluated the impact of a diagnostic test immediately after introduction into the clinical setting. As new medical technologies often take time to be adopted into clinical practice,15 we would expect the impact to increase over time.

CONCLUSIONS

In‐hospital EVPCR testing can improve the care of children with meningitis by reducing the length of unnecessary hospitalizations and parenteral antibiotics. Clinicians caring for children with meningitis should have access to in‐hospital EVPCR testing.

Non‐polio enteroviruses are the most common cause of aseptic meningitis in children.1 While bacterial meningitis requires parenteral antibiotics, aseptic meningitis requires only supportive care.1 Enteroviral reverse transcription polymerase chain reaction (EVPCR) testing of the cerebrospinal fluid (CSF) allows the virus to be detected with high sensitivity and specificity.2 Because children with a positive EVPCR test are at low risk of bacterial meningitis,3 access to rapid EVPCR results has the potential to impact the clinical management of children with meningitis.4, 5 We studied the impact of implementing an in‐hospital EVPCR testing protocol on the clinical management of children with meningitis in a single‐center retrospective cohort.

MATERIALS AND METHODS

RESULTS

Of the 593 children with meningitis, 152 (26%) were excluded for the reasons noted above. The 441 patients included in our analyses had the following final diagnoses: bacterial meningitis (1 patient with Streptococcus pneumoniae, 0.2%), pretreated culture‐negative meningitis (42 patients, 10%), and aseptic meningitis (398 patients, 90%).

We compared patient populations and EVPCR testing characteristics between the pre‐ and post‐study periods (Table 1). While CSF glucose differed between study periods, the difference was not felt to be clinically significant. However, during the post‐period, more children presented during enteroviral season. Clinicians were more likely to order an EVPCR test for children with aseptic, than bacterial, meningitis (213/370 [58%] vs 0/1 [0%]).

We evaluated the impact of the in‐hospital EVPCR test on the length of stay and duration of parenteral antibiotics for the 6 predefined patient groups (Table 2). Length of stay could be determined for 432 (98%) of study patients, and duration of parenteral antibiotics for 365 (83%). We found a clinically important decrease in both length of stay and duration of parenteral antibiotics for children with a positive EVPCR test in the post‐period. For every hour earlier the EVPCR results returned, length of stay was reduced by 0.3 hours ( = 0.3, 95% confidence interval [CI] 0.10.5), and parenteral antibiotics were reduced by 0.3 hours ( = 0.3, 95% CI 0.10.5). However, even in the post‐period, the median length of time from a positive EVPCR test result to hospital discharge was 14 hours (interquartile range, 533 hours).

We observed no trend in length of stay in either testing period ( = 0.17, 95% CI 3.9 to 3.6 pre vs = 1.64, 95% CI 6.3 to 3.0 post), with no change following the introduction of the faster EVPCR protocol (P = 0.52). We observed an increase in duration of parenteral antibiotics in the pre‐period ( = 5.4, 95% CI 0.3 to 10.6), with no trend in the post‐period ( = 1.7, 95% CI 5.2 to 1.8), but the difference was not significant (P = 0.08).

DISCUSSION

The in‐hospital EVPCR testing protocol reduced test turnaround time and increased testing. Children with a positive test had a shorter length of stay and duration of parenteral antibiotics. Decreasing the test turnaround time for EVPCR improved the care of children with enteroviral meningitis by reducing the length of unnecessary hospitalizations and parenteral antibiotics, with the potential for reducing the costs associated with these admissions.

Accurate identification of children with enteroviral meningitis, an often self‐limited infection requiring supportive care, can reduce hospitalization and unnecessary antibiotics. Previously, a positive EVPCR test result has been associated with a reduction in length of stay and of parenteral antibiotics,4, 5, 1012 with a direct correlation between test turnaround time and length of stay.12, 13 Moreover, positive EVPCR test results that were available prior to hospital discharge resulted in shorter length of hospital stay and duration of parenteral antibiotics.10

Our study is the largest to investigate the impact of implementing an in‐hospital EVPCR testing protocol, with the goal of making results available in a clinically useful time frame for all patients. Older EVPCR tests were typically performed in batches, or at centralized laboratories.4, 5, 1013 The in‐hospital EVPCR test utilized is a simple testing platform, which can be run multiple times daily. While there were higher charges associated with increased testing in the post‐period, these were more than offset by a reduced length of stay. Using study institution patient charges, we estimate that overall patient charges decreased approximately $80,000 in the post‐period, compared to the pre‐period (an average reduction of $375 per patient).

Many children were not discharged when a positive EVPCR test result became available. Some children with enteroviral meningitis will have persistent symptoms that require inpatient management. In addition, results that returned in the evening or nighttime were less likely to result in immediate hospital discharge. However, children with a positive EVPCR test are at very low risk for bacterial meningitis.3 As clinicians' knowledge of, and comfort with, the EVPCR test increase, this technology has the potential to further decrease the costs of caring for children with enteroviral meningitis.14

Our study had several limitations. First, it was retrospective; however, primary outcomes were objective measures accurately recorded in the medical record for most patients. Second, our study was single‐center, and findings may not be generalizable to other settings. Third, the management of children with meningitis may have been changing over the study period, independent of the in‐hospital EVPCR test. However, among children with a negative test, we observed no change in either of our primary outcomes. Fourth, given the large number of physicians involved with testing and treatment decisions, we could not adjust for clustering at the physician level. Fifth, we corrected CSF WBC in the case of a traumatic lumbar puncture (LP). Although use of this correction might underestimate the true CSF WBC count,6 the percentage of children with traumatic lumbar punctures was the same in both testing periods. Lastly, we evaluated the impact of a diagnostic test immediately after introduction into the clinical setting. As new medical technologies often take time to be adopted into clinical practice,15 we would expect the impact to increase over time.

CONCLUSIONS

In‐hospital EVPCR testing can improve the care of children with meningitis by reducing the length of unnecessary hospitalizations and parenteral antibiotics. Clinicians caring for children with meningitis should have access to in‐hospital EVPCR testing.

Non‐polio enteroviruses are the most common cause of aseptic meningitis in children.1 While bacterial meningitis requires parenteral antibiotics, aseptic meningitis requires only supportive care.1 Enteroviral reverse transcription polymerase chain reaction (EVPCR) testing of the cerebrospinal fluid (CSF) allows the virus to be detected with high sensitivity and specificity.2 Because children with a positive EVPCR test are at low risk of bacterial meningitis,3 access to rapid EVPCR results has the potential to impact the clinical management of children with meningitis.4, 5 We studied the impact of implementing an in‐hospital EVPCR testing protocol on the clinical management of children with meningitis in a single‐center retrospective cohort.

MATERIALS AND METHODS

RESULTS

Of the 593 children with meningitis, 152 (26%) were excluded for the reasons noted above. The 441 patients included in our analyses had the following final diagnoses: bacterial meningitis (1 patient with Streptococcus pneumoniae, 0.2%), pretreated culture‐negative meningitis (42 patients, 10%), and aseptic meningitis (398 patients, 90%).

We compared patient populations and EVPCR testing characteristics between the pre‐ and post‐study periods (Table 1). While CSF glucose differed between study periods, the difference was not felt to be clinically significant. However, during the post‐period, more children presented during enteroviral season. Clinicians were more likely to order an EVPCR test for children with aseptic, than bacterial, meningitis (213/370 [58%] vs 0/1 [0%]).

We evaluated the impact of the in‐hospital EVPCR test on the length of stay and duration of parenteral antibiotics for the 6 predefined patient groups (Table 2). Length of stay could be determined for 432 (98%) of study patients, and duration of parenteral antibiotics for 365 (83%). We found a clinically important decrease in both length of stay and duration of parenteral antibiotics for children with a positive EVPCR test in the post‐period. For every hour earlier the EVPCR results returned, length of stay was reduced by 0.3 hours ( = 0.3, 95% confidence interval [CI] 0.10.5), and parenteral antibiotics were reduced by 0.3 hours ( = 0.3, 95% CI 0.10.5). However, even in the post‐period, the median length of time from a positive EVPCR test result to hospital discharge was 14 hours (interquartile range, 533 hours).

We observed no trend in length of stay in either testing period ( = 0.17, 95% CI 3.9 to 3.6 pre vs = 1.64, 95% CI 6.3 to 3.0 post), with no change following the introduction of the faster EVPCR protocol (P = 0.52). We observed an increase in duration of parenteral antibiotics in the pre‐period ( = 5.4, 95% CI 0.3 to 10.6), with no trend in the post‐period ( = 1.7, 95% CI 5.2 to 1.8), but the difference was not significant (P = 0.08).

DISCUSSION

The in‐hospital EVPCR testing protocol reduced test turnaround time and increased testing. Children with a positive test had a shorter length of stay and duration of parenteral antibiotics. Decreasing the test turnaround time for EVPCR improved the care of children with enteroviral meningitis by reducing the length of unnecessary hospitalizations and parenteral antibiotics, with the potential for reducing the costs associated with these admissions.

Accurate identification of children with enteroviral meningitis, an often self‐limited infection requiring supportive care, can reduce hospitalization and unnecessary antibiotics. Previously, a positive EVPCR test result has been associated with a reduction in length of stay and of parenteral antibiotics,4, 5, 1012 with a direct correlation between test turnaround time and length of stay.12, 13 Moreover, positive EVPCR test results that were available prior to hospital discharge resulted in shorter length of hospital stay and duration of parenteral antibiotics.10

Our study is the largest to investigate the impact of implementing an in‐hospital EVPCR testing protocol, with the goal of making results available in a clinically useful time frame for all patients. Older EVPCR tests were typically performed in batches, or at centralized laboratories.4, 5, 1013 The in‐hospital EVPCR test utilized is a simple testing platform, which can be run multiple times daily. While there were higher charges associated with increased testing in the post‐period, these were more than offset by a reduced length of stay. Using study institution patient charges, we estimate that overall patient charges decreased approximately $80,000 in the post‐period, compared to the pre‐period (an average reduction of $375 per patient).

Many children were not discharged when a positive EVPCR test result became available. Some children with enteroviral meningitis will have persistent symptoms that require inpatient management. In addition, results that returned in the evening or nighttime were less likely to result in immediate hospital discharge. However, children with a positive EVPCR test are at very low risk for bacterial meningitis.3 As clinicians' knowledge of, and comfort with, the EVPCR test increase, this technology has the potential to further decrease the costs of caring for children with enteroviral meningitis.14

Our study had several limitations. First, it was retrospective; however, primary outcomes were objective measures accurately recorded in the medical record for most patients. Second, our study was single‐center, and findings may not be generalizable to other settings. Third, the management of children with meningitis may have been changing over the study period, independent of the in‐hospital EVPCR test. However, among children with a negative test, we observed no change in either of our primary outcomes. Fourth, given the large number of physicians involved with testing and treatment decisions, we could not adjust for clustering at the physician level. Fifth, we corrected CSF WBC in the case of a traumatic lumbar puncture (LP). Although use of this correction might underestimate the true CSF WBC count,6 the percentage of children with traumatic lumbar punctures was the same in both testing periods. Lastly, we evaluated the impact of a diagnostic test immediately after introduction into the clinical setting. As new medical technologies often take time to be adopted into clinical practice,15 we would expect the impact to increase over time.

CONCLUSIONS

In‐hospital EVPCR testing can improve the care of children with meningitis by reducing the length of unnecessary hospitalizations and parenteral antibiotics. Clinicians caring for children with meningitis should have access to in‐hospital EVPCR testing.

The chest radiograph (CXR) is the most commonly used diagnostic imaging modality in children, and is considered to be the gold standard for the diagnosis of pneumonia. As such, physicians in developed countries rely on chest radiography to establish the diagnosis of pneumonia.13 However, there are limited data investigating the reliability of this test for the diagnosis of pneumonia in children.2, 46

Prior investigations have noted poor overall agreement by emergency medicine, infectious diseases, and pulmonary medicine physicians, and even radiologists, in their interpretation of chest radiographs for the diagnosis of pneumonia.2, 5, 710 The World Health Organization (WHO) developed criteria to standardize CXR interpretation for the diagnosis of pneumonia in children for use in epidemiologic studies.11 These standardized definitions of pneumonia have been formally evaluated by the WHO6 and utilized in epidemiologic studies of vaccine efficacy,12 but the overall reliability of these radiographic criteria have not been studied outside of these forums.

We conducted this prospective case‐based study to evaluate the reliability of the radiographic diagnosis of pneumonia among children presenting to a pediatric emergency department with clinical suspicion of pneumonia. We were primarily interested in assessing the overall reliability in CXR interpretation for the diagnosis of pneumonia, and identifying which radiographic features of pneumonia were consistently identified by radiologists.

MATERIALS AND METHODS

RESULTS

Inter‐Rater Reliability

The kappa coefficients of inter‐rater reliability between the radiologists across the 6 clinical measures of interest are displayed in Table 1. As shown, the most reliable measure was that of alveolar infiltrate (Figure 1), which attained a substantial degree of agreement between the radiologists. Two other measures, any infiltrate and pleural effusion, attained moderate reliability, while bronchograms and hilar adenopathy were each classified as having fair reliability. However, interstitial infiltrate (Figure 2) was found to have the lowest kappa estimate, with a slight degree of reliability. When examining inter‐rater reliability among the radiologists separately from each institution, the pattern of results was similar.

Table 1.Inter‐Rater Reliability of Radiologists (n = 6) Evaluating Chest Radiographs in Children Presenting to the ED With Suspected Pneumonia (n = 100)

All Radiologists (n = 6)	Kappa	95% Confidence Interval
Abbreviation: ED, emergency department.
Any infiltrate	0.47	0.39, 0.56
Alveolar infiltrate	0.69	0.60, 0.78
Interstitial infiltrate	0.14	0.05, 0.23
Air bronchograms	0.32	0.24, 0.42
Hilar adenopathy	0.21	0.08, 0.39
Pleural effusion	0.45	0.29, 0.61

Figure 1

Figure 2

At least 4 of the 6 radiologists agreed on the presence or absence of an alveolar infiltrate for 95 of the 100 unique CXRs; all 6 radiologists agreed regarding the presence or absence of an alveolar infiltrate in 72 of the 100 unique CXRs. At least 4 of the 6 radiologists agreed on the presence or absence of any infiltrate and interstitial infiltrate 96% and 90% of the time, respectively. All 6 of the radiologists agreed on the presence or absence of any infiltrate and interstitial infiltrate 35% and 27% of the time, respectively.

DISCUSSION

The chest radiograph serves as an integral component of the reference standard for the diagnosis of childhood pneumonia. Few prior studies have assessed the reliability of chest radiograph findings in children.3, 5, 12, 14, 15 We found a high degree of agreement among radiologists for radiologic findings consistent with bacterial pneumonia when standardized interpretation criteria were applied. In this study, we identified radiographic features of pneumonia, such as alveolar infiltrate and pleural effusion, that were consistently identified by different radiologists reviewing the same radiograph and by the same radiologist reviewing the same radiograph. These data support the notion that radiographic features most suggestive of bacterial pneumonia are consistently identified by radiologists.16, 17 There was less consistency in the identification of other radiographic findings, such as interstitial infiltrates, air bronchograms, and hilar lymphadenopathy.

Prior studies have found high levels of disagreement among radiologists in the interpretation of chest radiographs.2, 3, 15, 18 Many of these prior studies emphasized variation in detection of radiographic findings that would not typically alter clinical management. We observed high intra‐rater, and inter‐rater reliability among radiologists for the findings of alveolar infiltrate and pleural effusion. These are the radiographic findings most consistent with a bacterial etiologic agent for pneumonia.19 Other studies have also found that the presence of an alveolar infiltrate is a reliable radiographic finding in children18 and adults.7, 9, 10 These findings support the use of the WHO definition of primary endpoint pneumonia for use in epidemiologic studies.4, 6, 11

This study also confirms a previous report by Cherian et al. that findings of many children with asthma, reactive airways disease, bronchiolitis, and viral infections interstitial infiltrates are less reliable.6 This is not surprising considering the fact that these patients often have radiographic findings due to small airway disease and atelectasis.19, 20 The differentiation between atelectasis and interstitial infiltrate is difficult, particularly in young children. A prior study conducted among neonates observed wide variability in the interpretation of chest radiographs, and that the differentiation of pneumonia from atelectasis was difficult for this patient population.5 The decisions around antimicrobial treatment of children with radiographic findings of interstitial infiltrates should be made in the context of the clinical history and physical examination findings, and clinicians should realize that these radiographic features demonstrate poor reliability for the diagnosis of pneumonia.

Overall reliability for the presence of any infiltrate, and its converse, no infiltrate was considered moderate. This is driven by the low reliability and variability around the radiographic diagnosis of interstitial infiltrates. Our findings are similar to those observed in adults with lower respiratory tract infections.9 The low reliability in identification of interstitial infiltrates may explain why prior studies have demonstrated that the CXR results rarely change management in children who have radiographs performed for suspicion of pneumonia.1, 21 Our study highlights the importance of quantifying CXR findings to include specific comments regarding the presence or absence of alveolar infiltrates, rather than the presence or absence of any infiltrate.

The WHO has procured definitions the radiographic diagnosis of pneumonia, and this definition has been utilized to help standardize the interpretation of chest radiographs for the conduct of epidemiological studies.6, 11 Specifically, the definitions utilized not only define the presence or absence of pneumonia, but also attempt to differentiate a primarily bacterial infection (consolidation or pleural effusion), from a viral or atypical presentation (interstitial pattern). Even under the best of circumstances, the differentiation of viral versus bacterial pneumonia is not always possible, and again, is often made by the treating physician by incorporating the clinical setting within which the radiograph was obtained.

This study had several limitations. Firstly, the included radiographs did not reflect the frequency with which certain radiographic findings would be identified in children evaluated for pneumonia in a pediatric emergency department setting. Radiographs were purposefully selected to encompass a broad spectrum of radiologic findings, including less common findings such as hilar lymphadenopathy and pleural effusions. Thus, the prevalence of pneumonia and other abnormal findings in this study was artificially higher than typically observed among a cohort of children for whom pneumonia is considered, a factor that may limit the generalizability of our results. Secondly, the clinical history was not provided to the radiologists to avoid bias by indication. For this study, we notified the radiologists that all radiographs were performed for clinical suspicion of pneumonia without providing details about the subjects' signs and symptoms. The absence of clinical history, however, does not mirror the real world scenario in which the interpretation of the chest radiograph is frequently made in the context of the clinical history. The relevance of this latter issue is unclear, as Tudor et al. found a nonstatistically significant improvement in the overall accuracy in chest radiograph interpretation when radiologists were provided clinical details.10 The radiologists recruited for this study all practice in an academic children's hospital setting, and thus, the generalizability of our findings may be limited to this type of practice setting. Finally, reproducibility does not imply accuracy, and reliability in identifying specific findings does not necessarily lead to improved or different management. Thus, while the reliability of radiographic findings of alveolar infiltrate and pleural effusion is reassuringly high, the validity of these radiographic features for bacterial pneumonia is not known. Ascertainment of validity can only be assessed through the use of invasive testing such as lung biopsy, as the yield from bacterial testing such as blood cultures is low, and the results of other studies such as viral testing of nasopharyngeal washings do not prove an etiologic cause of pneumonia.

CONCLUSIONS

Radiographic findings of alveolar infiltrates and pleural effusions are highly reliable among radiologists. Radiographic interpretation of interstitial infiltrates appears to be less reliable.

The chest radiograph (CXR) is the most commonly used diagnostic imaging modality in children, and is considered to be the gold standard for the diagnosis of pneumonia. As such, physicians in developed countries rely on chest radiography to establish the diagnosis of pneumonia.13 However, there are limited data investigating the reliability of this test for the diagnosis of pneumonia in children.2, 46

Prior investigations have noted poor overall agreement by emergency medicine, infectious diseases, and pulmonary medicine physicians, and even radiologists, in their interpretation of chest radiographs for the diagnosis of pneumonia.2, 5, 710 The World Health Organization (WHO) developed criteria to standardize CXR interpretation for the diagnosis of pneumonia in children for use in epidemiologic studies.11 These standardized definitions of pneumonia have been formally evaluated by the WHO6 and utilized in epidemiologic studies of vaccine efficacy,12 but the overall reliability of these radiographic criteria have not been studied outside of these forums.

We conducted this prospective case‐based study to evaluate the reliability of the radiographic diagnosis of pneumonia among children presenting to a pediatric emergency department with clinical suspicion of pneumonia. We were primarily interested in assessing the overall reliability in CXR interpretation for the diagnosis of pneumonia, and identifying which radiographic features of pneumonia were consistently identified by radiologists.

MATERIALS AND METHODS

RESULTS

Inter‐Rater Reliability

The kappa coefficients of inter‐rater reliability between the radiologists across the 6 clinical measures of interest are displayed in Table 1. As shown, the most reliable measure was that of alveolar infiltrate (Figure 1), which attained a substantial degree of agreement between the radiologists. Two other measures, any infiltrate and pleural effusion, attained moderate reliability, while bronchograms and hilar adenopathy were each classified as having fair reliability. However, interstitial infiltrate (Figure 2) was found to have the lowest kappa estimate, with a slight degree of reliability. When examining inter‐rater reliability among the radiologists separately from each institution, the pattern of results was similar.

Table 1.Inter‐Rater Reliability of Radiologists (n = 6) Evaluating Chest Radiographs in Children Presenting to the ED With Suspected Pneumonia (n = 100)

All Radiologists (n = 6)	Kappa	95% Confidence Interval
Abbreviation: ED, emergency department.
Any infiltrate	0.47	0.39, 0.56
Alveolar infiltrate	0.69	0.60, 0.78
Interstitial infiltrate	0.14	0.05, 0.23
Air bronchograms	0.32	0.24, 0.42
Hilar adenopathy	0.21	0.08, 0.39
Pleural effusion	0.45	0.29, 0.61

Figure 1

Figure 2

At least 4 of the 6 radiologists agreed on the presence or absence of an alveolar infiltrate for 95 of the 100 unique CXRs; all 6 radiologists agreed regarding the presence or absence of an alveolar infiltrate in 72 of the 100 unique CXRs. At least 4 of the 6 radiologists agreed on the presence or absence of any infiltrate and interstitial infiltrate 96% and 90% of the time, respectively. All 6 of the radiologists agreed on the presence or absence of any infiltrate and interstitial infiltrate 35% and 27% of the time, respectively.

DISCUSSION

The chest radiograph serves as an integral component of the reference standard for the diagnosis of childhood pneumonia. Few prior studies have assessed the reliability of chest radiograph findings in children.3, 5, 12, 14, 15 We found a high degree of agreement among radiologists for radiologic findings consistent with bacterial pneumonia when standardized interpretation criteria were applied. In this study, we identified radiographic features of pneumonia, such as alveolar infiltrate and pleural effusion, that were consistently identified by different radiologists reviewing the same radiograph and by the same radiologist reviewing the same radiograph. These data support the notion that radiographic features most suggestive of bacterial pneumonia are consistently identified by radiologists.16, 17 There was less consistency in the identification of other radiographic findings, such as interstitial infiltrates, air bronchograms, and hilar lymphadenopathy.

Prior studies have found high levels of disagreement among radiologists in the interpretation of chest radiographs.2, 3, 15, 18 Many of these prior studies emphasized variation in detection of radiographic findings that would not typically alter clinical management. We observed high intra‐rater, and inter‐rater reliability among radiologists for the findings of alveolar infiltrate and pleural effusion. These are the radiographic findings most consistent with a bacterial etiologic agent for pneumonia.19 Other studies have also found that the presence of an alveolar infiltrate is a reliable radiographic finding in children18 and adults.7, 9, 10 These findings support the use of the WHO definition of primary endpoint pneumonia for use in epidemiologic studies.4, 6, 11

This study also confirms a previous report by Cherian et al. that findings of many children with asthma, reactive airways disease, bronchiolitis, and viral infections interstitial infiltrates are less reliable.6 This is not surprising considering the fact that these patients often have radiographic findings due to small airway disease and atelectasis.19, 20 The differentiation between atelectasis and interstitial infiltrate is difficult, particularly in young children. A prior study conducted among neonates observed wide variability in the interpretation of chest radiographs, and that the differentiation of pneumonia from atelectasis was difficult for this patient population.5 The decisions around antimicrobial treatment of children with radiographic findings of interstitial infiltrates should be made in the context of the clinical history and physical examination findings, and clinicians should realize that these radiographic features demonstrate poor reliability for the diagnosis of pneumonia.

Overall reliability for the presence of any infiltrate, and its converse, no infiltrate was considered moderate. This is driven by the low reliability and variability around the radiographic diagnosis of interstitial infiltrates. Our findings are similar to those observed in adults with lower respiratory tract infections.9 The low reliability in identification of interstitial infiltrates may explain why prior studies have demonstrated that the CXR results rarely change management in children who have radiographs performed for suspicion of pneumonia.1, 21 Our study highlights the importance of quantifying CXR findings to include specific comments regarding the presence or absence of alveolar infiltrates, rather than the presence or absence of any infiltrate.

The WHO has procured definitions the radiographic diagnosis of pneumonia, and this definition has been utilized to help standardize the interpretation of chest radiographs for the conduct of epidemiological studies.6, 11 Specifically, the definitions utilized not only define the presence or absence of pneumonia, but also attempt to differentiate a primarily bacterial infection (consolidation or pleural effusion), from a viral or atypical presentation (interstitial pattern). Even under the best of circumstances, the differentiation of viral versus bacterial pneumonia is not always possible, and again, is often made by the treating physician by incorporating the clinical setting within which the radiograph was obtained.

This study had several limitations. Firstly, the included radiographs did not reflect the frequency with which certain radiographic findings would be identified in children evaluated for pneumonia in a pediatric emergency department setting. Radiographs were purposefully selected to encompass a broad spectrum of radiologic findings, including less common findings such as hilar lymphadenopathy and pleural effusions. Thus, the prevalence of pneumonia and other abnormal findings in this study was artificially higher than typically observed among a cohort of children for whom pneumonia is considered, a factor that may limit the generalizability of our results. Secondly, the clinical history was not provided to the radiologists to avoid bias by indication. For this study, we notified the radiologists that all radiographs were performed for clinical suspicion of pneumonia without providing details about the subjects' signs and symptoms. The absence of clinical history, however, does not mirror the real world scenario in which the interpretation of the chest radiograph is frequently made in the context of the clinical history. The relevance of this latter issue is unclear, as Tudor et al. found a nonstatistically significant improvement in the overall accuracy in chest radiograph interpretation when radiologists were provided clinical details.10 The radiologists recruited for this study all practice in an academic children's hospital setting, and thus, the generalizability of our findings may be limited to this type of practice setting. Finally, reproducibility does not imply accuracy, and reliability in identifying specific findings does not necessarily lead to improved or different management. Thus, while the reliability of radiographic findings of alveolar infiltrate and pleural effusion is reassuringly high, the validity of these radiographic features for bacterial pneumonia is not known. Ascertainment of validity can only be assessed through the use of invasive testing such as lung biopsy, as the yield from bacterial testing such as blood cultures is low, and the results of other studies such as viral testing of nasopharyngeal washings do not prove an etiologic cause of pneumonia.

CONCLUSIONS

Radiographic findings of alveolar infiltrates and pleural effusions are highly reliable among radiologists. Radiographic interpretation of interstitial infiltrates appears to be less reliable.

The chest radiograph (CXR) is the most commonly used diagnostic imaging modality in children, and is considered to be the gold standard for the diagnosis of pneumonia. As such, physicians in developed countries rely on chest radiography to establish the diagnosis of pneumonia.13 However, there are limited data investigating the reliability of this test for the diagnosis of pneumonia in children.2, 46

Prior investigations have noted poor overall agreement by emergency medicine, infectious diseases, and pulmonary medicine physicians, and even radiologists, in their interpretation of chest radiographs for the diagnosis of pneumonia.2, 5, 710 The World Health Organization (WHO) developed criteria to standardize CXR interpretation for the diagnosis of pneumonia in children for use in epidemiologic studies.11 These standardized definitions of pneumonia have been formally evaluated by the WHO6 and utilized in epidemiologic studies of vaccine efficacy,12 but the overall reliability of these radiographic criteria have not been studied outside of these forums.

We conducted this prospective case‐based study to evaluate the reliability of the radiographic diagnosis of pneumonia among children presenting to a pediatric emergency department with clinical suspicion of pneumonia. We were primarily interested in assessing the overall reliability in CXR interpretation for the diagnosis of pneumonia, and identifying which radiographic features of pneumonia were consistently identified by radiologists.

MATERIALS AND METHODS

RESULTS

Inter‐Rater Reliability

The kappa coefficients of inter‐rater reliability between the radiologists across the 6 clinical measures of interest are displayed in Table 1. As shown, the most reliable measure was that of alveolar infiltrate (Figure 1), which attained a substantial degree of agreement between the radiologists. Two other measures, any infiltrate and pleural effusion, attained moderate reliability, while bronchograms and hilar adenopathy were each classified as having fair reliability. However, interstitial infiltrate (Figure 2) was found to have the lowest kappa estimate, with a slight degree of reliability. When examining inter‐rater reliability among the radiologists separately from each institution, the pattern of results was similar.

Table 1.Inter‐Rater Reliability of Radiologists (n = 6) Evaluating Chest Radiographs in Children Presenting to the ED With Suspected Pneumonia (n = 100)

All Radiologists (n = 6)	Kappa	95% Confidence Interval
Abbreviation: ED, emergency department.
Any infiltrate	0.47	0.39, 0.56
Alveolar infiltrate	0.69	0.60, 0.78
Interstitial infiltrate	0.14	0.05, 0.23
Air bronchograms	0.32	0.24, 0.42
Hilar adenopathy	0.21	0.08, 0.39
Pleural effusion	0.45	0.29, 0.61

Figure 1

Figure 2

At least 4 of the 6 radiologists agreed on the presence or absence of an alveolar infiltrate for 95 of the 100 unique CXRs; all 6 radiologists agreed regarding the presence or absence of an alveolar infiltrate in 72 of the 100 unique CXRs. At least 4 of the 6 radiologists agreed on the presence or absence of any infiltrate and interstitial infiltrate 96% and 90% of the time, respectively. All 6 of the radiologists agreed on the presence or absence of any infiltrate and interstitial infiltrate 35% and 27% of the time, respectively.

DISCUSSION

The chest radiograph serves as an integral component of the reference standard for the diagnosis of childhood pneumonia. Few prior studies have assessed the reliability of chest radiograph findings in children.3, 5, 12, 14, 15 We found a high degree of agreement among radiologists for radiologic findings consistent with bacterial pneumonia when standardized interpretation criteria were applied. In this study, we identified radiographic features of pneumonia, such as alveolar infiltrate and pleural effusion, that were consistently identified by different radiologists reviewing the same radiograph and by the same radiologist reviewing the same radiograph. These data support the notion that radiographic features most suggestive of bacterial pneumonia are consistently identified by radiologists.16, 17 There was less consistency in the identification of other radiographic findings, such as interstitial infiltrates, air bronchograms, and hilar lymphadenopathy.

Prior studies have found high levels of disagreement among radiologists in the interpretation of chest radiographs.2, 3, 15, 18 Many of these prior studies emphasized variation in detection of radiographic findings that would not typically alter clinical management. We observed high intra‐rater, and inter‐rater reliability among radiologists for the findings of alveolar infiltrate and pleural effusion. These are the radiographic findings most consistent with a bacterial etiologic agent for pneumonia.19 Other studies have also found that the presence of an alveolar infiltrate is a reliable radiographic finding in children18 and adults.7, 9, 10 These findings support the use of the WHO definition of primary endpoint pneumonia for use in epidemiologic studies.4, 6, 11

This study also confirms a previous report by Cherian et al. that findings of many children with asthma, reactive airways disease, bronchiolitis, and viral infections interstitial infiltrates are less reliable.6 This is not surprising considering the fact that these patients often have radiographic findings due to small airway disease and atelectasis.19, 20 The differentiation between atelectasis and interstitial infiltrate is difficult, particularly in young children. A prior study conducted among neonates observed wide variability in the interpretation of chest radiographs, and that the differentiation of pneumonia from atelectasis was difficult for this patient population.5 The decisions around antimicrobial treatment of children with radiographic findings of interstitial infiltrates should be made in the context of the clinical history and physical examination findings, and clinicians should realize that these radiographic features demonstrate poor reliability for the diagnosis of pneumonia.

Overall reliability for the presence of any infiltrate, and its converse, no infiltrate was considered moderate. This is driven by the low reliability and variability around the radiographic diagnosis of interstitial infiltrates. Our findings are similar to those observed in adults with lower respiratory tract infections.9 The low reliability in identification of interstitial infiltrates may explain why prior studies have demonstrated that the CXR results rarely change management in children who have radiographs performed for suspicion of pneumonia.1, 21 Our study highlights the importance of quantifying CXR findings to include specific comments regarding the presence or absence of alveolar infiltrates, rather than the presence or absence of any infiltrate.

The WHO has procured definitions the radiographic diagnosis of pneumonia, and this definition has been utilized to help standardize the interpretation of chest radiographs for the conduct of epidemiological studies.6, 11 Specifically, the definitions utilized not only define the presence or absence of pneumonia, but also attempt to differentiate a primarily bacterial infection (consolidation or pleural effusion), from a viral or atypical presentation (interstitial pattern). Even under the best of circumstances, the differentiation of viral versus bacterial pneumonia is not always possible, and again, is often made by the treating physician by incorporating the clinical setting within which the radiograph was obtained.

This study had several limitations. Firstly, the included radiographs did not reflect the frequency with which certain radiographic findings would be identified in children evaluated for pneumonia in a pediatric emergency department setting. Radiographs were purposefully selected to encompass a broad spectrum of radiologic findings, including less common findings such as hilar lymphadenopathy and pleural effusions. Thus, the prevalence of pneumonia and other abnormal findings in this study was artificially higher than typically observed among a cohort of children for whom pneumonia is considered, a factor that may limit the generalizability of our results. Secondly, the clinical history was not provided to the radiologists to avoid bias by indication. For this study, we notified the radiologists that all radiographs were performed for clinical suspicion of pneumonia without providing details about the subjects' signs and symptoms. The absence of clinical history, however, does not mirror the real world scenario in which the interpretation of the chest radiograph is frequently made in the context of the clinical history. The relevance of this latter issue is unclear, as Tudor et al. found a nonstatistically significant improvement in the overall accuracy in chest radiograph interpretation when radiologists were provided clinical details.10 The radiologists recruited for this study all practice in an academic children's hospital setting, and thus, the generalizability of our findings may be limited to this type of practice setting. Finally, reproducibility does not imply accuracy, and reliability in identifying specific findings does not necessarily lead to improved or different management. Thus, while the reliability of radiographic findings of alveolar infiltrate and pleural effusion is reassuringly high, the validity of these radiographic features for bacterial pneumonia is not known. Ascertainment of validity can only be assessed through the use of invasive testing such as lung biopsy, as the yield from bacterial testing such as blood cultures is low, and the results of other studies such as viral testing of nasopharyngeal washings do not prove an etiologic cause of pneumonia.

CONCLUSIONS

Radiographic findings of alveolar infiltrates and pleural effusions are highly reliable among radiologists. Radiographic interpretation of interstitial infiltrates appears to be less reliable.