Matthew Hall, PhD

The ability to read and correctly interpret research is an essential skill, but most hospitalists—and physicians in general—do not receive formal training in biostatistics during their medical education.^1-3 In addition to straightforward statistical tests that compare a single exposure and outcome, researchers commonly use statistical models to identify and quantify complex relationships among many exposures (eg, demographics, clinical characteristics, interventions, or other variables) and an outcome. Understanding statistical models can be challenging. Still, it is important to recognize the advantages and limitations of statistical models, how to interpret their results, and the potential implications of findings on current clinical practice.

In the article “Rates and Characteristics of Medical Malpractice Claims Against Hospitalists” published in the July 2021 issue of the Journal of Hospital Medicine, Schaffer et al⁴ used the Comparative Benchmarking System database, which is maintained by a malpractice insurer, to characterize malpractice claims against hospitalists. The authors used multiple logistic regression models to understand the relationship among clinical factors and indemnity payments. In this Progress Note, we describe situations in which logistic regression is the proper statistical method to analyze a data set, explain results from logistic regression analyses, and equip readers with skills to critically appraise conclusions drawn from these models.

Statistical models often are used to describe the relationship among one or more exposure variables (ie, independent variables) and an outcome (ie, dependent variable). These models allow researchers to evaluate the effects of multiple exposure variables simultaneously, which in turn allows them to “isolate” the effect of each variable; in other words, models facilitate an understanding of the relationship between each exposure variable and the outcome, adjusted for (ie, independent of) the other exposure variables in the model.

Several statistical models can be used to quantify relationships within the data, but each type of model has certain assumptions that must be satisfied. Two important assumptions include characteristics of the outcome (eg, the type and distribution) and the nature of the relationships among the outcome and independent variables (eg, linear vs nonlinear). Simple linear regression, one of the most basic statistical models used in research,⁵ assumes that (a) the outcome is continuous (ie, any numeric value is possible) and normally distributed (ie, its histogram is a bell-shaped curve) and (b) the relationship between the independent variable and the outcome is linear (ie, follows a straight line). If an investigator wanted to understand how weight is related to height, a simple linear regression could be used to develop a mathematical equation that tells us how the outcome (weight) generally increases as the independent variable (height) increases.

Often, the outcome in a study is not a continuous variable but a simple success/failure variable (ie, dichotomous variable that can be one of two possible values). Schaffer et al⁴ examined the binary outcome of whether a malpractice claim case would end in an indemnity payment or no payment. Linear regression models are not equipped to handle dichotomous outcomes. Instead, we need to use a different statistical model: logistic regression. In logistic regression, the probability (p) of a defined outcome event is estimated by creating a regression model.

A probability (p) is a measure of how likely an event (eg, a malpractice claim ends in an indemnity payment or not) is to occur. It is always between 0 (ie, the event will definitely not occur) and 1 (ie, the event will definitely occur). A p of 0.5 means there is a 50/50 chance that the event will occur (ie, equivalent to a coin flip). Because p is a probability, we need to make sure it is always between 0 and 1. If we were to try to model p with a linear regression, the model would assume that p could extend beyond 0 and 1. What can we do?

Applying a transformation is a commonly used tool in statistics to make data work better within statistical models.⁶ In this case, we will transform the variable p. In logistic regression, we model the probability of experiencing the outcome through a transformation called a logit. The logit represents the natural logarithm (ln) of the ratio of the probability of experiencing the outcome (p) vs the probability of not experiencing the outcome (1 – p), with the ratio being the odds of the event occurring.

bettenhausen0393-1020e-f1.jpg

This transformation works well for dichotomous outcomes because the logit transformation approximates a straight line as long as p is not too large or too small (between 0.05 and 0.95).

If we are performing a logistic regression with only one independent variable (x) and want to understand the relationship between this variable (x) and the probability of an outcome event (p), then our model is the equation of a line. The equation for the base model of logistic regression with one independent variable (x) is

bettenhausen0393-1020e-f2.jpg

where β₀ is the y-intercept and β₁ is the slope of the line. Equation (2) is identical to the algebraic equation y = mx + b for a line, just rearranged slightly. In this algebraic equation, m is the slope (the same as β₁) and b is the y-intercept (the same as β₀). We will see that β₀ and β₁ are estimated (ie, assigned numeric values) from the data collected to help us understand how x and

bettenhausen0393-1020e-f3.jpg

are related and are the basis for estimating odds ratios.

We can build more complex models using multivariable logistic regression by adding more independent variables to the right side of equation (2). Essentially, this is what Schaffer et al⁴ did when, for example, they described clinical factors associated with indemnity payments (Schaffer et al, Table 3).

There are two notable techniques used frequently with multivariable logistic regression models. The first involves choosing which independent variables to include in the model. One way to select variables for multivariable models is defining them a priori, that is deciding which variables are clinically or conceptually associated with the outcome before looking at the data. With this approach, we can test specific hypotheses about the relationships between the independent variables and the outcome. Another common approach is to look at the data and identify the variables that vary significantly between the two outcome groups. Schaffer et al⁴ used an a priori approach to define variables in their multivariable model (ie, “variables for inclusion into the multivariable model were determined a priori”).

A second technique is the evaluation of collinearity, which helps us understand whether the independent variables are related to each other. It is important to consider collinearity between independent variables because the inclusion of two (or more) variables that are highly correlated can cause interference between the two and create misleading results from the model. There are techniques to assess collinear relationships before modeling or as part of the model-building process to determine which variables should be excluded. If there are two (or more) independent variables that are similar, one (or more) must be removed from the model.

Fitting the model is the process by which statistical software (eg, SAS, Stata, R, SPSS) estimates the relationships among independent variables in the model and the outcome within a specific dataset. In equation (2), this essentially means that the software will evaluate the data and provide us with the best estimates for β₀ (the y-intercept) and β₁ (the slope) that describe the relationship between the variable x and

bettenhausen0393-1020e-f4.jpg

Modeling can be iterative, and part of the process may include removing variables from the model that are not significantly associated with the outcome to create a simpler solution, a process known as model reduction. The results from models describe the independent association between a specific characteristic and the outcome, meaning that the relationship has been adjusted for all the other characteristics in the model.

The relationships among the independent variables and outcome are most often represented as an odds ratio (OR), which quantifies the strength of the association between two variables and is directly calculated from the β values in the model. As the name suggests, an OR is a ratio of odds. But what are odds? Simply, the odds of an outcome (such as mortality) is the probability of experiencing the event divided by the probability of not experiencing that event; in other words, it is the ratio:

bettenhausen0393-1020e-f5.jpg

The concept of odds is often unfamiliar, so it can be helpful to consider the definition in the context of games of chance. For example, in horse race betting, the outcome of interest is that a horse will lose a race. Imagine that the probability of a horse losing a race is 0.8 and the probability of winning is 0.2. The odds of losing are

bettenhausen0393-1020e-f6.jpg

These odds usually are listed as 4-to-1, meaning that out of 5 races (ie, 4 + 1) the horse is expected to lose 4 times and win once. When odds are listed this way, we can easily calculate the associated probability by recognizing that the total number of expected races is the sum of two numbers (probability of losing: 4 races out of 5, or 0.80 vs probability of winning: 1 race out of 5, or 0.20).

In medical research, the OR typically represents the odds for one group of patients (A) compared with the odds for another group of patients (B) experiencing an outcome. If the odds of the outcome are the same for group A and group B, then OR = 1.0, meaning that the probability of the outcome is the same between the two groups. If the patients in group A have greater odds of experiencing the outcome compared with group B patients (and a greater probability of the outcome), then the OR will be >1. If the opposite is true, then the OR will be <1.

Schaffer et al⁴ estimated that the OR of an indemnity payment in malpractice cases involving errors in clinical judgment as a contributing factor was 5.01 (95% CI, 3.37-7.45). This means that malpractice cases involving errors in clinical judgement had a 5.01 times greater odds of indemnity payment compared with those without these errors after adjusting for all other variables in the model (eg, age, severity). Note that the 95% CI does not include 1.0. This indicates that the OR is statistically >1, and we can conclude that there is a significant relationship between errors in clinical judgment and payment that is unlikely to be attributed to chance alone.

In logistic regression for categorical independent variables, all categories are compared with a reference group within that variable, with the reference group serving as the denominator of the OR. The authors⁴ did not incorporate continuous independent variables in their multivariable logistic regression model. However, if the authors examined length of hospitalization as a contributing factor in indemnity payments, for example, the OR would represent a 1-unit increase in this variable (eg, 1-day increase in length of stay).

Logistic regression describes the relationships in data and is an important statistical model across many types of research. This Progress Note emphasizes the importance of weighing the advantages and limitations of logistic regression, provides a common approach to data transformation, and guides the correct interpretation of logistic regression model results.

The ability to read and correctly interpret research is an essential skill, but most hospitalists—and physicians in general—do not receive formal training in biostatistics during their medical education.^1-3 In addition to straightforward statistical tests that compare a single exposure and outcome, researchers commonly use statistical models to identify and quantify complex relationships among many exposures (eg, demographics, clinical characteristics, interventions, or other variables) and an outcome. Understanding statistical models can be challenging. Still, it is important to recognize the advantages and limitations of statistical models, how to interpret their results, and the potential implications of findings on current clinical practice.

In the article “Rates and Characteristics of Medical Malpractice Claims Against Hospitalists” published in the July 2021 issue of the Journal of Hospital Medicine, Schaffer et al⁴ used the Comparative Benchmarking System database, which is maintained by a malpractice insurer, to characterize malpractice claims against hospitalists. The authors used multiple logistic regression models to understand the relationship among clinical factors and indemnity payments. In this Progress Note, we describe situations in which logistic regression is the proper statistical method to analyze a data set, explain results from logistic regression analyses, and equip readers with skills to critically appraise conclusions drawn from these models.

Statistical models often are used to describe the relationship among one or more exposure variables (ie, independent variables) and an outcome (ie, dependent variable). These models allow researchers to evaluate the effects of multiple exposure variables simultaneously, which in turn allows them to “isolate” the effect of each variable; in other words, models facilitate an understanding of the relationship between each exposure variable and the outcome, adjusted for (ie, independent of) the other exposure variables in the model.

Several statistical models can be used to quantify relationships within the data, but each type of model has certain assumptions that must be satisfied. Two important assumptions include characteristics of the outcome (eg, the type and distribution) and the nature of the relationships among the outcome and independent variables (eg, linear vs nonlinear). Simple linear regression, one of the most basic statistical models used in research,⁵ assumes that (a) the outcome is continuous (ie, any numeric value is possible) and normally distributed (ie, its histogram is a bell-shaped curve) and (b) the relationship between the independent variable and the outcome is linear (ie, follows a straight line). If an investigator wanted to understand how weight is related to height, a simple linear regression could be used to develop a mathematical equation that tells us how the outcome (weight) generally increases as the independent variable (height) increases.

Often, the outcome in a study is not a continuous variable but a simple success/failure variable (ie, dichotomous variable that can be one of two possible values). Schaffer et al⁴ examined the binary outcome of whether a malpractice claim case would end in an indemnity payment or no payment. Linear regression models are not equipped to handle dichotomous outcomes. Instead, we need to use a different statistical model: logistic regression. In logistic regression, the probability (p) of a defined outcome event is estimated by creating a regression model.

A probability (p) is a measure of how likely an event (eg, a malpractice claim ends in an indemnity payment or not) is to occur. It is always between 0 (ie, the event will definitely not occur) and 1 (ie, the event will definitely occur). A p of 0.5 means there is a 50/50 chance that the event will occur (ie, equivalent to a coin flip). Because p is a probability, we need to make sure it is always between 0 and 1. If we were to try to model p with a linear regression, the model would assume that p could extend beyond 0 and 1. What can we do?

Applying a transformation is a commonly used tool in statistics to make data work better within statistical models.⁶ In this case, we will transform the variable p. In logistic regression, we model the probability of experiencing the outcome through a transformation called a logit. The logit represents the natural logarithm (ln) of the ratio of the probability of experiencing the outcome (p) vs the probability of not experiencing the outcome (1 – p), with the ratio being the odds of the event occurring.

bettenhausen0393-1020e-f1.jpg

This transformation works well for dichotomous outcomes because the logit transformation approximates a straight line as long as p is not too large or too small (between 0.05 and 0.95).

If we are performing a logistic regression with only one independent variable (x) and want to understand the relationship between this variable (x) and the probability of an outcome event (p), then our model is the equation of a line. The equation for the base model of logistic regression with one independent variable (x) is

bettenhausen0393-1020e-f2.jpg

where β₀ is the y-intercept and β₁ is the slope of the line. Equation (2) is identical to the algebraic equation y = mx + b for a line, just rearranged slightly. In this algebraic equation, m is the slope (the same as β₁) and b is the y-intercept (the same as β₀). We will see that β₀ and β₁ are estimated (ie, assigned numeric values) from the data collected to help us understand how x and

bettenhausen0393-1020e-f3.jpg

are related and are the basis for estimating odds ratios.

We can build more complex models using multivariable logistic regression by adding more independent variables to the right side of equation (2). Essentially, this is what Schaffer et al⁴ did when, for example, they described clinical factors associated with indemnity payments (Schaffer et al, Table 3).

There are two notable techniques used frequently with multivariable logistic regression models. The first involves choosing which independent variables to include in the model. One way to select variables for multivariable models is defining them a priori, that is deciding which variables are clinically or conceptually associated with the outcome before looking at the data. With this approach, we can test specific hypotheses about the relationships between the independent variables and the outcome. Another common approach is to look at the data and identify the variables that vary significantly between the two outcome groups. Schaffer et al⁴ used an a priori approach to define variables in their multivariable model (ie, “variables for inclusion into the multivariable model were determined a priori”).

A second technique is the evaluation of collinearity, which helps us understand whether the independent variables are related to each other. It is important to consider collinearity between independent variables because the inclusion of two (or more) variables that are highly correlated can cause interference between the two and create misleading results from the model. There are techniques to assess collinear relationships before modeling or as part of the model-building process to determine which variables should be excluded. If there are two (or more) independent variables that are similar, one (or more) must be removed from the model.

Fitting the model is the process by which statistical software (eg, SAS, Stata, R, SPSS) estimates the relationships among independent variables in the model and the outcome within a specific dataset. In equation (2), this essentially means that the software will evaluate the data and provide us with the best estimates for β₀ (the y-intercept) and β₁ (the slope) that describe the relationship between the variable x and

bettenhausen0393-1020e-f4.jpg

Modeling can be iterative, and part of the process may include removing variables from the model that are not significantly associated with the outcome to create a simpler solution, a process known as model reduction. The results from models describe the independent association between a specific characteristic and the outcome, meaning that the relationship has been adjusted for all the other characteristics in the model.

The relationships among the independent variables and outcome are most often represented as an odds ratio (OR), which quantifies the strength of the association between two variables and is directly calculated from the β values in the model. As the name suggests, an OR is a ratio of odds. But what are odds? Simply, the odds of an outcome (such as mortality) is the probability of experiencing the event divided by the probability of not experiencing that event; in other words, it is the ratio:

bettenhausen0393-1020e-f5.jpg

The concept of odds is often unfamiliar, so it can be helpful to consider the definition in the context of games of chance. For example, in horse race betting, the outcome of interest is that a horse will lose a race. Imagine that the probability of a horse losing a race is 0.8 and the probability of winning is 0.2. The odds of losing are

bettenhausen0393-1020e-f6.jpg

These odds usually are listed as 4-to-1, meaning that out of 5 races (ie, 4 + 1) the horse is expected to lose 4 times and win once. When odds are listed this way, we can easily calculate the associated probability by recognizing that the total number of expected races is the sum of two numbers (probability of losing: 4 races out of 5, or 0.80 vs probability of winning: 1 race out of 5, or 0.20).

In medical research, the OR typically represents the odds for one group of patients (A) compared with the odds for another group of patients (B) experiencing an outcome. If the odds of the outcome are the same for group A and group B, then OR = 1.0, meaning that the probability of the outcome is the same between the two groups. If the patients in group A have greater odds of experiencing the outcome compared with group B patients (and a greater probability of the outcome), then the OR will be >1. If the opposite is true, then the OR will be <1.

Schaffer et al⁴ estimated that the OR of an indemnity payment in malpractice cases involving errors in clinical judgment as a contributing factor was 5.01 (95% CI, 3.37-7.45). This means that malpractice cases involving errors in clinical judgement had a 5.01 times greater odds of indemnity payment compared with those without these errors after adjusting for all other variables in the model (eg, age, severity). Note that the 95% CI does not include 1.0. This indicates that the OR is statistically >1, and we can conclude that there is a significant relationship between errors in clinical judgment and payment that is unlikely to be attributed to chance alone.

In logistic regression for categorical independent variables, all categories are compared with a reference group within that variable, with the reference group serving as the denominator of the OR. The authors⁴ did not incorporate continuous independent variables in their multivariable logistic regression model. However, if the authors examined length of hospitalization as a contributing factor in indemnity payments, for example, the OR would represent a 1-unit increase in this variable (eg, 1-day increase in length of stay).

Logistic regression describes the relationships in data and is an important statistical model across many types of research. This Progress Note emphasizes the importance of weighing the advantages and limitations of logistic regression, provides a common approach to data transformation, and guides the correct interpretation of logistic regression model results.

The ability to read and correctly interpret research is an essential skill, but most hospitalists—and physicians in general—do not receive formal training in biostatistics during their medical education.^1-3 In addition to straightforward statistical tests that compare a single exposure and outcome, researchers commonly use statistical models to identify and quantify complex relationships among many exposures (eg, demographics, clinical characteristics, interventions, or other variables) and an outcome. Understanding statistical models can be challenging. Still, it is important to recognize the advantages and limitations of statistical models, how to interpret their results, and the potential implications of findings on current clinical practice.

In the article “Rates and Characteristics of Medical Malpractice Claims Against Hospitalists” published in the July 2021 issue of the Journal of Hospital Medicine, Schaffer et al⁴ used the Comparative Benchmarking System database, which is maintained by a malpractice insurer, to characterize malpractice claims against hospitalists. The authors used multiple logistic regression models to understand the relationship among clinical factors and indemnity payments. In this Progress Note, we describe situations in which logistic regression is the proper statistical method to analyze a data set, explain results from logistic regression analyses, and equip readers with skills to critically appraise conclusions drawn from these models.

Statistical models often are used to describe the relationship among one or more exposure variables (ie, independent variables) and an outcome (ie, dependent variable). These models allow researchers to evaluate the effects of multiple exposure variables simultaneously, which in turn allows them to “isolate” the effect of each variable; in other words, models facilitate an understanding of the relationship between each exposure variable and the outcome, adjusted for (ie, independent of) the other exposure variables in the model.

Several statistical models can be used to quantify relationships within the data, but each type of model has certain assumptions that must be satisfied. Two important assumptions include characteristics of the outcome (eg, the type and distribution) and the nature of the relationships among the outcome and independent variables (eg, linear vs nonlinear). Simple linear regression, one of the most basic statistical models used in research,⁵ assumes that (a) the outcome is continuous (ie, any numeric value is possible) and normally distributed (ie, its histogram is a bell-shaped curve) and (b) the relationship between the independent variable and the outcome is linear (ie, follows a straight line). If an investigator wanted to understand how weight is related to height, a simple linear regression could be used to develop a mathematical equation that tells us how the outcome (weight) generally increases as the independent variable (height) increases.

Often, the outcome in a study is not a continuous variable but a simple success/failure variable (ie, dichotomous variable that can be one of two possible values). Schaffer et al⁴ examined the binary outcome of whether a malpractice claim case would end in an indemnity payment or no payment. Linear regression models are not equipped to handle dichotomous outcomes. Instead, we need to use a different statistical model: logistic regression. In logistic regression, the probability (p) of a defined outcome event is estimated by creating a regression model.

A probability (p) is a measure of how likely an event (eg, a malpractice claim ends in an indemnity payment or not) is to occur. It is always between 0 (ie, the event will definitely not occur) and 1 (ie, the event will definitely occur). A p of 0.5 means there is a 50/50 chance that the event will occur (ie, equivalent to a coin flip). Because p is a probability, we need to make sure it is always between 0 and 1. If we were to try to model p with a linear regression, the model would assume that p could extend beyond 0 and 1. What can we do?

Applying a transformation is a commonly used tool in statistics to make data work better within statistical models.⁶ In this case, we will transform the variable p. In logistic regression, we model the probability of experiencing the outcome through a transformation called a logit. The logit represents the natural logarithm (ln) of the ratio of the probability of experiencing the outcome (p) vs the probability of not experiencing the outcome (1 – p), with the ratio being the odds of the event occurring.

bettenhausen0393-1020e-f1.jpg

This transformation works well for dichotomous outcomes because the logit transformation approximates a straight line as long as p is not too large or too small (between 0.05 and 0.95).

If we are performing a logistic regression with only one independent variable (x) and want to understand the relationship between this variable (x) and the probability of an outcome event (p), then our model is the equation of a line. The equation for the base model of logistic regression with one independent variable (x) is

bettenhausen0393-1020e-f2.jpg

where β₀ is the y-intercept and β₁ is the slope of the line. Equation (2) is identical to the algebraic equation y = mx + b for a line, just rearranged slightly. In this algebraic equation, m is the slope (the same as β₁) and b is the y-intercept (the same as β₀). We will see that β₀ and β₁ are estimated (ie, assigned numeric values) from the data collected to help us understand how x and

bettenhausen0393-1020e-f3.jpg

are related and are the basis for estimating odds ratios.

We can build more complex models using multivariable logistic regression by adding more independent variables to the right side of equation (2). Essentially, this is what Schaffer et al⁴ did when, for example, they described clinical factors associated with indemnity payments (Schaffer et al, Table 3).

There are two notable techniques used frequently with multivariable logistic regression models. The first involves choosing which independent variables to include in the model. One way to select variables for multivariable models is defining them a priori, that is deciding which variables are clinically or conceptually associated with the outcome before looking at the data. With this approach, we can test specific hypotheses about the relationships between the independent variables and the outcome. Another common approach is to look at the data and identify the variables that vary significantly between the two outcome groups. Schaffer et al⁴ used an a priori approach to define variables in their multivariable model (ie, “variables for inclusion into the multivariable model were determined a priori”).

A second technique is the evaluation of collinearity, which helps us understand whether the independent variables are related to each other. It is important to consider collinearity between independent variables because the inclusion of two (or more) variables that are highly correlated can cause interference between the two and create misleading results from the model. There are techniques to assess collinear relationships before modeling or as part of the model-building process to determine which variables should be excluded. If there are two (or more) independent variables that are similar, one (or more) must be removed from the model.

Fitting the model is the process by which statistical software (eg, SAS, Stata, R, SPSS) estimates the relationships among independent variables in the model and the outcome within a specific dataset. In equation (2), this essentially means that the software will evaluate the data and provide us with the best estimates for β₀ (the y-intercept) and β₁ (the slope) that describe the relationship between the variable x and

bettenhausen0393-1020e-f4.jpg

Modeling can be iterative, and part of the process may include removing variables from the model that are not significantly associated with the outcome to create a simpler solution, a process known as model reduction. The results from models describe the independent association between a specific characteristic and the outcome, meaning that the relationship has been adjusted for all the other characteristics in the model.

The relationships among the independent variables and outcome are most often represented as an odds ratio (OR), which quantifies the strength of the association between two variables and is directly calculated from the β values in the model. As the name suggests, an OR is a ratio of odds. But what are odds? Simply, the odds of an outcome (such as mortality) is the probability of experiencing the event divided by the probability of not experiencing that event; in other words, it is the ratio:

bettenhausen0393-1020e-f5.jpg

The concept of odds is often unfamiliar, so it can be helpful to consider the definition in the context of games of chance. For example, in horse race betting, the outcome of interest is that a horse will lose a race. Imagine that the probability of a horse losing a race is 0.8 and the probability of winning is 0.2. The odds of losing are

bettenhausen0393-1020e-f6.jpg

These odds usually are listed as 4-to-1, meaning that out of 5 races (ie, 4 + 1) the horse is expected to lose 4 times and win once. When odds are listed this way, we can easily calculate the associated probability by recognizing that the total number of expected races is the sum of two numbers (probability of losing: 4 races out of 5, or 0.80 vs probability of winning: 1 race out of 5, or 0.20).

In medical research, the OR typically represents the odds for one group of patients (A) compared with the odds for another group of patients (B) experiencing an outcome. If the odds of the outcome are the same for group A and group B, then OR = 1.0, meaning that the probability of the outcome is the same between the two groups. If the patients in group A have greater odds of experiencing the outcome compared with group B patients (and a greater probability of the outcome), then the OR will be >1. If the opposite is true, then the OR will be <1.

Schaffer et al⁴ estimated that the OR of an indemnity payment in malpractice cases involving errors in clinical judgment as a contributing factor was 5.01 (95% CI, 3.37-7.45). This means that malpractice cases involving errors in clinical judgement had a 5.01 times greater odds of indemnity payment compared with those without these errors after adjusting for all other variables in the model (eg, age, severity). Note that the 95% CI does not include 1.0. This indicates that the OR is statistically >1, and we can conclude that there is a significant relationship between errors in clinical judgment and payment that is unlikely to be attributed to chance alone.

In logistic regression for categorical independent variables, all categories are compared with a reference group within that variable, with the reference group serving as the denominator of the OR. The authors⁴ did not incorporate continuous independent variables in their multivariable logistic regression model. However, if the authors examined length of hospitalization as a contributing factor in indemnity payments, for example, the OR would represent a 1-unit increase in this variable (eg, 1-day increase in length of stay).

Logistic regression describes the relationships in data and is an important statistical model across many types of research. This Progress Note emphasizes the importance of weighing the advantages and limitations of logistic regression, provides a common approach to data transformation, and guides the correct interpretation of logistic regression model results.

Hospital medicine research often asks the question whether an intervention, such as a policy or guideline, has improved quality of care and/or whether there were any unintended consequences. Alternatively, investigators may be interested in understanding the impact of an event, such as a natural disaster or a pandemic, on hospital care. The study design that provides the best estimate of the causal effect of the intervention is the randomized controlled trial (RCT). The goal of randomization, which can be implemented at the patient or cluster level (eg, hospitals), is attaining a balance of the known and unknown confounders between study groups.

However, an RCT may not be feasible for several reasons: complexity, insufficient setup time or funding, ethical barriers to randomization, unwillingness of funders or payers to withhold the intervention from patients (ie, the control group), or anticipated contamination of the intervention into the control group (eg, provider practice change interventions). In addition, it may be impossible to conduct an RCT because the investigator does not have control over the design of an intervention or because they are studying an event, such as a pandemic.

In the June 2020 issue of the Journal of Hospital Medicine, Coon et al¹ use a type of quasi-experimental design (QED)—specifically, the interrupted time series (ITS)—to examine the impact of the adoption of ward-based high-flow nasal cannula protocols on intensive care unit (ICU) admission for bronchiolitis at children’s hospitals. In this methodologic progress note, we discuss QEDs for evaluating the impact of healthcare interventions or events and focus on ITS, one of the strongest QEDs.

Quasi-experimental design refers to a broad range of nonrandomized or partially randomized pre- vs postintervention studies.² In order to test a causal hypothesis without randomization, QEDs define a comparison group or a time period in which an intervention has not been implemented, as well as at least one group or time period in which an intervention has been implemented. In a QED, the control may lack similarity with the intervention group or time period because of differences in the patients, sites, or time period (sometimes referred to as having a “nonequivalent control group”). Several design and analytic approaches are available to enhance the extent to which the study is able to make conclusions about the causal impact of the intervention.^2,3 Because randomization is not necessary, QEDs allow for inclusion of a broader population than that which is feasible by RCTs, which increases the applicability and generalizability of the results. Therefore, they are a powerful research design to test the effectiveness of interventions in real-world settings.

The choice of which QED depends on whether the investigators are conducting a prospective evaluation and have control over the study design (ie, the ordering of the intervention, selection of sites or individuals, and/or timing and frequency of the data collection) or whether the investigators do not have control over the intervention, which is also known as a “natural experiment.”^4,5 Some studies may also incorporate two QEDs in tandem.⁶ The Table provides a brief summary of different QEDs, ordered by methodologic strength, and distinguishes those that can be used to study natural experiments. In the study by Coon et al,¹ an ITS is used as opposed to a methodologically stronger QED, such as the stepped-wedge design, because the investigators did not have control over the rollout of heated high-flow nasal canula protocols across hospitals.

mahant0961_0521e_t1.png

Interrupted time series designs use repeated observations of an outcome over time. This method then divides, or “interrupts,” the series of data into two time periods: before the intervention or event and after. Using data from the preintervention period, an underlying trend in the outcome is estimated and assumed to continue forward into the postintervention period to estimate what would have occurred without the intervention. Any significant change in the outcome at the beginning of the postintervention period or change in the trend in the postintervention is then attributed to the intervention.

There are several important methodologic considerations when designing an ITS study, as detailed in other review papers.^2,3,7,8 An ITS design can be retrospective or prospective. It can be of a single center or include multiple sites, as in Coon et al. It can be conducted with or without a control. The inclusion of a control, when appropriately chosen, improves the strength of the study design because it can account for seasonal trends and potential confounders that vary over time. The control can be a different group of hospitals or participants that are similar but did not receive the intervention, or it can be a different outcome in the same group of hospitals or participants that are not expected to be affected by the intervention. The ITS design may also be set up to estimate the individual effects of multicomponent interventions. If the different components are phased in sequentially over time, then it may be possible to interrupt the time series at these points and estimate the impact of each intervention component.

Other examples of ITS studies in hospital medicine include those that evaluated the impact of a readmission-reduction program,⁹ of state sepsis regulations on in-hospital mortality,¹⁰ of resident duty-hour reform on mortality among hospitalized patients,¹¹ of a quality-improvement initiative on early discharge,¹² and of national guidelines on pediatric pneumonia antibiotic selection.¹³ There are several types of ITS analysis, and in this article, we focus on segmented regression without a control group.^7,8

Segmented regression is the statistical model used to measure (a) the immediate change in the outcome (level) at the start of the intervention and (b) the change in the trend of the outcome (slope) in the postintervention period vs that in the preintervention period. Therefore, the intervention effect size is expressed in terms of the level change and the slope change. To function properly, the models require several repeated (eg, monthly) measurements of the outcome before and after the intervention. Some experts suggest a minimum of 4 to 12 observations, depending on a number of factors including the stability of the outcome and seasonal variations.^7,8 If changes before and after more than one intervention are being examined, there should be the minimum number of observations separating them. Unlike typical regression models, time-series models can correct for autocorrelation if it is present in the data. Autocorrelation is the type of correlation that arises when data are collected over time, with those closest in time being more strongly correlated (there are also other types of autocorrelation, such as seasonal patterns). Using available statistical software, autocorrelation can be detected and, if present, it can be controlled for in the segmented regression models.

Coon et al present results of their ITS analysis in a panel of figures detailing each study outcome, ICU admission, ICU length of stay, total length of stay, and rates of mechanical ventilation. Each panel shows the rate of change in the outcome per season across hospitals, before and after adoption of heated high-flow nasal cannula protocols, and the level change at the time of adoption.

To further explain how segmented regression results are presented, in the Figure we detail the structure of a segmented regression figure evaluating the impact of an intervention without a control group. In addition to the regression figure, authors typically provide 95% CIs around the rates, level change, and the difference between the postintervention and preintervention periods, along with P values demonstrating whether the rates, level change, and the differences between period slopes differ significantly from zero.

mahant0961_0521e_tf.png

Segmented regression models assume a linear trend in the outcome. If the outcome follows a nonlinear pattern (eg, exponential spread of a disease during a pandemic), then using different distributions in the modeling or transformations of the data may be necessary. The validity of the comparison between the pre- and postintervention groups relies on the similarity between the populations. When there is imbalance, investigators can consider matching based on important characteristics or applying risk adjustment as necessary. Another important assumption is that the outcome of interest is unchanged in the absence of the intervention. Finally, the analysis assumes that the intervention is fully implemented at the time the postintervention period begins. Often, there is a washout period during which the old approach is stopped and the new approach (the intervention) is being implemented and can easily be taken into account.

There are several strengths of the ITS analysis and segmented regression.^7,8 First, this approach accounts for a possible secular trend in the outcome measure that may have been present prior to the intervention. For example, investigators might conclude that a readmissions program was effective in reducing readmissions if they found that the mean readmission percentage in the period after the intervention was significantly lower than before using a simple pre/post study design. However, what if the readmission rate was already going down prior to the intervention? Using an ITS approach, they may have found that the rate of readmissions simply continued to decrease after the intervention at the same rate that it was decreasing prior to the intervention and, therefore, conclude that the intervention was not effective. Second, because the ITS approach evaluates changes in rates of an outcome at a population level, confounding by individual-level variables will not introduce serious bias unless the confounding occurred at the same time as the intervention. Third, ITS can be used to measure the unintended consequences of interventions or events, and investigators can construct separate time-series analyses for different outcomes. Fourth, ITS can be used to evaluate the impact of the intervention on subpopulations (eg, those grouped by age, sex, race) by conducting stratified analysis. Fifth, ITS provides simple and clear graphical results that can be easily understood by various audiences.

By accounting for preintervention trends, ITS studies permit stronger causal inference than do cross-sectional or simple pre/post QEDs, but they may by prone to confounding by cointerventions or by changes in the population composition. Causal inference based on the ITS analysis is only valid to the extent to which the intervention was the only thing that changed at the point in time between the preintervention and postintervention periods. It is important for investigators to consider this in the design and discuss any coincident interventions. If there are multiple interventions over time, it is possible to account for these changes in the study design by creating multiple points of interruption provided there are sufficient measurements of the outcome between interventions. If the composition of the population changes at the same time as the intervention, this introduces bias. Changes in the ability to measure the outcome or changes to its definition also threaten the validity of the study’s inferences. Finally, it is also important to remember that when the outcome is a population-level measurement, inferences about individual-level outcomes are inappropriate due to ecological fallacies (ie, when inferences about individuals are deduced from inferences about the group to which those individuals belong). For example, Coon et al found that infants with bronchiolitis in the ward-based high-flow nasal cannula protocol group had greater ICU admission rates. It would be inappropriate to conclude that, based on this, an individual infant in a hospital on a ward-based protocol is more likely to be admitted to the ICU.

Studies evaluating interventions and events are important for informing healthcare practice, policy, and public health. While an RCT is the preferred method for such evaluations, investigators must often consider alternative study designs when an RCT is not feasible or when more real-world outcome evaluation is desired. Quasi-experimental designs are employed in studies that do not use randomization to study the impact of interventions in real-world settings, and an interrupted time series is a strong QED for the evaluation of interventions and natural experiments.

Hospital medicine research often asks the question whether an intervention, such as a policy or guideline, has improved quality of care and/or whether there were any unintended consequences. Alternatively, investigators may be interested in understanding the impact of an event, such as a natural disaster or a pandemic, on hospital care. The study design that provides the best estimate of the causal effect of the intervention is the randomized controlled trial (RCT). The goal of randomization, which can be implemented at the patient or cluster level (eg, hospitals), is attaining a balance of the known and unknown confounders between study groups.

However, an RCT may not be feasible for several reasons: complexity, insufficient setup time or funding, ethical barriers to randomization, unwillingness of funders or payers to withhold the intervention from patients (ie, the control group), or anticipated contamination of the intervention into the control group (eg, provider practice change interventions). In addition, it may be impossible to conduct an RCT because the investigator does not have control over the design of an intervention or because they are studying an event, such as a pandemic.

In the June 2020 issue of the Journal of Hospital Medicine, Coon et al¹ use a type of quasi-experimental design (QED)—specifically, the interrupted time series (ITS)—to examine the impact of the adoption of ward-based high-flow nasal cannula protocols on intensive care unit (ICU) admission for bronchiolitis at children’s hospitals. In this methodologic progress note, we discuss QEDs for evaluating the impact of healthcare interventions or events and focus on ITS, one of the strongest QEDs.

Quasi-experimental design refers to a broad range of nonrandomized or partially randomized pre- vs postintervention studies.² In order to test a causal hypothesis without randomization, QEDs define a comparison group or a time period in which an intervention has not been implemented, as well as at least one group or time period in which an intervention has been implemented. In a QED, the control may lack similarity with the intervention group or time period because of differences in the patients, sites, or time period (sometimes referred to as having a “nonequivalent control group”). Several design and analytic approaches are available to enhance the extent to which the study is able to make conclusions about the causal impact of the intervention.^2,3 Because randomization is not necessary, QEDs allow for inclusion of a broader population than that which is feasible by RCTs, which increases the applicability and generalizability of the results. Therefore, they are a powerful research design to test the effectiveness of interventions in real-world settings.

The choice of which QED depends on whether the investigators are conducting a prospective evaluation and have control over the study design (ie, the ordering of the intervention, selection of sites or individuals, and/or timing and frequency of the data collection) or whether the investigators do not have control over the intervention, which is also known as a “natural experiment.”^4,5 Some studies may also incorporate two QEDs in tandem.⁶ The Table provides a brief summary of different QEDs, ordered by methodologic strength, and distinguishes those that can be used to study natural experiments. In the study by Coon et al,¹ an ITS is used as opposed to a methodologically stronger QED, such as the stepped-wedge design, because the investigators did not have control over the rollout of heated high-flow nasal canula protocols across hospitals.

mahant0961_0521e_t1.png

Interrupted time series designs use repeated observations of an outcome over time. This method then divides, or “interrupts,” the series of data into two time periods: before the intervention or event and after. Using data from the preintervention period, an underlying trend in the outcome is estimated and assumed to continue forward into the postintervention period to estimate what would have occurred without the intervention. Any significant change in the outcome at the beginning of the postintervention period or change in the trend in the postintervention is then attributed to the intervention.

There are several important methodologic considerations when designing an ITS study, as detailed in other review papers.^2,3,7,8 An ITS design can be retrospective or prospective. It can be of a single center or include multiple sites, as in Coon et al. It can be conducted with or without a control. The inclusion of a control, when appropriately chosen, improves the strength of the study design because it can account for seasonal trends and potential confounders that vary over time. The control can be a different group of hospitals or participants that are similar but did not receive the intervention, or it can be a different outcome in the same group of hospitals or participants that are not expected to be affected by the intervention. The ITS design may also be set up to estimate the individual effects of multicomponent interventions. If the different components are phased in sequentially over time, then it may be possible to interrupt the time series at these points and estimate the impact of each intervention component.

Other examples of ITS studies in hospital medicine include those that evaluated the impact of a readmission-reduction program,⁹ of state sepsis regulations on in-hospital mortality,¹⁰ of resident duty-hour reform on mortality among hospitalized patients,¹¹ of a quality-improvement initiative on early discharge,¹² and of national guidelines on pediatric pneumonia antibiotic selection.¹³ There are several types of ITS analysis, and in this article, we focus on segmented regression without a control group.^7,8

Segmented regression is the statistical model used to measure (a) the immediate change in the outcome (level) at the start of the intervention and (b) the change in the trend of the outcome (slope) in the postintervention period vs that in the preintervention period. Therefore, the intervention effect size is expressed in terms of the level change and the slope change. To function properly, the models require several repeated (eg, monthly) measurements of the outcome before and after the intervention. Some experts suggest a minimum of 4 to 12 observations, depending on a number of factors including the stability of the outcome and seasonal variations.^7,8 If changes before and after more than one intervention are being examined, there should be the minimum number of observations separating them. Unlike typical regression models, time-series models can correct for autocorrelation if it is present in the data. Autocorrelation is the type of correlation that arises when data are collected over time, with those closest in time being more strongly correlated (there are also other types of autocorrelation, such as seasonal patterns). Using available statistical software, autocorrelation can be detected and, if present, it can be controlled for in the segmented regression models.

Coon et al present results of their ITS analysis in a panel of figures detailing each study outcome, ICU admission, ICU length of stay, total length of stay, and rates of mechanical ventilation. Each panel shows the rate of change in the outcome per season across hospitals, before and after adoption of heated high-flow nasal cannula protocols, and the level change at the time of adoption.

To further explain how segmented regression results are presented, in the Figure we detail the structure of a segmented regression figure evaluating the impact of an intervention without a control group. In addition to the regression figure, authors typically provide 95% CIs around the rates, level change, and the difference between the postintervention and preintervention periods, along with P values demonstrating whether the rates, level change, and the differences between period slopes differ significantly from zero.

mahant0961_0521e_tf.png

Segmented regression models assume a linear trend in the outcome. If the outcome follows a nonlinear pattern (eg, exponential spread of a disease during a pandemic), then using different distributions in the modeling or transformations of the data may be necessary. The validity of the comparison between the pre- and postintervention groups relies on the similarity between the populations. When there is imbalance, investigators can consider matching based on important characteristics or applying risk adjustment as necessary. Another important assumption is that the outcome of interest is unchanged in the absence of the intervention. Finally, the analysis assumes that the intervention is fully implemented at the time the postintervention period begins. Often, there is a washout period during which the old approach is stopped and the new approach (the intervention) is being implemented and can easily be taken into account.

There are several strengths of the ITS analysis and segmented regression.^7,8 First, this approach accounts for a possible secular trend in the outcome measure that may have been present prior to the intervention. For example, investigators might conclude that a readmissions program was effective in reducing readmissions if they found that the mean readmission percentage in the period after the intervention was significantly lower than before using a simple pre/post study design. However, what if the readmission rate was already going down prior to the intervention? Using an ITS approach, they may have found that the rate of readmissions simply continued to decrease after the intervention at the same rate that it was decreasing prior to the intervention and, therefore, conclude that the intervention was not effective. Second, because the ITS approach evaluates changes in rates of an outcome at a population level, confounding by individual-level variables will not introduce serious bias unless the confounding occurred at the same time as the intervention. Third, ITS can be used to measure the unintended consequences of interventions or events, and investigators can construct separate time-series analyses for different outcomes. Fourth, ITS can be used to evaluate the impact of the intervention on subpopulations (eg, those grouped by age, sex, race) by conducting stratified analysis. Fifth, ITS provides simple and clear graphical results that can be easily understood by various audiences.

By accounting for preintervention trends, ITS studies permit stronger causal inference than do cross-sectional or simple pre/post QEDs, but they may by prone to confounding by cointerventions or by changes in the population composition. Causal inference based on the ITS analysis is only valid to the extent to which the intervention was the only thing that changed at the point in time between the preintervention and postintervention periods. It is important for investigators to consider this in the design and discuss any coincident interventions. If there are multiple interventions over time, it is possible to account for these changes in the study design by creating multiple points of interruption provided there are sufficient measurements of the outcome between interventions. If the composition of the population changes at the same time as the intervention, this introduces bias. Changes in the ability to measure the outcome or changes to its definition also threaten the validity of the study’s inferences. Finally, it is also important to remember that when the outcome is a population-level measurement, inferences about individual-level outcomes are inappropriate due to ecological fallacies (ie, when inferences about individuals are deduced from inferences about the group to which those individuals belong). For example, Coon et al found that infants with bronchiolitis in the ward-based high-flow nasal cannula protocol group had greater ICU admission rates. It would be inappropriate to conclude that, based on this, an individual infant in a hospital on a ward-based protocol is more likely to be admitted to the ICU.

Studies evaluating interventions and events are important for informing healthcare practice, policy, and public health. While an RCT is the preferred method for such evaluations, investigators must often consider alternative study designs when an RCT is not feasible or when more real-world outcome evaluation is desired. Quasi-experimental designs are employed in studies that do not use randomization to study the impact of interventions in real-world settings, and an interrupted time series is a strong QED for the evaluation of interventions and natural experiments.

Hospital medicine research often asks the question whether an intervention, such as a policy or guideline, has improved quality of care and/or whether there were any unintended consequences. Alternatively, investigators may be interested in understanding the impact of an event, such as a natural disaster or a pandemic, on hospital care. The study design that provides the best estimate of the causal effect of the intervention is the randomized controlled trial (RCT). The goal of randomization, which can be implemented at the patient or cluster level (eg, hospitals), is attaining a balance of the known and unknown confounders between study groups.

However, an RCT may not be feasible for several reasons: complexity, insufficient setup time or funding, ethical barriers to randomization, unwillingness of funders or payers to withhold the intervention from patients (ie, the control group), or anticipated contamination of the intervention into the control group (eg, provider practice change interventions). In addition, it may be impossible to conduct an RCT because the investigator does not have control over the design of an intervention or because they are studying an event, such as a pandemic.

In the June 2020 issue of the Journal of Hospital Medicine, Coon et al¹ use a type of quasi-experimental design (QED)—specifically, the interrupted time series (ITS)—to examine the impact of the adoption of ward-based high-flow nasal cannula protocols on intensive care unit (ICU) admission for bronchiolitis at children’s hospitals. In this methodologic progress note, we discuss QEDs for evaluating the impact of healthcare interventions or events and focus on ITS, one of the strongest QEDs.

Quasi-experimental design refers to a broad range of nonrandomized or partially randomized pre- vs postintervention studies.² In order to test a causal hypothesis without randomization, QEDs define a comparison group or a time period in which an intervention has not been implemented, as well as at least one group or time period in which an intervention has been implemented. In a QED, the control may lack similarity with the intervention group or time period because of differences in the patients, sites, or time period (sometimes referred to as having a “nonequivalent control group”). Several design and analytic approaches are available to enhance the extent to which the study is able to make conclusions about the causal impact of the intervention.^2,3 Because randomization is not necessary, QEDs allow for inclusion of a broader population than that which is feasible by RCTs, which increases the applicability and generalizability of the results. Therefore, they are a powerful research design to test the effectiveness of interventions in real-world settings.

The choice of which QED depends on whether the investigators are conducting a prospective evaluation and have control over the study design (ie, the ordering of the intervention, selection of sites or individuals, and/or timing and frequency of the data collection) or whether the investigators do not have control over the intervention, which is also known as a “natural experiment.”^4,5 Some studies may also incorporate two QEDs in tandem.⁶ The Table provides a brief summary of different QEDs, ordered by methodologic strength, and distinguishes those that can be used to study natural experiments. In the study by Coon et al,¹ an ITS is used as opposed to a methodologically stronger QED, such as the stepped-wedge design, because the investigators did not have control over the rollout of heated high-flow nasal canula protocols across hospitals.

mahant0961_0521e_t1.png

Interrupted time series designs use repeated observations of an outcome over time. This method then divides, or “interrupts,” the series of data into two time periods: before the intervention or event and after. Using data from the preintervention period, an underlying trend in the outcome is estimated and assumed to continue forward into the postintervention period to estimate what would have occurred without the intervention. Any significant change in the outcome at the beginning of the postintervention period or change in the trend in the postintervention is then attributed to the intervention.

There are several important methodologic considerations when designing an ITS study, as detailed in other review papers.^2,3,7,8 An ITS design can be retrospective or prospective. It can be of a single center or include multiple sites, as in Coon et al. It can be conducted with or without a control. The inclusion of a control, when appropriately chosen, improves the strength of the study design because it can account for seasonal trends and potential confounders that vary over time. The control can be a different group of hospitals or participants that are similar but did not receive the intervention, or it can be a different outcome in the same group of hospitals or participants that are not expected to be affected by the intervention. The ITS design may also be set up to estimate the individual effects of multicomponent interventions. If the different components are phased in sequentially over time, then it may be possible to interrupt the time series at these points and estimate the impact of each intervention component.

Other examples of ITS studies in hospital medicine include those that evaluated the impact of a readmission-reduction program,⁹ of state sepsis regulations on in-hospital mortality,¹⁰ of resident duty-hour reform on mortality among hospitalized patients,¹¹ of a quality-improvement initiative on early discharge,¹² and of national guidelines on pediatric pneumonia antibiotic selection.¹³ There are several types of ITS analysis, and in this article, we focus on segmented regression without a control group.^7,8

Segmented regression is the statistical model used to measure (a) the immediate change in the outcome (level) at the start of the intervention and (b) the change in the trend of the outcome (slope) in the postintervention period vs that in the preintervention period. Therefore, the intervention effect size is expressed in terms of the level change and the slope change. To function properly, the models require several repeated (eg, monthly) measurements of the outcome before and after the intervention. Some experts suggest a minimum of 4 to 12 observations, depending on a number of factors including the stability of the outcome and seasonal variations.^7,8 If changes before and after more than one intervention are being examined, there should be the minimum number of observations separating them. Unlike typical regression models, time-series models can correct for autocorrelation if it is present in the data. Autocorrelation is the type of correlation that arises when data are collected over time, with those closest in time being more strongly correlated (there are also other types of autocorrelation, such as seasonal patterns). Using available statistical software, autocorrelation can be detected and, if present, it can be controlled for in the segmented regression models.

Coon et al present results of their ITS analysis in a panel of figures detailing each study outcome, ICU admission, ICU length of stay, total length of stay, and rates of mechanical ventilation. Each panel shows the rate of change in the outcome per season across hospitals, before and after adoption of heated high-flow nasal cannula protocols, and the level change at the time of adoption.

To further explain how segmented regression results are presented, in the Figure we detail the structure of a segmented regression figure evaluating the impact of an intervention without a control group. In addition to the regression figure, authors typically provide 95% CIs around the rates, level change, and the difference between the postintervention and preintervention periods, along with P values demonstrating whether the rates, level change, and the differences between period slopes differ significantly from zero.

mahant0961_0521e_tf.png

Segmented regression models assume a linear trend in the outcome. If the outcome follows a nonlinear pattern (eg, exponential spread of a disease during a pandemic), then using different distributions in the modeling or transformations of the data may be necessary. The validity of the comparison between the pre- and postintervention groups relies on the similarity between the populations. When there is imbalance, investigators can consider matching based on important characteristics or applying risk adjustment as necessary. Another important assumption is that the outcome of interest is unchanged in the absence of the intervention. Finally, the analysis assumes that the intervention is fully implemented at the time the postintervention period begins. Often, there is a washout period during which the old approach is stopped and the new approach (the intervention) is being implemented and can easily be taken into account.

There are several strengths of the ITS analysis and segmented regression.^7,8 First, this approach accounts for a possible secular trend in the outcome measure that may have been present prior to the intervention. For example, investigators might conclude that a readmissions program was effective in reducing readmissions if they found that the mean readmission percentage in the period after the intervention was significantly lower than before using a simple pre/post study design. However, what if the readmission rate was already going down prior to the intervention? Using an ITS approach, they may have found that the rate of readmissions simply continued to decrease after the intervention at the same rate that it was decreasing prior to the intervention and, therefore, conclude that the intervention was not effective. Second, because the ITS approach evaluates changes in rates of an outcome at a population level, confounding by individual-level variables will not introduce serious bias unless the confounding occurred at the same time as the intervention. Third, ITS can be used to measure the unintended consequences of interventions or events, and investigators can construct separate time-series analyses for different outcomes. Fourth, ITS can be used to evaluate the impact of the intervention on subpopulations (eg, those grouped by age, sex, race) by conducting stratified analysis. Fifth, ITS provides simple and clear graphical results that can be easily understood by various audiences.

By accounting for preintervention trends, ITS studies permit stronger causal inference than do cross-sectional or simple pre/post QEDs, but they may by prone to confounding by cointerventions or by changes in the population composition. Causal inference based on the ITS analysis is only valid to the extent to which the intervention was the only thing that changed at the point in time between the preintervention and postintervention periods. It is important for investigators to consider this in the design and discuss any coincident interventions. If there are multiple interventions over time, it is possible to account for these changes in the study design by creating multiple points of interruption provided there are sufficient measurements of the outcome between interventions. If the composition of the population changes at the same time as the intervention, this introduces bias. Changes in the ability to measure the outcome or changes to its definition also threaten the validity of the study’s inferences. Finally, it is also important to remember that when the outcome is a population-level measurement, inferences about individual-level outcomes are inappropriate due to ecological fallacies (ie, when inferences about individuals are deduced from inferences about the group to which those individuals belong). For example, Coon et al found that infants with bronchiolitis in the ward-based high-flow nasal cannula protocol group had greater ICU admission rates. It would be inappropriate to conclude that, based on this, an individual infant in a hospital on a ward-based protocol is more likely to be admitted to the ICU.

Studies evaluating interventions and events are important for informing healthcare practice, policy, and public health. While an RCT is the preferred method for such evaluations, investigators must often consider alternative study designs when an RCT is not feasible or when more real-world outcome evaluation is desired. Quasi-experimental designs are employed in studies that do not use randomization to study the impact of interventions in real-world settings, and an interrupted time series is a strong QED for the evaluation of interventions and natural experiments.

Staphylococcal scalded skin syndrome (SSSS) is an exfoliative toxin-mediated dermatitis that predominantly occurs in young children. Multiple recent reports indicate a rising incidence of this disease.^1-4 Recommended treatment for SSSS includes antistaphylococcal antibiotics and supportive care measures.^5,6 Elimination or reduction of the toxin-producing Staphylococcus aureus is thought to help limit disease progression and promote recovery. Experts advocate for the use of antibiotics even when there is no apparent focal source of infection, such as an abscess.^6,7

Several factors may affect antibiotic selection, including the desire to inhibit toxin production and to target the causative pathogen in a bactericidal fashion. Because SSSS is toxin mediated, clindamycin is often recommended because of its inhibition of toxin synthesis.^5,8 The clinical utility of adding other antibiotics to clindamycin for coverage of methicillin-sensitive S aureus (MSSA) or methicillin-resistant S aureus (MRSA) is uncertain. Several studies report MSSA to be the predominant pathogen identified by culture^2,9; however, SSSS caused by MRSA has been reported.^9-11 Additionally, bactericidal antibiotics (eg, nafcillin) have been considered to hold potential clinical advantage as compared with bacteriostatic antibiotics (eg, clindamycin), even though clinical studies have not clearly demonstrated this advantage in the general population.^12,13 Some experts recommend additional MRSA or MSSA coverage (such as vancomycin or nafcillin) in patients with high illness severity or nonresponse to therapy, or in areas where there is high prevalence of staphylococcal resistance to clindamycin.^5,7,9,14 Alternatively, for areas with low MRSA prevalence, monotherapy with an anti-MSSA antibiotic is another potential option. No recent studies have compared patient outcomes among antibiotic regimens in children with SSSS.

Knowledge of the outcomes associated with different antibiotic regimens for children hospitalized with SSSS is needed and could be used to improve patient outcomes and potentially promote antibiotic stewardship. In this study, our objectives were to (1) describe antibiotic regimens given to children hospitalized with SSSS, and (2) examine the association of three antibiotic regimens commonly used for SSSS (clindamycin monotherapy, clindamycin plus additional MSSA coverage, and clindamycin plus additional MRSA coverage) with patient outcomes of length of stay (LOS), treatment failure, and cost in a large cohort of children at US children’s hospitals.

We conducted a multicenter, retrospective cohort study utilizing data within the Pediatric Health Information System (PHIS) database from July 1, 2011, to June 30, 2016. Thirty-five free-standing tertiary care US children’s hospitals within 24 states were included. The Children’s Hospital Association (Lenexa, Kansas) maintains the PHIS database, which contains de-identified patient information, including diagnoses (with International Classification of Diseases, Ninth and Tenth Revision, Clinical Modification [ICD-9-CM, ICD-10-CM]), demographics, procedures, and daily billing records. Data quality and reliability are confirmed by participating institutions and the Children’s Hospital Association.¹⁵ The local institutional review board (IRB) deemed the study exempt from formal IRB review, as patient information was de-identified.

Study Population

We included hospitalized children aged newborn to 18 years with a primary or secondary diagnosis of SSSS (ICD-9, 695.81; ICD-10, L00). Children whose primary presentation and admission were to a PHIS hospital were included; children transferred from another hospital were excluded. The following exclusion criteria were based on previously published methodology.¹⁶ Children with complex chronic medical conditions as classified by Feudtner et al¹⁷ were excluded, since these children may require a different treatment approach than the general pediatric population. In order to decrease diagnostic ambiguity, we excluded children if an alternative dermatologic diagnosis was recorded as a principal or secondary diagnosis (eg, Stevens-Johnson syndrome or scarlet fever).¹⁶ Finally, hospitals with fewer than 10 children with SSSS during the study period were excluded.

Antibiotic Regimen Groups

We used PHIS daily billing codes to determine the antibiotics received by the study population. Children were classified into antibiotic regimen groups based on whether they received specific antibiotic combinations. Antibiotics received on any day during the hospitalization, including in the emergency department (ED), were used to assign patients to regimen groups. Antibiotics were classified into regimen groups based on consensus among study investigators, which included two board-certified pediatric infectious diseases specialists (A.C., R.M.). Antibiotic group definitions are listed in Table 1. Oral and intravenous (IV) therapies were grouped together for clindamycin, cephalexin/cefazolin, and linezolid because of good oral bioavailability in most situations.¹⁸ The three most common antistaphylococcal groups were chosen for further analysis: clindamycin alone, clindamycin plus MSSA coverage, and clindamycin plus MRSA coverage. The clindamycin group was defined as children with receipt of oral or IV clindamycin. Children who received clindamycin with additional MSSA coverage, such as cefazolin or nafcillin, were categorized as the clindamycin plus MSSA group. Children who received clindamycin with additional MRSA coverage, such as vancomycin or linezolid, were categorized as the clindamycin plus MRSA group. We chose not to include children who received the above regimens plus other antibiotics with partial antistaphylococcal activity, such as ampicillin, gentamicin, or ceftriaxone, in the clindamycin plus MSSA and clindamycin plus MRSA groups. We excluded these antibiotics to decrease the heterogeneity in the definition of regimen groups and allow a more direct comparison for effectiveness.

neubauer04830217e_t1.jpg

Covariates

Covariates included age, sex, ethnicity and/or race, payer type, level of care, illness severity, and region. The variable definitions below are in keeping with a prior study of SSSS.¹⁶ Age was categorized as: birth to 59 days, 2 to 11 months, 1 to 4 years (preschool age), 5 to 10 years (school age), and 11 to 18 years (adolescent). We examined infants younger than 60 days separately from older infants because this population may warrant additional treatment considerations. Race and ethnicity were categorized as White (non-Hispanic), African American (non-Hispanic), Hispanic, or other. Payer types included government, private, or other. Level of care was assigned as either intensive care or acute care. Illness severity was assigned using the All Patient Refined Diagnosis Related Group (APR-DRG; 3M Corporation, St. Paul, Minnesota) severity levels.¹⁹ In line with a prior study,¹⁶ we defined “low illness severity” as the APR-DRG minor (1) classification. The moderate (2), major (3), and extreme (4) classifications were defined as “moderate to high illness severity,” since there were very few classifications of major or extreme (<5%) illness severity. We categorized hospitals into the following US regions: Northeast, Midwest, South, and West.

Outcome Measures

The primary outcome was hospital LOS in days, and secondary outcomes were treatment failure and hospital costs. Hospital LOS was chosen as the primary outcome to represent the time needed for the child to show clinical improvement. Treatment failure was defined as a same-cause 14-day ED revisit or hospital readmission, and these were determined to be same-cause if a diagnosis for SSSS (ICD-9, 695.81; ICD-10, L00) was documented for the return encounter. The 14-day interval for readmission and ED revisit was chosen to measure any relapse of symptoms after completion of antibiotic therapy, similar to a prior study of treatment failure in skin and soft tissue infections.²⁰ Total costs of the hospitalization were estimated from charges using hospital- and year-specific cost-to-charge ratios. Subcategories of cost, including clinical, pharmacy, imaging, laboratory, supply, and other, were also compared among the three groups.

Statistical Analysis

Demographic and clinical characteristics of children were summarized using frequencies and percentages for categorical variables and medians with interquartile ranges (IQRs) for continuous variables. These were compared across antibiotic groups using chi-square and Kruskal–Wallis tests, respectively. In unadjusted analyses, outcomes were compared across antibiotic regimen groups using these same statistical tests. In order to account for patient clustering within hospitals, generalized linear mixed-effects models were used to model outcomes with a random intercept for each hospital. Models were adjusted for SSSS being listed as a principal or secondary diagnosis, race, illness severity, and level of care. We log-transformed LOS and cost data prior to modeling because of the nonnormal distributions for these data. Owing to the inability to measure the number of antibiotic doses, and to reduce the possibility of including children who received few regimen-defined combination antibiotics, a post hoc sensitivity analysis was performed. This analysis used an alternative definition for antibiotic regimen groups, for which children admitted for 2 or more calendar days must have received regimen-specified antibiotics on at least 2 days of the admission. Additionally, outcomes were stratified by low and moderate/high illness severity and compared across the three antibiotic regimen groups. All analyses were performed with SAS (SAS 9.4; SAS Institute, Cary, North Carolina), and P values of less than .05 were considered statistically significant.

Overall, 1,815 hospitalized children with SSSS were identified in the PHIS database, and after application of the exclusion criteria, 1,259 children remained, with 1,247 (99%) receiving antibiotics (Figure). The antibiotic regimens received by these children are described in Table 1. Of these, 828 children (66%) received one of the three most common antistaphylococcal regimens (clindamycin, clindamycin + MSSA, and clindamycin + MRSA) and were included for further analysis.

neubauer04830217e_f1.jpg

Characteristics of the 828 children are presented in Table 2. Most children (82%) were aged 4 years or younger, and distributions of age, sex, and insurance payer were similar among children receiving the three regimens. Thirty-two percent had moderate to high illness severity, and 3.5% required management in the intensive care setting. Of the three antibiotic regimens, clindamycin monotherapy was most common (47%), followed by clindamycin plus MSSA coverage (33%), and clindamycin plus MRSA coverage (20%). A higher proportion of children in the clindamycin plus MRSA group were African American and were hospitalized in the South. Children receiving clindamycin plus MRSA coverage had higher illness severity (44%) as compared with clindamycin monotherapy (28%) and clindamycin plus MSSA coverage (32%) (P = .001). Additionally, a larger proportion of children treated with clindamycin plus MRSA coverage were managed in the intensive care setting as compared with the clindamycin plus MSSA or clindamycin monotherapy groups.

neubauer04830217e_t2.jpg

Among the 828 children with SSSS, the median LOS was 2 days (IQR, 2-3), and treatment failure was 1.1% (95% CI, 0.4-1.8). After adjustment for illness severity, race, payer, and region (Table 3), the three antibiotic regimens were not associated with significant differences in LOS or treatment failure. Costs were significantly different among the three antibiotic regimens. Clindamycin plus MRSA coverage was associated with the greatest costs, whereas clindamycin monotherapy was associated with the lowest costs (mean, $5,348 vs $4,839, respectively; P < .001) (Table 3). In a sensitivity analysis using an alternative antibiotic regimen definition, we found results in line with the primary analysis, with no statistically significant differences in LOS (P = .44) or treatment failure (P = .54), but significant differences in cost (P < .001). Additionally, the same findings were present for LOS, treatment failure, and cost when outcomes were stratified by illness severity (Appendix Table). However, significant contributors to the higher cost in the clindamycin plus MRSA group did vary by illness severity stratification. Laboratory, supply, and pharmacy cost categories differed significantly among antibiotic groups for the low illness severity strata, whereas pharmacy was the only significant cost category difference in moderate/high illness severity.

neubauer04830217e_t3.jpg

Clindamycin monotherapy, clindamycin plus MSSA coverage, and clindamycin plus MRSA coverage are the most commonly administered antistaphylococcal antibiotic regimens for children hospitalized with SSSS at US children’s hospitals. Our multicenter study found that, across these antistaphylococcal antibiotic regimens, there were no associated differences in hospital LOS or treatment failure. However, the antibiotic regimens were associated with significant differences in overall hospital costs. These findings suggest that the use of clindamycin with additional MSSA or MRSA antibiotic coverage for children with SSSS may not be associated with additional clinical benefit, as compared with clindamycin monotherapy, and could potentially be more costly.

Prior literature describing LOS in relation to antibiotic use for children with SSSS is limited. Authors of a recent case series of 21 children in Philadelphia reported approximately 50% of children received clindamycin monotherapy or combination therapy, but patient outcomes such as LOS were not described.⁹ Clindamycin use and outcomes have been described in smaller studies and case reports of SSSS, which reported positive outcomes such as patient recovery and lack of disease recurrence.^2,9,21 A small retrospective, comparative effectiveness study of 30 neonates with SSSS examined beta-lactamase–resistant penicillin use with and without cephalosporins. They found no effect on LOS, but findings were limited by a small sample size.²² Our study cohort included relatively few neonates, and thus our findings may not be applicable to this population subgroup. We chose not to include regimens with third-generation cephalosporins or ampicillin, which may have limited the number of included neonates, because these antibiotics are frequently administered during evaluation for invasive bacterial infections.²³ We found a very low occurrence of treatment failure in our study cohort across all three groups, which is consistent with other studies of SSSS that report an overall good prognosis and low recurrence and/or readmission rates.^6,16,24 The low prevalence of treatment failure, however, precluded our ability to detect small differences among antibiotic regimen groups that may exist.

We observed that cost differed significantly across antibiotic regimen groups, with lowest cost associated with clindamycin monotherapy in adjusted analysis despite similar LOS. Even with our illness-severity adjustment, there may have been other unmeasured factors resulting in the higher cost associated with the combination groups. Hence, we also examined cost breakdown with a stratified analysis by illness severity. We found that pharmacy costs were significantly different among antibiotic groups in both illness severity strata, whereas those with low illness severity also differed by laboratory and supply costs. Thus, pharmacy cost differences may be the largest driver in the cost differential among groups. Lower cost in the clindamycin monotherapy group is likely due to administration of a single antibiotic. The reason for supply and laboratory cost differences is uncertain, but higher cost in the clindamycin plus MRSA group could possibly be from laboratory testing related to drug monitoring (eg, renal function testing or drug levels). While other studies have reported costs for hospitalized children with SSSS associated with different patient characteristics or diagnostic testing,^1,16 to our knowledge, no other studies have reported cost related to antibiotic regimens for SSSS. As healthcare reimbursements shift to value-based models, identifying treatment regimens with equal efficacy but lower cost will become increasingly important. Future studies should also examine other covariates and outcomes, such as oral vs parenteral antibiotic use, use of monitoring laboratories related to antibiotic choice, and adverse drug effects.

Several strengths and additional limitations apply to our study. Our study is one of the few to describe outcomes associated with antibiotic regimens for children with SSSS. With the PHIS database, we were able to include a large number of children with SSSS from children’s hospitals across the United States. Although the PHIS database affords these strengths, there are limitations inherent to administrative data. Children with SSSS were identified by documented ICD-9 and ICD-10 diagnostic codes, which might lead to misclassification. However, misclassification is less likely because only one ICD-9 and ICD-10 code exists for SSSS, and the characteristics of this condition are specific. Also, diagnostic codes for other dermatologic conditions (eg, scarlet fever) were excluded to further reduce the chance of misclassification. A limitation to our use of PHIS billing codes was the inability to confirm the dosage of antibiotics given, the number of doses, or whether antibiotics were prescribed upon discharge. Another limitation is that children whose antibiotic therapy was changed during hospitalization (eg, from clindamycin monotherapy to cefazolin monotherapy) were categorized into the combination groups. However, the sensitivity analysis performed based on a stricter antibiotic group definition (receipt of both antibiotics on at least 2 calendar days) did not alter the outcomes, which is reassuring. We were unable to assess the use of targeted antibiotic therapy because clinical data (eg, microbiology results) were not available. However, this may be less important because some literature suggests that cultures for S aureus are obtained infrequently² and may be difficult to interpret when obtained,²⁵ since culture growth can represent colonization rather than causative strains. An additional limitation is that administrative data do not include certain clinical outcomes, such as fever duration or degree of skin involvement, which could have differed among the groups. Last, the PHIS database only captures revisits or readmissions to PHIS hospitals, and so we are unable to exclude the possibility of a child being seen at or readmitted to another hospital.

Due to the observational design of this study and potential for incomplete measurement of illness severity, we recommend a future prospective trial with randomization to confirm these findings. One possible reason that LOS did not differ among groups is that the burden of clindamycin-resistant strains in our cohort could be low, and the addition of MSSA or MRSA coverage does not result in a clinically important increase in S aureus coverage. However, pooled pediatric hospital antibiogram data suggest the overall rate of clindamycin resistance is close to 20% in hospitals located in all US regions.²⁶ Limited studies also suggest that MSSA may be the predominant pathogen associated with SSSS.^2,9 To address this, future randomized trials could compare the effectiveness of clindamycin monotherapy to MSSA-specific agents like cefazolin or nafcillin. Unfortunately, anti-MSSA monotherapy was not evaluated in our study because very few children received this treatment. Using monotherapy as opposed to multiple antibiotics has the potential to promote antibiotic stewardship for antistaphylococcal antibiotics in the management of SSSS. Reducing unnecessary antibiotic use not only potentially affects antibiotic resistance, but could also benefit patients in reducing possible side effects, cost, and IV catheter complications.²⁷ However, acknowledging our study limitations, our findings should be applied cautiously in clinical settings, in the context of local antibiogram data, individual culture results, and specific patient factors. The local clindamycin resistance rate for both MSSA and MRSA should be considered. Many antibiotics chosen to treat MRSA—such as vancomycin and trimethoprim/sulfamethoxazole—will also have anti-MSSA activity and may have lower local resistance rates than clindamycin. Practitioners may also consider how each antibiotic kills bacteria; for example, beta-lactams rely on bacterial replication, but clindamycin does not. Each factor should influence how empiric treatment, whether monotherapy or combination, is chosen for children with SSSS.

In this large, multicenter cohort of hospitalized children with SSSS, we found that the addition of MSSA or MRSA coverage to clindamycin monotherapy was not associated with differences in outcomes of hospital LOS and treatment failure. Furthermore, clindamycin monotherapy was associated with lower overall cost. Prospective randomized studies are needed to confirm these findings and assess whether clindamycin monotherapy, monotherapy with an anti-MSSA antibiotic, or alternative regimens are most effective for treatment of children with SSSS.

Staphylococcal scalded skin syndrome (SSSS) is an exfoliative toxin-mediated dermatitis that predominantly occurs in young children. Multiple recent reports indicate a rising incidence of this disease.^1-4 Recommended treatment for SSSS includes antistaphylococcal antibiotics and supportive care measures.^5,6 Elimination or reduction of the toxin-producing Staphylococcus aureus is thought to help limit disease progression and promote recovery. Experts advocate for the use of antibiotics even when there is no apparent focal source of infection, such as an abscess.^6,7

Several factors may affect antibiotic selection, including the desire to inhibit toxin production and to target the causative pathogen in a bactericidal fashion. Because SSSS is toxin mediated, clindamycin is often recommended because of its inhibition of toxin synthesis.^5,8 The clinical utility of adding other antibiotics to clindamycin for coverage of methicillin-sensitive S aureus (MSSA) or methicillin-resistant S aureus (MRSA) is uncertain. Several studies report MSSA to be the predominant pathogen identified by culture^2,9; however, SSSS caused by MRSA has been reported.^9-11 Additionally, bactericidal antibiotics (eg, nafcillin) have been considered to hold potential clinical advantage as compared with bacteriostatic antibiotics (eg, clindamycin), even though clinical studies have not clearly demonstrated this advantage in the general population.^12,13 Some experts recommend additional MRSA or MSSA coverage (such as vancomycin or nafcillin) in patients with high illness severity or nonresponse to therapy, or in areas where there is high prevalence of staphylococcal resistance to clindamycin.^5,7,9,14 Alternatively, for areas with low MRSA prevalence, monotherapy with an anti-MSSA antibiotic is another potential option. No recent studies have compared patient outcomes among antibiotic regimens in children with SSSS.

Knowledge of the outcomes associated with different antibiotic regimens for children hospitalized with SSSS is needed and could be used to improve patient outcomes and potentially promote antibiotic stewardship. In this study, our objectives were to (1) describe antibiotic regimens given to children hospitalized with SSSS, and (2) examine the association of three antibiotic regimens commonly used for SSSS (clindamycin monotherapy, clindamycin plus additional MSSA coverage, and clindamycin plus additional MRSA coverage) with patient outcomes of length of stay (LOS), treatment failure, and cost in a large cohort of children at US children’s hospitals.

We conducted a multicenter, retrospective cohort study utilizing data within the Pediatric Health Information System (PHIS) database from July 1, 2011, to June 30, 2016. Thirty-five free-standing tertiary care US children’s hospitals within 24 states were included. The Children’s Hospital Association (Lenexa, Kansas) maintains the PHIS database, which contains de-identified patient information, including diagnoses (with International Classification of Diseases, Ninth and Tenth Revision, Clinical Modification [ICD-9-CM, ICD-10-CM]), demographics, procedures, and daily billing records. Data quality and reliability are confirmed by participating institutions and the Children’s Hospital Association.¹⁵ The local institutional review board (IRB) deemed the study exempt from formal IRB review, as patient information was de-identified.

Study Population

We included hospitalized children aged newborn to 18 years with a primary or secondary diagnosis of SSSS (ICD-9, 695.81; ICD-10, L00). Children whose primary presentation and admission were to a PHIS hospital were included; children transferred from another hospital were excluded. The following exclusion criteria were based on previously published methodology.¹⁶ Children with complex chronic medical conditions as classified by Feudtner et al¹⁷ were excluded, since these children may require a different treatment approach than the general pediatric population. In order to decrease diagnostic ambiguity, we excluded children if an alternative dermatologic diagnosis was recorded as a principal or secondary diagnosis (eg, Stevens-Johnson syndrome or scarlet fever).¹⁶ Finally, hospitals with fewer than 10 children with SSSS during the study period were excluded.

Antibiotic Regimen Groups

We used PHIS daily billing codes to determine the antibiotics received by the study population. Children were classified into antibiotic regimen groups based on whether they received specific antibiotic combinations. Antibiotics received on any day during the hospitalization, including in the emergency department (ED), were used to assign patients to regimen groups. Antibiotics were classified into regimen groups based on consensus among study investigators, which included two board-certified pediatric infectious diseases specialists (A.C., R.M.). Antibiotic group definitions are listed in Table 1. Oral and intravenous (IV) therapies were grouped together for clindamycin, cephalexin/cefazolin, and linezolid because of good oral bioavailability in most situations.¹⁸ The three most common antistaphylococcal groups were chosen for further analysis: clindamycin alone, clindamycin plus MSSA coverage, and clindamycin plus MRSA coverage. The clindamycin group was defined as children with receipt of oral or IV clindamycin. Children who received clindamycin with additional MSSA coverage, such as cefazolin or nafcillin, were categorized as the clindamycin plus MSSA group. Children who received clindamycin with additional MRSA coverage, such as vancomycin or linezolid, were categorized as the clindamycin plus MRSA group. We chose not to include children who received the above regimens plus other antibiotics with partial antistaphylococcal activity, such as ampicillin, gentamicin, or ceftriaxone, in the clindamycin plus MSSA and clindamycin plus MRSA groups. We excluded these antibiotics to decrease the heterogeneity in the definition of regimen groups and allow a more direct comparison for effectiveness.

neubauer04830217e_t1.jpg

Covariates

Covariates included age, sex, ethnicity and/or race, payer type, level of care, illness severity, and region. The variable definitions below are in keeping with a prior study of SSSS.¹⁶ Age was categorized as: birth to 59 days, 2 to 11 months, 1 to 4 years (preschool age), 5 to 10 years (school age), and 11 to 18 years (adolescent). We examined infants younger than 60 days separately from older infants because this population may warrant additional treatment considerations. Race and ethnicity were categorized as White (non-Hispanic), African American (non-Hispanic), Hispanic, or other. Payer types included government, private, or other. Level of care was assigned as either intensive care or acute care. Illness severity was assigned using the All Patient Refined Diagnosis Related Group (APR-DRG; 3M Corporation, St. Paul, Minnesota) severity levels.¹⁹ In line with a prior study,¹⁶ we defined “low illness severity” as the APR-DRG minor (1) classification. The moderate (2), major (3), and extreme (4) classifications were defined as “moderate to high illness severity,” since there were very few classifications of major or extreme (<5%) illness severity. We categorized hospitals into the following US regions: Northeast, Midwest, South, and West.

Outcome Measures

The primary outcome was hospital LOS in days, and secondary outcomes were treatment failure and hospital costs. Hospital LOS was chosen as the primary outcome to represent the time needed for the child to show clinical improvement. Treatment failure was defined as a same-cause 14-day ED revisit or hospital readmission, and these were determined to be same-cause if a diagnosis for SSSS (ICD-9, 695.81; ICD-10, L00) was documented for the return encounter. The 14-day interval for readmission and ED revisit was chosen to measure any relapse of symptoms after completion of antibiotic therapy, similar to a prior study of treatment failure in skin and soft tissue infections.²⁰ Total costs of the hospitalization were estimated from charges using hospital- and year-specific cost-to-charge ratios. Subcategories of cost, including clinical, pharmacy, imaging, laboratory, supply, and other, were also compared among the three groups.

Statistical Analysis

Demographic and clinical characteristics of children were summarized using frequencies and percentages for categorical variables and medians with interquartile ranges (IQRs) for continuous variables. These were compared across antibiotic groups using chi-square and Kruskal–Wallis tests, respectively. In unadjusted analyses, outcomes were compared across antibiotic regimen groups using these same statistical tests. In order to account for patient clustering within hospitals, generalized linear mixed-effects models were used to model outcomes with a random intercept for each hospital. Models were adjusted for SSSS being listed as a principal or secondary diagnosis, race, illness severity, and level of care. We log-transformed LOS and cost data prior to modeling because of the nonnormal distributions for these data. Owing to the inability to measure the number of antibiotic doses, and to reduce the possibility of including children who received few regimen-defined combination antibiotics, a post hoc sensitivity analysis was performed. This analysis used an alternative definition for antibiotic regimen groups, for which children admitted for 2 or more calendar days must have received regimen-specified antibiotics on at least 2 days of the admission. Additionally, outcomes were stratified by low and moderate/high illness severity and compared across the three antibiotic regimen groups. All analyses were performed with SAS (SAS 9.4; SAS Institute, Cary, North Carolina), and P values of less than .05 were considered statistically significant.

Overall, 1,815 hospitalized children with SSSS were identified in the PHIS database, and after application of the exclusion criteria, 1,259 children remained, with 1,247 (99%) receiving antibiotics (Figure). The antibiotic regimens received by these children are described in Table 1. Of these, 828 children (66%) received one of the three most common antistaphylococcal regimens (clindamycin, clindamycin + MSSA, and clindamycin + MRSA) and were included for further analysis.

neubauer04830217e_f1.jpg

Characteristics of the 828 children are presented in Table 2. Most children (82%) were aged 4 years or younger, and distributions of age, sex, and insurance payer were similar among children receiving the three regimens. Thirty-two percent had moderate to high illness severity, and 3.5% required management in the intensive care setting. Of the three antibiotic regimens, clindamycin monotherapy was most common (47%), followed by clindamycin plus MSSA coverage (33%), and clindamycin plus MRSA coverage (20%). A higher proportion of children in the clindamycin plus MRSA group were African American and were hospitalized in the South. Children receiving clindamycin plus MRSA coverage had higher illness severity (44%) as compared with clindamycin monotherapy (28%) and clindamycin plus MSSA coverage (32%) (P = .001). Additionally, a larger proportion of children treated with clindamycin plus MRSA coverage were managed in the intensive care setting as compared with the clindamycin plus MSSA or clindamycin monotherapy groups.

neubauer04830217e_t2.jpg

Among the 828 children with SSSS, the median LOS was 2 days (IQR, 2-3), and treatment failure was 1.1% (95% CI, 0.4-1.8). After adjustment for illness severity, race, payer, and region (Table 3), the three antibiotic regimens were not associated with significant differences in LOS or treatment failure. Costs were significantly different among the three antibiotic regimens. Clindamycin plus MRSA coverage was associated with the greatest costs, whereas clindamycin monotherapy was associated with the lowest costs (mean, $5,348 vs $4,839, respectively; P < .001) (Table 3). In a sensitivity analysis using an alternative antibiotic regimen definition, we found results in line with the primary analysis, with no statistically significant differences in LOS (P = .44) or treatment failure (P = .54), but significant differences in cost (P < .001). Additionally, the same findings were present for LOS, treatment failure, and cost when outcomes were stratified by illness severity (Appendix Table). However, significant contributors to the higher cost in the clindamycin plus MRSA group did vary by illness severity stratification. Laboratory, supply, and pharmacy cost categories differed significantly among antibiotic groups for the low illness severity strata, whereas pharmacy was the only significant cost category difference in moderate/high illness severity.

neubauer04830217e_t3.jpg

Clindamycin monotherapy, clindamycin plus MSSA coverage, and clindamycin plus MRSA coverage are the most commonly administered antistaphylococcal antibiotic regimens for children hospitalized with SSSS at US children’s hospitals. Our multicenter study found that, across these antistaphylococcal antibiotic regimens, there were no associated differences in hospital LOS or treatment failure. However, the antibiotic regimens were associated with significant differences in overall hospital costs. These findings suggest that the use of clindamycin with additional MSSA or MRSA antibiotic coverage for children with SSSS may not be associated with additional clinical benefit, as compared with clindamycin monotherapy, and could potentially be more costly.

Prior literature describing LOS in relation to antibiotic use for children with SSSS is limited. Authors of a recent case series of 21 children in Philadelphia reported approximately 50% of children received clindamycin monotherapy or combination therapy, but patient outcomes such as LOS were not described.⁹ Clindamycin use and outcomes have been described in smaller studies and case reports of SSSS, which reported positive outcomes such as patient recovery and lack of disease recurrence.^2,9,21 A small retrospective, comparative effectiveness study of 30 neonates with SSSS examined beta-lactamase–resistant penicillin use with and without cephalosporins. They found no effect on LOS, but findings were limited by a small sample size.²² Our study cohort included relatively few neonates, and thus our findings may not be applicable to this population subgroup. We chose not to include regimens with third-generation cephalosporins or ampicillin, which may have limited the number of included neonates, because these antibiotics are frequently administered during evaluation for invasive bacterial infections.²³ We found a very low occurrence of treatment failure in our study cohort across all three groups, which is consistent with other studies of SSSS that report an overall good prognosis and low recurrence and/or readmission rates.^6,16,24 The low prevalence of treatment failure, however, precluded our ability to detect small differences among antibiotic regimen groups that may exist.

We observed that cost differed significantly across antibiotic regimen groups, with lowest cost associated with clindamycin monotherapy in adjusted analysis despite similar LOS. Even with our illness-severity adjustment, there may have been other unmeasured factors resulting in the higher cost associated with the combination groups. Hence, we also examined cost breakdown with a stratified analysis by illness severity. We found that pharmacy costs were significantly different among antibiotic groups in both illness severity strata, whereas those with low illness severity also differed by laboratory and supply costs. Thus, pharmacy cost differences may be the largest driver in the cost differential among groups. Lower cost in the clindamycin monotherapy group is likely due to administration of a single antibiotic. The reason for supply and laboratory cost differences is uncertain, but higher cost in the clindamycin plus MRSA group could possibly be from laboratory testing related to drug monitoring (eg, renal function testing or drug levels). While other studies have reported costs for hospitalized children with SSSS associated with different patient characteristics or diagnostic testing,^1,16 to our knowledge, no other studies have reported cost related to antibiotic regimens for SSSS. As healthcare reimbursements shift to value-based models, identifying treatment regimens with equal efficacy but lower cost will become increasingly important. Future studies should also examine other covariates and outcomes, such as oral vs parenteral antibiotic use, use of monitoring laboratories related to antibiotic choice, and adverse drug effects.

Several strengths and additional limitations apply to our study. Our study is one of the few to describe outcomes associated with antibiotic regimens for children with SSSS. With the PHIS database, we were able to include a large number of children with SSSS from children’s hospitals across the United States. Although the PHIS database affords these strengths, there are limitations inherent to administrative data. Children with SSSS were identified by documented ICD-9 and ICD-10 diagnostic codes, which might lead to misclassification. However, misclassification is less likely because only one ICD-9 and ICD-10 code exists for SSSS, and the characteristics of this condition are specific. Also, diagnostic codes for other dermatologic conditions (eg, scarlet fever) were excluded to further reduce the chance of misclassification. A limitation to our use of PHIS billing codes was the inability to confirm the dosage of antibiotics given, the number of doses, or whether antibiotics were prescribed upon discharge. Another limitation is that children whose antibiotic therapy was changed during hospitalization (eg, from clindamycin monotherapy to cefazolin monotherapy) were categorized into the combination groups. However, the sensitivity analysis performed based on a stricter antibiotic group definition (receipt of both antibiotics on at least 2 calendar days) did not alter the outcomes, which is reassuring. We were unable to assess the use of targeted antibiotic therapy because clinical data (eg, microbiology results) were not available. However, this may be less important because some literature suggests that cultures for S aureus are obtained infrequently² and may be difficult to interpret when obtained,²⁵ since culture growth can represent colonization rather than causative strains. An additional limitation is that administrative data do not include certain clinical outcomes, such as fever duration or degree of skin involvement, which could have differed among the groups. Last, the PHIS database only captures revisits or readmissions to PHIS hospitals, and so we are unable to exclude the possibility of a child being seen at or readmitted to another hospital.

Due to the observational design of this study and potential for incomplete measurement of illness severity, we recommend a future prospective trial with randomization to confirm these findings. One possible reason that LOS did not differ among groups is that the burden of clindamycin-resistant strains in our cohort could be low, and the addition of MSSA or MRSA coverage does not result in a clinically important increase in S aureus coverage. However, pooled pediatric hospital antibiogram data suggest the overall rate of clindamycin resistance is close to 20% in hospitals located in all US regions.²⁶ Limited studies also suggest that MSSA may be the predominant pathogen associated with SSSS.^2,9 To address this, future randomized trials could compare the effectiveness of clindamycin monotherapy to MSSA-specific agents like cefazolin or nafcillin. Unfortunately, anti-MSSA monotherapy was not evaluated in our study because very few children received this treatment. Using monotherapy as opposed to multiple antibiotics has the potential to promote antibiotic stewardship for antistaphylococcal antibiotics in the management of SSSS. Reducing unnecessary antibiotic use not only potentially affects antibiotic resistance, but could also benefit patients in reducing possible side effects, cost, and IV catheter complications.²⁷ However, acknowledging our study limitations, our findings should be applied cautiously in clinical settings, in the context of local antibiogram data, individual culture results, and specific patient factors. The local clindamycin resistance rate for both MSSA and MRSA should be considered. Many antibiotics chosen to treat MRSA—such as vancomycin and trimethoprim/sulfamethoxazole—will also have anti-MSSA activity and may have lower local resistance rates than clindamycin. Practitioners may also consider how each antibiotic kills bacteria; for example, beta-lactams rely on bacterial replication, but clindamycin does not. Each factor should influence how empiric treatment, whether monotherapy or combination, is chosen for children with SSSS.

In this large, multicenter cohort of hospitalized children with SSSS, we found that the addition of MSSA or MRSA coverage to clindamycin monotherapy was not associated with differences in outcomes of hospital LOS and treatment failure. Furthermore, clindamycin monotherapy was associated with lower overall cost. Prospective randomized studies are needed to confirm these findings and assess whether clindamycin monotherapy, monotherapy with an anti-MSSA antibiotic, or alternative regimens are most effective for treatment of children with SSSS.

Staphylococcal scalded skin syndrome (SSSS) is an exfoliative toxin-mediated dermatitis that predominantly occurs in young children. Multiple recent reports indicate a rising incidence of this disease.^1-4 Recommended treatment for SSSS includes antistaphylococcal antibiotics and supportive care measures.^5,6 Elimination or reduction of the toxin-producing Staphylococcus aureus is thought to help limit disease progression and promote recovery. Experts advocate for the use of antibiotics even when there is no apparent focal source of infection, such as an abscess.^6,7

Several factors may affect antibiotic selection, including the desire to inhibit toxin production and to target the causative pathogen in a bactericidal fashion. Because SSSS is toxin mediated, clindamycin is often recommended because of its inhibition of toxin synthesis.^5,8 The clinical utility of adding other antibiotics to clindamycin for coverage of methicillin-sensitive S aureus (MSSA) or methicillin-resistant S aureus (MRSA) is uncertain. Several studies report MSSA to be the predominant pathogen identified by culture^2,9; however, SSSS caused by MRSA has been reported.^9-11 Additionally, bactericidal antibiotics (eg, nafcillin) have been considered to hold potential clinical advantage as compared with bacteriostatic antibiotics (eg, clindamycin), even though clinical studies have not clearly demonstrated this advantage in the general population.^12,13 Some experts recommend additional MRSA or MSSA coverage (such as vancomycin or nafcillin) in patients with high illness severity or nonresponse to therapy, or in areas where there is high prevalence of staphylococcal resistance to clindamycin.^5,7,9,14 Alternatively, for areas with low MRSA prevalence, monotherapy with an anti-MSSA antibiotic is another potential option. No recent studies have compared patient outcomes among antibiotic regimens in children with SSSS.

Knowledge of the outcomes associated with different antibiotic regimens for children hospitalized with SSSS is needed and could be used to improve patient outcomes and potentially promote antibiotic stewardship. In this study, our objectives were to (1) describe antibiotic regimens given to children hospitalized with SSSS, and (2) examine the association of three antibiotic regimens commonly used for SSSS (clindamycin monotherapy, clindamycin plus additional MSSA coverage, and clindamycin plus additional MRSA coverage) with patient outcomes of length of stay (LOS), treatment failure, and cost in a large cohort of children at US children’s hospitals.

We conducted a multicenter, retrospective cohort study utilizing data within the Pediatric Health Information System (PHIS) database from July 1, 2011, to June 30, 2016. Thirty-five free-standing tertiary care US children’s hospitals within 24 states were included. The Children’s Hospital Association (Lenexa, Kansas) maintains the PHIS database, which contains de-identified patient information, including diagnoses (with International Classification of Diseases, Ninth and Tenth Revision, Clinical Modification [ICD-9-CM, ICD-10-CM]), demographics, procedures, and daily billing records. Data quality and reliability are confirmed by participating institutions and the Children’s Hospital Association.¹⁵ The local institutional review board (IRB) deemed the study exempt from formal IRB review, as patient information was de-identified.

Study Population

We included hospitalized children aged newborn to 18 years with a primary or secondary diagnosis of SSSS (ICD-9, 695.81; ICD-10, L00). Children whose primary presentation and admission were to a PHIS hospital were included; children transferred from another hospital were excluded. The following exclusion criteria were based on previously published methodology.¹⁶ Children with complex chronic medical conditions as classified by Feudtner et al¹⁷ were excluded, since these children may require a different treatment approach than the general pediatric population. In order to decrease diagnostic ambiguity, we excluded children if an alternative dermatologic diagnosis was recorded as a principal or secondary diagnosis (eg, Stevens-Johnson syndrome or scarlet fever).¹⁶ Finally, hospitals with fewer than 10 children with SSSS during the study period were excluded.

Antibiotic Regimen Groups

We used PHIS daily billing codes to determine the antibiotics received by the study population. Children were classified into antibiotic regimen groups based on whether they received specific antibiotic combinations. Antibiotics received on any day during the hospitalization, including in the emergency department (ED), were used to assign patients to regimen groups. Antibiotics were classified into regimen groups based on consensus among study investigators, which included two board-certified pediatric infectious diseases specialists (A.C., R.M.). Antibiotic group definitions are listed in Table 1. Oral and intravenous (IV) therapies were grouped together for clindamycin, cephalexin/cefazolin, and linezolid because of good oral bioavailability in most situations.¹⁸ The three most common antistaphylococcal groups were chosen for further analysis: clindamycin alone, clindamycin plus MSSA coverage, and clindamycin plus MRSA coverage. The clindamycin group was defined as children with receipt of oral or IV clindamycin. Children who received clindamycin with additional MSSA coverage, such as cefazolin or nafcillin, were categorized as the clindamycin plus MSSA group. Children who received clindamycin with additional MRSA coverage, such as vancomycin or linezolid, were categorized as the clindamycin plus MRSA group. We chose not to include children who received the above regimens plus other antibiotics with partial antistaphylococcal activity, such as ampicillin, gentamicin, or ceftriaxone, in the clindamycin plus MSSA and clindamycin plus MRSA groups. We excluded these antibiotics to decrease the heterogeneity in the definition of regimen groups and allow a more direct comparison for effectiveness.

neubauer04830217e_t1.jpg

Covariates

Covariates included age, sex, ethnicity and/or race, payer type, level of care, illness severity, and region. The variable definitions below are in keeping with a prior study of SSSS.¹⁶ Age was categorized as: birth to 59 days, 2 to 11 months, 1 to 4 years (preschool age), 5 to 10 years (school age), and 11 to 18 years (adolescent). We examined infants younger than 60 days separately from older infants because this population may warrant additional treatment considerations. Race and ethnicity were categorized as White (non-Hispanic), African American (non-Hispanic), Hispanic, or other. Payer types included government, private, or other. Level of care was assigned as either intensive care or acute care. Illness severity was assigned using the All Patient Refined Diagnosis Related Group (APR-DRG; 3M Corporation, St. Paul, Minnesota) severity levels.¹⁹ In line with a prior study,¹⁶ we defined “low illness severity” as the APR-DRG minor (1) classification. The moderate (2), major (3), and extreme (4) classifications were defined as “moderate to high illness severity,” since there were very few classifications of major or extreme (<5%) illness severity. We categorized hospitals into the following US regions: Northeast, Midwest, South, and West.

Outcome Measures

The primary outcome was hospital LOS in days, and secondary outcomes were treatment failure and hospital costs. Hospital LOS was chosen as the primary outcome to represent the time needed for the child to show clinical improvement. Treatment failure was defined as a same-cause 14-day ED revisit or hospital readmission, and these were determined to be same-cause if a diagnosis for SSSS (ICD-9, 695.81; ICD-10, L00) was documented for the return encounter. The 14-day interval for readmission and ED revisit was chosen to measure any relapse of symptoms after completion of antibiotic therapy, similar to a prior study of treatment failure in skin and soft tissue infections.²⁰ Total costs of the hospitalization were estimated from charges using hospital- and year-specific cost-to-charge ratios. Subcategories of cost, including clinical, pharmacy, imaging, laboratory, supply, and other, were also compared among the three groups.

Statistical Analysis

Demographic and clinical characteristics of children were summarized using frequencies and percentages for categorical variables and medians with interquartile ranges (IQRs) for continuous variables. These were compared across antibiotic groups using chi-square and Kruskal–Wallis tests, respectively. In unadjusted analyses, outcomes were compared across antibiotic regimen groups using these same statistical tests. In order to account for patient clustering within hospitals, generalized linear mixed-effects models were used to model outcomes with a random intercept for each hospital. Models were adjusted for SSSS being listed as a principal or secondary diagnosis, race, illness severity, and level of care. We log-transformed LOS and cost data prior to modeling because of the nonnormal distributions for these data. Owing to the inability to measure the number of antibiotic doses, and to reduce the possibility of including children who received few regimen-defined combination antibiotics, a post hoc sensitivity analysis was performed. This analysis used an alternative definition for antibiotic regimen groups, for which children admitted for 2 or more calendar days must have received regimen-specified antibiotics on at least 2 days of the admission. Additionally, outcomes were stratified by low and moderate/high illness severity and compared across the three antibiotic regimen groups. All analyses were performed with SAS (SAS 9.4; SAS Institute, Cary, North Carolina), and P values of less than .05 were considered statistically significant.

Overall, 1,815 hospitalized children with SSSS were identified in the PHIS database, and after application of the exclusion criteria, 1,259 children remained, with 1,247 (99%) receiving antibiotics (Figure). The antibiotic regimens received by these children are described in Table 1. Of these, 828 children (66%) received one of the three most common antistaphylococcal regimens (clindamycin, clindamycin + MSSA, and clindamycin + MRSA) and were included for further analysis.

neubauer04830217e_f1.jpg

Characteristics of the 828 children are presented in Table 2. Most children (82%) were aged 4 years or younger, and distributions of age, sex, and insurance payer were similar among children receiving the three regimens. Thirty-two percent had moderate to high illness severity, and 3.5% required management in the intensive care setting. Of the three antibiotic regimens, clindamycin monotherapy was most common (47%), followed by clindamycin plus MSSA coverage (33%), and clindamycin plus MRSA coverage (20%). A higher proportion of children in the clindamycin plus MRSA group were African American and were hospitalized in the South. Children receiving clindamycin plus MRSA coverage had higher illness severity (44%) as compared with clindamycin monotherapy (28%) and clindamycin plus MSSA coverage (32%) (P = .001). Additionally, a larger proportion of children treated with clindamycin plus MRSA coverage were managed in the intensive care setting as compared with the clindamycin plus MSSA or clindamycin monotherapy groups.

neubauer04830217e_t2.jpg

Among the 828 children with SSSS, the median LOS was 2 days (IQR, 2-3), and treatment failure was 1.1% (95% CI, 0.4-1.8). After adjustment for illness severity, race, payer, and region (Table 3), the three antibiotic regimens were not associated with significant differences in LOS or treatment failure. Costs were significantly different among the three antibiotic regimens. Clindamycin plus MRSA coverage was associated with the greatest costs, whereas clindamycin monotherapy was associated with the lowest costs (mean, $5,348 vs $4,839, respectively; P < .001) (Table 3). In a sensitivity analysis using an alternative antibiotic regimen definition, we found results in line with the primary analysis, with no statistically significant differences in LOS (P = .44) or treatment failure (P = .54), but significant differences in cost (P < .001). Additionally, the same findings were present for LOS, treatment failure, and cost when outcomes were stratified by illness severity (Appendix Table). However, significant contributors to the higher cost in the clindamycin plus MRSA group did vary by illness severity stratification. Laboratory, supply, and pharmacy cost categories differed significantly among antibiotic groups for the low illness severity strata, whereas pharmacy was the only significant cost category difference in moderate/high illness severity.

neubauer04830217e_t3.jpg

Clindamycin monotherapy, clindamycin plus MSSA coverage, and clindamycin plus MRSA coverage are the most commonly administered antistaphylococcal antibiotic regimens for children hospitalized with SSSS at US children’s hospitals. Our multicenter study found that, across these antistaphylococcal antibiotic regimens, there were no associated differences in hospital LOS or treatment failure. However, the antibiotic regimens were associated with significant differences in overall hospital costs. These findings suggest that the use of clindamycin with additional MSSA or MRSA antibiotic coverage for children with SSSS may not be associated with additional clinical benefit, as compared with clindamycin monotherapy, and could potentially be more costly.

Prior literature describing LOS in relation to antibiotic use for children with SSSS is limited. Authors of a recent case series of 21 children in Philadelphia reported approximately 50% of children received clindamycin monotherapy or combination therapy, but patient outcomes such as LOS were not described.⁹ Clindamycin use and outcomes have been described in smaller studies and case reports of SSSS, which reported positive outcomes such as patient recovery and lack of disease recurrence.^2,9,21 A small retrospective, comparative effectiveness study of 30 neonates with SSSS examined beta-lactamase–resistant penicillin use with and without cephalosporins. They found no effect on LOS, but findings were limited by a small sample size.²² Our study cohort included relatively few neonates, and thus our findings may not be applicable to this population subgroup. We chose not to include regimens with third-generation cephalosporins or ampicillin, which may have limited the number of included neonates, because these antibiotics are frequently administered during evaluation for invasive bacterial infections.²³ We found a very low occurrence of treatment failure in our study cohort across all three groups, which is consistent with other studies of SSSS that report an overall good prognosis and low recurrence and/or readmission rates.^6,16,24 The low prevalence of treatment failure, however, precluded our ability to detect small differences among antibiotic regimen groups that may exist.

We observed that cost differed significantly across antibiotic regimen groups, with lowest cost associated with clindamycin monotherapy in adjusted analysis despite similar LOS. Even with our illness-severity adjustment, there may have been other unmeasured factors resulting in the higher cost associated with the combination groups. Hence, we also examined cost breakdown with a stratified analysis by illness severity. We found that pharmacy costs were significantly different among antibiotic groups in both illness severity strata, whereas those with low illness severity also differed by laboratory and supply costs. Thus, pharmacy cost differences may be the largest driver in the cost differential among groups. Lower cost in the clindamycin monotherapy group is likely due to administration of a single antibiotic. The reason for supply and laboratory cost differences is uncertain, but higher cost in the clindamycin plus MRSA group could possibly be from laboratory testing related to drug monitoring (eg, renal function testing or drug levels). While other studies have reported costs for hospitalized children with SSSS associated with different patient characteristics or diagnostic testing,^1,16 to our knowledge, no other studies have reported cost related to antibiotic regimens for SSSS. As healthcare reimbursements shift to value-based models, identifying treatment regimens with equal efficacy but lower cost will become increasingly important. Future studies should also examine other covariates and outcomes, such as oral vs parenteral antibiotic use, use of monitoring laboratories related to antibiotic choice, and adverse drug effects.

Several strengths and additional limitations apply to our study. Our study is one of the few to describe outcomes associated with antibiotic regimens for children with SSSS. With the PHIS database, we were able to include a large number of children with SSSS from children’s hospitals across the United States. Although the PHIS database affords these strengths, there are limitations inherent to administrative data. Children with SSSS were identified by documented ICD-9 and ICD-10 diagnostic codes, which might lead to misclassification. However, misclassification is less likely because only one ICD-9 and ICD-10 code exists for SSSS, and the characteristics of this condition are specific. Also, diagnostic codes for other dermatologic conditions (eg, scarlet fever) were excluded to further reduce the chance of misclassification. A limitation to our use of PHIS billing codes was the inability to confirm the dosage of antibiotics given, the number of doses, or whether antibiotics were prescribed upon discharge. Another limitation is that children whose antibiotic therapy was changed during hospitalization (eg, from clindamycin monotherapy to cefazolin monotherapy) were categorized into the combination groups. However, the sensitivity analysis performed based on a stricter antibiotic group definition (receipt of both antibiotics on at least 2 calendar days) did not alter the outcomes, which is reassuring. We were unable to assess the use of targeted antibiotic therapy because clinical data (eg, microbiology results) were not available. However, this may be less important because some literature suggests that cultures for S aureus are obtained infrequently² and may be difficult to interpret when obtained,²⁵ since culture growth can represent colonization rather than causative strains. An additional limitation is that administrative data do not include certain clinical outcomes, such as fever duration or degree of skin involvement, which could have differed among the groups. Last, the PHIS database only captures revisits or readmissions to PHIS hospitals, and so we are unable to exclude the possibility of a child being seen at or readmitted to another hospital.

Due to the observational design of this study and potential for incomplete measurement of illness severity, we recommend a future prospective trial with randomization to confirm these findings. One possible reason that LOS did not differ among groups is that the burden of clindamycin-resistant strains in our cohort could be low, and the addition of MSSA or MRSA coverage does not result in a clinically important increase in S aureus coverage. However, pooled pediatric hospital antibiogram data suggest the overall rate of clindamycin resistance is close to 20% in hospitals located in all US regions.²⁶ Limited studies also suggest that MSSA may be the predominant pathogen associated with SSSS.^2,9 To address this, future randomized trials could compare the effectiveness of clindamycin monotherapy to MSSA-specific agents like cefazolin or nafcillin. Unfortunately, anti-MSSA monotherapy was not evaluated in our study because very few children received this treatment. Using monotherapy as opposed to multiple antibiotics has the potential to promote antibiotic stewardship for antistaphylococcal antibiotics in the management of SSSS. Reducing unnecessary antibiotic use not only potentially affects antibiotic resistance, but could also benefit patients in reducing possible side effects, cost, and IV catheter complications.²⁷ However, acknowledging our study limitations, our findings should be applied cautiously in clinical settings, in the context of local antibiogram data, individual culture results, and specific patient factors. The local clindamycin resistance rate for both MSSA and MRSA should be considered. Many antibiotics chosen to treat MRSA—such as vancomycin and trimethoprim/sulfamethoxazole—will also have anti-MSSA activity and may have lower local resistance rates than clindamycin. Practitioners may also consider how each antibiotic kills bacteria; for example, beta-lactams rely on bacterial replication, but clindamycin does not. Each factor should influence how empiric treatment, whether monotherapy or combination, is chosen for children with SSSS.

In this large, multicenter cohort of hospitalized children with SSSS, we found that the addition of MSSA or MRSA coverage to clindamycin monotherapy was not associated with differences in outcomes of hospital LOS and treatment failure. Furthermore, clindamycin monotherapy was associated with lower overall cost. Prospective randomized studies are needed to confirm these findings and assess whether clindamycin monotherapy, monotherapy with an anti-MSSA antibiotic, or alternative regimens are most effective for treatment of children with SSSS.

Pediatric hospital medicine (PHM) is one of the most rapidly growing disciplines in pediatrics,¹ with 8% of pediatric residency graduates each year entering the field.² Research plays an important role in advancing care in the field and is a critical component for board certification and fellowship accreditation.^3-6 The annual PHM conference, which has been jointly sponsored by the Academic Pediatric Association, the American Academy of Pediatrics, and the Society of Hospital Medicine, is an important venue for the dissemination of research findings. Abstract selection is determined by peer review; however, reviewers are provided with only a brief snapshot of the research, which may not contain sufficient information to fully evaluate the methodological quality of the work.^7-10 Additionally, while instructions are provided, reviewers often lack formal training in abstract review. Consequently, scores may vary.⁹

Publication in a peer-reviewed journal is considered a measure of research success because it requires more rigorous peer review than the abstract selection process at scientific meetings.^11-16 Rates of subsequent journal publication differ based on specialty and meeting, and they have been reported at 23% to 78%.^10,12,14-18 In pediatrics, publication rates after presentation at scientific meetings range from 36% to 63%, with mean time to publication ranging from 20 to 26 months following the meeting.^11,19,20 No studies have reviewed abstract submissions to the annual PHM meeting to determine if selection or presentation format is associated with subsequent publication in a peer-reviewed journal.

We sought to identify the publication rate of abstracts submitted to the 2014 PHM conference and determine whether presentation format was associated with the likelihood of subsequent journal publication or time to publication.

Study Design

Data for this retrospective cohort study were obtained from a database of all abstracts submitted for presentation at the 2014 PHM conference in Lake Buena Vista, Florida.

Main Exposures

The main exposure was presentation format, which was categorized as not presented (ie, rejected), poster presentation, or oral presentation. PHM has a blinded abstract peer-review process; in 2014, an average of 10 reviewers scored each abstract. Reviewers graded abstracts on a scale of 1 (best in category) to 7 (unacceptable for presentation) according to the following criteria: originality, scientific importance, methodological rigor, and quality of presentation. Abstracts with the lowest average scores in each content area, usually less than or equal to 3, were accepted as oral presentations while most abstracts with scores greater than 5 were rejected. For this study, information collected from each abstract included authors, if the primary author was a trainee, title, content area, and presentation format. Content areas included clinical research, educational research, health services research (HSR) and/or epidemiology, practice management research, and quality improvement. Abstracts were then grouped by presentation format and content area for analysis. The Pediatric Academic Societies (PAS) annual meeting, another common venue for the presentation of pediatric research, precedes the PHM conference. Because acceptance for PAS presentation may represent more strongly developed abstract submissions for PHM, we identified which abstracts had also been presented at the PAS conference that same year by cross-referencing authors and abstract titles with the PAS 2014 program.

Main Outcome Measures

All submissions were assessed for subsequent publication in peer-reviewed journals through January 2017 (30 months following the July 2014 PHM conference). To identify abstracts that went on to full publication, 2 authors (JC and LEH) independently searched for the lead author’s name and the presentation title in PubMed, Google Scholar, and MedEdPORTAL in January 2017. PubMed was searched using both the general search box and an advanced search for author and title. Google Scholar was added to capture manuscripts that may not have been indexed in PubMed at the time of our search. MedEdPORTAL, a common venue for the publication of educational initiatives that are not currently indexed in PubMed, was searched by lead author name via the general search box. If a full manuscript was published discussing similar outcomes or results and was written by the same authors who had submitted a PHM conference abstract, it was considered to have been published. The journal, month, and year of publication were recorded. For journals published every 2 months, the date of publication was recorded as falling between the 2 months. For those journals with biannual publication in the spring and fall, the months of March and October were used, respectively. The impact factor of the publication journal was also recorded for the year preceding publication. A journal’s impact factor is frequently used as a quantitative measure of journal quality and reflects the frequency with which a journal’s articles are cited in the scientific literature.²¹ Journals without an impact factor (eg, newer journals) were assigned a 0.

Data Analysis

All abstracts submitted to the PHM conference were analyzed based on content area and presentation format. The proportion of all abstracts subsequently published was determined for each format type and content area, and the odds ratio (OR) for publication after abstract submission was calculated using logistic regression. We calculated an adjusted OR for subsequent publication controlling for PAS presentation and the trainee status of the primary author. The journals most frequently publishing abstracts submitted to the PHM conference were identified. Median time to publication was calculated using the number of months elapsed between the PHM conference and publication date and compared across all abstract formats using Cox proportional hazards models adjusted for PAS presentation and trainee status. Kaplan-Meier survival curves were also generated for each of the 3 formats and compared using log-rank tests. The median impact factor was determined for each abstract format and compared using Wilcoxon rank-sum tests. Median impact factor by content area was compared using a Kruskal-Wallis test. All statistical analyses were performed using SAS version 9.2 (SAS Institute, Cary, NC). P values < 0.05 were considered statistically significant. In accordance with the Common Rule²² and the policies of the Cincinnati Children’s Hospital Medical Center Institutional Review Board, this research was not considered human subjects research.

herrmann1004e_t1.jpg

For the 2014 PHM meeting, 226 abstracts were submitted, of which 183 (81.0%) were selected for presentation, including 154 (68.0%) as poster presentations and 29 (12.8%) as oral presentations. Of all submitted abstracts, 82 (36.3%) were published within 30 months following the meeting. Eighty-one of these (98.8%) were identified via PubMed, and 1 was found only in MedEdPORTAL. No additional publications were found via Google Scholar. The presenting author for the PHM abstract was the first author for 87.8% (n = 72) of the publications. A trainee was the presenting author for only 2 of these abstracts. For the publications in which the first author was not the presenting author, the presenting author was the senior author in 2 of the publications and the second or third author on the remaining 8. Of the abstracts accepted for presentation, 70 (38.3%) were subsequently published. Abstracts accepted for oral presentation had almost 7-fold greater odds of subsequent publication than those that were rejected (Table 1; OR 6.8; 95% confidence interval [CI], 2.4-19.4). Differences in the odds of publication for rejected abstracts compared with those accepted for poster presentation were not statistically significant (OR 1.2; 95% CI, 0.5-2.5).

herrmann1004e_f1.jpg

herrmann1004e_t2.jpg

herrmann1004e_t3.jpg

About one-third of abstracts submitted to the 2014 PHM conference were subsequently published in peer-reviewed journals within 30 months of the conference. Compared with rejected abstracts, the rate of publication was significantly higher for abstracts selected for oral presentation but not for those selected for poster presentation. For abstracts ultimately published in journals, selection for oral presentation was significantly associated with both a shorter time to publication and a higher median journal impact factor compared with rejected abstracts. Time to publication and median journal impact factor were similar between rejected abstracts and those accepted for poster presentation. Our findings suggest that abstract reviewers may be able to identify which abstracts will ultimately withstand more stringent peer review in the publication process when accepting abstracts for oral presentation. However, the selection for poster presentation versus rejection may not be indicative of future publication or the impact factor of the subsequent publication journal.

Previous studies have reviewed publication rates after meetings of the European Society for Pediatric Urology (publication rate of 47%),¹¹ the Ambulatory Pediatric Association (now the Academic Pediatric Association; publication rate of 47%), the American Pediatric Society/Society for Pediatric Research (publication rate of 54%), and the PAS (publication rate of 45%).^19,20 Our lower publication rate of 36.3% may be attributed to the shorter follow-up time in our study (30 months from the PHM conference), whereas prior studies monitored for publication up to 60 months after the PAS conference.²⁰ Factors associated with subsequent publication include statistically significant results, a large sample size, and a randomized controlled trial study design.^15,16 The primary reason for nonpublication for up to 80% of abstracts is failure to submit a manuscript for publication.²³ A lack of time and fear of rejection after peer review are commonly cited explanations.^18,23,24 Individuals may view acceptance for an oral presentation as positive reinforcement and be more motivated to pursue subsequent manuscript publication than individuals whose abstracts are offered poster presentations or are rejected. Trainees frequently present abstracts at scientific meetings, representing 40.3% of primary authors submitting abstracts to PHM in 2014, but may not have sufficient time or mentorship to develop a complete manuscript.¹⁸ To our knowledge, there have been no publications that assess the impact of trainee status on subsequent publication after conference submission.

Our study demonstrated that selection for oral presentation was associated with subsequent publication, shorter time to publication, and publication in journals with higher impact factors. A 2005 Cochrane review also demonstrated that selection for oral presentation was associated with subsequent journal publication.¹⁶ Abstracts accepted for oral publication may represent work further along in the research process, with more developed methodology and results. The shorter time to publication for abstracts accepted for oral presentation could also reflect feedback provided by conference attendees after the presentation, whereas poster sessions frequently lack a formalized process for critique.

Carroll et al. found no difference in time to publication between abstracts accepted for presentation at the PAS and rejected abstracts.²⁰ Previous studies demonstrate that most abstracts presented at scientific meetings that are subsequently accepted for publication are published within 2 to 3 years of the meeting,¹² with publication rates as high as 98% within 3 years of presentation.¹⁷ In contrast to Carroll et al., we found that abstracts accepted for oral presentation had a 4-fold greater likelihood of publication at each month than rejected abstracts. However, abstracts accepted for poster presentation did not have a significant difference in the proportional hazard ratio models for publication compared with rejected abstracts. Because space considerations limit the number of abstracts that can be accepted for presentation at a conference, some abstracts that are suitable for future publication may have been rejected due to a lack of space. Because researchers often use scientific meetings as a forum to receive peer feedback,¹² authors who present at conferences may take more time to write a manuscript in order to incorporate this feedback.

The most common journal in which submitted abstracts were subsequently published was Hospital Pediatrics, representing twice as many published manuscripts as the second most frequent journal, Pediatrics. Hospital Pediatrics, which was first published in 2011, did not have an impact factor assigned during the study period. Yet, as a peer-reviewed journal dedicated to the field of PHM, it is well aligned with the research presented at the PHM meeting. It is unclear if Hospital Pediatrics is a journal to which pediatric hospitalists tend to submit manuscripts initially or if manuscripts are frequently submitted elsewhere prior to their publication in Hospital Pediatrics. Submission to other journals first likely extends the time to publication, especially for abstracts accepted for poster presentation, which may describe studies with less developed methods or results.

This study has several limitations. Previous studies have demonstrated mean time to publication of 12 to 32 months following abstract presentation with a median time of 19.6 months.¹⁶ Because we only have a 30-month follow-up, there may be abstracts still in the review process that are yet to be published, especially because the length of the review process varies by journal. We based our literature search on the first author of each PHM conference abstract submission, assuming that this presenting author would be one of the publishing authors even if not remaining first author; if this was not the case, we may have missed some abstracts that were subsequently published in full. Likewise, if a presenting author’s last name changed prior to the publication of a manuscript, a publication may have been missed. This limitation would cause us to underestimate the overall publication rate. It is not clear whether this would differentially affect the method of presentation. However, in this study, there was concordance between the presenting author and the publication’s first author in 87.8% of the abstracts subsequently published in full. Presenting authors who did not remain the first author on the published manuscript maintained authorship as either the senior author or second or third author, which may represent changes in the degree of involvement or a division of responsibilities for individuals working on a project together. While our search methods were comprehensive, there is a possibility that abstracts may have been published in a venue that was not searched. Additionally, we only reviewed abstracts submitted to PHM for 1 year. As the field matures and the number of fellowship programs increases, the quality of submitted abstracts may increase, leading to higher publication rates or shorter times to publication. It is also possible that the publication rate may not be reflective of PHM as a field because hospitalists may submit their work to conferences other than the PHM. Lastly, it may be more challenging to interpret any differences in impact factor because some journals, including Hospital Pediatrics (which represented a plurality of poster presentation abstracts that were subsequently published and is a relatively new journal), did not have an impact factor assigned during the study period. Assigning a 0 to journals without an impact factor may artificially lower the average impact factor reported. Furthermore, an impact factor, which is based on the frequency with which an individual journal’s articles are cited in scientific or medical publications, may not necessarily reflect a journal’s quality.

Of the 226 abstracts submitted to the 2014 PHM conference, approximately one-third were published in peer-reviewed journals within 30 months of the conference. Selection for oral presentation was found to be associated with subsequent publication as well as publication in journals with higher impact factors. The overall low publication rate may indicate a need for increased mentorship and resources for research development in this growing specialty. Improved mechanisms for author feedback at poster sessions may provide constructive suggestions for further development of these projects into full manuscripts or opportunities for trainees and early-career hospitalists to network with more experienced researchers in the field.

Disclosure

Drs. Herrmann, Hall, Kyler, Andrews, Williams, and Shah and Mr. Cochran have nothing to disclose. Dr. Wilson reports personal fees from the American Academy of Pediatrics during the conduct of the study. The authors have no financial relationships relevant to this article to disclose.

Pediatric hospital medicine (PHM) is one of the most rapidly growing disciplines in pediatrics,¹ with 8% of pediatric residency graduates each year entering the field.² Research plays an important role in advancing care in the field and is a critical component for board certification and fellowship accreditation.^3-6 The annual PHM conference, which has been jointly sponsored by the Academic Pediatric Association, the American Academy of Pediatrics, and the Society of Hospital Medicine, is an important venue for the dissemination of research findings. Abstract selection is determined by peer review; however, reviewers are provided with only a brief snapshot of the research, which may not contain sufficient information to fully evaluate the methodological quality of the work.^7-10 Additionally, while instructions are provided, reviewers often lack formal training in abstract review. Consequently, scores may vary.⁹

Publication in a peer-reviewed journal is considered a measure of research success because it requires more rigorous peer review than the abstract selection process at scientific meetings.^11-16 Rates of subsequent journal publication differ based on specialty and meeting, and they have been reported at 23% to 78%.^10,12,14-18 In pediatrics, publication rates after presentation at scientific meetings range from 36% to 63%, with mean time to publication ranging from 20 to 26 months following the meeting.^11,19,20 No studies have reviewed abstract submissions to the annual PHM meeting to determine if selection or presentation format is associated with subsequent publication in a peer-reviewed journal.

We sought to identify the publication rate of abstracts submitted to the 2014 PHM conference and determine whether presentation format was associated with the likelihood of subsequent journal publication or time to publication.

Study Design

Data for this retrospective cohort study were obtained from a database of all abstracts submitted for presentation at the 2014 PHM conference in Lake Buena Vista, Florida.

Main Exposures

The main exposure was presentation format, which was categorized as not presented (ie, rejected), poster presentation, or oral presentation. PHM has a blinded abstract peer-review process; in 2014, an average of 10 reviewers scored each abstract. Reviewers graded abstracts on a scale of 1 (best in category) to 7 (unacceptable for presentation) according to the following criteria: originality, scientific importance, methodological rigor, and quality of presentation. Abstracts with the lowest average scores in each content area, usually less than or equal to 3, were accepted as oral presentations while most abstracts with scores greater than 5 were rejected. For this study, information collected from each abstract included authors, if the primary author was a trainee, title, content area, and presentation format. Content areas included clinical research, educational research, health services research (HSR) and/or epidemiology, practice management research, and quality improvement. Abstracts were then grouped by presentation format and content area for analysis. The Pediatric Academic Societies (PAS) annual meeting, another common venue for the presentation of pediatric research, precedes the PHM conference. Because acceptance for PAS presentation may represent more strongly developed abstract submissions for PHM, we identified which abstracts had also been presented at the PAS conference that same year by cross-referencing authors and abstract titles with the PAS 2014 program.

Main Outcome Measures

All submissions were assessed for subsequent publication in peer-reviewed journals through January 2017 (30 months following the July 2014 PHM conference). To identify abstracts that went on to full publication, 2 authors (JC and LEH) independently searched for the lead author’s name and the presentation title in PubMed, Google Scholar, and MedEdPORTAL in January 2017. PubMed was searched using both the general search box and an advanced search for author and title. Google Scholar was added to capture manuscripts that may not have been indexed in PubMed at the time of our search. MedEdPORTAL, a common venue for the publication of educational initiatives that are not currently indexed in PubMed, was searched by lead author name via the general search box. If a full manuscript was published discussing similar outcomes or results and was written by the same authors who had submitted a PHM conference abstract, it was considered to have been published. The journal, month, and year of publication were recorded. For journals published every 2 months, the date of publication was recorded as falling between the 2 months. For those journals with biannual publication in the spring and fall, the months of March and October were used, respectively. The impact factor of the publication journal was also recorded for the year preceding publication. A journal’s impact factor is frequently used as a quantitative measure of journal quality and reflects the frequency with which a journal’s articles are cited in the scientific literature.²¹ Journals without an impact factor (eg, newer journals) were assigned a 0.

Data Analysis

All abstracts submitted to the PHM conference were analyzed based on content area and presentation format. The proportion of all abstracts subsequently published was determined for each format type and content area, and the odds ratio (OR) for publication after abstract submission was calculated using logistic regression. We calculated an adjusted OR for subsequent publication controlling for PAS presentation and the trainee status of the primary author. The journals most frequently publishing abstracts submitted to the PHM conference were identified. Median time to publication was calculated using the number of months elapsed between the PHM conference and publication date and compared across all abstract formats using Cox proportional hazards models adjusted for PAS presentation and trainee status. Kaplan-Meier survival curves were also generated for each of the 3 formats and compared using log-rank tests. The median impact factor was determined for each abstract format and compared using Wilcoxon rank-sum tests. Median impact factor by content area was compared using a Kruskal-Wallis test. All statistical analyses were performed using SAS version 9.2 (SAS Institute, Cary, NC). P values < 0.05 were considered statistically significant. In accordance with the Common Rule²² and the policies of the Cincinnati Children’s Hospital Medical Center Institutional Review Board, this research was not considered human subjects research.

herrmann1004e_t1.jpg

For the 2014 PHM meeting, 226 abstracts were submitted, of which 183 (81.0%) were selected for presentation, including 154 (68.0%) as poster presentations and 29 (12.8%) as oral presentations. Of all submitted abstracts, 82 (36.3%) were published within 30 months following the meeting. Eighty-one of these (98.8%) were identified via PubMed, and 1 was found only in MedEdPORTAL. No additional publications were found via Google Scholar. The presenting author for the PHM abstract was the first author for 87.8% (n = 72) of the publications. A trainee was the presenting author for only 2 of these abstracts. For the publications in which the first author was not the presenting author, the presenting author was the senior author in 2 of the publications and the second or third author on the remaining 8. Of the abstracts accepted for presentation, 70 (38.3%) were subsequently published. Abstracts accepted for oral presentation had almost 7-fold greater odds of subsequent publication than those that were rejected (Table 1; OR 6.8; 95% confidence interval [CI], 2.4-19.4). Differences in the odds of publication for rejected abstracts compared with those accepted for poster presentation were not statistically significant (OR 1.2; 95% CI, 0.5-2.5).

herrmann1004e_f1.jpg

herrmann1004e_t2.jpg

herrmann1004e_t3.jpg

About one-third of abstracts submitted to the 2014 PHM conference were subsequently published in peer-reviewed journals within 30 months of the conference. Compared with rejected abstracts, the rate of publication was significantly higher for abstracts selected for oral presentation but not for those selected for poster presentation. For abstracts ultimately published in journals, selection for oral presentation was significantly associated with both a shorter time to publication and a higher median journal impact factor compared with rejected abstracts. Time to publication and median journal impact factor were similar between rejected abstracts and those accepted for poster presentation. Our findings suggest that abstract reviewers may be able to identify which abstracts will ultimately withstand more stringent peer review in the publication process when accepting abstracts for oral presentation. However, the selection for poster presentation versus rejection may not be indicative of future publication or the impact factor of the subsequent publication journal.

Previous studies have reviewed publication rates after meetings of the European Society for Pediatric Urology (publication rate of 47%),¹¹ the Ambulatory Pediatric Association (now the Academic Pediatric Association; publication rate of 47%), the American Pediatric Society/Society for Pediatric Research (publication rate of 54%), and the PAS (publication rate of 45%).^19,20 Our lower publication rate of 36.3% may be attributed to the shorter follow-up time in our study (30 months from the PHM conference), whereas prior studies monitored for publication up to 60 months after the PAS conference.²⁰ Factors associated with subsequent publication include statistically significant results, a large sample size, and a randomized controlled trial study design.^15,16 The primary reason for nonpublication for up to 80% of abstracts is failure to submit a manuscript for publication.²³ A lack of time and fear of rejection after peer review are commonly cited explanations.^18,23,24 Individuals may view acceptance for an oral presentation as positive reinforcement and be more motivated to pursue subsequent manuscript publication than individuals whose abstracts are offered poster presentations or are rejected. Trainees frequently present abstracts at scientific meetings, representing 40.3% of primary authors submitting abstracts to PHM in 2014, but may not have sufficient time or mentorship to develop a complete manuscript.¹⁸ To our knowledge, there have been no publications that assess the impact of trainee status on subsequent publication after conference submission.

Our study demonstrated that selection for oral presentation was associated with subsequent publication, shorter time to publication, and publication in journals with higher impact factors. A 2005 Cochrane review also demonstrated that selection for oral presentation was associated with subsequent journal publication.¹⁶ Abstracts accepted for oral publication may represent work further along in the research process, with more developed methodology and results. The shorter time to publication for abstracts accepted for oral presentation could also reflect feedback provided by conference attendees after the presentation, whereas poster sessions frequently lack a formalized process for critique.

Carroll et al. found no difference in time to publication between abstracts accepted for presentation at the PAS and rejected abstracts.²⁰ Previous studies demonstrate that most abstracts presented at scientific meetings that are subsequently accepted for publication are published within 2 to 3 years of the meeting,¹² with publication rates as high as 98% within 3 years of presentation.¹⁷ In contrast to Carroll et al., we found that abstracts accepted for oral presentation had a 4-fold greater likelihood of publication at each month than rejected abstracts. However, abstracts accepted for poster presentation did not have a significant difference in the proportional hazard ratio models for publication compared with rejected abstracts. Because space considerations limit the number of abstracts that can be accepted for presentation at a conference, some abstracts that are suitable for future publication may have been rejected due to a lack of space. Because researchers often use scientific meetings as a forum to receive peer feedback,¹² authors who present at conferences may take more time to write a manuscript in order to incorporate this feedback.

The most common journal in which submitted abstracts were subsequently published was Hospital Pediatrics, representing twice as many published manuscripts as the second most frequent journal, Pediatrics. Hospital Pediatrics, which was first published in 2011, did not have an impact factor assigned during the study period. Yet, as a peer-reviewed journal dedicated to the field of PHM, it is well aligned with the research presented at the PHM meeting. It is unclear if Hospital Pediatrics is a journal to which pediatric hospitalists tend to submit manuscripts initially or if manuscripts are frequently submitted elsewhere prior to their publication in Hospital Pediatrics. Submission to other journals first likely extends the time to publication, especially for abstracts accepted for poster presentation, which may describe studies with less developed methods or results.

This study has several limitations. Previous studies have demonstrated mean time to publication of 12 to 32 months following abstract presentation with a median time of 19.6 months.¹⁶ Because we only have a 30-month follow-up, there may be abstracts still in the review process that are yet to be published, especially because the length of the review process varies by journal. We based our literature search on the first author of each PHM conference abstract submission, assuming that this presenting author would be one of the publishing authors even if not remaining first author; if this was not the case, we may have missed some abstracts that were subsequently published in full. Likewise, if a presenting author’s last name changed prior to the publication of a manuscript, a publication may have been missed. This limitation would cause us to underestimate the overall publication rate. It is not clear whether this would differentially affect the method of presentation. However, in this study, there was concordance between the presenting author and the publication’s first author in 87.8% of the abstracts subsequently published in full. Presenting authors who did not remain the first author on the published manuscript maintained authorship as either the senior author or second or third author, which may represent changes in the degree of involvement or a division of responsibilities for individuals working on a project together. While our search methods were comprehensive, there is a possibility that abstracts may have been published in a venue that was not searched. Additionally, we only reviewed abstracts submitted to PHM for 1 year. As the field matures and the number of fellowship programs increases, the quality of submitted abstracts may increase, leading to higher publication rates or shorter times to publication. It is also possible that the publication rate may not be reflective of PHM as a field because hospitalists may submit their work to conferences other than the PHM. Lastly, it may be more challenging to interpret any differences in impact factor because some journals, including Hospital Pediatrics (which represented a plurality of poster presentation abstracts that were subsequently published and is a relatively new journal), did not have an impact factor assigned during the study period. Assigning a 0 to journals without an impact factor may artificially lower the average impact factor reported. Furthermore, an impact factor, which is based on the frequency with which an individual journal’s articles are cited in scientific or medical publications, may not necessarily reflect a journal’s quality.

Of the 226 abstracts submitted to the 2014 PHM conference, approximately one-third were published in peer-reviewed journals within 30 months of the conference. Selection for oral presentation was found to be associated with subsequent publication as well as publication in journals with higher impact factors. The overall low publication rate may indicate a need for increased mentorship and resources for research development in this growing specialty. Improved mechanisms for author feedback at poster sessions may provide constructive suggestions for further development of these projects into full manuscripts or opportunities for trainees and early-career hospitalists to network with more experienced researchers in the field.

Disclosure

Drs. Herrmann, Hall, Kyler, Andrews, Williams, and Shah and Mr. Cochran have nothing to disclose. Dr. Wilson reports personal fees from the American Academy of Pediatrics during the conduct of the study. The authors have no financial relationships relevant to this article to disclose.

Pediatric hospital medicine (PHM) is one of the most rapidly growing disciplines in pediatrics,¹ with 8% of pediatric residency graduates each year entering the field.² Research plays an important role in advancing care in the field and is a critical component for board certification and fellowship accreditation.^3-6 The annual PHM conference, which has been jointly sponsored by the Academic Pediatric Association, the American Academy of Pediatrics, and the Society of Hospital Medicine, is an important venue for the dissemination of research findings. Abstract selection is determined by peer review; however, reviewers are provided with only a brief snapshot of the research, which may not contain sufficient information to fully evaluate the methodological quality of the work.^7-10 Additionally, while instructions are provided, reviewers often lack formal training in abstract review. Consequently, scores may vary.⁹

Publication in a peer-reviewed journal is considered a measure of research success because it requires more rigorous peer review than the abstract selection process at scientific meetings.^11-16 Rates of subsequent journal publication differ based on specialty and meeting, and they have been reported at 23% to 78%.^10,12,14-18 In pediatrics, publication rates after presentation at scientific meetings range from 36% to 63%, with mean time to publication ranging from 20 to 26 months following the meeting.^11,19,20 No studies have reviewed abstract submissions to the annual PHM meeting to determine if selection or presentation format is associated with subsequent publication in a peer-reviewed journal.

We sought to identify the publication rate of abstracts submitted to the 2014 PHM conference and determine whether presentation format was associated with the likelihood of subsequent journal publication or time to publication.

Study Design

Data for this retrospective cohort study were obtained from a database of all abstracts submitted for presentation at the 2014 PHM conference in Lake Buena Vista, Florida.

Main Exposures

The main exposure was presentation format, which was categorized as not presented (ie, rejected), poster presentation, or oral presentation. PHM has a blinded abstract peer-review process; in 2014, an average of 10 reviewers scored each abstract. Reviewers graded abstracts on a scale of 1 (best in category) to 7 (unacceptable for presentation) according to the following criteria: originality, scientific importance, methodological rigor, and quality of presentation. Abstracts with the lowest average scores in each content area, usually less than or equal to 3, were accepted as oral presentations while most abstracts with scores greater than 5 were rejected. For this study, information collected from each abstract included authors, if the primary author was a trainee, title, content area, and presentation format. Content areas included clinical research, educational research, health services research (HSR) and/or epidemiology, practice management research, and quality improvement. Abstracts were then grouped by presentation format and content area for analysis. The Pediatric Academic Societies (PAS) annual meeting, another common venue for the presentation of pediatric research, precedes the PHM conference. Because acceptance for PAS presentation may represent more strongly developed abstract submissions for PHM, we identified which abstracts had also been presented at the PAS conference that same year by cross-referencing authors and abstract titles with the PAS 2014 program.

Main Outcome Measures

All submissions were assessed for subsequent publication in peer-reviewed journals through January 2017 (30 months following the July 2014 PHM conference). To identify abstracts that went on to full publication, 2 authors (JC and LEH) independently searched for the lead author’s name and the presentation title in PubMed, Google Scholar, and MedEdPORTAL in January 2017. PubMed was searched using both the general search box and an advanced search for author and title. Google Scholar was added to capture manuscripts that may not have been indexed in PubMed at the time of our search. MedEdPORTAL, a common venue for the publication of educational initiatives that are not currently indexed in PubMed, was searched by lead author name via the general search box. If a full manuscript was published discussing similar outcomes or results and was written by the same authors who had submitted a PHM conference abstract, it was considered to have been published. The journal, month, and year of publication were recorded. For journals published every 2 months, the date of publication was recorded as falling between the 2 months. For those journals with biannual publication in the spring and fall, the months of March and October were used, respectively. The impact factor of the publication journal was also recorded for the year preceding publication. A journal’s impact factor is frequently used as a quantitative measure of journal quality and reflects the frequency with which a journal’s articles are cited in the scientific literature.²¹ Journals without an impact factor (eg, newer journals) were assigned a 0.

Data Analysis

All abstracts submitted to the PHM conference were analyzed based on content area and presentation format. The proportion of all abstracts subsequently published was determined for each format type and content area, and the odds ratio (OR) for publication after abstract submission was calculated using logistic regression. We calculated an adjusted OR for subsequent publication controlling for PAS presentation and the trainee status of the primary author. The journals most frequently publishing abstracts submitted to the PHM conference were identified. Median time to publication was calculated using the number of months elapsed between the PHM conference and publication date and compared across all abstract formats using Cox proportional hazards models adjusted for PAS presentation and trainee status. Kaplan-Meier survival curves were also generated for each of the 3 formats and compared using log-rank tests. The median impact factor was determined for each abstract format and compared using Wilcoxon rank-sum tests. Median impact factor by content area was compared using a Kruskal-Wallis test. All statistical analyses were performed using SAS version 9.2 (SAS Institute, Cary, NC). P values < 0.05 were considered statistically significant. In accordance with the Common Rule²² and the policies of the Cincinnati Children’s Hospital Medical Center Institutional Review Board, this research was not considered human subjects research.

herrmann1004e_t1.jpg

For the 2014 PHM meeting, 226 abstracts were submitted, of which 183 (81.0%) were selected for presentation, including 154 (68.0%) as poster presentations and 29 (12.8%) as oral presentations. Of all submitted abstracts, 82 (36.3%) were published within 30 months following the meeting. Eighty-one of these (98.8%) were identified via PubMed, and 1 was found only in MedEdPORTAL. No additional publications were found via Google Scholar. The presenting author for the PHM abstract was the first author for 87.8% (n = 72) of the publications. A trainee was the presenting author for only 2 of these abstracts. For the publications in which the first author was not the presenting author, the presenting author was the senior author in 2 of the publications and the second or third author on the remaining 8. Of the abstracts accepted for presentation, 70 (38.3%) were subsequently published. Abstracts accepted for oral presentation had almost 7-fold greater odds of subsequent publication than those that were rejected (Table 1; OR 6.8; 95% confidence interval [CI], 2.4-19.4). Differences in the odds of publication for rejected abstracts compared with those accepted for poster presentation were not statistically significant (OR 1.2; 95% CI, 0.5-2.5).

herrmann1004e_f1.jpg

herrmann1004e_t2.jpg

herrmann1004e_t3.jpg

About one-third of abstracts submitted to the 2014 PHM conference were subsequently published in peer-reviewed journals within 30 months of the conference. Compared with rejected abstracts, the rate of publication was significantly higher for abstracts selected for oral presentation but not for those selected for poster presentation. For abstracts ultimately published in journals, selection for oral presentation was significantly associated with both a shorter time to publication and a higher median journal impact factor compared with rejected abstracts. Time to publication and median journal impact factor were similar between rejected abstracts and those accepted for poster presentation. Our findings suggest that abstract reviewers may be able to identify which abstracts will ultimately withstand more stringent peer review in the publication process when accepting abstracts for oral presentation. However, the selection for poster presentation versus rejection may not be indicative of future publication or the impact factor of the subsequent publication journal.

Previous studies have reviewed publication rates after meetings of the European Society for Pediatric Urology (publication rate of 47%),¹¹ the Ambulatory Pediatric Association (now the Academic Pediatric Association; publication rate of 47%), the American Pediatric Society/Society for Pediatric Research (publication rate of 54%), and the PAS (publication rate of 45%).^19,20 Our lower publication rate of 36.3% may be attributed to the shorter follow-up time in our study (30 months from the PHM conference), whereas prior studies monitored for publication up to 60 months after the PAS conference.²⁰ Factors associated with subsequent publication include statistically significant results, a large sample size, and a randomized controlled trial study design.^15,16 The primary reason for nonpublication for up to 80% of abstracts is failure to submit a manuscript for publication.²³ A lack of time and fear of rejection after peer review are commonly cited explanations.^18,23,24 Individuals may view acceptance for an oral presentation as positive reinforcement and be more motivated to pursue subsequent manuscript publication than individuals whose abstracts are offered poster presentations or are rejected. Trainees frequently present abstracts at scientific meetings, representing 40.3% of primary authors submitting abstracts to PHM in 2014, but may not have sufficient time or mentorship to develop a complete manuscript.¹⁸ To our knowledge, there have been no publications that assess the impact of trainee status on subsequent publication after conference submission.

Our study demonstrated that selection for oral presentation was associated with subsequent publication, shorter time to publication, and publication in journals with higher impact factors. A 2005 Cochrane review also demonstrated that selection for oral presentation was associated with subsequent journal publication.¹⁶ Abstracts accepted for oral publication may represent work further along in the research process, with more developed methodology and results. The shorter time to publication for abstracts accepted for oral presentation could also reflect feedback provided by conference attendees after the presentation, whereas poster sessions frequently lack a formalized process for critique.

Carroll et al. found no difference in time to publication between abstracts accepted for presentation at the PAS and rejected abstracts.²⁰ Previous studies demonstrate that most abstracts presented at scientific meetings that are subsequently accepted for publication are published within 2 to 3 years of the meeting,¹² with publication rates as high as 98% within 3 years of presentation.¹⁷ In contrast to Carroll et al., we found that abstracts accepted for oral presentation had a 4-fold greater likelihood of publication at each month than rejected abstracts. However, abstracts accepted for poster presentation did not have a significant difference in the proportional hazard ratio models for publication compared with rejected abstracts. Because space considerations limit the number of abstracts that can be accepted for presentation at a conference, some abstracts that are suitable for future publication may have been rejected due to a lack of space. Because researchers often use scientific meetings as a forum to receive peer feedback,¹² authors who present at conferences may take more time to write a manuscript in order to incorporate this feedback.

The most common journal in which submitted abstracts were subsequently published was Hospital Pediatrics, representing twice as many published manuscripts as the second most frequent journal, Pediatrics. Hospital Pediatrics, which was first published in 2011, did not have an impact factor assigned during the study period. Yet, as a peer-reviewed journal dedicated to the field of PHM, it is well aligned with the research presented at the PHM meeting. It is unclear if Hospital Pediatrics is a journal to which pediatric hospitalists tend to submit manuscripts initially or if manuscripts are frequently submitted elsewhere prior to their publication in Hospital Pediatrics. Submission to other journals first likely extends the time to publication, especially for abstracts accepted for poster presentation, which may describe studies with less developed methods or results.

This study has several limitations. Previous studies have demonstrated mean time to publication of 12 to 32 months following abstract presentation with a median time of 19.6 months.¹⁶ Because we only have a 30-month follow-up, there may be abstracts still in the review process that are yet to be published, especially because the length of the review process varies by journal. We based our literature search on the first author of each PHM conference abstract submission, assuming that this presenting author would be one of the publishing authors even if not remaining first author; if this was not the case, we may have missed some abstracts that were subsequently published in full. Likewise, if a presenting author’s last name changed prior to the publication of a manuscript, a publication may have been missed. This limitation would cause us to underestimate the overall publication rate. It is not clear whether this would differentially affect the method of presentation. However, in this study, there was concordance between the presenting author and the publication’s first author in 87.8% of the abstracts subsequently published in full. Presenting authors who did not remain the first author on the published manuscript maintained authorship as either the senior author or second or third author, which may represent changes in the degree of involvement or a division of responsibilities for individuals working on a project together. While our search methods were comprehensive, there is a possibility that abstracts may have been published in a venue that was not searched. Additionally, we only reviewed abstracts submitted to PHM for 1 year. As the field matures and the number of fellowship programs increases, the quality of submitted abstracts may increase, leading to higher publication rates or shorter times to publication. It is also possible that the publication rate may not be reflective of PHM as a field because hospitalists may submit their work to conferences other than the PHM. Lastly, it may be more challenging to interpret any differences in impact factor because some journals, including Hospital Pediatrics (which represented a plurality of poster presentation abstracts that were subsequently published and is a relatively new journal), did not have an impact factor assigned during the study period. Assigning a 0 to journals without an impact factor may artificially lower the average impact factor reported. Furthermore, an impact factor, which is based on the frequency with which an individual journal’s articles are cited in scientific or medical publications, may not necessarily reflect a journal’s quality.

Of the 226 abstracts submitted to the 2014 PHM conference, approximately one-third were published in peer-reviewed journals within 30 months of the conference. Selection for oral presentation was found to be associated with subsequent publication as well as publication in journals with higher impact factors. The overall low publication rate may indicate a need for increased mentorship and resources for research development in this growing specialty. Improved mechanisms for author feedback at poster sessions may provide constructive suggestions for further development of these projects into full manuscripts or opportunities for trainees and early-career hospitalists to network with more experienced researchers in the field.

Disclosure

Drs. Herrmann, Hall, Kyler, Andrews, Williams, and Shah and Mr. Cochran have nothing to disclose. Dr. Wilson reports personal fees from the American Academy of Pediatrics during the conduct of the study. The authors have no financial relationships relevant to this article to disclose.

With some areas of the country spending close to 3 times more on healthcare than others, regional variation in healthcare spending has been the focus of national attention.^1-7 Since 1973, the Dartmouth Institute has studied regional variation in healthcare utilization and spending and concluded that variation is “unwarranted” because it is driven by providers’ practice patterns rather than differences in medical need, patient preferences, or evidence-based medicine.^8-11 However, critics of the Dartmouth Institute’s findings argue that their approach does not adequately adjust for community-level income, and that higher costs in some areas reflect greater patient needs that are not reflected in illness acuity alone.^12-14

While Medicare data have made it possible to study variations in spending for the senior population, fragmentation of insurance coverage and nonstandardized data structures make studying the pediatric population more difficult. However, the Children’s Hospital Association’s (CHA) Pediatric Health Information System (PHIS) has made large-scale comparisons more feasible. To overcome challenges associated with using charges and nonuniform cost data, PHIS-derived standardized costs provide new opportunities for comparisons.^15,16 Initial analyses using PHIS data showed significant interhospital variations in costs of care,¹⁵ but they did not adjust for differences in populations and assess the drivers of variation. A more recent study that controlled for payer status, comorbidities, and illness severity found that intensive care unit (ICU) utilization varied significantly for children hospitalized for asthma, suggesting that hospital practice patterns drive differences in cost.¹⁷

This study uses PHIS data to analyze regional variations in standardized costs of care for 3 conditions for which children are hospitalized. To assess potential drivers of variation, the study investigates the effects of patient-level demographic and illness-severity variables as well as encounter-level variables on costs of care. It also estimates cost savings from reducing variation.

Data Source

This retrospective cohort study uses the PHIS database (CHA, Overland Park, KS), which includes 48 freestanding children’s hospitals located in noncompeting markets across the United States and accounts for approximately 20% of pediatric hospitalizations. PHIS includes patient demographics, International Classification of Diseases, 9th Revision (ICD-9) diagnosis and procedure codes, as well as hospital charges. In addition to total charges, PHIS reports imaging, laboratory, pharmacy, and “other” charges. The “other” category aggregates clinical, supply, room, and nursing charges (including facility fees and ancillary staff services).

Inclusion Criteria

Inpatient- and observation-status hospitalizations for asthma, diabetic ketoacidosis (DKA), and acute gastroenteritis (AGE) at 46 PHIS hospitals from October 2014 to September 2015 were included. Two hospitals were excluded because of missing data. Hospitalizations for patients >18 years were excluded.

Hospitalizations were categorized by using All Patient Refined-Diagnosis Related Groups (APR-DRGs) version 24 (3M Health Information Systems, St. Paul, MN)¹⁸ based on the ICD-9 diagnosis and procedure codes assigned during the episode of care. Analyses included APR-DRG 141 (asthma), primary diagnosis ICD-9 codes 250.11 and 250.13 (DKA), and APR-DRG 249 (AGE). ICD-9 codes were used for DKA for increased specificity.¹⁹ These conditions were chosen to represent 3 clinical scenarios: (1) a diagnosis for which hospitals differ on whether certain aspects of care are provided in the ICU (asthma), (2) a diagnosis that frequently includes care in an ICU (DKA), and (3) a diagnosis that typically does not include ICU care (AGE).¹⁹

Study Design

To focus the analysis on variation in resource utilization across hospitals rather than variations in hospital item charges, each billed resource was assigned a standardized cost.^15,16 For each clinical transaction code (CTC), the median unit cost was calculated for each hospital. The median of the hospital medians was defined as the standardized unit cost for that CTC.

The primary outcome variable was the total standardized cost for the hospitalization adjusted for patient-level demographic and illness-severity variables. Patient demographic and illness-severity covariates included age, race, gender, ZIP code-based median annual household income (HHI), rural-urban location, distance from home ZIP code to the hospital, chronic condition indicator (CCI), and severity-of-illness (SOI). When assessing drivers of variation, encounter-level covariates were added, including length of stay (LOS) in hours, ICU utilization, and 7-day readmission (an imprecise measure to account for quality of care during the index visit). The contribution of imaging, laboratory, pharmacy, and “other” costs was also considered.

Median annual HHI for patients’ home ZIP code was obtained from 2010 US Census data. Community-level HHI, a proxy for socioeconomic status (SES),^20,21 was classified into categories based on the 2015 US federal poverty level (FPL) for a family of 4²²: HHI-1 = ≤ 1.5 × FPL; HHI-2 = 1.5 to 2 × FPL; HHI-3 = 2 to 3 × FPL; HHI-4 = ≥ 3 × FPL. Rural-urban commuting area (RUCA) codes were used to determine the rural-urban classification of the patient’s home.²³ The distance from home ZIP code to the hospital was included as an additional control for illness severity because patients traveling longer distances are often more sick and require more resources.²⁴

The Agency for Healthcare Research and Quality CCI classification system was used to identify the presence of a chronic condition.²⁵ For asthma, CCI was flagged if the patient had a chronic condition other than asthma; for DKA, CCI was flagged if the patient had a chronic condition other than DKA; and for AGE, CCI was flagged if the patient had any chronic condition.

The APR-DRG system provides a 4-level SOI score with each APR-DRG category. Patient factors, such as comorbid diagnoses, are considered in severity scores generated through 3M’s proprietary algorithms.¹⁸

For the first analysis, the 46 hospitals were categorized into 7 geographic regions based on 2010 US Census Divisions.²⁶ To overcome small hospital sample sizes, Mountain and Pacific were combined into West, and Middle Atlantic and New England were combined into North East. Because PHIS hospitals are located in noncompeting geographic regions, for the second analysis, we examined hospital-level variation (considering each hospital as its own region).

Data Analysis

To focus the analysis on “typical” patients and produce more robust estimates of central tendencies, the top and bottom 5% of hospitalizations with the most extreme standardized costs by condition were trimmed.²⁷ Standardized costs were log-transformed because of their nonnormal distribution and analyzed by using linear mixed models. Covariates were added stepwise to assess the proportion of the variance explained by each predictor. Post-hoc tests with conservative single-step stepwise mutation model corrections for multiple testing were used to compare adjusted costs. Statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC). P values < 0.05 were considered significant. The Children’s Hospital of Philadelphia Institutional Review Board did not classify this study as human subjects research.

During the study period, there were 26,430 hospitalizations for asthma, 5056 for DKA, and 16,274 for AGE (Table 1).

Variation Across Census Regions

After adjusting for patient-level demographic and illness-severity variables, differences in adjusted total standardized costs remained between regions (P < 0.001). Although no region was an outlier compared to the overall mean for any of the conditions, regions were statistically different in pairwise comparison. The East North Central, South Atlantic, and West South Central regions had the highest adjusted total standardized costs for each of the conditions. The East South Central and West North Central regions had the lowest costs for each of the conditions. Adjusted total standardized costs were 120% higher for asthma ($1920 vs $4227), 46% higher for DKA ($7429 vs $10,881), and 150% higher for AGE ($3316 vs $8292) in the highest-cost region compared with the lowest-cost region (Table 2A).

jhm012100818_t2.jpg

Variation Within Census Regions

After controlling for patient-level demographic and illness-severity variables, standardized costs were different across hospitals in the same region (P < 0.001; panel A in Figure). This was true for all conditions in each region. Differences between the lowest- and highest-cost hospitals within the same region ranged from 111% to 420% for asthma, 101% to 398% for DKA, and 166% to 787% for AGE (Table 3).

jhm012100818_t3.jpg

Variation Across Hospitals (Each Hospital as Its Own Region)

One hospital had the highest adjusted standardized costs for all 3 conditions ($9087 for asthma, $28,564 for DKA, and $23,387 for AGE) and was outside of the 95% confidence interval compared with the overall means. The second highest-cost hospitals for asthma ($5977) and AGE ($18,780) were also outside of the 95% confidence interval. After removing these outliers, the difference between the highest- and lowest-cost hospitals was 549% for asthma ($721 vs $4678), 491% for DKA ($2738 vs $16,192), and 681% for AGE ($1317 vs $10,281; Table 2B).

Drivers of Variation Across Census Regions

Patient-level demographic and illness-severity variables explained very little of the variation in standardized costs across regions. For each of the conditions, age, race, gender, community-level HHI, RUCA, and distance from home to the hospital each accounted for <1.5% of variation, while SOI and CCI each accounted for <5%. Overall, patient-level variables explained 5.5%, 3.7%, and 6.7% of variation for asthma, DKA, and AGE.

Encounter-level variables explained a much larger percentage of the variation in costs. LOS accounted for 17.8% of the variation for asthma, 9.8% for DKA, and 8.7% for AGE. ICU utilization explained 6.9% of the variation for asthma and 12.5% for DKA; ICU use was not a major driver for AGE. Seven-day readmissions accounted for <0.5% for each of the conditions. The combination of patient-level and encounter-level variables explained 27%, 24%, and 15% of the variation for asthma, DKA, and AGE.

Drivers of Variation Across Hospitals

For each of the conditions, patient-level demographic variables each accounted for <2% of variation in costs between hospitals. SOI accounted for 4.5% of the variation for asthma and CCI accounted for 5.2% for AGE. Overall, patient-level variables explained 6.9%, 5.3%, and 7.3% of variation for asthma, DKA, and AGE.

Encounter-level variables accounted for a much larger percentage of the variation in cost. LOS explained 25.4% for asthma, 13.3% for DKA, and 14.2% for AGE. ICU utilization accounted for 13.4% for asthma and 21.9% for DKA; ICU use was not a major driver for AGE. Seven-day readmissions accounted for <0.5% for each of the conditions. Together, patient-level and encounter-level variables explained 40%, 36%, and 22% of variation for asthma, DKA, and AGE.

Imaging, Laboratory, Pharmacy, and “Other” Costs

jhm012100818_f1.jpg

The largest contributor to total costs adjusted for patient-level factors for all conditions was “other,” which aggregates room, nursing, clinical, and supply charges (panel B in Figure). When considering drivers of variation, this category explained >50% for each of the conditions. The next largest contributor to total costs was laboratory charges, which accounted for 15% of the variation across regions for asthma and 11% for DKA. Differences in imaging accounted for 18% of the variation for DKA and 15% for AGE. Differences in pharmacy charges accounted for <4% of the variation for each of the conditions. Adding the 4 cost components to the other patient- and encounter-level covariates, the model explained 81%, 78%, and 72% of the variation across census regions for asthma, DKA, and AGE.

For the hospital-level analysis, differences in “other” remained the largest driver of cost variation. For asthma, “other” explained 61% of variation, while pharmacy, laboratory, and imaging each accounted for <8%. For DKA, differences in imaging accounted for 18% of the variation and laboratory charges accounted for 12%. For AGE, imaging accounted for 15% of the variation. Adding the 4 cost components to the other patient- and encounter-level covariates, the model explained 81%, 72%, and 67% of the variation for asthma, DKA, and AGE.

Cost Savings

If all hospitals in this cohort with adjusted standardized costs above the national PHIS average achieved costs equal to the national PHIS average, estimated annual savings in adjusted standardized costs for these 3 conditions would be $69.1 million. If each hospital with adjusted costs above the average within its census region achieved costs equal to its regional average, estimated annual savings in adjusted standardized costs for these conditions would be $25.2 million.

This study reported on the regional variation in costs of care for 3 conditions treated at 46 children’s hospitals across 7 geographic regions, and it demonstrated that variations in costs of care exist in pediatrics. This study used standardized costs to compare utilization patterns across hospitals and adjusted for several patient-level demographic and illness-severity factors, and it found that differences in costs of care for children hospitalized with asthma, DKA, and AGE remained both between and within regions.

These variations are noteworthy, as hospitals strive to improve the value of healthcare. If the higher-cost hospitals in this cohort could achieve costs equal to the national PHIS averages, estimated annual savings in adjusted standardized costs for these conditions alone would equal $69.1 million. If higher-cost hospitals relative to the average in their own region reduced costs to their regional averages, annual standardized cost savings could equal $25.2 million for these conditions.

The differences observed are also significant in that they provide a foundation for exploring whether lower-cost regions or lower-cost hospitals achieve comparable quality outcomes.²⁸ If so, studying what those hospitals do to achieve outcomes more efficiently can serve as the basis for the establishment of best practices.²⁹ Standardizing best practices through protocols, pathways, and care-model redesign can reduce potentially unnecessary spending.³⁰

Our findings showed that patient-level demographic and illness-severity covariates, including community-level HHI and SOI, did not consistently explain cost differences. Instead, LOS and ICU utilization were associated with higher costs.^17,19 When considering the effect of the 4 cost components on the variation in total standardized costs between regions and between hospitals, the fact that the “other” category accounted for the largest percent of the variation is not surprising, because the cost of room occupancy and nursing services increases with longer LOS and more time in the ICU. Other individual cost components that were major drivers of variation were laboratory utilization for asthma and imaging for DKA and AGE³¹ (though they accounted for a much smaller proportion of total adjusted costs).¹⁹

To determine if these factors are modifiable, more information is needed to explain why practices differ. Many factors may contribute to varying utilization patterns, including differences in capabilities and resources (in the hospital and in the community) and patient volumes. For example, some hospitals provide continuous albuterol for status asthmaticus only in ICUs, while others provide it on regular units.³² But if certain hospitals do not have adequate resources or volumes to effectively care for certain populations outside of the ICU, their higher-value approach (considering quality and cost) may be to utilize ICU beds, even if some other hospitals care for those patients on non-ICU floors. Another possibility is that family preferences about care delivery (such as how long children stay in the hospital) may vary across regions.³³

Other evidence suggests that physician practice and spending patterns are strongly influenced by the practices of the region where they trained.³⁴ Because physicians often practice close to where they trained,^35,36 this may partially explain how regional patterns are reinforced.

Even considering all mentioned covariates, our model did not fully explain variation in standardized costs. After adding the cost components as covariates, between one-third and one-fifth of the variation remained unexplained. It is possible that this unexplained variation stemmed from unmeasured patient-level factors.

In addition, while proxies for SES, including community-level HHI, did not significantly predict differences in costs across regions, it is possible that SES affected LOS differently in different regions. Previous studies have suggested that lower SES is associated with longer LOS.³⁷ If this effect is more pronounced in certain regions (potentially because of differences in social service infrastructures), SES may be contributing to variations in cost through LOS.

Our findings were subject to limitations. First, this study only examined 3 diagnoses and did not include surgical or less common conditions. Second, while PHIS includes tertiary care, academic, and freestanding children’s hospitals, it does not include general hospitals, which is where most pediatric patients receive care.³⁸ Third, we used ZIP code-based median annual HHI to account for SES, and we used ZIP codes to determine the distance to the hospital and rural-urban location of patients’ homes. These approximations lack precision because SES and distances vary within ZIP codes.³⁹ Fourth, while adjusted standardized costs allow for comparisons between hospitals, they do not represent actual costs to patients or individual hospitals. Additionally, when determining whether variation remained after controlling for patient-level variables, we included SOI as a reflection of illness-severity at presentation. However, in practice, SOI scores may be assigned partially based on factors determined during the hospitalization.¹⁸ Finally, the use of other regional boundaries or the selection of different hospitals may yield different results.

This study reveals regional variations in costs of care for 3 inpatient pediatric conditions. Future studies should explore whether lower-cost regions or lower-cost hospitals achieve comparable quality outcomes. To the extent that variation is driven by modifiable factors and lower spending does not compromise outcomes, these data may prompt reviews of care models to reduce unwarranted variation and improve the value of care delivery at local, regional, and national levels.

Disclosure

Internal funds from the CHA and The Children’s Hospital of Philadelphia supported the conduct of this work. The authors have no financial interests, relationships, or affiliations relevant to the subject matter or materials discussed in the manuscript to disclose. The authors have no potential conflicts of interest relevant to the subject matter or materials discussed in the manuscript to disclose

With some areas of the country spending close to 3 times more on healthcare than others, regional variation in healthcare spending has been the focus of national attention.^1-7 Since 1973, the Dartmouth Institute has studied regional variation in healthcare utilization and spending and concluded that variation is “unwarranted” because it is driven by providers’ practice patterns rather than differences in medical need, patient preferences, or evidence-based medicine.^8-11 However, critics of the Dartmouth Institute’s findings argue that their approach does not adequately adjust for community-level income, and that higher costs in some areas reflect greater patient needs that are not reflected in illness acuity alone.^12-14

While Medicare data have made it possible to study variations in spending for the senior population, fragmentation of insurance coverage and nonstandardized data structures make studying the pediatric population more difficult. However, the Children’s Hospital Association’s (CHA) Pediatric Health Information System (PHIS) has made large-scale comparisons more feasible. To overcome challenges associated with using charges and nonuniform cost data, PHIS-derived standardized costs provide new opportunities for comparisons.^15,16 Initial analyses using PHIS data showed significant interhospital variations in costs of care,¹⁵ but they did not adjust for differences in populations and assess the drivers of variation. A more recent study that controlled for payer status, comorbidities, and illness severity found that intensive care unit (ICU) utilization varied significantly for children hospitalized for asthma, suggesting that hospital practice patterns drive differences in cost.¹⁷

This study uses PHIS data to analyze regional variations in standardized costs of care for 3 conditions for which children are hospitalized. To assess potential drivers of variation, the study investigates the effects of patient-level demographic and illness-severity variables as well as encounter-level variables on costs of care. It also estimates cost savings from reducing variation.

Data Source

This retrospective cohort study uses the PHIS database (CHA, Overland Park, KS), which includes 48 freestanding children’s hospitals located in noncompeting markets across the United States and accounts for approximately 20% of pediatric hospitalizations. PHIS includes patient demographics, International Classification of Diseases, 9th Revision (ICD-9) diagnosis and procedure codes, as well as hospital charges. In addition to total charges, PHIS reports imaging, laboratory, pharmacy, and “other” charges. The “other” category aggregates clinical, supply, room, and nursing charges (including facility fees and ancillary staff services).

Inclusion Criteria

Inpatient- and observation-status hospitalizations for asthma, diabetic ketoacidosis (DKA), and acute gastroenteritis (AGE) at 46 PHIS hospitals from October 2014 to September 2015 were included. Two hospitals were excluded because of missing data. Hospitalizations for patients >18 years were excluded.

Hospitalizations were categorized by using All Patient Refined-Diagnosis Related Groups (APR-DRGs) version 24 (3M Health Information Systems, St. Paul, MN)¹⁸ based on the ICD-9 diagnosis and procedure codes assigned during the episode of care. Analyses included APR-DRG 141 (asthma), primary diagnosis ICD-9 codes 250.11 and 250.13 (DKA), and APR-DRG 249 (AGE). ICD-9 codes were used for DKA for increased specificity.¹⁹ These conditions were chosen to represent 3 clinical scenarios: (1) a diagnosis for which hospitals differ on whether certain aspects of care are provided in the ICU (asthma), (2) a diagnosis that frequently includes care in an ICU (DKA), and (3) a diagnosis that typically does not include ICU care (AGE).¹⁹

Study Design

To focus the analysis on variation in resource utilization across hospitals rather than variations in hospital item charges, each billed resource was assigned a standardized cost.^15,16 For each clinical transaction code (CTC), the median unit cost was calculated for each hospital. The median of the hospital medians was defined as the standardized unit cost for that CTC.

The primary outcome variable was the total standardized cost for the hospitalization adjusted for patient-level demographic and illness-severity variables. Patient demographic and illness-severity covariates included age, race, gender, ZIP code-based median annual household income (HHI), rural-urban location, distance from home ZIP code to the hospital, chronic condition indicator (CCI), and severity-of-illness (SOI). When assessing drivers of variation, encounter-level covariates were added, including length of stay (LOS) in hours, ICU utilization, and 7-day readmission (an imprecise measure to account for quality of care during the index visit). The contribution of imaging, laboratory, pharmacy, and “other” costs was also considered.

Median annual HHI for patients’ home ZIP code was obtained from 2010 US Census data. Community-level HHI, a proxy for socioeconomic status (SES),^20,21 was classified into categories based on the 2015 US federal poverty level (FPL) for a family of 4²²: HHI-1 = ≤ 1.5 × FPL; HHI-2 = 1.5 to 2 × FPL; HHI-3 = 2 to 3 × FPL; HHI-4 = ≥ 3 × FPL. Rural-urban commuting area (RUCA) codes were used to determine the rural-urban classification of the patient’s home.²³ The distance from home ZIP code to the hospital was included as an additional control for illness severity because patients traveling longer distances are often more sick and require more resources.²⁴

The Agency for Healthcare Research and Quality CCI classification system was used to identify the presence of a chronic condition.²⁵ For asthma, CCI was flagged if the patient had a chronic condition other than asthma; for DKA, CCI was flagged if the patient had a chronic condition other than DKA; and for AGE, CCI was flagged if the patient had any chronic condition.

The APR-DRG system provides a 4-level SOI score with each APR-DRG category. Patient factors, such as comorbid diagnoses, are considered in severity scores generated through 3M’s proprietary algorithms.¹⁸

For the first analysis, the 46 hospitals were categorized into 7 geographic regions based on 2010 US Census Divisions.²⁶ To overcome small hospital sample sizes, Mountain and Pacific were combined into West, and Middle Atlantic and New England were combined into North East. Because PHIS hospitals are located in noncompeting geographic regions, for the second analysis, we examined hospital-level variation (considering each hospital as its own region).

Data Analysis

To focus the analysis on “typical” patients and produce more robust estimates of central tendencies, the top and bottom 5% of hospitalizations with the most extreme standardized costs by condition were trimmed.²⁷ Standardized costs were log-transformed because of their nonnormal distribution and analyzed by using linear mixed models. Covariates were added stepwise to assess the proportion of the variance explained by each predictor. Post-hoc tests with conservative single-step stepwise mutation model corrections for multiple testing were used to compare adjusted costs. Statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC). P values < 0.05 were considered significant. The Children’s Hospital of Philadelphia Institutional Review Board did not classify this study as human subjects research.

During the study period, there were 26,430 hospitalizations for asthma, 5056 for DKA, and 16,274 for AGE (Table 1).

Variation Across Census Regions

After adjusting for patient-level demographic and illness-severity variables, differences in adjusted total standardized costs remained between regions (P < 0.001). Although no region was an outlier compared to the overall mean for any of the conditions, regions were statistically different in pairwise comparison. The East North Central, South Atlantic, and West South Central regions had the highest adjusted total standardized costs for each of the conditions. The East South Central and West North Central regions had the lowest costs for each of the conditions. Adjusted total standardized costs were 120% higher for asthma ($1920 vs $4227), 46% higher for DKA ($7429 vs $10,881), and 150% higher for AGE ($3316 vs $8292) in the highest-cost region compared with the lowest-cost region (Table 2A).

jhm012100818_t2.jpg

Variation Within Census Regions

After controlling for patient-level demographic and illness-severity variables, standardized costs were different across hospitals in the same region (P < 0.001; panel A in Figure). This was true for all conditions in each region. Differences between the lowest- and highest-cost hospitals within the same region ranged from 111% to 420% for asthma, 101% to 398% for DKA, and 166% to 787% for AGE (Table 3).

jhm012100818_t3.jpg

Variation Across Hospitals (Each Hospital as Its Own Region)

One hospital had the highest adjusted standardized costs for all 3 conditions ($9087 for asthma, $28,564 for DKA, and $23,387 for AGE) and was outside of the 95% confidence interval compared with the overall means. The second highest-cost hospitals for asthma ($5977) and AGE ($18,780) were also outside of the 95% confidence interval. After removing these outliers, the difference between the highest- and lowest-cost hospitals was 549% for asthma ($721 vs $4678), 491% for DKA ($2738 vs $16,192), and 681% for AGE ($1317 vs $10,281; Table 2B).

Drivers of Variation Across Census Regions

Patient-level demographic and illness-severity variables explained very little of the variation in standardized costs across regions. For each of the conditions, age, race, gender, community-level HHI, RUCA, and distance from home to the hospital each accounted for <1.5% of variation, while SOI and CCI each accounted for <5%. Overall, patient-level variables explained 5.5%, 3.7%, and 6.7% of variation for asthma, DKA, and AGE.

Encounter-level variables explained a much larger percentage of the variation in costs. LOS accounted for 17.8% of the variation for asthma, 9.8% for DKA, and 8.7% for AGE. ICU utilization explained 6.9% of the variation for asthma and 12.5% for DKA; ICU use was not a major driver for AGE. Seven-day readmissions accounted for <0.5% for each of the conditions. The combination of patient-level and encounter-level variables explained 27%, 24%, and 15% of the variation for asthma, DKA, and AGE.

Drivers of Variation Across Hospitals

For each of the conditions, patient-level demographic variables each accounted for <2% of variation in costs between hospitals. SOI accounted for 4.5% of the variation for asthma and CCI accounted for 5.2% for AGE. Overall, patient-level variables explained 6.9%, 5.3%, and 7.3% of variation for asthma, DKA, and AGE.

Encounter-level variables accounted for a much larger percentage of the variation in cost. LOS explained 25.4% for asthma, 13.3% for DKA, and 14.2% for AGE. ICU utilization accounted for 13.4% for asthma and 21.9% for DKA; ICU use was not a major driver for AGE. Seven-day readmissions accounted for <0.5% for each of the conditions. Together, patient-level and encounter-level variables explained 40%, 36%, and 22% of variation for asthma, DKA, and AGE.

Imaging, Laboratory, Pharmacy, and “Other” Costs

jhm012100818_f1.jpg

The largest contributor to total costs adjusted for patient-level factors for all conditions was “other,” which aggregates room, nursing, clinical, and supply charges (panel B in Figure). When considering drivers of variation, this category explained >50% for each of the conditions. The next largest contributor to total costs was laboratory charges, which accounted for 15% of the variation across regions for asthma and 11% for DKA. Differences in imaging accounted for 18% of the variation for DKA and 15% for AGE. Differences in pharmacy charges accounted for <4% of the variation for each of the conditions. Adding the 4 cost components to the other patient- and encounter-level covariates, the model explained 81%, 78%, and 72% of the variation across census regions for asthma, DKA, and AGE.

For the hospital-level analysis, differences in “other” remained the largest driver of cost variation. For asthma, “other” explained 61% of variation, while pharmacy, laboratory, and imaging each accounted for <8%. For DKA, differences in imaging accounted for 18% of the variation and laboratory charges accounted for 12%. For AGE, imaging accounted for 15% of the variation. Adding the 4 cost components to the other patient- and encounter-level covariates, the model explained 81%, 72%, and 67% of the variation for asthma, DKA, and AGE.

Cost Savings

If all hospitals in this cohort with adjusted standardized costs above the national PHIS average achieved costs equal to the national PHIS average, estimated annual savings in adjusted standardized costs for these 3 conditions would be $69.1 million. If each hospital with adjusted costs above the average within its census region achieved costs equal to its regional average, estimated annual savings in adjusted standardized costs for these conditions would be $25.2 million.

This study reported on the regional variation in costs of care for 3 conditions treated at 46 children’s hospitals across 7 geographic regions, and it demonstrated that variations in costs of care exist in pediatrics. This study used standardized costs to compare utilization patterns across hospitals and adjusted for several patient-level demographic and illness-severity factors, and it found that differences in costs of care for children hospitalized with asthma, DKA, and AGE remained both between and within regions.

These variations are noteworthy, as hospitals strive to improve the value of healthcare. If the higher-cost hospitals in this cohort could achieve costs equal to the national PHIS averages, estimated annual savings in adjusted standardized costs for these conditions alone would equal $69.1 million. If higher-cost hospitals relative to the average in their own region reduced costs to their regional averages, annual standardized cost savings could equal $25.2 million for these conditions.

The differences observed are also significant in that they provide a foundation for exploring whether lower-cost regions or lower-cost hospitals achieve comparable quality outcomes.²⁸ If so, studying what those hospitals do to achieve outcomes more efficiently can serve as the basis for the establishment of best practices.²⁹ Standardizing best practices through protocols, pathways, and care-model redesign can reduce potentially unnecessary spending.³⁰

Our findings showed that patient-level demographic and illness-severity covariates, including community-level HHI and SOI, did not consistently explain cost differences. Instead, LOS and ICU utilization were associated with higher costs.^17,19 When considering the effect of the 4 cost components on the variation in total standardized costs between regions and between hospitals, the fact that the “other” category accounted for the largest percent of the variation is not surprising, because the cost of room occupancy and nursing services increases with longer LOS and more time in the ICU. Other individual cost components that were major drivers of variation were laboratory utilization for asthma and imaging for DKA and AGE³¹ (though they accounted for a much smaller proportion of total adjusted costs).¹⁹

To determine if these factors are modifiable, more information is needed to explain why practices differ. Many factors may contribute to varying utilization patterns, including differences in capabilities and resources (in the hospital and in the community) and patient volumes. For example, some hospitals provide continuous albuterol for status asthmaticus only in ICUs, while others provide it on regular units.³² But if certain hospitals do not have adequate resources or volumes to effectively care for certain populations outside of the ICU, their higher-value approach (considering quality and cost) may be to utilize ICU beds, even if some other hospitals care for those patients on non-ICU floors. Another possibility is that family preferences about care delivery (such as how long children stay in the hospital) may vary across regions.³³

Other evidence suggests that physician practice and spending patterns are strongly influenced by the practices of the region where they trained.³⁴ Because physicians often practice close to where they trained,^35,36 this may partially explain how regional patterns are reinforced.

Even considering all mentioned covariates, our model did not fully explain variation in standardized costs. After adding the cost components as covariates, between one-third and one-fifth of the variation remained unexplained. It is possible that this unexplained variation stemmed from unmeasured patient-level factors.

In addition, while proxies for SES, including community-level HHI, did not significantly predict differences in costs across regions, it is possible that SES affected LOS differently in different regions. Previous studies have suggested that lower SES is associated with longer LOS.³⁷ If this effect is more pronounced in certain regions (potentially because of differences in social service infrastructures), SES may be contributing to variations in cost through LOS.

Our findings were subject to limitations. First, this study only examined 3 diagnoses and did not include surgical or less common conditions. Second, while PHIS includes tertiary care, academic, and freestanding children’s hospitals, it does not include general hospitals, which is where most pediatric patients receive care.³⁸ Third, we used ZIP code-based median annual HHI to account for SES, and we used ZIP codes to determine the distance to the hospital and rural-urban location of patients’ homes. These approximations lack precision because SES and distances vary within ZIP codes.³⁹ Fourth, while adjusted standardized costs allow for comparisons between hospitals, they do not represent actual costs to patients or individual hospitals. Additionally, when determining whether variation remained after controlling for patient-level variables, we included SOI as a reflection of illness-severity at presentation. However, in practice, SOI scores may be assigned partially based on factors determined during the hospitalization.¹⁸ Finally, the use of other regional boundaries or the selection of different hospitals may yield different results.

This study reveals regional variations in costs of care for 3 inpatient pediatric conditions. Future studies should explore whether lower-cost regions or lower-cost hospitals achieve comparable quality outcomes. To the extent that variation is driven by modifiable factors and lower spending does not compromise outcomes, these data may prompt reviews of care models to reduce unwarranted variation and improve the value of care delivery at local, regional, and national levels.

Disclosure

Internal funds from the CHA and The Children’s Hospital of Philadelphia supported the conduct of this work. The authors have no financial interests, relationships, or affiliations relevant to the subject matter or materials discussed in the manuscript to disclose. The authors have no potential conflicts of interest relevant to the subject matter or materials discussed in the manuscript to disclose

With some areas of the country spending close to 3 times more on healthcare than others, regional variation in healthcare spending has been the focus of national attention.^1-7 Since 1973, the Dartmouth Institute has studied regional variation in healthcare utilization and spending and concluded that variation is “unwarranted” because it is driven by providers’ practice patterns rather than differences in medical need, patient preferences, or evidence-based medicine.^8-11 However, critics of the Dartmouth Institute’s findings argue that their approach does not adequately adjust for community-level income, and that higher costs in some areas reflect greater patient needs that are not reflected in illness acuity alone.^12-14

While Medicare data have made it possible to study variations in spending for the senior population, fragmentation of insurance coverage and nonstandardized data structures make studying the pediatric population more difficult. However, the Children’s Hospital Association’s (CHA) Pediatric Health Information System (PHIS) has made large-scale comparisons more feasible. To overcome challenges associated with using charges and nonuniform cost data, PHIS-derived standardized costs provide new opportunities for comparisons.^15,16 Initial analyses using PHIS data showed significant interhospital variations in costs of care,¹⁵ but they did not adjust for differences in populations and assess the drivers of variation. A more recent study that controlled for payer status, comorbidities, and illness severity found that intensive care unit (ICU) utilization varied significantly for children hospitalized for asthma, suggesting that hospital practice patterns drive differences in cost.¹⁷

This study uses PHIS data to analyze regional variations in standardized costs of care for 3 conditions for which children are hospitalized. To assess potential drivers of variation, the study investigates the effects of patient-level demographic and illness-severity variables as well as encounter-level variables on costs of care. It also estimates cost savings from reducing variation.

Data Source

This retrospective cohort study uses the PHIS database (CHA, Overland Park, KS), which includes 48 freestanding children’s hospitals located in noncompeting markets across the United States and accounts for approximately 20% of pediatric hospitalizations. PHIS includes patient demographics, International Classification of Diseases, 9th Revision (ICD-9) diagnosis and procedure codes, as well as hospital charges. In addition to total charges, PHIS reports imaging, laboratory, pharmacy, and “other” charges. The “other” category aggregates clinical, supply, room, and nursing charges (including facility fees and ancillary staff services).

Inclusion Criteria

Inpatient- and observation-status hospitalizations for asthma, diabetic ketoacidosis (DKA), and acute gastroenteritis (AGE) at 46 PHIS hospitals from October 2014 to September 2015 were included. Two hospitals were excluded because of missing data. Hospitalizations for patients >18 years were excluded.

Hospitalizations were categorized by using All Patient Refined-Diagnosis Related Groups (APR-DRGs) version 24 (3M Health Information Systems, St. Paul, MN)¹⁸ based on the ICD-9 diagnosis and procedure codes assigned during the episode of care. Analyses included APR-DRG 141 (asthma), primary diagnosis ICD-9 codes 250.11 and 250.13 (DKA), and APR-DRG 249 (AGE). ICD-9 codes were used for DKA for increased specificity.¹⁹ These conditions were chosen to represent 3 clinical scenarios: (1) a diagnosis for which hospitals differ on whether certain aspects of care are provided in the ICU (asthma), (2) a diagnosis that frequently includes care in an ICU (DKA), and (3) a diagnosis that typically does not include ICU care (AGE).¹⁹

Study Design

To focus the analysis on variation in resource utilization across hospitals rather than variations in hospital item charges, each billed resource was assigned a standardized cost.^15,16 For each clinical transaction code (CTC), the median unit cost was calculated for each hospital. The median of the hospital medians was defined as the standardized unit cost for that CTC.

The primary outcome variable was the total standardized cost for the hospitalization adjusted for patient-level demographic and illness-severity variables. Patient demographic and illness-severity covariates included age, race, gender, ZIP code-based median annual household income (HHI), rural-urban location, distance from home ZIP code to the hospital, chronic condition indicator (CCI), and severity-of-illness (SOI). When assessing drivers of variation, encounter-level covariates were added, including length of stay (LOS) in hours, ICU utilization, and 7-day readmission (an imprecise measure to account for quality of care during the index visit). The contribution of imaging, laboratory, pharmacy, and “other” costs was also considered.

Median annual HHI for patients’ home ZIP code was obtained from 2010 US Census data. Community-level HHI, a proxy for socioeconomic status (SES),^20,21 was classified into categories based on the 2015 US federal poverty level (FPL) for a family of 4²²: HHI-1 = ≤ 1.5 × FPL; HHI-2 = 1.5 to 2 × FPL; HHI-3 = 2 to 3 × FPL; HHI-4 = ≥ 3 × FPL. Rural-urban commuting area (RUCA) codes were used to determine the rural-urban classification of the patient’s home.²³ The distance from home ZIP code to the hospital was included as an additional control for illness severity because patients traveling longer distances are often more sick and require more resources.²⁴

The Agency for Healthcare Research and Quality CCI classification system was used to identify the presence of a chronic condition.²⁵ For asthma, CCI was flagged if the patient had a chronic condition other than asthma; for DKA, CCI was flagged if the patient had a chronic condition other than DKA; and for AGE, CCI was flagged if the patient had any chronic condition.

The APR-DRG system provides a 4-level SOI score with each APR-DRG category. Patient factors, such as comorbid diagnoses, are considered in severity scores generated through 3M’s proprietary algorithms.¹⁸

For the first analysis, the 46 hospitals were categorized into 7 geographic regions based on 2010 US Census Divisions.²⁶ To overcome small hospital sample sizes, Mountain and Pacific were combined into West, and Middle Atlantic and New England were combined into North East. Because PHIS hospitals are located in noncompeting geographic regions, for the second analysis, we examined hospital-level variation (considering each hospital as its own region).

Data Analysis

To focus the analysis on “typical” patients and produce more robust estimates of central tendencies, the top and bottom 5% of hospitalizations with the most extreme standardized costs by condition were trimmed.²⁷ Standardized costs were log-transformed because of their nonnormal distribution and analyzed by using linear mixed models. Covariates were added stepwise to assess the proportion of the variance explained by each predictor. Post-hoc tests with conservative single-step stepwise mutation model corrections for multiple testing were used to compare adjusted costs. Statistical analyses were performed using SAS version 9.3 (SAS Institute, Cary, NC). P values < 0.05 were considered significant. The Children’s Hospital of Philadelphia Institutional Review Board did not classify this study as human subjects research.

During the study period, there were 26,430 hospitalizations for asthma, 5056 for DKA, and 16,274 for AGE (Table 1).

Variation Across Census Regions

After adjusting for patient-level demographic and illness-severity variables, differences in adjusted total standardized costs remained between regions (P < 0.001). Although no region was an outlier compared to the overall mean for any of the conditions, regions were statistically different in pairwise comparison. The East North Central, South Atlantic, and West South Central regions had the highest adjusted total standardized costs for each of the conditions. The East South Central and West North Central regions had the lowest costs for each of the conditions. Adjusted total standardized costs were 120% higher for asthma ($1920 vs $4227), 46% higher for DKA ($7429 vs $10,881), and 150% higher for AGE ($3316 vs $8292) in the highest-cost region compared with the lowest-cost region (Table 2A).

jhm012100818_t2.jpg

Variation Within Census Regions

After controlling for patient-level demographic and illness-severity variables, standardized costs were different across hospitals in the same region (P < 0.001; panel A in Figure). This was true for all conditions in each region. Differences between the lowest- and highest-cost hospitals within the same region ranged from 111% to 420% for asthma, 101% to 398% for DKA, and 166% to 787% for AGE (Table 3).

jhm012100818_t3.jpg

Variation Across Hospitals (Each Hospital as Its Own Region)

One hospital had the highest adjusted standardized costs for all 3 conditions ($9087 for asthma, $28,564 for DKA, and $23,387 for AGE) and was outside of the 95% confidence interval compared with the overall means. The second highest-cost hospitals for asthma ($5977) and AGE ($18,780) were also outside of the 95% confidence interval. After removing these outliers, the difference between the highest- and lowest-cost hospitals was 549% for asthma ($721 vs $4678), 491% for DKA ($2738 vs $16,192), and 681% for AGE ($1317 vs $10,281; Table 2B).

Drivers of Variation Across Census Regions

Patient-level demographic and illness-severity variables explained very little of the variation in standardized costs across regions. For each of the conditions, age, race, gender, community-level HHI, RUCA, and distance from home to the hospital each accounted for <1.5% of variation, while SOI and CCI each accounted for <5%. Overall, patient-level variables explained 5.5%, 3.7%, and 6.7% of variation for asthma, DKA, and AGE.

Encounter-level variables explained a much larger percentage of the variation in costs. LOS accounted for 17.8% of the variation for asthma, 9.8% for DKA, and 8.7% for AGE. ICU utilization explained 6.9% of the variation for asthma and 12.5% for DKA; ICU use was not a major driver for AGE. Seven-day readmissions accounted for <0.5% for each of the conditions. The combination of patient-level and encounter-level variables explained 27%, 24%, and 15% of the variation for asthma, DKA, and AGE.

Drivers of Variation Across Hospitals

For each of the conditions, patient-level demographic variables each accounted for <2% of variation in costs between hospitals. SOI accounted for 4.5% of the variation for asthma and CCI accounted for 5.2% for AGE. Overall, patient-level variables explained 6.9%, 5.3%, and 7.3% of variation for asthma, DKA, and AGE.

Encounter-level variables accounted for a much larger percentage of the variation in cost. LOS explained 25.4% for asthma, 13.3% for DKA, and 14.2% for AGE. ICU utilization accounted for 13.4% for asthma and 21.9% for DKA; ICU use was not a major driver for AGE. Seven-day readmissions accounted for <0.5% for each of the conditions. Together, patient-level and encounter-level variables explained 40%, 36%, and 22% of variation for asthma, DKA, and AGE.

Imaging, Laboratory, Pharmacy, and “Other” Costs

jhm012100818_f1.jpg

The largest contributor to total costs adjusted for patient-level factors for all conditions was “other,” which aggregates room, nursing, clinical, and supply charges (panel B in Figure). When considering drivers of variation, this category explained >50% for each of the conditions. The next largest contributor to total costs was laboratory charges, which accounted for 15% of the variation across regions for asthma and 11% for DKA. Differences in imaging accounted for 18% of the variation for DKA and 15% for AGE. Differences in pharmacy charges accounted for <4% of the variation for each of the conditions. Adding the 4 cost components to the other patient- and encounter-level covariates, the model explained 81%, 78%, and 72% of the variation across census regions for asthma, DKA, and AGE.

For the hospital-level analysis, differences in “other” remained the largest driver of cost variation. For asthma, “other” explained 61% of variation, while pharmacy, laboratory, and imaging each accounted for <8%. For DKA, differences in imaging accounted for 18% of the variation and laboratory charges accounted for 12%. For AGE, imaging accounted for 15% of the variation. Adding the 4 cost components to the other patient- and encounter-level covariates, the model explained 81%, 72%, and 67% of the variation for asthma, DKA, and AGE.

Cost Savings

If all hospitals in this cohort with adjusted standardized costs above the national PHIS average achieved costs equal to the national PHIS average, estimated annual savings in adjusted standardized costs for these 3 conditions would be $69.1 million. If each hospital with adjusted costs above the average within its census region achieved costs equal to its regional average, estimated annual savings in adjusted standardized costs for these conditions would be $25.2 million.

This study reported on the regional variation in costs of care for 3 conditions treated at 46 children’s hospitals across 7 geographic regions, and it demonstrated that variations in costs of care exist in pediatrics. This study used standardized costs to compare utilization patterns across hospitals and adjusted for several patient-level demographic and illness-severity factors, and it found that differences in costs of care for children hospitalized with asthma, DKA, and AGE remained both between and within regions.

These variations are noteworthy, as hospitals strive to improve the value of healthcare. If the higher-cost hospitals in this cohort could achieve costs equal to the national PHIS averages, estimated annual savings in adjusted standardized costs for these conditions alone would equal $69.1 million. If higher-cost hospitals relative to the average in their own region reduced costs to their regional averages, annual standardized cost savings could equal $25.2 million for these conditions.

The differences observed are also significant in that they provide a foundation for exploring whether lower-cost regions or lower-cost hospitals achieve comparable quality outcomes.²⁸ If so, studying what those hospitals do to achieve outcomes more efficiently can serve as the basis for the establishment of best practices.²⁹ Standardizing best practices through protocols, pathways, and care-model redesign can reduce potentially unnecessary spending.³⁰

Our findings showed that patient-level demographic and illness-severity covariates, including community-level HHI and SOI, did not consistently explain cost differences. Instead, LOS and ICU utilization were associated with higher costs.^17,19 When considering the effect of the 4 cost components on the variation in total standardized costs between regions and between hospitals, the fact that the “other” category accounted for the largest percent of the variation is not surprising, because the cost of room occupancy and nursing services increases with longer LOS and more time in the ICU. Other individual cost components that were major drivers of variation were laboratory utilization for asthma and imaging for DKA and AGE³¹ (though they accounted for a much smaller proportion of total adjusted costs).¹⁹

To determine if these factors are modifiable, more information is needed to explain why practices differ. Many factors may contribute to varying utilization patterns, including differences in capabilities and resources (in the hospital and in the community) and patient volumes. For example, some hospitals provide continuous albuterol for status asthmaticus only in ICUs, while others provide it on regular units.³² But if certain hospitals do not have adequate resources or volumes to effectively care for certain populations outside of the ICU, their higher-value approach (considering quality and cost) may be to utilize ICU beds, even if some other hospitals care for those patients on non-ICU floors. Another possibility is that family preferences about care delivery (such as how long children stay in the hospital) may vary across regions.³³

Other evidence suggests that physician practice and spending patterns are strongly influenced by the practices of the region where they trained.³⁴ Because physicians often practice close to where they trained,^35,36 this may partially explain how regional patterns are reinforced.

Even considering all mentioned covariates, our model did not fully explain variation in standardized costs. After adding the cost components as covariates, between one-third and one-fifth of the variation remained unexplained. It is possible that this unexplained variation stemmed from unmeasured patient-level factors.

In addition, while proxies for SES, including community-level HHI, did not significantly predict differences in costs across regions, it is possible that SES affected LOS differently in different regions. Previous studies have suggested that lower SES is associated with longer LOS.³⁷ If this effect is more pronounced in certain regions (potentially because of differences in social service infrastructures), SES may be contributing to variations in cost through LOS.

Our findings were subject to limitations. First, this study only examined 3 diagnoses and did not include surgical or less common conditions. Second, while PHIS includes tertiary care, academic, and freestanding children’s hospitals, it does not include general hospitals, which is where most pediatric patients receive care.³⁸ Third, we used ZIP code-based median annual HHI to account for SES, and we used ZIP codes to determine the distance to the hospital and rural-urban location of patients’ homes. These approximations lack precision because SES and distances vary within ZIP codes.³⁹ Fourth, while adjusted standardized costs allow for comparisons between hospitals, they do not represent actual costs to patients or individual hospitals. Additionally, when determining whether variation remained after controlling for patient-level variables, we included SOI as a reflection of illness-severity at presentation. However, in practice, SOI scores may be assigned partially based on factors determined during the hospitalization.¹⁸ Finally, the use of other regional boundaries or the selection of different hospitals may yield different results.

This study reveals regional variations in costs of care for 3 inpatient pediatric conditions. Future studies should explore whether lower-cost regions or lower-cost hospitals achieve comparable quality outcomes. To the extent that variation is driven by modifiable factors and lower spending does not compromise outcomes, these data may prompt reviews of care models to reduce unwarranted variation and improve the value of care delivery at local, regional, and national levels.

Disclosure

Internal funds from the CHA and The Children’s Hospital of Philadelphia supported the conduct of this work. The authors have no financial interests, relationships, or affiliations relevant to the subject matter or materials discussed in the manuscript to disclose. The authors have no potential conflicts of interest relevant to the subject matter or materials discussed in the manuscript to disclose