Troy Richardson, PhD

The ability to read and correctly interpret research is an essential skill, but most hospitalists—and physicians in general—do not receive formal training in biostatistics during their medical education.^1-3 In addition to straightforward statistical tests that compare a single exposure and outcome, researchers commonly use statistical models to identify and quantify complex relationships among many exposures (eg, demographics, clinical characteristics, interventions, or other variables) and an outcome. Understanding statistical models can be challenging. Still, it is important to recognize the advantages and limitations of statistical models, how to interpret their results, and the potential implications of findings on current clinical practice.

In the article “Rates and Characteristics of Medical Malpractice Claims Against Hospitalists” published in the July 2021 issue of the Journal of Hospital Medicine, Schaffer et al⁴ used the Comparative Benchmarking System database, which is maintained by a malpractice insurer, to characterize malpractice claims against hospitalists. The authors used multiple logistic regression models to understand the relationship among clinical factors and indemnity payments. In this Progress Note, we describe situations in which logistic regression is the proper statistical method to analyze a data set, explain results from logistic regression analyses, and equip readers with skills to critically appraise conclusions drawn from these models.

Statistical models often are used to describe the relationship among one or more exposure variables (ie, independent variables) and an outcome (ie, dependent variable). These models allow researchers to evaluate the effects of multiple exposure variables simultaneously, which in turn allows them to “isolate” the effect of each variable; in other words, models facilitate an understanding of the relationship between each exposure variable and the outcome, adjusted for (ie, independent of) the other exposure variables in the model.

Several statistical models can be used to quantify relationships within the data, but each type of model has certain assumptions that must be satisfied. Two important assumptions include characteristics of the outcome (eg, the type and distribution) and the nature of the relationships among the outcome and independent variables (eg, linear vs nonlinear). Simple linear regression, one of the most basic statistical models used in research,⁵ assumes that (a) the outcome is continuous (ie, any numeric value is possible) and normally distributed (ie, its histogram is a bell-shaped curve) and (b) the relationship between the independent variable and the outcome is linear (ie, follows a straight line). If an investigator wanted to understand how weight is related to height, a simple linear regression could be used to develop a mathematical equation that tells us how the outcome (weight) generally increases as the independent variable (height) increases.

Often, the outcome in a study is not a continuous variable but a simple success/failure variable (ie, dichotomous variable that can be one of two possible values). Schaffer et al⁴ examined the binary outcome of whether a malpractice claim case would end in an indemnity payment or no payment. Linear regression models are not equipped to handle dichotomous outcomes. Instead, we need to use a different statistical model: logistic regression. In logistic regression, the probability (p) of a defined outcome event is estimated by creating a regression model.

A probability (p) is a measure of how likely an event (eg, a malpractice claim ends in an indemnity payment or not) is to occur. It is always between 0 (ie, the event will definitely not occur) and 1 (ie, the event will definitely occur). A p of 0.5 means there is a 50/50 chance that the event will occur (ie, equivalent to a coin flip). Because p is a probability, we need to make sure it is always between 0 and 1. If we were to try to model p with a linear regression, the model would assume that p could extend beyond 0 and 1. What can we do?

Applying a transformation is a commonly used tool in statistics to make data work better within statistical models.⁶ In this case, we will transform the variable p. In logistic regression, we model the probability of experiencing the outcome through a transformation called a logit. The logit represents the natural logarithm (ln) of the ratio of the probability of experiencing the outcome (p) vs the probability of not experiencing the outcome (1 – p), with the ratio being the odds of the event occurring.

bettenhausen0393-1020e-f1.jpg

This transformation works well for dichotomous outcomes because the logit transformation approximates a straight line as long as p is not too large or too small (between 0.05 and 0.95).

If we are performing a logistic regression with only one independent variable (x) and want to understand the relationship between this variable (x) and the probability of an outcome event (p), then our model is the equation of a line. The equation for the base model of logistic regression with one independent variable (x) is

bettenhausen0393-1020e-f2.jpg

where β₀ is the y-intercept and β₁ is the slope of the line. Equation (2) is identical to the algebraic equation y = mx + b for a line, just rearranged slightly. In this algebraic equation, m is the slope (the same as β₁) and b is the y-intercept (the same as β₀). We will see that β₀ and β₁ are estimated (ie, assigned numeric values) from the data collected to help us understand how x and

bettenhausen0393-1020e-f3.jpg

are related and are the basis for estimating odds ratios.

We can build more complex models using multivariable logistic regression by adding more independent variables to the right side of equation (2). Essentially, this is what Schaffer et al⁴ did when, for example, they described clinical factors associated with indemnity payments (Schaffer et al, Table 3).

There are two notable techniques used frequently with multivariable logistic regression models. The first involves choosing which independent variables to include in the model. One way to select variables for multivariable models is defining them a priori, that is deciding which variables are clinically or conceptually associated with the outcome before looking at the data. With this approach, we can test specific hypotheses about the relationships between the independent variables and the outcome. Another common approach is to look at the data and identify the variables that vary significantly between the two outcome groups. Schaffer et al⁴ used an a priori approach to define variables in their multivariable model (ie, “variables for inclusion into the multivariable model were determined a priori”).

A second technique is the evaluation of collinearity, which helps us understand whether the independent variables are related to each other. It is important to consider collinearity between independent variables because the inclusion of two (or more) variables that are highly correlated can cause interference between the two and create misleading results from the model. There are techniques to assess collinear relationships before modeling or as part of the model-building process to determine which variables should be excluded. If there are two (or more) independent variables that are similar, one (or more) must be removed from the model.

Fitting the model is the process by which statistical software (eg, SAS, Stata, R, SPSS) estimates the relationships among independent variables in the model and the outcome within a specific dataset. In equation (2), this essentially means that the software will evaluate the data and provide us with the best estimates for β₀ (the y-intercept) and β₁ (the slope) that describe the relationship between the variable x and

bettenhausen0393-1020e-f4.jpg

Modeling can be iterative, and part of the process may include removing variables from the model that are not significantly associated with the outcome to create a simpler solution, a process known as model reduction. The results from models describe the independent association between a specific characteristic and the outcome, meaning that the relationship has been adjusted for all the other characteristics in the model.

The relationships among the independent variables and outcome are most often represented as an odds ratio (OR), which quantifies the strength of the association between two variables and is directly calculated from the β values in the model. As the name suggests, an OR is a ratio of odds. But what are odds? Simply, the odds of an outcome (such as mortality) is the probability of experiencing the event divided by the probability of not experiencing that event; in other words, it is the ratio:

bettenhausen0393-1020e-f5.jpg

The concept of odds is often unfamiliar, so it can be helpful to consider the definition in the context of games of chance. For example, in horse race betting, the outcome of interest is that a horse will lose a race. Imagine that the probability of a horse losing a race is 0.8 and the probability of winning is 0.2. The odds of losing are

bettenhausen0393-1020e-f6.jpg

These odds usually are listed as 4-to-1, meaning that out of 5 races (ie, 4 + 1) the horse is expected to lose 4 times and win once. When odds are listed this way, we can easily calculate the associated probability by recognizing that the total number of expected races is the sum of two numbers (probability of losing: 4 races out of 5, or 0.80 vs probability of winning: 1 race out of 5, or 0.20).

In medical research, the OR typically represents the odds for one group of patients (A) compared with the odds for another group of patients (B) experiencing an outcome. If the odds of the outcome are the same for group A and group B, then OR = 1.0, meaning that the probability of the outcome is the same between the two groups. If the patients in group A have greater odds of experiencing the outcome compared with group B patients (and a greater probability of the outcome), then the OR will be >1. If the opposite is true, then the OR will be <1.

Schaffer et al⁴ estimated that the OR of an indemnity payment in malpractice cases involving errors in clinical judgment as a contributing factor was 5.01 (95% CI, 3.37-7.45). This means that malpractice cases involving errors in clinical judgement had a 5.01 times greater odds of indemnity payment compared with those without these errors after adjusting for all other variables in the model (eg, age, severity). Note that the 95% CI does not include 1.0. This indicates that the OR is statistically >1, and we can conclude that there is a significant relationship between errors in clinical judgment and payment that is unlikely to be attributed to chance alone.

In logistic regression for categorical independent variables, all categories are compared with a reference group within that variable, with the reference group serving as the denominator of the OR. The authors⁴ did not incorporate continuous independent variables in their multivariable logistic regression model. However, if the authors examined length of hospitalization as a contributing factor in indemnity payments, for example, the OR would represent a 1-unit increase in this variable (eg, 1-day increase in length of stay).

Logistic regression describes the relationships in data and is an important statistical model across many types of research. This Progress Note emphasizes the importance of weighing the advantages and limitations of logistic regression, provides a common approach to data transformation, and guides the correct interpretation of logistic regression model results.

The ability to read and correctly interpret research is an essential skill, but most hospitalists—and physicians in general—do not receive formal training in biostatistics during their medical education.^1-3 In addition to straightforward statistical tests that compare a single exposure and outcome, researchers commonly use statistical models to identify and quantify complex relationships among many exposures (eg, demographics, clinical characteristics, interventions, or other variables) and an outcome. Understanding statistical models can be challenging. Still, it is important to recognize the advantages and limitations of statistical models, how to interpret their results, and the potential implications of findings on current clinical practice.

In the article “Rates and Characteristics of Medical Malpractice Claims Against Hospitalists” published in the July 2021 issue of the Journal of Hospital Medicine, Schaffer et al⁴ used the Comparative Benchmarking System database, which is maintained by a malpractice insurer, to characterize malpractice claims against hospitalists. The authors used multiple logistic regression models to understand the relationship among clinical factors and indemnity payments. In this Progress Note, we describe situations in which logistic regression is the proper statistical method to analyze a data set, explain results from logistic regression analyses, and equip readers with skills to critically appraise conclusions drawn from these models.

Statistical models often are used to describe the relationship among one or more exposure variables (ie, independent variables) and an outcome (ie, dependent variable). These models allow researchers to evaluate the effects of multiple exposure variables simultaneously, which in turn allows them to “isolate” the effect of each variable; in other words, models facilitate an understanding of the relationship between each exposure variable and the outcome, adjusted for (ie, independent of) the other exposure variables in the model.

Several statistical models can be used to quantify relationships within the data, but each type of model has certain assumptions that must be satisfied. Two important assumptions include characteristics of the outcome (eg, the type and distribution) and the nature of the relationships among the outcome and independent variables (eg, linear vs nonlinear). Simple linear regression, one of the most basic statistical models used in research,⁵ assumes that (a) the outcome is continuous (ie, any numeric value is possible) and normally distributed (ie, its histogram is a bell-shaped curve) and (b) the relationship between the independent variable and the outcome is linear (ie, follows a straight line). If an investigator wanted to understand how weight is related to height, a simple linear regression could be used to develop a mathematical equation that tells us how the outcome (weight) generally increases as the independent variable (height) increases.

Often, the outcome in a study is not a continuous variable but a simple success/failure variable (ie, dichotomous variable that can be one of two possible values). Schaffer et al⁴ examined the binary outcome of whether a malpractice claim case would end in an indemnity payment or no payment. Linear regression models are not equipped to handle dichotomous outcomes. Instead, we need to use a different statistical model: logistic regression. In logistic regression, the probability (p) of a defined outcome event is estimated by creating a regression model.

A probability (p) is a measure of how likely an event (eg, a malpractice claim ends in an indemnity payment or not) is to occur. It is always between 0 (ie, the event will definitely not occur) and 1 (ie, the event will definitely occur). A p of 0.5 means there is a 50/50 chance that the event will occur (ie, equivalent to a coin flip). Because p is a probability, we need to make sure it is always between 0 and 1. If we were to try to model p with a linear regression, the model would assume that p could extend beyond 0 and 1. What can we do?

Applying a transformation is a commonly used tool in statistics to make data work better within statistical models.⁶ In this case, we will transform the variable p. In logistic regression, we model the probability of experiencing the outcome through a transformation called a logit. The logit represents the natural logarithm (ln) of the ratio of the probability of experiencing the outcome (p) vs the probability of not experiencing the outcome (1 – p), with the ratio being the odds of the event occurring.

bettenhausen0393-1020e-f1.jpg

This transformation works well for dichotomous outcomes because the logit transformation approximates a straight line as long as p is not too large or too small (between 0.05 and 0.95).

If we are performing a logistic regression with only one independent variable (x) and want to understand the relationship between this variable (x) and the probability of an outcome event (p), then our model is the equation of a line. The equation for the base model of logistic regression with one independent variable (x) is

bettenhausen0393-1020e-f2.jpg

where β₀ is the y-intercept and β₁ is the slope of the line. Equation (2) is identical to the algebraic equation y = mx + b for a line, just rearranged slightly. In this algebraic equation, m is the slope (the same as β₁) and b is the y-intercept (the same as β₀). We will see that β₀ and β₁ are estimated (ie, assigned numeric values) from the data collected to help us understand how x and

bettenhausen0393-1020e-f3.jpg

are related and are the basis for estimating odds ratios.

We can build more complex models using multivariable logistic regression by adding more independent variables to the right side of equation (2). Essentially, this is what Schaffer et al⁴ did when, for example, they described clinical factors associated with indemnity payments (Schaffer et al, Table 3).

There are two notable techniques used frequently with multivariable logistic regression models. The first involves choosing which independent variables to include in the model. One way to select variables for multivariable models is defining them a priori, that is deciding which variables are clinically or conceptually associated with the outcome before looking at the data. With this approach, we can test specific hypotheses about the relationships between the independent variables and the outcome. Another common approach is to look at the data and identify the variables that vary significantly between the two outcome groups. Schaffer et al⁴ used an a priori approach to define variables in their multivariable model (ie, “variables for inclusion into the multivariable model were determined a priori”).

A second technique is the evaluation of collinearity, which helps us understand whether the independent variables are related to each other. It is important to consider collinearity between independent variables because the inclusion of two (or more) variables that are highly correlated can cause interference between the two and create misleading results from the model. There are techniques to assess collinear relationships before modeling or as part of the model-building process to determine which variables should be excluded. If there are two (or more) independent variables that are similar, one (or more) must be removed from the model.

Fitting the model is the process by which statistical software (eg, SAS, Stata, R, SPSS) estimates the relationships among independent variables in the model and the outcome within a specific dataset. In equation (2), this essentially means that the software will evaluate the data and provide us with the best estimates for β₀ (the y-intercept) and β₁ (the slope) that describe the relationship between the variable x and

bettenhausen0393-1020e-f4.jpg

Modeling can be iterative, and part of the process may include removing variables from the model that are not significantly associated with the outcome to create a simpler solution, a process known as model reduction. The results from models describe the independent association between a specific characteristic and the outcome, meaning that the relationship has been adjusted for all the other characteristics in the model.

The relationships among the independent variables and outcome are most often represented as an odds ratio (OR), which quantifies the strength of the association between two variables and is directly calculated from the β values in the model. As the name suggests, an OR is a ratio of odds. But what are odds? Simply, the odds of an outcome (such as mortality) is the probability of experiencing the event divided by the probability of not experiencing that event; in other words, it is the ratio:

bettenhausen0393-1020e-f5.jpg

The concept of odds is often unfamiliar, so it can be helpful to consider the definition in the context of games of chance. For example, in horse race betting, the outcome of interest is that a horse will lose a race. Imagine that the probability of a horse losing a race is 0.8 and the probability of winning is 0.2. The odds of losing are

bettenhausen0393-1020e-f6.jpg

These odds usually are listed as 4-to-1, meaning that out of 5 races (ie, 4 + 1) the horse is expected to lose 4 times and win once. When odds are listed this way, we can easily calculate the associated probability by recognizing that the total number of expected races is the sum of two numbers (probability of losing: 4 races out of 5, or 0.80 vs probability of winning: 1 race out of 5, or 0.20).

In medical research, the OR typically represents the odds for one group of patients (A) compared with the odds for another group of patients (B) experiencing an outcome. If the odds of the outcome are the same for group A and group B, then OR = 1.0, meaning that the probability of the outcome is the same between the two groups. If the patients in group A have greater odds of experiencing the outcome compared with group B patients (and a greater probability of the outcome), then the OR will be >1. If the opposite is true, then the OR will be <1.

Schaffer et al⁴ estimated that the OR of an indemnity payment in malpractice cases involving errors in clinical judgment as a contributing factor was 5.01 (95% CI, 3.37-7.45). This means that malpractice cases involving errors in clinical judgement had a 5.01 times greater odds of indemnity payment compared with those without these errors after adjusting for all other variables in the model (eg, age, severity). Note that the 95% CI does not include 1.0. This indicates that the OR is statistically >1, and we can conclude that there is a significant relationship between errors in clinical judgment and payment that is unlikely to be attributed to chance alone.

In logistic regression for categorical independent variables, all categories are compared with a reference group within that variable, with the reference group serving as the denominator of the OR. The authors⁴ did not incorporate continuous independent variables in their multivariable logistic regression model. However, if the authors examined length of hospitalization as a contributing factor in indemnity payments, for example, the OR would represent a 1-unit increase in this variable (eg, 1-day increase in length of stay).

Logistic regression describes the relationships in data and is an important statistical model across many types of research. This Progress Note emphasizes the importance of weighing the advantages and limitations of logistic regression, provides a common approach to data transformation, and guides the correct interpretation of logistic regression model results.

The ability to read and correctly interpret research is an essential skill, but most hospitalists—and physicians in general—do not receive formal training in biostatistics during their medical education.^1-3 In addition to straightforward statistical tests that compare a single exposure and outcome, researchers commonly use statistical models to identify and quantify complex relationships among many exposures (eg, demographics, clinical characteristics, interventions, or other variables) and an outcome. Understanding statistical models can be challenging. Still, it is important to recognize the advantages and limitations of statistical models, how to interpret their results, and the potential implications of findings on current clinical practice.

In the article “Rates and Characteristics of Medical Malpractice Claims Against Hospitalists” published in the July 2021 issue of the Journal of Hospital Medicine, Schaffer et al⁴ used the Comparative Benchmarking System database, which is maintained by a malpractice insurer, to characterize malpractice claims against hospitalists. The authors used multiple logistic regression models to understand the relationship among clinical factors and indemnity payments. In this Progress Note, we describe situations in which logistic regression is the proper statistical method to analyze a data set, explain results from logistic regression analyses, and equip readers with skills to critically appraise conclusions drawn from these models.

Statistical models often are used to describe the relationship among one or more exposure variables (ie, independent variables) and an outcome (ie, dependent variable). These models allow researchers to evaluate the effects of multiple exposure variables simultaneously, which in turn allows them to “isolate” the effect of each variable; in other words, models facilitate an understanding of the relationship between each exposure variable and the outcome, adjusted for (ie, independent of) the other exposure variables in the model.

Several statistical models can be used to quantify relationships within the data, but each type of model has certain assumptions that must be satisfied. Two important assumptions include characteristics of the outcome (eg, the type and distribution) and the nature of the relationships among the outcome and independent variables (eg, linear vs nonlinear). Simple linear regression, one of the most basic statistical models used in research,⁵ assumes that (a) the outcome is continuous (ie, any numeric value is possible) and normally distributed (ie, its histogram is a bell-shaped curve) and (b) the relationship between the independent variable and the outcome is linear (ie, follows a straight line). If an investigator wanted to understand how weight is related to height, a simple linear regression could be used to develop a mathematical equation that tells us how the outcome (weight) generally increases as the independent variable (height) increases.

Often, the outcome in a study is not a continuous variable but a simple success/failure variable (ie, dichotomous variable that can be one of two possible values). Schaffer et al⁴ examined the binary outcome of whether a malpractice claim case would end in an indemnity payment or no payment. Linear regression models are not equipped to handle dichotomous outcomes. Instead, we need to use a different statistical model: logistic regression. In logistic regression, the probability (p) of a defined outcome event is estimated by creating a regression model.

A probability (p) is a measure of how likely an event (eg, a malpractice claim ends in an indemnity payment or not) is to occur. It is always between 0 (ie, the event will definitely not occur) and 1 (ie, the event will definitely occur). A p of 0.5 means there is a 50/50 chance that the event will occur (ie, equivalent to a coin flip). Because p is a probability, we need to make sure it is always between 0 and 1. If we were to try to model p with a linear regression, the model would assume that p could extend beyond 0 and 1. What can we do?

Applying a transformation is a commonly used tool in statistics to make data work better within statistical models.⁶ In this case, we will transform the variable p. In logistic regression, we model the probability of experiencing the outcome through a transformation called a logit. The logit represents the natural logarithm (ln) of the ratio of the probability of experiencing the outcome (p) vs the probability of not experiencing the outcome (1 – p), with the ratio being the odds of the event occurring.

bettenhausen0393-1020e-f1.jpg

This transformation works well for dichotomous outcomes because the logit transformation approximates a straight line as long as p is not too large or too small (between 0.05 and 0.95).

If we are performing a logistic regression with only one independent variable (x) and want to understand the relationship between this variable (x) and the probability of an outcome event (p), then our model is the equation of a line. The equation for the base model of logistic regression with one independent variable (x) is

bettenhausen0393-1020e-f2.jpg

where β₀ is the y-intercept and β₁ is the slope of the line. Equation (2) is identical to the algebraic equation y = mx + b for a line, just rearranged slightly. In this algebraic equation, m is the slope (the same as β₁) and b is the y-intercept (the same as β₀). We will see that β₀ and β₁ are estimated (ie, assigned numeric values) from the data collected to help us understand how x and

bettenhausen0393-1020e-f3.jpg

are related and are the basis for estimating odds ratios.

We can build more complex models using multivariable logistic regression by adding more independent variables to the right side of equation (2). Essentially, this is what Schaffer et al⁴ did when, for example, they described clinical factors associated with indemnity payments (Schaffer et al, Table 3).

There are two notable techniques used frequently with multivariable logistic regression models. The first involves choosing which independent variables to include in the model. One way to select variables for multivariable models is defining them a priori, that is deciding which variables are clinically or conceptually associated with the outcome before looking at the data. With this approach, we can test specific hypotheses about the relationships between the independent variables and the outcome. Another common approach is to look at the data and identify the variables that vary significantly between the two outcome groups. Schaffer et al⁴ used an a priori approach to define variables in their multivariable model (ie, “variables for inclusion into the multivariable model were determined a priori”).

A second technique is the evaluation of collinearity, which helps us understand whether the independent variables are related to each other. It is important to consider collinearity between independent variables because the inclusion of two (or more) variables that are highly correlated can cause interference between the two and create misleading results from the model. There are techniques to assess collinear relationships before modeling or as part of the model-building process to determine which variables should be excluded. If there are two (or more) independent variables that are similar, one (or more) must be removed from the model.

Fitting the model is the process by which statistical software (eg, SAS, Stata, R, SPSS) estimates the relationships among independent variables in the model and the outcome within a specific dataset. In equation (2), this essentially means that the software will evaluate the data and provide us with the best estimates for β₀ (the y-intercept) and β₁ (the slope) that describe the relationship between the variable x and

bettenhausen0393-1020e-f4.jpg

Modeling can be iterative, and part of the process may include removing variables from the model that are not significantly associated with the outcome to create a simpler solution, a process known as model reduction. The results from models describe the independent association between a specific characteristic and the outcome, meaning that the relationship has been adjusted for all the other characteristics in the model.

The relationships among the independent variables and outcome are most often represented as an odds ratio (OR), which quantifies the strength of the association between two variables and is directly calculated from the β values in the model. As the name suggests, an OR is a ratio of odds. But what are odds? Simply, the odds of an outcome (such as mortality) is the probability of experiencing the event divided by the probability of not experiencing that event; in other words, it is the ratio:

bettenhausen0393-1020e-f5.jpg

The concept of odds is often unfamiliar, so it can be helpful to consider the definition in the context of games of chance. For example, in horse race betting, the outcome of interest is that a horse will lose a race. Imagine that the probability of a horse losing a race is 0.8 and the probability of winning is 0.2. The odds of losing are

bettenhausen0393-1020e-f6.jpg

These odds usually are listed as 4-to-1, meaning that out of 5 races (ie, 4 + 1) the horse is expected to lose 4 times and win once. When odds are listed this way, we can easily calculate the associated probability by recognizing that the total number of expected races is the sum of two numbers (probability of losing: 4 races out of 5, or 0.80 vs probability of winning: 1 race out of 5, or 0.20).

In medical research, the OR typically represents the odds for one group of patients (A) compared with the odds for another group of patients (B) experiencing an outcome. If the odds of the outcome are the same for group A and group B, then OR = 1.0, meaning that the probability of the outcome is the same between the two groups. If the patients in group A have greater odds of experiencing the outcome compared with group B patients (and a greater probability of the outcome), then the OR will be >1. If the opposite is true, then the OR will be <1.

Schaffer et al⁴ estimated that the OR of an indemnity payment in malpractice cases involving errors in clinical judgment as a contributing factor was 5.01 (95% CI, 3.37-7.45). This means that malpractice cases involving errors in clinical judgement had a 5.01 times greater odds of indemnity payment compared with those without these errors after adjusting for all other variables in the model (eg, age, severity). Note that the 95% CI does not include 1.0. This indicates that the OR is statistically >1, and we can conclude that there is a significant relationship between errors in clinical judgment and payment that is unlikely to be attributed to chance alone.

In logistic regression for categorical independent variables, all categories are compared with a reference group within that variable, with the reference group serving as the denominator of the OR. The authors⁴ did not incorporate continuous independent variables in their multivariable logistic regression model. However, if the authors examined length of hospitalization as a contributing factor in indemnity payments, for example, the OR would represent a 1-unit increase in this variable (eg, 1-day increase in length of stay).

Logistic regression describes the relationships in data and is an important statistical model across many types of research. This Progress Note emphasizes the importance of weighing the advantages and limitations of logistic regression, provides a common approach to data transformation, and guides the correct interpretation of logistic regression model results.

The COVID-19 pandemic has led to more than 40 million infections and more than 650,000 deaths in the United States alone.¹ Morbidity and mortality have disproportionately affected older adults.^2-4 However, acute infection and delayed effects, such as multisystem inflammatory syndrome in children (MIS-C), occur and can lead to severe complications, hospitalization, and death in pediatric patients.^5,6 Due to higher clinical disease prevalence and morbidity in the adult population, we have learned much about the clinical factors associated with severe adult COVID-19 disease.^5,7-9 Such clinical factors include older age, concurrent comorbidities, smoke exposure, and Black race or Hispanic ethnicity, among others.^5,7-10 However, there is a paucity of data on severe COVID-19 disease in pediatric patients.^5,11,12 In addition, most immunization strategies and pharmacologic treatments for COVID-19 have not been evaluated or approved for use in children.¹³ To guide targeted prevention and treatment strategies, there is a critical need to identify children and adolescents—who are among the most vulnerable patient populations—at high risk for severe disease.

Identifying the clinical factors associated with severe COVID-19 disease will help with prioritizing and allocating vaccines when they are approved for use in patients younger than 12 years. It also can provide insight for clinicians and families faced with decisions wherein individual risk assessment is crucial (eg, in-person schooling, other group activities). The objective of this study was to determine the clinical factors associated with severe COVID-19 among children and adolescents in the United States.

Study Design

We conducted a multicenter retrospective cohort study of patients presenting for care at pediatric hospitals that report data to the Pediatric Health Information System (PHIS) database. The PHIS administrative database includes billing and utilization data from 45 US tertiary care hospitals affiliated with the Children’s Hospital Association (Lenexa, Kansas). Data quality and reliability are ensured through a joint validation effort between the Children’s Hospital Association and participating hospitals. Hospitals submit discharge data, including demographics, diagnoses, and procedures using International Classification of Diseases, 10th Revision (ICD-10) codes, along with daily detailed information on pharmacy, location of care, and other services.

Study Population

Patients 30 days to 18 years of age discharged from the emergency department (ED) or inpatient setting with a primary diagnosis of COVID-19 (ICD-10 codes U.071 and U.072) between April 1, 2020, and September 30, 2020, were eligible for inclusion.¹⁴ In a prior study, the positive predictive value of an ICD-10–coded diagnosis of COVID-19 among hospitalized pediatric patients was 95.5%, compared with reverse transcription polymerase reaction results or presence of MIS-C.¹⁵ The diagnostic code for COVID-19 (ICD-10-CM) also had a high sensitivity (98.0%) in the hospitalized population.¹⁶ Acknowledging the increasing practice of screening patients upon admission, and in an attempt to minimize potential misclassification, we did not include encounters with secondary diagnoses of COVID-19 in our primary analyses. Pediatric patients with surgical diagnoses and neonates who never left the hospital were also excluded.

Factors Associated With Severe COVID-19 Disease

Exposures of interest were determined a priori based on current evidence in the literature and included patient age (0-4 years, 5-11 years, and 12-18 years), sex, race and ethnicity (non-Hispanic White, non-Hispanic Black, Hispanic, Asian, other non-White race [defined as Pacific Islander, Native American, or other]), payor type, cardiovascular complex chronic conditions (CCC), neuromuscular CCC, obesity/type 2 diabetes mellitus (DM), pulmonary CCC, asthma (defined using ICD-10 codes¹⁷), and immunocompromised CCC. Race and ethnicity were included as covariates based on previous studies reporting differences in COVID-19 outcomes among racial and ethnic groups.⁹ The CCC covariates were defined using the pediatric CCC ICD-10 classification system version 2.¹⁸

Pediatric Complications and Conditions Associated With COVID-19

Based on current evidence and expert opinion of study members, associated diagnoses and complications co-occurring with a COVID-19 diagnosis were defined a priori and identified through ICD-10 codes (Appendix Table 1). These included acute kidney injury, acute liver injury, aseptic meningitis, asthma exacerbation, bronchiolitis, cerebral infarction, croup, encephalitis, encephalopathy, infant fever, febrile seizure, gastroenteritis/dehydration, Kawasaki disease/MIS-C, myocarditis/pericarditis, pneumonia, lung effusion or empyema, respiratory failure, sepsis, nonfebrile seizure, pancreatitis, sickle cell complications, and thrombotic complications.

Outcomes

COVID-19 severity outcomes were assessed as follows: (1) mild = ED discharge; (2) moderate = inpatient admission; (3) severe = intensive care unit (ICU) admission without mechanical ventilation, shock, or death; and (4) very severe = ICU admission with mechanical ventilation, shock, or death.¹⁹ This ordinal ranking system did not violate the proportional odds assumption. Potential reasons for admission to the ICU without mechanical ventilation, shock, or death include, but are not limited to, need for noninvasive ventilation, vital sign instability, dysrhythmias, respiratory insufficiency, or complications arising from concurrent conditions (eg, thrombotic events, need for continuous albuterol therapy). We examined several secondary, hospital-based outcomes, including associated diagnoses and complications, all-cause 30-day healthcare reutilization (ED visit or rehospitalization), length of stay (LOS), and ICU LOS.

Statistical Analysis

Demographic characteristics were summarized using frequencies and percentages for categorical variables and geometric means with SD and medians with interquartile ranges (IQR) for continuous variables, as appropriate. Factors associated with hospitalization (encompassing severity levels 2-4) vs ED discharge (severity level 1) were assessed using logistic regression. Factors associated with increasing severity among hospitalized pediatric patients (severity levels 2, 3, and 4) were assessed using ordinal logistic regression. Covariates in these analyses included race and ethnicity, age, sex, payor, cardiovascular CCC, neurologic/neuromuscular CCC, obesity/type 2 DM, pulmonary CCC, asthma, and immunocompromised CCC. Adjusted odds ratios (aOR) and corresponding 95% CI for each risk factor were generated using generalized linear mixed effects models and random intercepts for each hospital. Given the potential for diagnostic misclassification of pediatric patients with COVID-19 based on primary vs secondary diagnoses, we performed sensitivity analyses defining the study population as those with a primary diagnosis of COVID-19 and those with a secondary diagnosis of COVID-19 plus a concurrent primary diagnosis of a condition associated with COVID-19 (Appendix Table 1).

All analyses were performed using SAS version 9.4 (SAS Institute, Inc), and P < .05 was considered statistically significant. The Institutional Review Board at Vanderbilt University Medical Center determined that this study of de-identified data did not meet the criteria for human subjects research.

Study Population

A total of 19,976 encounters were included in the study. Of those, 15,913 (79.7%) were discharged from the ED and 4063 (20.3%) were hospitalized (Table 1). The most common race/ethnicity was Hispanic (9741, 48.8%), followed by non-Hispanic White (4217, 21.1%). Reference race/ethnicity data for the overall 2019 PHIS population can be found in Appendix Table 2.

JHMVol16No9_Antoon03490915e_t1.JPG

The severity distribution among the hospitalized population was moderate (3222, 79.3%), severe (431, 11.3%), and very severe (380, 9.4%). The frequency of COVID-19 diagnoses increased late in the study period (Figure). Among those hospitalized, the median LOS for the index admission was 2 days (IQR, 1-4), while among those admitted to the ICU, the median LOS was 3 days (IQR, 2-5).

JHMVol16No9_Antoon03490915e_f1.JPG

Overall, 10.1% (n = 2020) of the study population had an all-cause repeat encounter (ie, subsequent ED encounter or hospitalization) within 30 days following the index discharge. Repeat encounters were more frequent among patients hospitalized than among those discharged from the ED (Appendix Table 3).

Prevalence of Conditions and Complications Associated With COVID-19

Overall, 3257 (16.3%) patients had one or more co-occurring diagnoses categorized as a COVID-19–associated condition or complication. The most frequent diagnoses included lower respiratory tract disease (pneumonia, lung effusion, or empyema; n = 1415, 7.1%), gastroenteritis/dehydration (n = 1068, 5.3%), respiratory failure (n = 731, 3.7%), febrile infant (n = 413, 2.1%), and nonfebrile seizure (n = 425, 2.1%). Aside from nonfebrile seizure, neurological complications were less frequent and included febrile seizure (n = 155, 0.8%), encephalopathy (n = 63, 0.3%), aseptic meningitis (n = 16, 0.1%), encephalitis (n = 11, 0.1%), and cerebral infarction (n = 6, <0.1%). Kawasaki disease and MIS-C comprised 1.7% (n = 346) of diagnoses. Thrombotic complications occurred in 0.1% (n = 13) of patients. Overall, these conditions and complications associated with COVID-19 were more frequent in hospitalized patients than in those discharged from the ED (P < .001) (Table 2).

JHMVol16No9_Antoon03490915e_t2.JPG

Factors Associated With COVID-19 Disease Severity

Compared to pediatric patients with COVID-19 discharged from the ED, factors associated with increased odds of hospitalization included private payor insurance; obesity/type 2 DM; asthma; and cardiovascular, immunocompromised, neurologic/neuromuscular, and pulmonary CCCs (Table 3). Factors associated with decreased risk of hospitalization included Black race or Hispanic ethnicity compared with White race; female sex; and age 5 to 11 years and age 12 to 17 years (vs age 0-4 years). Among children and adolescents hospitalized with COVID-19, factors associated with greater disease severity included Black or other non-White race; age 5 to 11 years; age 12 to 17 years; obesity/type 2 DM; immunocompromised conditions; and cardiovascular, neurologic/neuromuscular, and pulmonary CCCs (Table 3).

JHMVol16No9_Antoon03490915e_t3.JPG

Sensitivity Analysis

We performed a sensitivity analysis that expanded the study population to include those with a secondary diagnosis of COVID-19 plus a diagnosis of a COVID-19–associated condition or complication. Analyses using the expanded population (N = 21,247) were similar to the primary analyses (Appendix Table 4 and Appendix Table 5).

In this large multicenter study evaluating COVID-19 disease severity in more than 19,000 patients presenting for emergency care at US pediatric hospitals, approximately 20% were hospitalized, and among those hospitalized almost a quarter required ICU care. Clinical risk factors associated with increased risk of hospitalization include private payor status and selected comorbidities (obesity/type 2 DM; asthma; and cardiovascular, pulmonary, immunocompromised, neurologic/neuromuscular CCCs), while those associated with decreased risk of hospitalization include older age, female sex, and Black race or Hispanic ethnicity. Factors associated with severe disease among hospitalized pediatric patients include Black or other non-White race, school age (≥5 years), and certain chronic conditions (cardiovascular disease, obesity/type 2 DM, neurologic or neuromuscular disease). Sixteen percent of patients had a concurrent diagnosis for a condition or complication associated with COVID-19.

While the study population (ie, children and adolescents presenting to the ED) represents a small fraction of children and adolescents in the community with SARS-CoV-2 infection, the results provide important insight into factors of severe COVID-19 in the pediatric population. A report from France suggested ventilatory or hemodynamic support or death were independently associated with older age (≥10 years), elevated C-reactive protein, and hypoxemia.¹² An Italian study found that younger age (0-4 years) was associated with less severe disease, while preexisting conditions were more likely in patients with severe disease.¹¹ A single-center case series of 50 patients (aged ≤21 years) hospitalized at a children’s hospital in New York City found respiratory failure (n = 9) was more common in children older than 1 year, patients with elevated inflammatory markers, and patients with obesity.²⁰

Our study confirms several factors for severe COVID-19 found in these studies, including older age,^11,12,20 obesity,²⁰ and preexisting conditions.¹¹ Our findings also expand on these reports, including identification of factors associated with hospitalization. Given the rate of 30-day re-encounters among pediatric patients with COVID-19 (10.1%), identifying risk factors for hospitalization may aid ED providers in determining optimal disposition (eg, home, hospital admission, ICU). We also identified specific comorbidities associated with more severe disease in those hospitalized with COVID-19, such as cardiovascular disease, obesity/type 2 DM, and pulmonary, neurologic, or neuromuscular conditions. We also found that asthma increased the risk for hospitalization but not more severe disease among those hospitalized. This latter finding also aligns with recent single-center studies,^21,22 whereas a Turkish study of pediatric patients aged 0 to 18 years found no association between asthma and COVID-19 hospitalizations.²³We also examined payor type and racial/ethnic factors in our analysis. In 2019, patients who identified as Black or Hispanic comprised 52.3% of all encounters and 40.7% of hospitalizations recorded in the PHIS database. During the same year, encounters for influenza among Black or Hispanic pediatric patients comprised 58.7% of all influenza diagnoses and 47.0% of pediatric influenza hospitalizations (Appendix Table 2). In this study, patients who identified as Black or Hispanic race represented a disproportionately large share of patients presenting to children’s hospitals (68.5%) and of those hospitalized (60.8%). Hispanic ethnicity, in particular, represented a disproportionate share of patients seeking care for COVID-19 compared to the overall PHIS population (47.7% and 27.1%, respectively). After accounting for other factors, we found Black and other non-White race—but not of Hispanic ethnicity—were independently associated with more disease severity among those hospitalized. This contrasts with findings from a recent adult study by Yehia et al,²⁴ who found (after adjusting for other clinical factors) no significant difference in mortality between Black patients and White patients among adults hospitalized due to COVID-19. It also contrasts with a recent large population-based UK study wherein pediatric patients identifying as Asian, but not Black or mixed race or ethnicity, had an increased risk of hospital admission and admission to the ICU compared to children identifying as White. Children identifying as Black or mixed race had longer hospital admissions.²⁵ However, as the authors of the study note, residual confounders and ascertainment bias due to differences in COVID testing may have influenced these findings.

Our findings of differences in hospitalization and disease severity among those hospitalized by race and ethnicity should be interpreted carefully. These may reflect a constellation of factors that are difficult to measure, including differences in healthcare access, inequalities in care (including hospital admission inequalities), and implicit bias—all of which may reflect structural racism. For example, it is possible that children who identify as Black or Hispanic have different access to care compared to children who identify as White, and this may affect disease severity on presentation.² Alternatively, it is possible that White pediatric patients are more likely to be hospitalized as compared to non-White pediatric patients with similar illness severity. Our finding that pediatric patients who identify as Hispanic or Black had a lower risk of hospitalization should be also interpreted carefully, as this may reflect higher utilization of the ED for SARS-CoV-2 testing, increased use of nonemergency services among those without access to primary care, or systematic differences in provider decision-making among this segment of the population.² Further study is needed to determine specific drivers for racial and ethnic differences in healthcare utilization in children and adolescents with COVID-19.²⁶

Complications and co-occurring diagnoses in adults with COVID-19 are well documented.^27-30 However, there is little information to date on the co-occurring diagnoses and complications associated with COVID-19 in children and adolescents. We found that complications and co-occurring conditions occurred in 16.3% of the study population, with the most frequent conditions including known complications of viral infections such as pneumonia, respiratory failure, and seizures. Acute kidney and liver injury, as well as thrombotic complications, occurred less commonly than in adults.^26-29 Interestingly, neurologic complications were also uncommon compared to adult reports^8,31 and less frequent than in other viral illnesses in children and adolescents. For example, neurologic complications occur in approximately 7.5% of children and adolescents hospitalized with influenza.³²

Limitations of the present study include the retrospective design, as well as incomplete patient-level clinical data in the PHIS database. The PHIS database only includes children’s hospitals, which may limit the generalizability of findings to community hospitals. We also excluded newborns, and our findings may not be generalizable to this population. We only included children and adolescents with a primary diagnosis of COVID-19, which has the potential for misclassification in cases where COVID-19 was a secondary diagnosis. However, results of our sensitivity analysis, which incorporated secondary diagnoses of COVID-19, were consistent with findings from our main analyses. Our study was designed to examine associations between certain prespecified factors and COVID-19 severity among pediatric patients who visited the ED or were admitted to the hospital during the COVID-19 pandemic. Thus, our findings must be interpreted in light of these considerations and may not be generalizable outside the ED or hospital setting. For example, it could be that some segments of the population utilized ED resources for testing, whereas others avoided the ED and other healthcare settings for fear of exposure to SARS-CoV-2. We also relied on diagnosis codes to identify concurrent diagnoses, as well as mechanical ventilation in our very severe outcome cohort, which resulted in this classification for some of these diagnoses. Despite these limitations, our findings represent an important step in understanding the risk factors associated with severe clinical COVID-19 disease in pediatric patients.

Our findings may inform future research and clinical interventions. Future studies on antiviral therapies and immune modulators targeting SARS-CoV-2 infection in children and adolescents should focus on high-risk populations, such as those identified in the study, as these patients are most likely to benefit from therapeutic interventions. Similarly, vaccine-development efforts may benefit from additional evaluation in high-risk populations, some of which may have altered immune responses. Furthermore, with increasing vaccination among adults and changes in recommendations, societal mitigation efforts (eg, masking, physical distancing) will diminish. Continued vigilance and COVID-19–mitigation efforts among high-risk children, for whom vaccines are not yet available, are critical during this transition.

Among children with COVID-19 who received care at children’s hospitals and EDs, 20% were hospitalized, and, of those, 21% were admitted to the ICU. Older children and adolescent patients had a lower risk of hospitalization; however, when hospitalized, they had greater illness severity. Those with selected comorbidities (eg, cardiovascular, obesity/type 2 DM, pulmonary and neurologic or neuromuscular disease) had both increased odds of hospitalization and in-hospital illness severity. While there were observed differences in COVID-19 severity by race and ethnicity, additional research is needed to clarify the drivers of such disparities. These factors should be considered when prioritizing mitigation strategies to prevent infection (eg, remote learning, avoidance of group activities, prioritization of COVID-19 vaccine when approved for children aged <12 years).

The COVID-19 pandemic has led to more than 40 million infections and more than 650,000 deaths in the United States alone.¹ Morbidity and mortality have disproportionately affected older adults.^2-4 However, acute infection and delayed effects, such as multisystem inflammatory syndrome in children (MIS-C), occur and can lead to severe complications, hospitalization, and death in pediatric patients.^5,6 Due to higher clinical disease prevalence and morbidity in the adult population, we have learned much about the clinical factors associated with severe adult COVID-19 disease.^5,7-9 Such clinical factors include older age, concurrent comorbidities, smoke exposure, and Black race or Hispanic ethnicity, among others.^5,7-10 However, there is a paucity of data on severe COVID-19 disease in pediatric patients.^5,11,12 In addition, most immunization strategies and pharmacologic treatments for COVID-19 have not been evaluated or approved for use in children.¹³ To guide targeted prevention and treatment strategies, there is a critical need to identify children and adolescents—who are among the most vulnerable patient populations—at high risk for severe disease.

Identifying the clinical factors associated with severe COVID-19 disease will help with prioritizing and allocating vaccines when they are approved for use in patients younger than 12 years. It also can provide insight for clinicians and families faced with decisions wherein individual risk assessment is crucial (eg, in-person schooling, other group activities). The objective of this study was to determine the clinical factors associated with severe COVID-19 among children and adolescents in the United States.

Study Design

We conducted a multicenter retrospective cohort study of patients presenting for care at pediatric hospitals that report data to the Pediatric Health Information System (PHIS) database. The PHIS administrative database includes billing and utilization data from 45 US tertiary care hospitals affiliated with the Children’s Hospital Association (Lenexa, Kansas). Data quality and reliability are ensured through a joint validation effort between the Children’s Hospital Association and participating hospitals. Hospitals submit discharge data, including demographics, diagnoses, and procedures using International Classification of Diseases, 10th Revision (ICD-10) codes, along with daily detailed information on pharmacy, location of care, and other services.

Study Population

Patients 30 days to 18 years of age discharged from the emergency department (ED) or inpatient setting with a primary diagnosis of COVID-19 (ICD-10 codes U.071 and U.072) between April 1, 2020, and September 30, 2020, were eligible for inclusion.¹⁴ In a prior study, the positive predictive value of an ICD-10–coded diagnosis of COVID-19 among hospitalized pediatric patients was 95.5%, compared with reverse transcription polymerase reaction results or presence of MIS-C.¹⁵ The diagnostic code for COVID-19 (ICD-10-CM) also had a high sensitivity (98.0%) in the hospitalized population.¹⁶ Acknowledging the increasing practice of screening patients upon admission, and in an attempt to minimize potential misclassification, we did not include encounters with secondary diagnoses of COVID-19 in our primary analyses. Pediatric patients with surgical diagnoses and neonates who never left the hospital were also excluded.

Factors Associated With Severe COVID-19 Disease

Exposures of interest were determined a priori based on current evidence in the literature and included patient age (0-4 years, 5-11 years, and 12-18 years), sex, race and ethnicity (non-Hispanic White, non-Hispanic Black, Hispanic, Asian, other non-White race [defined as Pacific Islander, Native American, or other]), payor type, cardiovascular complex chronic conditions (CCC), neuromuscular CCC, obesity/type 2 diabetes mellitus (DM), pulmonary CCC, asthma (defined using ICD-10 codes¹⁷), and immunocompromised CCC. Race and ethnicity were included as covariates based on previous studies reporting differences in COVID-19 outcomes among racial and ethnic groups.⁹ The CCC covariates were defined using the pediatric CCC ICD-10 classification system version 2.¹⁸

Pediatric Complications and Conditions Associated With COVID-19

Based on current evidence and expert opinion of study members, associated diagnoses and complications co-occurring with a COVID-19 diagnosis were defined a priori and identified through ICD-10 codes (Appendix Table 1). These included acute kidney injury, acute liver injury, aseptic meningitis, asthma exacerbation, bronchiolitis, cerebral infarction, croup, encephalitis, encephalopathy, infant fever, febrile seizure, gastroenteritis/dehydration, Kawasaki disease/MIS-C, myocarditis/pericarditis, pneumonia, lung effusion or empyema, respiratory failure, sepsis, nonfebrile seizure, pancreatitis, sickle cell complications, and thrombotic complications.

Outcomes

COVID-19 severity outcomes were assessed as follows: (1) mild = ED discharge; (2) moderate = inpatient admission; (3) severe = intensive care unit (ICU) admission without mechanical ventilation, shock, or death; and (4) very severe = ICU admission with mechanical ventilation, shock, or death.¹⁹ This ordinal ranking system did not violate the proportional odds assumption. Potential reasons for admission to the ICU without mechanical ventilation, shock, or death include, but are not limited to, need for noninvasive ventilation, vital sign instability, dysrhythmias, respiratory insufficiency, or complications arising from concurrent conditions (eg, thrombotic events, need for continuous albuterol therapy). We examined several secondary, hospital-based outcomes, including associated diagnoses and complications, all-cause 30-day healthcare reutilization (ED visit or rehospitalization), length of stay (LOS), and ICU LOS.

Statistical Analysis

Demographic characteristics were summarized using frequencies and percentages for categorical variables and geometric means with SD and medians with interquartile ranges (IQR) for continuous variables, as appropriate. Factors associated with hospitalization (encompassing severity levels 2-4) vs ED discharge (severity level 1) were assessed using logistic regression. Factors associated with increasing severity among hospitalized pediatric patients (severity levels 2, 3, and 4) were assessed using ordinal logistic regression. Covariates in these analyses included race and ethnicity, age, sex, payor, cardiovascular CCC, neurologic/neuromuscular CCC, obesity/type 2 DM, pulmonary CCC, asthma, and immunocompromised CCC. Adjusted odds ratios (aOR) and corresponding 95% CI for each risk factor were generated using generalized linear mixed effects models and random intercepts for each hospital. Given the potential for diagnostic misclassification of pediatric patients with COVID-19 based on primary vs secondary diagnoses, we performed sensitivity analyses defining the study population as those with a primary diagnosis of COVID-19 and those with a secondary diagnosis of COVID-19 plus a concurrent primary diagnosis of a condition associated with COVID-19 (Appendix Table 1).

All analyses were performed using SAS version 9.4 (SAS Institute, Inc), and P < .05 was considered statistically significant. The Institutional Review Board at Vanderbilt University Medical Center determined that this study of de-identified data did not meet the criteria for human subjects research.

Study Population

A total of 19,976 encounters were included in the study. Of those, 15,913 (79.7%) were discharged from the ED and 4063 (20.3%) were hospitalized (Table 1). The most common race/ethnicity was Hispanic (9741, 48.8%), followed by non-Hispanic White (4217, 21.1%). Reference race/ethnicity data for the overall 2019 PHIS population can be found in Appendix Table 2.

JHMVol16No9_Antoon03490915e_t1.JPG

The severity distribution among the hospitalized population was moderate (3222, 79.3%), severe (431, 11.3%), and very severe (380, 9.4%). The frequency of COVID-19 diagnoses increased late in the study period (Figure). Among those hospitalized, the median LOS for the index admission was 2 days (IQR, 1-4), while among those admitted to the ICU, the median LOS was 3 days (IQR, 2-5).

JHMVol16No9_Antoon03490915e_f1.JPG

Overall, 10.1% (n = 2020) of the study population had an all-cause repeat encounter (ie, subsequent ED encounter or hospitalization) within 30 days following the index discharge. Repeat encounters were more frequent among patients hospitalized than among those discharged from the ED (Appendix Table 3).

Prevalence of Conditions and Complications Associated With COVID-19

Overall, 3257 (16.3%) patients had one or more co-occurring diagnoses categorized as a COVID-19–associated condition or complication. The most frequent diagnoses included lower respiratory tract disease (pneumonia, lung effusion, or empyema; n = 1415, 7.1%), gastroenteritis/dehydration (n = 1068, 5.3%), respiratory failure (n = 731, 3.7%), febrile infant (n = 413, 2.1%), and nonfebrile seizure (n = 425, 2.1%). Aside from nonfebrile seizure, neurological complications were less frequent and included febrile seizure (n = 155, 0.8%), encephalopathy (n = 63, 0.3%), aseptic meningitis (n = 16, 0.1%), encephalitis (n = 11, 0.1%), and cerebral infarction (n = 6, <0.1%). Kawasaki disease and MIS-C comprised 1.7% (n = 346) of diagnoses. Thrombotic complications occurred in 0.1% (n = 13) of patients. Overall, these conditions and complications associated with COVID-19 were more frequent in hospitalized patients than in those discharged from the ED (P < .001) (Table 2).

JHMVol16No9_Antoon03490915e_t2.JPG

Factors Associated With COVID-19 Disease Severity

Compared to pediatric patients with COVID-19 discharged from the ED, factors associated with increased odds of hospitalization included private payor insurance; obesity/type 2 DM; asthma; and cardiovascular, immunocompromised, neurologic/neuromuscular, and pulmonary CCCs (Table 3). Factors associated with decreased risk of hospitalization included Black race or Hispanic ethnicity compared with White race; female sex; and age 5 to 11 years and age 12 to 17 years (vs age 0-4 years). Among children and adolescents hospitalized with COVID-19, factors associated with greater disease severity included Black or other non-White race; age 5 to 11 years; age 12 to 17 years; obesity/type 2 DM; immunocompromised conditions; and cardiovascular, neurologic/neuromuscular, and pulmonary CCCs (Table 3).

JHMVol16No9_Antoon03490915e_t3.JPG

Sensitivity Analysis

We performed a sensitivity analysis that expanded the study population to include those with a secondary diagnosis of COVID-19 plus a diagnosis of a COVID-19–associated condition or complication. Analyses using the expanded population (N = 21,247) were similar to the primary analyses (Appendix Table 4 and Appendix Table 5).

In this large multicenter study evaluating COVID-19 disease severity in more than 19,000 patients presenting for emergency care at US pediatric hospitals, approximately 20% were hospitalized, and among those hospitalized almost a quarter required ICU care. Clinical risk factors associated with increased risk of hospitalization include private payor status and selected comorbidities (obesity/type 2 DM; asthma; and cardiovascular, pulmonary, immunocompromised, neurologic/neuromuscular CCCs), while those associated with decreased risk of hospitalization include older age, female sex, and Black race or Hispanic ethnicity. Factors associated with severe disease among hospitalized pediatric patients include Black or other non-White race, school age (≥5 years), and certain chronic conditions (cardiovascular disease, obesity/type 2 DM, neurologic or neuromuscular disease). Sixteen percent of patients had a concurrent diagnosis for a condition or complication associated with COVID-19.

While the study population (ie, children and adolescents presenting to the ED) represents a small fraction of children and adolescents in the community with SARS-CoV-2 infection, the results provide important insight into factors of severe COVID-19 in the pediatric population. A report from France suggested ventilatory or hemodynamic support or death were independently associated with older age (≥10 years), elevated C-reactive protein, and hypoxemia.¹² An Italian study found that younger age (0-4 years) was associated with less severe disease, while preexisting conditions were more likely in patients with severe disease.¹¹ A single-center case series of 50 patients (aged ≤21 years) hospitalized at a children’s hospital in New York City found respiratory failure (n = 9) was more common in children older than 1 year, patients with elevated inflammatory markers, and patients with obesity.²⁰

Our study confirms several factors for severe COVID-19 found in these studies, including older age,^11,12,20 obesity,²⁰ and preexisting conditions.¹¹ Our findings also expand on these reports, including identification of factors associated with hospitalization. Given the rate of 30-day re-encounters among pediatric patients with COVID-19 (10.1%), identifying risk factors for hospitalization may aid ED providers in determining optimal disposition (eg, home, hospital admission, ICU). We also identified specific comorbidities associated with more severe disease in those hospitalized with COVID-19, such as cardiovascular disease, obesity/type 2 DM, and pulmonary, neurologic, or neuromuscular conditions. We also found that asthma increased the risk for hospitalization but not more severe disease among those hospitalized. This latter finding also aligns with recent single-center studies,^21,22 whereas a Turkish study of pediatric patients aged 0 to 18 years found no association between asthma and COVID-19 hospitalizations.²³We also examined payor type and racial/ethnic factors in our analysis. In 2019, patients who identified as Black or Hispanic comprised 52.3% of all encounters and 40.7% of hospitalizations recorded in the PHIS database. During the same year, encounters for influenza among Black or Hispanic pediatric patients comprised 58.7% of all influenza diagnoses and 47.0% of pediatric influenza hospitalizations (Appendix Table 2). In this study, patients who identified as Black or Hispanic race represented a disproportionately large share of patients presenting to children’s hospitals (68.5%) and of those hospitalized (60.8%). Hispanic ethnicity, in particular, represented a disproportionate share of patients seeking care for COVID-19 compared to the overall PHIS population (47.7% and 27.1%, respectively). After accounting for other factors, we found Black and other non-White race—but not of Hispanic ethnicity—were independently associated with more disease severity among those hospitalized. This contrasts with findings from a recent adult study by Yehia et al,²⁴ who found (after adjusting for other clinical factors) no significant difference in mortality between Black patients and White patients among adults hospitalized due to COVID-19. It also contrasts with a recent large population-based UK study wherein pediatric patients identifying as Asian, but not Black or mixed race or ethnicity, had an increased risk of hospital admission and admission to the ICU compared to children identifying as White. Children identifying as Black or mixed race had longer hospital admissions.²⁵ However, as the authors of the study note, residual confounders and ascertainment bias due to differences in COVID testing may have influenced these findings.

Our findings of differences in hospitalization and disease severity among those hospitalized by race and ethnicity should be interpreted carefully. These may reflect a constellation of factors that are difficult to measure, including differences in healthcare access, inequalities in care (including hospital admission inequalities), and implicit bias—all of which may reflect structural racism. For example, it is possible that children who identify as Black or Hispanic have different access to care compared to children who identify as White, and this may affect disease severity on presentation.² Alternatively, it is possible that White pediatric patients are more likely to be hospitalized as compared to non-White pediatric patients with similar illness severity. Our finding that pediatric patients who identify as Hispanic or Black had a lower risk of hospitalization should be also interpreted carefully, as this may reflect higher utilization of the ED for SARS-CoV-2 testing, increased use of nonemergency services among those without access to primary care, or systematic differences in provider decision-making among this segment of the population.² Further study is needed to determine specific drivers for racial and ethnic differences in healthcare utilization in children and adolescents with COVID-19.²⁶

Complications and co-occurring diagnoses in adults with COVID-19 are well documented.^27-30 However, there is little information to date on the co-occurring diagnoses and complications associated with COVID-19 in children and adolescents. We found that complications and co-occurring conditions occurred in 16.3% of the study population, with the most frequent conditions including known complications of viral infections such as pneumonia, respiratory failure, and seizures. Acute kidney and liver injury, as well as thrombotic complications, occurred less commonly than in adults.^26-29 Interestingly, neurologic complications were also uncommon compared to adult reports^8,31 and less frequent than in other viral illnesses in children and adolescents. For example, neurologic complications occur in approximately 7.5% of children and adolescents hospitalized with influenza.³²

Limitations of the present study include the retrospective design, as well as incomplete patient-level clinical data in the PHIS database. The PHIS database only includes children’s hospitals, which may limit the generalizability of findings to community hospitals. We also excluded newborns, and our findings may not be generalizable to this population. We only included children and adolescents with a primary diagnosis of COVID-19, which has the potential for misclassification in cases where COVID-19 was a secondary diagnosis. However, results of our sensitivity analysis, which incorporated secondary diagnoses of COVID-19, were consistent with findings from our main analyses. Our study was designed to examine associations between certain prespecified factors and COVID-19 severity among pediatric patients who visited the ED or were admitted to the hospital during the COVID-19 pandemic. Thus, our findings must be interpreted in light of these considerations and may not be generalizable outside the ED or hospital setting. For example, it could be that some segments of the population utilized ED resources for testing, whereas others avoided the ED and other healthcare settings for fear of exposure to SARS-CoV-2. We also relied on diagnosis codes to identify concurrent diagnoses, as well as mechanical ventilation in our very severe outcome cohort, which resulted in this classification for some of these diagnoses. Despite these limitations, our findings represent an important step in understanding the risk factors associated with severe clinical COVID-19 disease in pediatric patients.

Our findings may inform future research and clinical interventions. Future studies on antiviral therapies and immune modulators targeting SARS-CoV-2 infection in children and adolescents should focus on high-risk populations, such as those identified in the study, as these patients are most likely to benefit from therapeutic interventions. Similarly, vaccine-development efforts may benefit from additional evaluation in high-risk populations, some of which may have altered immune responses. Furthermore, with increasing vaccination among adults and changes in recommendations, societal mitigation efforts (eg, masking, physical distancing) will diminish. Continued vigilance and COVID-19–mitigation efforts among high-risk children, for whom vaccines are not yet available, are critical during this transition.

Among children with COVID-19 who received care at children’s hospitals and EDs, 20% were hospitalized, and, of those, 21% were admitted to the ICU. Older children and adolescent patients had a lower risk of hospitalization; however, when hospitalized, they had greater illness severity. Those with selected comorbidities (eg, cardiovascular, obesity/type 2 DM, pulmonary and neurologic or neuromuscular disease) had both increased odds of hospitalization and in-hospital illness severity. While there were observed differences in COVID-19 severity by race and ethnicity, additional research is needed to clarify the drivers of such disparities. These factors should be considered when prioritizing mitigation strategies to prevent infection (eg, remote learning, avoidance of group activities, prioritization of COVID-19 vaccine when approved for children aged <12 years).

The COVID-19 pandemic has led to more than 40 million infections and more than 650,000 deaths in the United States alone.¹ Morbidity and mortality have disproportionately affected older adults.^2-4 However, acute infection and delayed effects, such as multisystem inflammatory syndrome in children (MIS-C), occur and can lead to severe complications, hospitalization, and death in pediatric patients.^5,6 Due to higher clinical disease prevalence and morbidity in the adult population, we have learned much about the clinical factors associated with severe adult COVID-19 disease.^5,7-9 Such clinical factors include older age, concurrent comorbidities, smoke exposure, and Black race or Hispanic ethnicity, among others.^5,7-10 However, there is a paucity of data on severe COVID-19 disease in pediatric patients.^5,11,12 In addition, most immunization strategies and pharmacologic treatments for COVID-19 have not been evaluated or approved for use in children.¹³ To guide targeted prevention and treatment strategies, there is a critical need to identify children and adolescents—who are among the most vulnerable patient populations—at high risk for severe disease.

Identifying the clinical factors associated with severe COVID-19 disease will help with prioritizing and allocating vaccines when they are approved for use in patients younger than 12 years. It also can provide insight for clinicians and families faced with decisions wherein individual risk assessment is crucial (eg, in-person schooling, other group activities). The objective of this study was to determine the clinical factors associated with severe COVID-19 among children and adolescents in the United States.

Study Design

We conducted a multicenter retrospective cohort study of patients presenting for care at pediatric hospitals that report data to the Pediatric Health Information System (PHIS) database. The PHIS administrative database includes billing and utilization data from 45 US tertiary care hospitals affiliated with the Children’s Hospital Association (Lenexa, Kansas). Data quality and reliability are ensured through a joint validation effort between the Children’s Hospital Association and participating hospitals. Hospitals submit discharge data, including demographics, diagnoses, and procedures using International Classification of Diseases, 10th Revision (ICD-10) codes, along with daily detailed information on pharmacy, location of care, and other services.

Study Population

Patients 30 days to 18 years of age discharged from the emergency department (ED) or inpatient setting with a primary diagnosis of COVID-19 (ICD-10 codes U.071 and U.072) between April 1, 2020, and September 30, 2020, were eligible for inclusion.¹⁴ In a prior study, the positive predictive value of an ICD-10–coded diagnosis of COVID-19 among hospitalized pediatric patients was 95.5%, compared with reverse transcription polymerase reaction results or presence of MIS-C.¹⁵ The diagnostic code for COVID-19 (ICD-10-CM) also had a high sensitivity (98.0%) in the hospitalized population.¹⁶ Acknowledging the increasing practice of screening patients upon admission, and in an attempt to minimize potential misclassification, we did not include encounters with secondary diagnoses of COVID-19 in our primary analyses. Pediatric patients with surgical diagnoses and neonates who never left the hospital were also excluded.

Factors Associated With Severe COVID-19 Disease

Exposures of interest were determined a priori based on current evidence in the literature and included patient age (0-4 years, 5-11 years, and 12-18 years), sex, race and ethnicity (non-Hispanic White, non-Hispanic Black, Hispanic, Asian, other non-White race [defined as Pacific Islander, Native American, or other]), payor type, cardiovascular complex chronic conditions (CCC), neuromuscular CCC, obesity/type 2 diabetes mellitus (DM), pulmonary CCC, asthma (defined using ICD-10 codes¹⁷), and immunocompromised CCC. Race and ethnicity were included as covariates based on previous studies reporting differences in COVID-19 outcomes among racial and ethnic groups.⁹ The CCC covariates were defined using the pediatric CCC ICD-10 classification system version 2.¹⁸

Pediatric Complications and Conditions Associated With COVID-19

Based on current evidence and expert opinion of study members, associated diagnoses and complications co-occurring with a COVID-19 diagnosis were defined a priori and identified through ICD-10 codes (Appendix Table 1). These included acute kidney injury, acute liver injury, aseptic meningitis, asthma exacerbation, bronchiolitis, cerebral infarction, croup, encephalitis, encephalopathy, infant fever, febrile seizure, gastroenteritis/dehydration, Kawasaki disease/MIS-C, myocarditis/pericarditis, pneumonia, lung effusion or empyema, respiratory failure, sepsis, nonfebrile seizure, pancreatitis, sickle cell complications, and thrombotic complications.

Outcomes

COVID-19 severity outcomes were assessed as follows: (1) mild = ED discharge; (2) moderate = inpatient admission; (3) severe = intensive care unit (ICU) admission without mechanical ventilation, shock, or death; and (4) very severe = ICU admission with mechanical ventilation, shock, or death.¹⁹ This ordinal ranking system did not violate the proportional odds assumption. Potential reasons for admission to the ICU without mechanical ventilation, shock, or death include, but are not limited to, need for noninvasive ventilation, vital sign instability, dysrhythmias, respiratory insufficiency, or complications arising from concurrent conditions (eg, thrombotic events, need for continuous albuterol therapy). We examined several secondary, hospital-based outcomes, including associated diagnoses and complications, all-cause 30-day healthcare reutilization (ED visit or rehospitalization), length of stay (LOS), and ICU LOS.

Statistical Analysis

Demographic characteristics were summarized using frequencies and percentages for categorical variables and geometric means with SD and medians with interquartile ranges (IQR) for continuous variables, as appropriate. Factors associated with hospitalization (encompassing severity levels 2-4) vs ED discharge (severity level 1) were assessed using logistic regression. Factors associated with increasing severity among hospitalized pediatric patients (severity levels 2, 3, and 4) were assessed using ordinal logistic regression. Covariates in these analyses included race and ethnicity, age, sex, payor, cardiovascular CCC, neurologic/neuromuscular CCC, obesity/type 2 DM, pulmonary CCC, asthma, and immunocompromised CCC. Adjusted odds ratios (aOR) and corresponding 95% CI for each risk factor were generated using generalized linear mixed effects models and random intercepts for each hospital. Given the potential for diagnostic misclassification of pediatric patients with COVID-19 based on primary vs secondary diagnoses, we performed sensitivity analyses defining the study population as those with a primary diagnosis of COVID-19 and those with a secondary diagnosis of COVID-19 plus a concurrent primary diagnosis of a condition associated with COVID-19 (Appendix Table 1).

All analyses were performed using SAS version 9.4 (SAS Institute, Inc), and P < .05 was considered statistically significant. The Institutional Review Board at Vanderbilt University Medical Center determined that this study of de-identified data did not meet the criteria for human subjects research.

Study Population

A total of 19,976 encounters were included in the study. Of those, 15,913 (79.7%) were discharged from the ED and 4063 (20.3%) were hospitalized (Table 1). The most common race/ethnicity was Hispanic (9741, 48.8%), followed by non-Hispanic White (4217, 21.1%). Reference race/ethnicity data for the overall 2019 PHIS population can be found in Appendix Table 2.

JHMVol16No9_Antoon03490915e_t1.JPG

The severity distribution among the hospitalized population was moderate (3222, 79.3%), severe (431, 11.3%), and very severe (380, 9.4%). The frequency of COVID-19 diagnoses increased late in the study period (Figure). Among those hospitalized, the median LOS for the index admission was 2 days (IQR, 1-4), while among those admitted to the ICU, the median LOS was 3 days (IQR, 2-5).

JHMVol16No9_Antoon03490915e_f1.JPG

Overall, 10.1% (n = 2020) of the study population had an all-cause repeat encounter (ie, subsequent ED encounter or hospitalization) within 30 days following the index discharge. Repeat encounters were more frequent among patients hospitalized than among those discharged from the ED (Appendix Table 3).

Prevalence of Conditions and Complications Associated With COVID-19

Overall, 3257 (16.3%) patients had one or more co-occurring diagnoses categorized as a COVID-19–associated condition or complication. The most frequent diagnoses included lower respiratory tract disease (pneumonia, lung effusion, or empyema; n = 1415, 7.1%), gastroenteritis/dehydration (n = 1068, 5.3%), respiratory failure (n = 731, 3.7%), febrile infant (n = 413, 2.1%), and nonfebrile seizure (n = 425, 2.1%). Aside from nonfebrile seizure, neurological complications were less frequent and included febrile seizure (n = 155, 0.8%), encephalopathy (n = 63, 0.3%), aseptic meningitis (n = 16, 0.1%), encephalitis (n = 11, 0.1%), and cerebral infarction (n = 6, <0.1%). Kawasaki disease and MIS-C comprised 1.7% (n = 346) of diagnoses. Thrombotic complications occurred in 0.1% (n = 13) of patients. Overall, these conditions and complications associated with COVID-19 were more frequent in hospitalized patients than in those discharged from the ED (P < .001) (Table 2).

JHMVol16No9_Antoon03490915e_t2.JPG

Factors Associated With COVID-19 Disease Severity

Compared to pediatric patients with COVID-19 discharged from the ED, factors associated with increased odds of hospitalization included private payor insurance; obesity/type 2 DM; asthma; and cardiovascular, immunocompromised, neurologic/neuromuscular, and pulmonary CCCs (Table 3). Factors associated with decreased risk of hospitalization included Black race or Hispanic ethnicity compared with White race; female sex; and age 5 to 11 years and age 12 to 17 years (vs age 0-4 years). Among children and adolescents hospitalized with COVID-19, factors associated with greater disease severity included Black or other non-White race; age 5 to 11 years; age 12 to 17 years; obesity/type 2 DM; immunocompromised conditions; and cardiovascular, neurologic/neuromuscular, and pulmonary CCCs (Table 3).

JHMVol16No9_Antoon03490915e_t3.JPG

Sensitivity Analysis

We performed a sensitivity analysis that expanded the study population to include those with a secondary diagnosis of COVID-19 plus a diagnosis of a COVID-19–associated condition or complication. Analyses using the expanded population (N = 21,247) were similar to the primary analyses (Appendix Table 4 and Appendix Table 5).

In this large multicenter study evaluating COVID-19 disease severity in more than 19,000 patients presenting for emergency care at US pediatric hospitals, approximately 20% were hospitalized, and among those hospitalized almost a quarter required ICU care. Clinical risk factors associated with increased risk of hospitalization include private payor status and selected comorbidities (obesity/type 2 DM; asthma; and cardiovascular, pulmonary, immunocompromised, neurologic/neuromuscular CCCs), while those associated with decreased risk of hospitalization include older age, female sex, and Black race or Hispanic ethnicity. Factors associated with severe disease among hospitalized pediatric patients include Black or other non-White race, school age (≥5 years), and certain chronic conditions (cardiovascular disease, obesity/type 2 DM, neurologic or neuromuscular disease). Sixteen percent of patients had a concurrent diagnosis for a condition or complication associated with COVID-19.

While the study population (ie, children and adolescents presenting to the ED) represents a small fraction of children and adolescents in the community with SARS-CoV-2 infection, the results provide important insight into factors of severe COVID-19 in the pediatric population. A report from France suggested ventilatory or hemodynamic support or death were independently associated with older age (≥10 years), elevated C-reactive protein, and hypoxemia.¹² An Italian study found that younger age (0-4 years) was associated with less severe disease, while preexisting conditions were more likely in patients with severe disease.¹¹ A single-center case series of 50 patients (aged ≤21 years) hospitalized at a children’s hospital in New York City found respiratory failure (n = 9) was more common in children older than 1 year, patients with elevated inflammatory markers, and patients with obesity.²⁰

Our study confirms several factors for severe COVID-19 found in these studies, including older age,^11,12,20 obesity,²⁰ and preexisting conditions.¹¹ Our findings also expand on these reports, including identification of factors associated with hospitalization. Given the rate of 30-day re-encounters among pediatric patients with COVID-19 (10.1%), identifying risk factors for hospitalization may aid ED providers in determining optimal disposition (eg, home, hospital admission, ICU). We also identified specific comorbidities associated with more severe disease in those hospitalized with COVID-19, such as cardiovascular disease, obesity/type 2 DM, and pulmonary, neurologic, or neuromuscular conditions. We also found that asthma increased the risk for hospitalization but not more severe disease among those hospitalized. This latter finding also aligns with recent single-center studies,^21,22 whereas a Turkish study of pediatric patients aged 0 to 18 years found no association between asthma and COVID-19 hospitalizations.²³We also examined payor type and racial/ethnic factors in our analysis. In 2019, patients who identified as Black or Hispanic comprised 52.3% of all encounters and 40.7% of hospitalizations recorded in the PHIS database. During the same year, encounters for influenza among Black or Hispanic pediatric patients comprised 58.7% of all influenza diagnoses and 47.0% of pediatric influenza hospitalizations (Appendix Table 2). In this study, patients who identified as Black or Hispanic race represented a disproportionately large share of patients presenting to children’s hospitals (68.5%) and of those hospitalized (60.8%). Hispanic ethnicity, in particular, represented a disproportionate share of patients seeking care for COVID-19 compared to the overall PHIS population (47.7% and 27.1%, respectively). After accounting for other factors, we found Black and other non-White race—but not of Hispanic ethnicity—were independently associated with more disease severity among those hospitalized. This contrasts with findings from a recent adult study by Yehia et al,²⁴ who found (after adjusting for other clinical factors) no significant difference in mortality between Black patients and White patients among adults hospitalized due to COVID-19. It also contrasts with a recent large population-based UK study wherein pediatric patients identifying as Asian, but not Black or mixed race or ethnicity, had an increased risk of hospital admission and admission to the ICU compared to children identifying as White. Children identifying as Black or mixed race had longer hospital admissions.²⁵ However, as the authors of the study note, residual confounders and ascertainment bias due to differences in COVID testing may have influenced these findings.

Our findings of differences in hospitalization and disease severity among those hospitalized by race and ethnicity should be interpreted carefully. These may reflect a constellation of factors that are difficult to measure, including differences in healthcare access, inequalities in care (including hospital admission inequalities), and implicit bias—all of which may reflect structural racism. For example, it is possible that children who identify as Black or Hispanic have different access to care compared to children who identify as White, and this may affect disease severity on presentation.² Alternatively, it is possible that White pediatric patients are more likely to be hospitalized as compared to non-White pediatric patients with similar illness severity. Our finding that pediatric patients who identify as Hispanic or Black had a lower risk of hospitalization should be also interpreted carefully, as this may reflect higher utilization of the ED for SARS-CoV-2 testing, increased use of nonemergency services among those without access to primary care, or systematic differences in provider decision-making among this segment of the population.² Further study is needed to determine specific drivers for racial and ethnic differences in healthcare utilization in children and adolescents with COVID-19.²⁶

Complications and co-occurring diagnoses in adults with COVID-19 are well documented.^27-30 However, there is little information to date on the co-occurring diagnoses and complications associated with COVID-19 in children and adolescents. We found that complications and co-occurring conditions occurred in 16.3% of the study population, with the most frequent conditions including known complications of viral infections such as pneumonia, respiratory failure, and seizures. Acute kidney and liver injury, as well as thrombotic complications, occurred less commonly than in adults.^26-29 Interestingly, neurologic complications were also uncommon compared to adult reports^8,31 and less frequent than in other viral illnesses in children and adolescents. For example, neurologic complications occur in approximately 7.5% of children and adolescents hospitalized with influenza.³²

Limitations of the present study include the retrospective design, as well as incomplete patient-level clinical data in the PHIS database. The PHIS database only includes children’s hospitals, which may limit the generalizability of findings to community hospitals. We also excluded newborns, and our findings may not be generalizable to this population. We only included children and adolescents with a primary diagnosis of COVID-19, which has the potential for misclassification in cases where COVID-19 was a secondary diagnosis. However, results of our sensitivity analysis, which incorporated secondary diagnoses of COVID-19, were consistent with findings from our main analyses. Our study was designed to examine associations between certain prespecified factors and COVID-19 severity among pediatric patients who visited the ED or were admitted to the hospital during the COVID-19 pandemic. Thus, our findings must be interpreted in light of these considerations and may not be generalizable outside the ED or hospital setting. For example, it could be that some segments of the population utilized ED resources for testing, whereas others avoided the ED and other healthcare settings for fear of exposure to SARS-CoV-2. We also relied on diagnosis codes to identify concurrent diagnoses, as well as mechanical ventilation in our very severe outcome cohort, which resulted in this classification for some of these diagnoses. Despite these limitations, our findings represent an important step in understanding the risk factors associated with severe clinical COVID-19 disease in pediatric patients.

Our findings may inform future research and clinical interventions. Future studies on antiviral therapies and immune modulators targeting SARS-CoV-2 infection in children and adolescents should focus on high-risk populations, such as those identified in the study, as these patients are most likely to benefit from therapeutic interventions. Similarly, vaccine-development efforts may benefit from additional evaluation in high-risk populations, some of which may have altered immune responses. Furthermore, with increasing vaccination among adults and changes in recommendations, societal mitigation efforts (eg, masking, physical distancing) will diminish. Continued vigilance and COVID-19–mitigation efforts among high-risk children, for whom vaccines are not yet available, are critical during this transition.

Among children with COVID-19 who received care at children’s hospitals and EDs, 20% were hospitalized, and, of those, 21% were admitted to the ICU. Older children and adolescent patients had a lower risk of hospitalization; however, when hospitalized, they had greater illness severity. Those with selected comorbidities (eg, cardiovascular, obesity/type 2 DM, pulmonary and neurologic or neuromuscular disease) had both increased odds of hospitalization and in-hospital illness severity. While there were observed differences in COVID-19 severity by race and ethnicity, additional research is needed to clarify the drivers of such disparities. These factors should be considered when prioritizing mitigation strategies to prevent infection (eg, remote learning, avoidance of group activities, prioritization of COVID-19 vaccine when approved for children aged <12 years).

Increasingly, metrics such as length of stay (LOS) and readmissions are being utilized in the United States to assess quality of healthcare because these factors may represent opportunities to reduce cost and improve healthcare delivery.^1-8 However, the relatively low rate of pediatric readmissions,⁹ coupled with limited data regarding recommended LOS or best practices to prevent readmissions in children, challenges the ability of hospitals to safely reduce LOS and readmission rates for children.^10–12

In adults, weekend admission is associated with prolonged LOS, increased readmission rates, and increased risk of mortality.^13-21 This association is referred to as the “weekend effect.” While the weekend effect has been examined in children, the results of these studies have been variable, with some studies supporting this association and others refuting it.^22-31 In contrast to patient demographic and clinical characteristics that are known to affect LOS and readmissions,³² the weekend effect represents a potentially modifiable aspect of a hospitalization that could be targeted to improve healthcare delivery.

With increasing national attention toward improving quality of care and reducing LOS and healthcare costs, more definitive evidence of the weekend effect is necessary to prioritize resource use at both the local and national levels. Therefore, we sought to determine the association of weekend admission and weekend discharge on LOS and 30-day readmissions, respectively, among a national cohort of children. We hypothesized that children admitted on the weekend would have longer LOS, whereas those discharged on the weekend would have higher readmission rates.

Study Design and Data Source

We conducted a multicenter, retrospective, cross-sectional study. Data were obtained from the Pediatric Health Information System (PHIS), an administrative and billing database of 46 free-standing tertiary care pediatric hospitals affiliated with the Children’s Hospital Association (Lenexa, Kansas). Patient data are de-identified within PHIS; however, encrypted patient identifiers allow individual patients to be followed across visits. This study was not considered human subjects research by the policies of the Cincinnati Children’s Hospital Institutional Review Board.

Participants

We included hospitalizations to a PHIS-participating hospital for children aged 0-17 years between October 1, 2014 and September 30, 2015. We excluded children who were transferred from/to another institution, left against medical advice, or died in the hospital because these indications may result in incomplete LOS information and would not consistently contribute to readmission rates. We also excluded birth hospitalizations and children admitted for planned procedures. Birth hospitalizations were defined as hospitalizations that began on the day of birth. Planned procedures were identified using methodology previously described by Berry et al.⁹ With the use of this methodology, a planned procedure was identified if the coded primary procedure was one in which >80% of cases (eg, spinal fusion) are scheduled in advance. Finally, we excluded data from three hospitals due to incomplete data (eg, no admission or discharge time recorded).

Main Exposures

No standard definition of weekend admission or discharge was identified in the literature.³³ Thus, we defined a weekend admission as an admission between 3:00 pm Friday and 2:59 pm Sunday and a weekend discharge as a discharge between 3:00 pm Friday and 11:59 pm Sunday. These times were chosen by group consensus to account for the potential differences in hospital care during weekend hours (eg, decreased levels of provider staffing, access to ancillary services). To allow for a full 30-day readmission window, we defined an index admission as a hospitalization with no admission within the preceding 30 days. Individual children may contribute more than one index hospitalization to the dataset.

Main Outcomes

Our outcomes included LOS for weekend admission and 30-day readmissions for weekend discharge. LOS, measured in hours, was defined using the reported admission and discharge times. Readmissions were defined as a return to the same hospital within the subsequent 30 days following discharge.

Patient Demographics and Other Study Variables

Patient demographics included age, gender, race/ethnicity, payer, and median household income quartile based on the patient’s home ZIP code. Other study variables included presence of a complex chronic condition (CCC),³⁴ technology dependence,³⁴ number of chronic conditions of any complexity, admission through the emergency department, intensive care unit (ICU) admission, and case mix index. ICU admission and case mix index were chosen as markers for severity of illness. ICU admission was defined as any child who incurred ICU charges at any time following admission. Case mix index in PHIS is a relative weight assigned to each discharge based on the All-Patient Refined Diagnostic Group (APR-DRG; 3M) assignment and APR-DRG severity of illness, which ranges from 1 (minor) to 4 (extreme). The weights are derived by the Children’s Hospital Association from the HCUP KID 2012 database as the ratio of the average cost for discharges within a specific APR-DRG severity of illness combination to the average cost for all discharges in the database.

Statistical Analysis

Continuous variables were summarized with medians and interquartile ranges, while categorical variables were summarized with frequencies and percentages. Differences in admission and discharge characteristics between weekend and weekday were assessed using Wilcoxon rank sum tests for continuous variables and chi-square tests of association for categorical variables. We used generalized linear mixed modeling (GLMM) techniques to assess the impact of weekend admission on LOS and weekend discharge on readmission, adjusting for important patient demographic and clinical characteristics. Furthermore, we used GLMM point estimates to describe the variation across hospitals of the impact of weekday versus weekend care on LOS and readmissions. We assumed an underlying log-normal distribution for LOS and an underlying binomial distribution for 30-day readmission. All GLMMs included a random intercept for each hospital to account for patient clustering within a hospital. All statistical analyses were performed using SAS v.9.4 (SAS Institute, Cary, North Carolina), and P values <.05 were considered statistically significant.

We identified 390,745 hospitalizations that met inclusion criteria (Supplementary Figure 1). The median LOS among our cohort was 41 hours (interquartile range [IQR] 24-71) and the median 30-day readmission rate was 8.2% (IQR 7.2-9.4).

Admission Demographics for Weekends and Weekdays

Among the included hospitalizations, 92,266 (23.6%) admissions occurred on a weekend (Supplementary Table 1). Overall, a higher percentage of children <5 years of age were admitted on a weekend compared with those admitted on a weekday (53.3% vs 49.1%, P < .001). We observed a small but statistically significant difference in the proportion of weekend versus weekday admissions according to gender, race/ethnicity, payer, and median household income quartile. Children with medical complexity and those with technology dependence were admitted less frequently on a weekend. A higher proportion of children were admitted through the emergency department on a weekend and a higher frequency of ICU utilization was observed for children admitted on a weekend compared with those admitted on a weekday.

Association Between Study Variables and Length of Stay

In comparing adjusted LOS for weekend versus weekday admissions across 43 hospitals, not only did LOS vary across hospitals (P < .001), but the association between LOS and weekend versus weekday care also varied across hospitals (P < .001) (Figure 1). Weekend admission was associated with a significantly longer LOS at eight (18.6%) hospitals and a significantly shorter LOS at four (9.3%) hospitals with nonstatistically significant differences at the remaining hospitals.

jhm014020075_f1.jpg

In adjusted analyses, we observed that infants ≤30 days of age, on average, had an adjusted LOS that was 24% longer than that of 15- to 17-year-olds, while children aged 1-14 years had an adjusted LOS that was 6%-18% shorter (Table 1). ICU utilization, admission through the emergency department, and number of chronic conditions had the greatest association with LOS. As the number of chronic conditions increased, the LOS increased. No association was found between weekend versus weekday admission and LOS (adjusted LOS [95% CI]: weekend 63.70 [61.01-66.52] hours versus weekday 63.40 [60.73-66.19] hours, P = .112).

jhm014020075_t1.jpg

Discharge Demographics for Weekends and Weekdays

Of the included hospitalizations, 127,421 (32.6%) discharges occurred on a weekend (Supplementary Table 2). Overall, a greater percentage of weekend discharges comprised children <5 years of age compared with the percentage of weekday discharges for children <5 years of age (51.5% vs 49.5%, P < .001). No statistically significant differences were found in gender, payer, or median household income quartile between those children discharged on a weekend versus those discharged on a weekday. We found small, statistically significant differences in the proportion of weekend versus weekday discharges according to race/ethnicity, with fewer non-Hispanic white children being discharged on the weekend versus weekday. Children with medical complexity, technology dependence, and patients with ICU utilization were less frequently discharged on a weekend compared with those discharged on a weekday.

Association Between Study Variables and Readmissions

In comparing the adjusted odds of readmissions for weekend versus weekday discharges across 43 PHIS hospitals, we observed significant variation (P < .001) in readmission rates from hospital to hospital (Figure 2). However, the direction of impact of weekend care on readmissions was similar (P = .314) across hospitals (ie, for 37 of 43 hospitals, the readmission rate was greater for weekend discharges compared with that for weekday discharges). For 17 (39.5%) of 43 hospitals, weekend discharge was associated with a significantly higher readmission rate, while the differences between weekday and weekend discharge were not statistically significant for the remaining hospitals.

jhm014020075_f2.jpg

In adjusted analyses, we observed that infants <1 year were more likely to be readmitted compared with 15- to 17-year-olds, while children 5-14 years of age were less likely to be readmitted (Table 2). Medical complexity and the number of chronic conditions had the greatest association with readmissions, with increased likelihood of readmission observed as the number of chronic conditions increased. Weekend discharge was associated with increased probability of readmission compared with weekday discharge (adjusted probability of readmission [95% CI]: weekend 0.13 [0.12-0.13] vs weekday 0.11 [0.11-0.12], P < .001).

jhm014020075_t2.jpg

In this multicenter retrospective study, we observed substantial variation across hospitals in the relationship between weekend admission and LOS and weekend discharge and readmission rates. Overall, we did not observe an association between weekend admission and LOS. However, significant associations were noted between weekend admission and LOS at some hospitals, although the magnitude and direction of the effect varied. We observed a modestly increased risk of readmission among those discharged on the weekend. At the hospital level, the association between weekend discharge and increased readmissions was statistically significant at 39.5% of hospitals. Not surprisingly, certain patient demographic and clinical characteristics, including medical complexity and number of chronic conditions, were also associated with LOS and readmission risk. Taken together, our findings demonstrate that among a large sample of children, the degree to which a weekend admission or discharge impacts LOS or readmission risk varies considerably according to specific patient characteristics and individual hospital.

While the reasons for the weekend effect are unclear, data supporting this difference have been observed across many diverse patient groups and health systems both nationally and internationally.^13-27,31 Weekend care is thought to differ from weekday care because of differences in physician and nurse staffing, availability of ancillary services, access to diagnostic testing and therapeutic interventions, ability to arrange outpatient follow-up, and individual patient clinical factors, including acuity of illness. Few studies have assessed the effect of weekend discharges on patient or system outcomes. Among children within a single health system, readmission risk was associated with weekend admission but not with weekend discharge.²² This observation suggests that if differential care exists, then it occurs during initial clinical management rather than during discharge planning. Consequently, understanding the interaction of weekend admission and LOS is important. In addition, the relative paucity of pediatric data examining a weekend discharge effect limits the ability to generalize these findings across other hospitals or health systems.

In contrast to prior work, we observed a modest increased risk for readmission among those discharged on the weekend in a large sample of children. Auger and Davis reported a lack of association between weekend discharge and readmissions at one tertiary care children’s hospital, citing reduced discharge volumes on the weekend, especially among children with medical complexity, as a possible driver for their observation.²² The inclusion of a much larger population across 43 hospitals in our study may explain our different findings compared with previous research. In addition, the inclusion/exclusion criteria differed between the two studies; we excluded index admissions for planned procedures in this study (which are more likely to occur during the week), which may have contributed to the differing conclusions. Although Auger and Davis suggest that differences in initial clinical management may be responsible for the weekend effect,²² our observations suggest that discharge planning practices may also contribute to readmission risk. For example, a family’s inability to access compounded medications at a local pharmacy or to access primary care following discharge could reasonably contribute to treatment failure and increased readmission risk. Attention to improving and standardizing discharge practices may alleviate differences in readmission risk among children discharged on a weekend.

Individual patient characteristics greatly influence LOS and readmission risk. Congruent with prior studies, medical complexity and technology dependence were among the factors in our study that had the strongest association with LOS and readmission risk.³² As with prior studies²², we observed that children with medical complexity and technology dependence were less frequently admitted and discharged on a weekend than on a weekday, which suggests that physicians may avoid complicated discharges on the weekend. Children with medical complexity present a unique challenge to physicians when assessing discharge readiness, given that these children frequently require careful coordination of durable medical equipment, obtainment of special medication preparations, and possibly the resumption or establishment of home health services. Notably, we cannot discern from our data what proportion of discharges may be delayed over the weekend secondary to challenges involved in coordinating care for children with medical complexity. Future investigations aimed at assessing physician decision making and discharge readiness in relation to discharge timing among children with medical complexity may establish this relationship more clearly.

We observed substantial variation in LOS and readmission risk across 43 tertiary care children’s hospitals. Since the 1970s, numerous studies have reported worse outcomes among patients admitted on the weekend. While the majority of studies support the weekend effect, several recent studies suggest that patients admitted on the weekend are at no greater risk of adverse outcomes than those admitted during the week.^35-37 Our work builds on the existing literature, demonstrating a complex and variable relationship between weekend admission/discharge, LOS, and readmission risk across hospitals. Notably, while many hospitals in our study experienced a significant weekend effect in LOS or readmission risk, only four hospitals experienced a statistically significant weekend effect for both LOS and readmission risk (three hospitals experienced increased risk for both, while one hospital experienced increased readmission risk but decreased LOS). Future investigations of the weekend effect should focus on exploring the differences in admission/discharge practices and staffing patterns of hospitals that did or did not experience a weekend effect.

This study has several limitations. We may have underestimated the total number of readmissions because we are unable to capture readmissions to other institutions by using the PHIS database. Our definition of a weekend admission or discharge did not account for three-day weekends or other holidays where staffing issues would be expected to be similar to that on weekends; consequently, our approach would be expected to bias the results toward null. Thus, a possible (but unlikely) result is that our approach masked a weekend effect that might have been more prominent had holidays been included. Although prior studies suggest that low physician/nurse staffing volumes and high patient workload are associated with worse patient outcomes,^38,39 we are unable to discern the role of differential staffing patterns, patient workload, or service availability in our observations using the PHIS database. Moreover, the PHIS database does not allow for any assessment of the preventability of a readmission or the impact of patient/family preference on the decision to admit or discharge, factors that could reasonably contribute to some of the observed variation. Finally, the PHIS database contains administrative data only, thus limiting our ability to fully adjust for patient severity of illness and sociodemographic factors that may have affected clinical decision making, including discharge decision making.

In a study of 43 children’s hospitals, children discharged on the weekend had a slightly increased readmission risk compared with children discharged on a weekday. Wide variation in the weekend effect on LOS and readmission risk was evident across hospitals. Individual patient characteristics had a greater impact on LOS and readmission risk than the weekend effect. Future investigations aimed at understanding which factors contribute most strongly to a weekend effect within individual hospitals (eg, differences in institutional admission/discharge practices) may help alleviate the weekend effect and improve healthcare quality.

Acknowledgments

This manuscript resulted from “Paper in a Day,” a Pediatric Research in Inpatient Settings (PRIS) Network-sponsored workshop presented at the Pediatric Hospital Medicine 2017 annual meeting. Workshop participants learned how to ask and answer a health services research question and efficiently prepare a manuscript for publication. The following are the members of the PRIS Network who contributed to this work: Jessica L. Bettenhausen, MD; Rebecca M. Cantu, MD, MPH; Jillian M Cotter, MD; Megan Deisz, MD; Teresa Frazer, MD; Pratichi Goenka, MD; Ashley Jenkins, MD; Kathryn E. Kyler, MD; Janet T. Lau, MD; Brian E. Lee, MD; Christiane Lenzen, MD; Trisha Marshall, MD; John M. Morrison MD, PhD; Lauren Nassetta, MD; Raymond Parlar-Chun, MD; Sonya Tang Girdwood MD, PhD; Tony R Tarchichi, MD; Irina G. Trifonova, MD; Jacqueline M. Walker, MD, MHPE; and Susan C. Walley, MD. See appendix for contact information for members of the PRIS Network.

Funding

The authors have no financial relationships relevant to this article to disclose.

Disclosures

The authors have no conflicts of interest to disclose.

Increasingly, metrics such as length of stay (LOS) and readmissions are being utilized in the United States to assess quality of healthcare because these factors may represent opportunities to reduce cost and improve healthcare delivery.^1-8 However, the relatively low rate of pediatric readmissions,⁹ coupled with limited data regarding recommended LOS or best practices to prevent readmissions in children, challenges the ability of hospitals to safely reduce LOS and readmission rates for children.^10–12

In adults, weekend admission is associated with prolonged LOS, increased readmission rates, and increased risk of mortality.^13-21 This association is referred to as the “weekend effect.” While the weekend effect has been examined in children, the results of these studies have been variable, with some studies supporting this association and others refuting it.^22-31 In contrast to patient demographic and clinical characteristics that are known to affect LOS and readmissions,³² the weekend effect represents a potentially modifiable aspect of a hospitalization that could be targeted to improve healthcare delivery.

With increasing national attention toward improving quality of care and reducing LOS and healthcare costs, more definitive evidence of the weekend effect is necessary to prioritize resource use at both the local and national levels. Therefore, we sought to determine the association of weekend admission and weekend discharge on LOS and 30-day readmissions, respectively, among a national cohort of children. We hypothesized that children admitted on the weekend would have longer LOS, whereas those discharged on the weekend would have higher readmission rates.

Study Design and Data Source

We conducted a multicenter, retrospective, cross-sectional study. Data were obtained from the Pediatric Health Information System (PHIS), an administrative and billing database of 46 free-standing tertiary care pediatric hospitals affiliated with the Children’s Hospital Association (Lenexa, Kansas). Patient data are de-identified within PHIS; however, encrypted patient identifiers allow individual patients to be followed across visits. This study was not considered human subjects research by the policies of the Cincinnati Children’s Hospital Institutional Review Board.

Participants

We included hospitalizations to a PHIS-participating hospital for children aged 0-17 years between October 1, 2014 and September 30, 2015. We excluded children who were transferred from/to another institution, left against medical advice, or died in the hospital because these indications may result in incomplete LOS information and would not consistently contribute to readmission rates. We also excluded birth hospitalizations and children admitted for planned procedures. Birth hospitalizations were defined as hospitalizations that began on the day of birth. Planned procedures were identified using methodology previously described by Berry et al.⁹ With the use of this methodology, a planned procedure was identified if the coded primary procedure was one in which >80% of cases (eg, spinal fusion) are scheduled in advance. Finally, we excluded data from three hospitals due to incomplete data (eg, no admission or discharge time recorded).

Main Exposures

No standard definition of weekend admission or discharge was identified in the literature.³³ Thus, we defined a weekend admission as an admission between 3:00 pm Friday and 2:59 pm Sunday and a weekend discharge as a discharge between 3:00 pm Friday and 11:59 pm Sunday. These times were chosen by group consensus to account for the potential differences in hospital care during weekend hours (eg, decreased levels of provider staffing, access to ancillary services). To allow for a full 30-day readmission window, we defined an index admission as a hospitalization with no admission within the preceding 30 days. Individual children may contribute more than one index hospitalization to the dataset.

Main Outcomes

Our outcomes included LOS for weekend admission and 30-day readmissions for weekend discharge. LOS, measured in hours, was defined using the reported admission and discharge times. Readmissions were defined as a return to the same hospital within the subsequent 30 days following discharge.

Patient Demographics and Other Study Variables

Patient demographics included age, gender, race/ethnicity, payer, and median household income quartile based on the patient’s home ZIP code. Other study variables included presence of a complex chronic condition (CCC),³⁴ technology dependence,³⁴ number of chronic conditions of any complexity, admission through the emergency department, intensive care unit (ICU) admission, and case mix index. ICU admission and case mix index were chosen as markers for severity of illness. ICU admission was defined as any child who incurred ICU charges at any time following admission. Case mix index in PHIS is a relative weight assigned to each discharge based on the All-Patient Refined Diagnostic Group (APR-DRG; 3M) assignment and APR-DRG severity of illness, which ranges from 1 (minor) to 4 (extreme). The weights are derived by the Children’s Hospital Association from the HCUP KID 2012 database as the ratio of the average cost for discharges within a specific APR-DRG severity of illness combination to the average cost for all discharges in the database.

Statistical Analysis

Continuous variables were summarized with medians and interquartile ranges, while categorical variables were summarized with frequencies and percentages. Differences in admission and discharge characteristics between weekend and weekday were assessed using Wilcoxon rank sum tests for continuous variables and chi-square tests of association for categorical variables. We used generalized linear mixed modeling (GLMM) techniques to assess the impact of weekend admission on LOS and weekend discharge on readmission, adjusting for important patient demographic and clinical characteristics. Furthermore, we used GLMM point estimates to describe the variation across hospitals of the impact of weekday versus weekend care on LOS and readmissions. We assumed an underlying log-normal distribution for LOS and an underlying binomial distribution for 30-day readmission. All GLMMs included a random intercept for each hospital to account for patient clustering within a hospital. All statistical analyses were performed using SAS v.9.4 (SAS Institute, Cary, North Carolina), and P values <.05 were considered statistically significant.

We identified 390,745 hospitalizations that met inclusion criteria (Supplementary Figure 1). The median LOS among our cohort was 41 hours (interquartile range [IQR] 24-71) and the median 30-day readmission rate was 8.2% (IQR 7.2-9.4).

Admission Demographics for Weekends and Weekdays

Among the included hospitalizations, 92,266 (23.6%) admissions occurred on a weekend (Supplementary Table 1). Overall, a higher percentage of children <5 years of age were admitted on a weekend compared with those admitted on a weekday (53.3% vs 49.1%, P < .001). We observed a small but statistically significant difference in the proportion of weekend versus weekday admissions according to gender, race/ethnicity, payer, and median household income quartile. Children with medical complexity and those with technology dependence were admitted less frequently on a weekend. A higher proportion of children were admitted through the emergency department on a weekend and a higher frequency of ICU utilization was observed for children admitted on a weekend compared with those admitted on a weekday.

Association Between Study Variables and Length of Stay

In comparing adjusted LOS for weekend versus weekday admissions across 43 hospitals, not only did LOS vary across hospitals (P < .001), but the association between LOS and weekend versus weekday care also varied across hospitals (P < .001) (Figure 1). Weekend admission was associated with a significantly longer LOS at eight (18.6%) hospitals and a significantly shorter LOS at four (9.3%) hospitals with nonstatistically significant differences at the remaining hospitals.

jhm014020075_f1.jpg

In adjusted analyses, we observed that infants ≤30 days of age, on average, had an adjusted LOS that was 24% longer than that of 15- to 17-year-olds, while children aged 1-14 years had an adjusted LOS that was 6%-18% shorter (Table 1). ICU utilization, admission through the emergency department, and number of chronic conditions had the greatest association with LOS. As the number of chronic conditions increased, the LOS increased. No association was found between weekend versus weekday admission and LOS (adjusted LOS [95% CI]: weekend 63.70 [61.01-66.52] hours versus weekday 63.40 [60.73-66.19] hours, P = .112).

jhm014020075_t1.jpg

Discharge Demographics for Weekends and Weekdays

Of the included hospitalizations, 127,421 (32.6%) discharges occurred on a weekend (Supplementary Table 2). Overall, a greater percentage of weekend discharges comprised children <5 years of age compared with the percentage of weekday discharges for children <5 years of age (51.5% vs 49.5%, P < .001). No statistically significant differences were found in gender, payer, or median household income quartile between those children discharged on a weekend versus those discharged on a weekday. We found small, statistically significant differences in the proportion of weekend versus weekday discharges according to race/ethnicity, with fewer non-Hispanic white children being discharged on the weekend versus weekday. Children with medical complexity, technology dependence, and patients with ICU utilization were less frequently discharged on a weekend compared with those discharged on a weekday.

Association Between Study Variables and Readmissions

In comparing the adjusted odds of readmissions for weekend versus weekday discharges across 43 PHIS hospitals, we observed significant variation (P < .001) in readmission rates from hospital to hospital (Figure 2). However, the direction of impact of weekend care on readmissions was similar (P = .314) across hospitals (ie, for 37 of 43 hospitals, the readmission rate was greater for weekend discharges compared with that for weekday discharges). For 17 (39.5%) of 43 hospitals, weekend discharge was associated with a significantly higher readmission rate, while the differences between weekday and weekend discharge were not statistically significant for the remaining hospitals.

jhm014020075_f2.jpg

In adjusted analyses, we observed that infants <1 year were more likely to be readmitted compared with 15- to 17-year-olds, while children 5-14 years of age were less likely to be readmitted (Table 2). Medical complexity and the number of chronic conditions had the greatest association with readmissions, with increased likelihood of readmission observed as the number of chronic conditions increased. Weekend discharge was associated with increased probability of readmission compared with weekday discharge (adjusted probability of readmission [95% CI]: weekend 0.13 [0.12-0.13] vs weekday 0.11 [0.11-0.12], P < .001).

jhm014020075_t2.jpg

In this multicenter retrospective study, we observed substantial variation across hospitals in the relationship between weekend admission and LOS and weekend discharge and readmission rates. Overall, we did not observe an association between weekend admission and LOS. However, significant associations were noted between weekend admission and LOS at some hospitals, although the magnitude and direction of the effect varied. We observed a modestly increased risk of readmission among those discharged on the weekend. At the hospital level, the association between weekend discharge and increased readmissions was statistically significant at 39.5% of hospitals. Not surprisingly, certain patient demographic and clinical characteristics, including medical complexity and number of chronic conditions, were also associated with LOS and readmission risk. Taken together, our findings demonstrate that among a large sample of children, the degree to which a weekend admission or discharge impacts LOS or readmission risk varies considerably according to specific patient characteristics and individual hospital.

While the reasons for the weekend effect are unclear, data supporting this difference have been observed across many diverse patient groups and health systems both nationally and internationally.^13-27,31 Weekend care is thought to differ from weekday care because of differences in physician and nurse staffing, availability of ancillary services, access to diagnostic testing and therapeutic interventions, ability to arrange outpatient follow-up, and individual patient clinical factors, including acuity of illness. Few studies have assessed the effect of weekend discharges on patient or system outcomes. Among children within a single health system, readmission risk was associated with weekend admission but not with weekend discharge.²² This observation suggests that if differential care exists, then it occurs during initial clinical management rather than during discharge planning. Consequently, understanding the interaction of weekend admission and LOS is important. In addition, the relative paucity of pediatric data examining a weekend discharge effect limits the ability to generalize these findings across other hospitals or health systems.

In contrast to prior work, we observed a modest increased risk for readmission among those discharged on the weekend in a large sample of children. Auger and Davis reported a lack of association between weekend discharge and readmissions at one tertiary care children’s hospital, citing reduced discharge volumes on the weekend, especially among children with medical complexity, as a possible driver for their observation.²² The inclusion of a much larger population across 43 hospitals in our study may explain our different findings compared with previous research. In addition, the inclusion/exclusion criteria differed between the two studies; we excluded index admissions for planned procedures in this study (which are more likely to occur during the week), which may have contributed to the differing conclusions. Although Auger and Davis suggest that differences in initial clinical management may be responsible for the weekend effect,²² our observations suggest that discharge planning practices may also contribute to readmission risk. For example, a family’s inability to access compounded medications at a local pharmacy or to access primary care following discharge could reasonably contribute to treatment failure and increased readmission risk. Attention to improving and standardizing discharge practices may alleviate differences in readmission risk among children discharged on a weekend.

Individual patient characteristics greatly influence LOS and readmission risk. Congruent with prior studies, medical complexity and technology dependence were among the factors in our study that had the strongest association with LOS and readmission risk.³² As with prior studies²², we observed that children with medical complexity and technology dependence were less frequently admitted and discharged on a weekend than on a weekday, which suggests that physicians may avoid complicated discharges on the weekend. Children with medical complexity present a unique challenge to physicians when assessing discharge readiness, given that these children frequently require careful coordination of durable medical equipment, obtainment of special medication preparations, and possibly the resumption or establishment of home health services. Notably, we cannot discern from our data what proportion of discharges may be delayed over the weekend secondary to challenges involved in coordinating care for children with medical complexity. Future investigations aimed at assessing physician decision making and discharge readiness in relation to discharge timing among children with medical complexity may establish this relationship more clearly.

We observed substantial variation in LOS and readmission risk across 43 tertiary care children’s hospitals. Since the 1970s, numerous studies have reported worse outcomes among patients admitted on the weekend. While the majority of studies support the weekend effect, several recent studies suggest that patients admitted on the weekend are at no greater risk of adverse outcomes than those admitted during the week.^35-37 Our work builds on the existing literature, demonstrating a complex and variable relationship between weekend admission/discharge, LOS, and readmission risk across hospitals. Notably, while many hospitals in our study experienced a significant weekend effect in LOS or readmission risk, only four hospitals experienced a statistically significant weekend effect for both LOS and readmission risk (three hospitals experienced increased risk for both, while one hospital experienced increased readmission risk but decreased LOS). Future investigations of the weekend effect should focus on exploring the differences in admission/discharge practices and staffing patterns of hospitals that did or did not experience a weekend effect.

This study has several limitations. We may have underestimated the total number of readmissions because we are unable to capture readmissions to other institutions by using the PHIS database. Our definition of a weekend admission or discharge did not account for three-day weekends or other holidays where staffing issues would be expected to be similar to that on weekends; consequently, our approach would be expected to bias the results toward null. Thus, a possible (but unlikely) result is that our approach masked a weekend effect that might have been more prominent had holidays been included. Although prior studies suggest that low physician/nurse staffing volumes and high patient workload are associated with worse patient outcomes,^38,39 we are unable to discern the role of differential staffing patterns, patient workload, or service availability in our observations using the PHIS database. Moreover, the PHIS database does not allow for any assessment of the preventability of a readmission or the impact of patient/family preference on the decision to admit or discharge, factors that could reasonably contribute to some of the observed variation. Finally, the PHIS database contains administrative data only, thus limiting our ability to fully adjust for patient severity of illness and sociodemographic factors that may have affected clinical decision making, including discharge decision making.

In a study of 43 children’s hospitals, children discharged on the weekend had a slightly increased readmission risk compared with children discharged on a weekday. Wide variation in the weekend effect on LOS and readmission risk was evident across hospitals. Individual patient characteristics had a greater impact on LOS and readmission risk than the weekend effect. Future investigations aimed at understanding which factors contribute most strongly to a weekend effect within individual hospitals (eg, differences in institutional admission/discharge practices) may help alleviate the weekend effect and improve healthcare quality.

Acknowledgments

This manuscript resulted from “Paper in a Day,” a Pediatric Research in Inpatient Settings (PRIS) Network-sponsored workshop presented at the Pediatric Hospital Medicine 2017 annual meeting. Workshop participants learned how to ask and answer a health services research question and efficiently prepare a manuscript for publication. The following are the members of the PRIS Network who contributed to this work: Jessica L. Bettenhausen, MD; Rebecca M. Cantu, MD, MPH; Jillian M Cotter, MD; Megan Deisz, MD; Teresa Frazer, MD; Pratichi Goenka, MD; Ashley Jenkins, MD; Kathryn E. Kyler, MD; Janet T. Lau, MD; Brian E. Lee, MD; Christiane Lenzen, MD; Trisha Marshall, MD; John M. Morrison MD, PhD; Lauren Nassetta, MD; Raymond Parlar-Chun, MD; Sonya Tang Girdwood MD, PhD; Tony R Tarchichi, MD; Irina G. Trifonova, MD; Jacqueline M. Walker, MD, MHPE; and Susan C. Walley, MD. See appendix for contact information for members of the PRIS Network.

Funding

The authors have no financial relationships relevant to this article to disclose.

Disclosures

The authors have no conflicts of interest to disclose.

Increasingly, metrics such as length of stay (LOS) and readmissions are being utilized in the United States to assess quality of healthcare because these factors may represent opportunities to reduce cost and improve healthcare delivery.^1-8 However, the relatively low rate of pediatric readmissions,⁹ coupled with limited data regarding recommended LOS or best practices to prevent readmissions in children, challenges the ability of hospitals to safely reduce LOS and readmission rates for children.^10–12

In adults, weekend admission is associated with prolonged LOS, increased readmission rates, and increased risk of mortality.^13-21 This association is referred to as the “weekend effect.” While the weekend effect has been examined in children, the results of these studies have been variable, with some studies supporting this association and others refuting it.^22-31 In contrast to patient demographic and clinical characteristics that are known to affect LOS and readmissions,³² the weekend effect represents a potentially modifiable aspect of a hospitalization that could be targeted to improve healthcare delivery.

With increasing national attention toward improving quality of care and reducing LOS and healthcare costs, more definitive evidence of the weekend effect is necessary to prioritize resource use at both the local and national levels. Therefore, we sought to determine the association of weekend admission and weekend discharge on LOS and 30-day readmissions, respectively, among a national cohort of children. We hypothesized that children admitted on the weekend would have longer LOS, whereas those discharged on the weekend would have higher readmission rates.

Study Design and Data Source

We conducted a multicenter, retrospective, cross-sectional study. Data were obtained from the Pediatric Health Information System (PHIS), an administrative and billing database of 46 free-standing tertiary care pediatric hospitals affiliated with the Children’s Hospital Association (Lenexa, Kansas). Patient data are de-identified within PHIS; however, encrypted patient identifiers allow individual patients to be followed across visits. This study was not considered human subjects research by the policies of the Cincinnati Children’s Hospital Institutional Review Board.

Participants

We included hospitalizations to a PHIS-participating hospital for children aged 0-17 years between October 1, 2014 and September 30, 2015. We excluded children who were transferred from/to another institution, left against medical advice, or died in the hospital because these indications may result in incomplete LOS information and would not consistently contribute to readmission rates. We also excluded birth hospitalizations and children admitted for planned procedures. Birth hospitalizations were defined as hospitalizations that began on the day of birth. Planned procedures were identified using methodology previously described by Berry et al.⁹ With the use of this methodology, a planned procedure was identified if the coded primary procedure was one in which >80% of cases (eg, spinal fusion) are scheduled in advance. Finally, we excluded data from three hospitals due to incomplete data (eg, no admission or discharge time recorded).

Main Exposures

No standard definition of weekend admission or discharge was identified in the literature.³³ Thus, we defined a weekend admission as an admission between 3:00 pm Friday and 2:59 pm Sunday and a weekend discharge as a discharge between 3:00 pm Friday and 11:59 pm Sunday. These times were chosen by group consensus to account for the potential differences in hospital care during weekend hours (eg, decreased levels of provider staffing, access to ancillary services). To allow for a full 30-day readmission window, we defined an index admission as a hospitalization with no admission within the preceding 30 days. Individual children may contribute more than one index hospitalization to the dataset.

Main Outcomes

Our outcomes included LOS for weekend admission and 30-day readmissions for weekend discharge. LOS, measured in hours, was defined using the reported admission and discharge times. Readmissions were defined as a return to the same hospital within the subsequent 30 days following discharge.

Patient Demographics and Other Study Variables

Patient demographics included age, gender, race/ethnicity, payer, and median household income quartile based on the patient’s home ZIP code. Other study variables included presence of a complex chronic condition (CCC),³⁴ technology dependence,³⁴ number of chronic conditions of any complexity, admission through the emergency department, intensive care unit (ICU) admission, and case mix index. ICU admission and case mix index were chosen as markers for severity of illness. ICU admission was defined as any child who incurred ICU charges at any time following admission. Case mix index in PHIS is a relative weight assigned to each discharge based on the All-Patient Refined Diagnostic Group (APR-DRG; 3M) assignment and APR-DRG severity of illness, which ranges from 1 (minor) to 4 (extreme). The weights are derived by the Children’s Hospital Association from the HCUP KID 2012 database as the ratio of the average cost for discharges within a specific APR-DRG severity of illness combination to the average cost for all discharges in the database.

Statistical Analysis

Continuous variables were summarized with medians and interquartile ranges, while categorical variables were summarized with frequencies and percentages. Differences in admission and discharge characteristics between weekend and weekday were assessed using Wilcoxon rank sum tests for continuous variables and chi-square tests of association for categorical variables. We used generalized linear mixed modeling (GLMM) techniques to assess the impact of weekend admission on LOS and weekend discharge on readmission, adjusting for important patient demographic and clinical characteristics. Furthermore, we used GLMM point estimates to describe the variation across hospitals of the impact of weekday versus weekend care on LOS and readmissions. We assumed an underlying log-normal distribution for LOS and an underlying binomial distribution for 30-day readmission. All GLMMs included a random intercept for each hospital to account for patient clustering within a hospital. All statistical analyses were performed using SAS v.9.4 (SAS Institute, Cary, North Carolina), and P values <.05 were considered statistically significant.

We identified 390,745 hospitalizations that met inclusion criteria (Supplementary Figure 1). The median LOS among our cohort was 41 hours (interquartile range [IQR] 24-71) and the median 30-day readmission rate was 8.2% (IQR 7.2-9.4).

Admission Demographics for Weekends and Weekdays

Among the included hospitalizations, 92,266 (23.6%) admissions occurred on a weekend (Supplementary Table 1). Overall, a higher percentage of children <5 years of age were admitted on a weekend compared with those admitted on a weekday (53.3% vs 49.1%, P < .001). We observed a small but statistically significant difference in the proportion of weekend versus weekday admissions according to gender, race/ethnicity, payer, and median household income quartile. Children with medical complexity and those with technology dependence were admitted less frequently on a weekend. A higher proportion of children were admitted through the emergency department on a weekend and a higher frequency of ICU utilization was observed for children admitted on a weekend compared with those admitted on a weekday.

Association Between Study Variables and Length of Stay

In comparing adjusted LOS for weekend versus weekday admissions across 43 hospitals, not only did LOS vary across hospitals (P < .001), but the association between LOS and weekend versus weekday care also varied across hospitals (P < .001) (Figure 1). Weekend admission was associated with a significantly longer LOS at eight (18.6%) hospitals and a significantly shorter LOS at four (9.3%) hospitals with nonstatistically significant differences at the remaining hospitals.

jhm014020075_f1.jpg

In adjusted analyses, we observed that infants ≤30 days of age, on average, had an adjusted LOS that was 24% longer than that of 15- to 17-year-olds, while children aged 1-14 years had an adjusted LOS that was 6%-18% shorter (Table 1). ICU utilization, admission through the emergency department, and number of chronic conditions had the greatest association with LOS. As the number of chronic conditions increased, the LOS increased. No association was found between weekend versus weekday admission and LOS (adjusted LOS [95% CI]: weekend 63.70 [61.01-66.52] hours versus weekday 63.40 [60.73-66.19] hours, P = .112).

jhm014020075_t1.jpg

Discharge Demographics for Weekends and Weekdays

Of the included hospitalizations, 127,421 (32.6%) discharges occurred on a weekend (Supplementary Table 2). Overall, a greater percentage of weekend discharges comprised children <5 years of age compared with the percentage of weekday discharges for children <5 years of age (51.5% vs 49.5%, P < .001). No statistically significant differences were found in gender, payer, or median household income quartile between those children discharged on a weekend versus those discharged on a weekday. We found small, statistically significant differences in the proportion of weekend versus weekday discharges according to race/ethnicity, with fewer non-Hispanic white children being discharged on the weekend versus weekday. Children with medical complexity, technology dependence, and patients with ICU utilization were less frequently discharged on a weekend compared with those discharged on a weekday.

Association Between Study Variables and Readmissions

In comparing the adjusted odds of readmissions for weekend versus weekday discharges across 43 PHIS hospitals, we observed significant variation (P < .001) in readmission rates from hospital to hospital (Figure 2). However, the direction of impact of weekend care on readmissions was similar (P = .314) across hospitals (ie, for 37 of 43 hospitals, the readmission rate was greater for weekend discharges compared with that for weekday discharges). For 17 (39.5%) of 43 hospitals, weekend discharge was associated with a significantly higher readmission rate, while the differences between weekday and weekend discharge were not statistically significant for the remaining hospitals.

jhm014020075_f2.jpg

In adjusted analyses, we observed that infants <1 year were more likely to be readmitted compared with 15- to 17-year-olds, while children 5-14 years of age were less likely to be readmitted (Table 2). Medical complexity and the number of chronic conditions had the greatest association with readmissions, with increased likelihood of readmission observed as the number of chronic conditions increased. Weekend discharge was associated with increased probability of readmission compared with weekday discharge (adjusted probability of readmission [95% CI]: weekend 0.13 [0.12-0.13] vs weekday 0.11 [0.11-0.12], P < .001).

jhm014020075_t2.jpg

In this multicenter retrospective study, we observed substantial variation across hospitals in the relationship between weekend admission and LOS and weekend discharge and readmission rates. Overall, we did not observe an association between weekend admission and LOS. However, significant associations were noted between weekend admission and LOS at some hospitals, although the magnitude and direction of the effect varied. We observed a modestly increased risk of readmission among those discharged on the weekend. At the hospital level, the association between weekend discharge and increased readmissions was statistically significant at 39.5% of hospitals. Not surprisingly, certain patient demographic and clinical characteristics, including medical complexity and number of chronic conditions, were also associated with LOS and readmission risk. Taken together, our findings demonstrate that among a large sample of children, the degree to which a weekend admission or discharge impacts LOS or readmission risk varies considerably according to specific patient characteristics and individual hospital.

While the reasons for the weekend effect are unclear, data supporting this difference have been observed across many diverse patient groups and health systems both nationally and internationally.^13-27,31 Weekend care is thought to differ from weekday care because of differences in physician and nurse staffing, availability of ancillary services, access to diagnostic testing and therapeutic interventions, ability to arrange outpatient follow-up, and individual patient clinical factors, including acuity of illness. Few studies have assessed the effect of weekend discharges on patient or system outcomes. Among children within a single health system, readmission risk was associated with weekend admission but not with weekend discharge.²² This observation suggests that if differential care exists, then it occurs during initial clinical management rather than during discharge planning. Consequently, understanding the interaction of weekend admission and LOS is important. In addition, the relative paucity of pediatric data examining a weekend discharge effect limits the ability to generalize these findings across other hospitals or health systems.

In contrast to prior work, we observed a modest increased risk for readmission among those discharged on the weekend in a large sample of children. Auger and Davis reported a lack of association between weekend discharge and readmissions at one tertiary care children’s hospital, citing reduced discharge volumes on the weekend, especially among children with medical complexity, as a possible driver for their observation.²² The inclusion of a much larger population across 43 hospitals in our study may explain our different findings compared with previous research. In addition, the inclusion/exclusion criteria differed between the two studies; we excluded index admissions for planned procedures in this study (which are more likely to occur during the week), which may have contributed to the differing conclusions. Although Auger and Davis suggest that differences in initial clinical management may be responsible for the weekend effect,²² our observations suggest that discharge planning practices may also contribute to readmission risk. For example, a family’s inability to access compounded medications at a local pharmacy or to access primary care following discharge could reasonably contribute to treatment failure and increased readmission risk. Attention to improving and standardizing discharge practices may alleviate differences in readmission risk among children discharged on a weekend.

Individual patient characteristics greatly influence LOS and readmission risk. Congruent with prior studies, medical complexity and technology dependence were among the factors in our study that had the strongest association with LOS and readmission risk.³² As with prior studies²², we observed that children with medical complexity and technology dependence were less frequently admitted and discharged on a weekend than on a weekday, which suggests that physicians may avoid complicated discharges on the weekend. Children with medical complexity present a unique challenge to physicians when assessing discharge readiness, given that these children frequently require careful coordination of durable medical equipment, obtainment of special medication preparations, and possibly the resumption or establishment of home health services. Notably, we cannot discern from our data what proportion of discharges may be delayed over the weekend secondary to challenges involved in coordinating care for children with medical complexity. Future investigations aimed at assessing physician decision making and discharge readiness in relation to discharge timing among children with medical complexity may establish this relationship more clearly.

We observed substantial variation in LOS and readmission risk across 43 tertiary care children’s hospitals. Since the 1970s, numerous studies have reported worse outcomes among patients admitted on the weekend. While the majority of studies support the weekend effect, several recent studies suggest that patients admitted on the weekend are at no greater risk of adverse outcomes than those admitted during the week.^35-37 Our work builds on the existing literature, demonstrating a complex and variable relationship between weekend admission/discharge, LOS, and readmission risk across hospitals. Notably, while many hospitals in our study experienced a significant weekend effect in LOS or readmission risk, only four hospitals experienced a statistically significant weekend effect for both LOS and readmission risk (three hospitals experienced increased risk for both, while one hospital experienced increased readmission risk but decreased LOS). Future investigations of the weekend effect should focus on exploring the differences in admission/discharge practices and staffing patterns of hospitals that did or did not experience a weekend effect.

This study has several limitations. We may have underestimated the total number of readmissions because we are unable to capture readmissions to other institutions by using the PHIS database. Our definition of a weekend admission or discharge did not account for three-day weekends or other holidays where staffing issues would be expected to be similar to that on weekends; consequently, our approach would be expected to bias the results toward null. Thus, a possible (but unlikely) result is that our approach masked a weekend effect that might have been more prominent had holidays been included. Although prior studies suggest that low physician/nurse staffing volumes and high patient workload are associated with worse patient outcomes,^38,39 we are unable to discern the role of differential staffing patterns, patient workload, or service availability in our observations using the PHIS database. Moreover, the PHIS database does not allow for any assessment of the preventability of a readmission or the impact of patient/family preference on the decision to admit or discharge, factors that could reasonably contribute to some of the observed variation. Finally, the PHIS database contains administrative data only, thus limiting our ability to fully adjust for patient severity of illness and sociodemographic factors that may have affected clinical decision making, including discharge decision making.

In a study of 43 children’s hospitals, children discharged on the weekend had a slightly increased readmission risk compared with children discharged on a weekday. Wide variation in the weekend effect on LOS and readmission risk was evident across hospitals. Individual patient characteristics had a greater impact on LOS and readmission risk than the weekend effect. Future investigations aimed at understanding which factors contribute most strongly to a weekend effect within individual hospitals (eg, differences in institutional admission/discharge practices) may help alleviate the weekend effect and improve healthcare quality.

Acknowledgments

This manuscript resulted from “Paper in a Day,” a Pediatric Research in Inpatient Settings (PRIS) Network-sponsored workshop presented at the Pediatric Hospital Medicine 2017 annual meeting. Workshop participants learned how to ask and answer a health services research question and efficiently prepare a manuscript for publication. The following are the members of the PRIS Network who contributed to this work: Jessica L. Bettenhausen, MD; Rebecca M. Cantu, MD, MPH; Jillian M Cotter, MD; Megan Deisz, MD; Teresa Frazer, MD; Pratichi Goenka, MD; Ashley Jenkins, MD; Kathryn E. Kyler, MD; Janet T. Lau, MD; Brian E. Lee, MD; Christiane Lenzen, MD; Trisha Marshall, MD; John M. Morrison MD, PhD; Lauren Nassetta, MD; Raymond Parlar-Chun, MD; Sonya Tang Girdwood MD, PhD; Tony R Tarchichi, MD; Irina G. Trifonova, MD; Jacqueline M. Walker, MD, MHPE; and Susan C. Walley, MD. See appendix for contact information for members of the PRIS Network.

Funding

The authors have no financial relationships relevant to this article to disclose.

Disclosures

The authors have no conflicts of interest to disclose.

Hospitals are increasingly assessed comparatively in terms of costs and quality for benchmarking purposes. These comparisons can be used by patients and families to determine where to seek care, to report compliance and grant certifications by oversight organizations (eg, Leapfrog, Magnet, Joint Commission), and by payers, to determine reimbursement models and/or to assess financial penalty or bonuses for underperforming or overperforming hospitals. As these efforts can cause substantial reputational and financial consequences for hospitals, these metrics must be contextualized within the population of patients that each hospital serves.

In adult Medicare patient populations, methods have been developed to assess the relative severity of a hospital’s full complement of patients.^1,2 These methods assume a relationship between severity and hospital resource intensity (ie, cost) and typically assume the form of relative weights (RWs), which are developed for clinically similar groups of patients (eg, Medicare Diagnosis Related Groups; MS-DRG) from a reference population. A RW for each MS-DRG is calculated as the average cost of patients within the group divided by the average cost for all patients in the reference population. These weights are then applied to a hospital’s discharges over a specific time period and averaged to obtain a hospital-level case-mix index (CMI). A value of 1 indicates that a hospital serves a mix of patients with similar severity (or resource intensity) to that of an “average” hospital discharge in the reference population, whereas a value of 1.2 indicates that a hospital serves a population of patients with 20% more severity than that of an “average” hospital discharge. Since 1983, the Centers for Medicare and Medicaid Services (CMS) has used RWs in their inpatient prospective payment system.³

Similar pediatric methods are less developed and necessitate special consideration as the use of existing weights may be inappropriate for a pediatric population. First, MS-DRGs were developed primarily for the Medicare population and lack sufficient granularity for pediatric populations, specifically newborns. Second, a severity stratification which incorporates important patient characteristics, such as age in pediatrics, does not exist in the MS-DRG system . Finally, although the reference populations that are used to develop MS-DRG weights do not explicitly exclude children, children typically account for approximately 15% of hospitalizations (6% excluding neonatal/maternal) and possibly feature different utilization patterns than adults with similar conditions. Thus, weights developed from a combined pediatric/adult reference population primarily reflect an adult population.

With valid pediatric RWs, stakeholders can assess a hospital’s severity mix of patients in a comparable fashion and contextualize outcome metrics. Additionally, these same weights can be used to estimate expected costs for hospitalizations or for risk adjusting various outcomes at the discharge- or hospital-level. Thus, we sought to develop hospitalization resource intensity scores for kids (H-RISK) using pediatric-specific weights and compare hospital-level CMIs across various hospital types and locations as an example of the application of this novel methodology.

Dataset

Data for this analysis were obtained from the 2012 Healthcare Cost and Utilization Project (HCUP) Kids’ Inpatient Database (KID).⁴ KID is the largest publicly available all-payer inpatient administrative database in the United States and is sponsored by the Agency for Healthcare Research and Quality as part of the HCUP. The 2012 KID included a sample of approximately 3.2 million discharge records of children <21 years old from 44 states and 4,179 community, nonrehabilitation hospitals weighted for national estimates.

Hospital discharge costs were estimated from charges using cost-to-charge ratios (CCR) provided by HCUP as a supplement to the 2012 KID.⁵ Cost estimates associated with a specific discharge were estimated by multiplying the total charges reported in the data by the appropriate hospital-specific CCR and then adjusted for price factors beyond a hospital’s control using the area wage index also provided by HCUP as a supplement.

H-RISK and Case-Mix Index Calculations

We calculated H-RISK as pediatric-specific RWs based on version 30 of 3M’s All Patient Refined DRG (APR-DRG; 3M Health Information Systems, Salt Lake City, Utah) system as a measure of resource intensity. The APR-DRG system classifies hospital discharges into over 300 base DRGs based on demographic, diagnostic, and therapeutic characteristics. Each APR-DRG is further sub-divided into 4 subclasses of severity of illness (SOI; eg, minor, moderate, major, and extreme) to indicate the intensity of resource utilization during hospitalization. However, SOI levels for differing APR-DRGs are not comparable.

For every APR-DRG SOI combinations available in the 2012 KID, calculation of RW was based on the ratio of the mean cost for patients assigned to a particular APR-DRG SOI compared with the mean cost for all patients in the database. Inpatient costs less than $0.50 were set to missing and removed from analysis. Mortalities and discharges with missing CCR and wage index values were also excluded from analysis. We required that estimates for RWs be based on a reasonable set of data (ie, 10 or more discharges) for each APR-DRG SOI, and that estimates across the 4 SOI levels within an APR-DRG be monotonically nondecreasing (ie, as SOI level increases, weights must either be the same or increasing). Winsorized means were used as point estimates for mean cost in both the numerator and denominator of RW computation. Winsorizing refers to an analytic transformation by which the influence of outliers (eg, values beyond a certain threshold) is mitigated by replacing the value of outliers with the value of the threshold. We used the 5^th and 95^th percentiles as thresholds for Winsorizing our point estimates.

Winsorized point estimates failing to meet the minimum sample size of 10 or nondecreasing monotonicity requirement were modified by one of the two following methods:

• Cost data were modeled using a generalized linear model assuming an exponential distribution. Covariates in the model included APR-DRG and SOI within APR-DRG as a continuous variable. Where applicable, Winsorized estimates of the mean were replaced with modeled estimates.
• Data from an APR-DRG SOI in question were combined with other SOIs within the same APR-DRG with the closest Winsorized mean value. Once data were combined, a common Winsorized value was re-computed and values across SOIs were checked to ensure that nondecreasing monotonicity was maintained. In some APR-DRGs with sparse data, this involved combining pairs of severity levels; in others, it involved combining three or four severity levels together.

For APR-DRGs in which no discharges at any SOI were recorded in the 2012 KID, we used the Winsorized mean of all encounters with a common major diagnostic category (MDC) as the missing APR-DRG as point estimate for all 4 SOI levels.

To calculate the CMI for a set of discharges (eg, discharges at a hospital in a year), RWs were assigned to each discharge based on APR-DRG SOI designation. Consequently, all discharges from a specific APR-DRG SOI were assigned the same RW. Once RWs were assigned, CMI was calculated as the mean RW across all discharges. To compare hospital types based on acute-care hospital stays which are usually considered with the realm of pediatric care, we excluded RWs for normal newborns, defined as APR-DRG 626 (neonate birthweight of 2000–2499 g, normal newborn or neonate with other problems) and 640 (neonate birthweight >2499 g, normal newborn or neonate with other problems), and maternal hospitalizations, defined as APR-DRG 540 (cesarean delivery) and 560 (vaginal delivery), from our CMI calculations.

Statistical Methodology

Categorical variables were summarized using frequencies and percentages; continuous variables were summarized using medians and interquartile ranges. Differences between hospital

types (eg, rural, urban nonteaching, urban teaching, and

free-standing) were assessed using a Chi-square test for association for categorical variables. Differences in continuous variables including comparisons of neonatal (MDC 15) and nonneonatal discharges, and medical versus procedural discharges as defined by the APR-DRG grouper were assessed using a Kruskal–Wallis test. All analyses were performed using SAS, Version 9.4 (SAS Institute, Cary, North Carolina); P values <.05 were considered statistically significant.

This study was considered nonhuman subjects research by the Institutional Review Board of Vanderbilt University Medical Center.

Patient Population

Table 1 summarizes the patient characteristics for all 4 hospital types. All comparisons of patient characteristics across the four hospital types are significant (P < .001). Of the 6,675,222 weighted discharges in HCUP KID 2012, almost two-thirds were less than one year old (4,269,984). Three-quarters of those infant discharges (3,733,760) were in-hospital births. The South was the Census region with the most number of discharges (38.8%), and over half of discharges (53.2%) included patients who lived in metro areas with more than 1 million residents. Patients disproportionately originated from lower-income areas with 30.9% living in zip codes with median incomes in the first quartile.

jhm013090602_t1.jpg

richardson0425e_t2.jpg

H-RISK Generation

richardson0425e_t3.jpg

The weighted Winsorized mean cost of all discharges was $6,135 per discharge. The majority of cost-based H-RISK were higher than 1, with 1,038 (82.5%) of APR-DRG SOIs incurring an estimated cost higher than $6,135. Solid organ and bone marrow transplantations represented 4 of the 10 highest cost-based RWs for procedural APR-DRG SOIs (Table 3). Neonatal APR-DRG SOIs accounted for 8 of the 10 highest medical RWs. A list of all APR-DRG SOIs and H-RISK can be found in Appendix A.

Hospital-Level Case-Mix Index for Acute Hospitalizations

After excluding normal newborn and maternal hospitalizations, median CMI of the 3117 hospitals with at least 20 unweighted discharges was 1.0 (interquartile range [IQR]: 0.8, 1.7). CMI varied significantly across hospital types (P < .001). Free-standing children’s hospitals exhibited the highest cost-based CMI (median: 2.7, IQR: 2.2–3.1), followed by urban teaching hospitals (median: 1.8, IQR: 1.3–2.6), urban nonteaching hospitals (median: 1.1, IQR: 0.9–1.5), and rural hospitals (median: 0.9, IQR: 0.7–0.9).

richardson0425e_f1.jpg

Currently, no widely available measures can compare the relative intensity of hospital care specific for inpatient pediatric populations. To meet this important need, we have developed a methodology to determine valid pediatric RWs (H-RISK) which can be used to estimate the intensity of care for applications across entire hospital patient populations and specific subpopulations. H-RISK allow calculation of CMIs for risk adjustment of various outcomes at the discharge- or hospital-level and for comparisons among hospitals and populations. Using this methodology, we demonstrated that the CMI for free-standing children’s hospitals was significantly higher than those of rural, urban, nonteaching and urban teaching hospitals for all discharges and medical or procedural subgroups.

CMS has used RWs based on DRGs since the inception of the prospective payment system in 1983. The sequence of DRGs used by CMS has purposely focused on older adult Medicare population, and CMS itself recommends applying Medicare-focused DRGs (MS-DRGs being the current iteration) only for the >65 year population.⁶ Nevertheless, many payers, both government and commercial, utilize MS-DRGs and their RWs for payment purposes when reimbursing children’s hospitals. The validity of using weights developed using this grouper in hospitals treating large numbers of pediatric patients and childhood illnesses has been called into question, particularly when such weights are used in reimbursement of children’s hospitals.⁷

Several factors contribute to the validity of a model for developing RWs. First, the system used to describe patient hospitalizations and illnesses should be appropriate to the population in question. As described above, the original DRG system and its subsequent iterations were designed to describe hospitalizations for adults >65 years of age.^{8, 9} Over the years, CMS DRGs incorporated rudimentary categories for neonatal and obstetrical hospitalizations. Still, the current MS-DRGs lack sufficient focus on common inpatient pediatric conditions to adequately describe pediatric hospitalizations, particularly those in free-standing children’s hospitals delivering tertiary and quaternary care. Thus, a more appropriate classification schema for developing RWs specific for pediatric hospitalization should include patients across the entire age spectrum. APR-DRGs represent one such classification system.

Once an appropriate patient classification system is selected, then the population of hospitalized patients to be used as the reference group becomes important. For a system targeting a pediatric inpatient population, a hospital discharge database representing a broad sample of pediatric hospitalizations offers the best basis for developing a system of weights applicable to different types of hospitals providing care for children. For this purpose, we selected the 2012 KID database, a nationally representative dataset containing data on newborn and pediatric discharges from the majority of states within the US. This choice assured that the RWs developed were based on and applicable to pediatric hospitalizations across the entire spectrum of SOI and resource intensity.

A number of measures of hospital performance and quality have been developed and are used by various entities, including individual hospitals, CMS, Leapfrog, Magnet, Joint Commission, and payers, for purposes ranging from benchmarking for improvement to payment models to reimbursement penalties. However, SOI of a hospital’s patient population influences not only the intensity of care that a hospital provides but also presents a potential impact on process and outcome measures. Thus, fair and appropriate measures must consider differences in SOI when comparing hospital performances. Using the weights derived in this paper, these adjustments can be possibly made at either the discharge- or hospital-level, depending on the application, and may include comparisons by hospital location, ownership, payer mix, or socioeconomic strata.

It is also common for hospitals to quantitatively express the uniqueness of services that they deliver to payers or the general public. A hospital-level CMI (derived as the average discharge weight for patients within a hospital) is one way that hospitals may differentiate themselves. This can be accomplished by considering the ratio of one hospital’s CMI to another hospital’s (or an average of a group of hospitals) as an expression of the relative intensity of services. For example, if hospital x has a CMI of 2.3, and hospital y has a CMI of 1.4, the population of children hospitalized at hospital x was 64.3% (1–2.3/1.4) more resource intensive than the children seen at hospital y.

This study should be considered in terms of several limitations. We used costs as the basis for determining intensity of service. Thus, the difference in cost structure among children’s hospitals and between children’s hospitals and other hospital types in the KID could have affected the final calculated weights. Also, the RWs calculated in this study rely on hospital discharge data. Thus, complications which were not “present on admission” and occurred during a hospitalization could have reflected poor quality of care yet still increase resource intensity as measured by total costs. Future studies should examine the potential impact of using present-on-admission diagnoses only for the APR-DRG grouping on the values of RWs. Significant variation may have existed among hospitals in resource utilization, and some hospitals may have exhibited significant overutilization of resources for the same conditions. However, as we used Winsorized means, the impact of potential outliers should have been reduced. Some APR-DRG-SOI combinations were seen mainly at children’s hospitals. Thus, cost structure and resource utilization practices of this subset of hospitals would have been the only contributors to weights for these patients. Given that the 2012 KID contained a broad representation of pediatric hospitalizations, with age 0–20 years, newborns accounted for the majority of total cases in the database. While providing a full range of pediatric weights, inclusion of these patients lowered the overall average RW. For this reason, we excluded normal newborn categories and maternal categories from analysis of CMI across hospital types and focused on acute-care hospitalizations. Lastly, as with any study relying on administrative data, there is always the possibility of coding errors or data entry errors in the reference dataset.

H-RISK can be used to risk adjust measures to account for severity differences across populations. These weights can also be averaged across hospitals’ patient populations to compare relative resource intensities of the patients served.

Disclosures

The authors have nothing to disclose.

Hospitals are increasingly assessed comparatively in terms of costs and quality for benchmarking purposes. These comparisons can be used by patients and families to determine where to seek care, to report compliance and grant certifications by oversight organizations (eg, Leapfrog, Magnet, Joint Commission), and by payers, to determine reimbursement models and/or to assess financial penalty or bonuses for underperforming or overperforming hospitals. As these efforts can cause substantial reputational and financial consequences for hospitals, these metrics must be contextualized within the population of patients that each hospital serves.

In adult Medicare patient populations, methods have been developed to assess the relative severity of a hospital’s full complement of patients.^1,2 These methods assume a relationship between severity and hospital resource intensity (ie, cost) and typically assume the form of relative weights (RWs), which are developed for clinically similar groups of patients (eg, Medicare Diagnosis Related Groups; MS-DRG) from a reference population. A RW for each MS-DRG is calculated as the average cost of patients within the group divided by the average cost for all patients in the reference population. These weights are then applied to a hospital’s discharges over a specific time period and averaged to obtain a hospital-level case-mix index (CMI). A value of 1 indicates that a hospital serves a mix of patients with similar severity (or resource intensity) to that of an “average” hospital discharge in the reference population, whereas a value of 1.2 indicates that a hospital serves a population of patients with 20% more severity than that of an “average” hospital discharge. Since 1983, the Centers for Medicare and Medicaid Services (CMS) has used RWs in their inpatient prospective payment system.³

Similar pediatric methods are less developed and necessitate special consideration as the use of existing weights may be inappropriate for a pediatric population. First, MS-DRGs were developed primarily for the Medicare population and lack sufficient granularity for pediatric populations, specifically newborns. Second, a severity stratification which incorporates important patient characteristics, such as age in pediatrics, does not exist in the MS-DRG system . Finally, although the reference populations that are used to develop MS-DRG weights do not explicitly exclude children, children typically account for approximately 15% of hospitalizations (6% excluding neonatal/maternal) and possibly feature different utilization patterns than adults with similar conditions. Thus, weights developed from a combined pediatric/adult reference population primarily reflect an adult population.

With valid pediatric RWs, stakeholders can assess a hospital’s severity mix of patients in a comparable fashion and contextualize outcome metrics. Additionally, these same weights can be used to estimate expected costs for hospitalizations or for risk adjusting various outcomes at the discharge- or hospital-level. Thus, we sought to develop hospitalization resource intensity scores for kids (H-RISK) using pediatric-specific weights and compare hospital-level CMIs across various hospital types and locations as an example of the application of this novel methodology.

Dataset

Data for this analysis were obtained from the 2012 Healthcare Cost and Utilization Project (HCUP) Kids’ Inpatient Database (KID).⁴ KID is the largest publicly available all-payer inpatient administrative database in the United States and is sponsored by the Agency for Healthcare Research and Quality as part of the HCUP. The 2012 KID included a sample of approximately 3.2 million discharge records of children <21 years old from 44 states and 4,179 community, nonrehabilitation hospitals weighted for national estimates.

Hospital discharge costs were estimated from charges using cost-to-charge ratios (CCR) provided by HCUP as a supplement to the 2012 KID.⁵ Cost estimates associated with a specific discharge were estimated by multiplying the total charges reported in the data by the appropriate hospital-specific CCR and then adjusted for price factors beyond a hospital’s control using the area wage index also provided by HCUP as a supplement.

H-RISK and Case-Mix Index Calculations

We calculated H-RISK as pediatric-specific RWs based on version 30 of 3M’s All Patient Refined DRG (APR-DRG; 3M Health Information Systems, Salt Lake City, Utah) system as a measure of resource intensity. The APR-DRG system classifies hospital discharges into over 300 base DRGs based on demographic, diagnostic, and therapeutic characteristics. Each APR-DRG is further sub-divided into 4 subclasses of severity of illness (SOI; eg, minor, moderate, major, and extreme) to indicate the intensity of resource utilization during hospitalization. However, SOI levels for differing APR-DRGs are not comparable.

For every APR-DRG SOI combinations available in the 2012 KID, calculation of RW was based on the ratio of the mean cost for patients assigned to a particular APR-DRG SOI compared with the mean cost for all patients in the database. Inpatient costs less than $0.50 were set to missing and removed from analysis. Mortalities and discharges with missing CCR and wage index values were also excluded from analysis. We required that estimates for RWs be based on a reasonable set of data (ie, 10 or more discharges) for each APR-DRG SOI, and that estimates across the 4 SOI levels within an APR-DRG be monotonically nondecreasing (ie, as SOI level increases, weights must either be the same or increasing). Winsorized means were used as point estimates for mean cost in both the numerator and denominator of RW computation. Winsorizing refers to an analytic transformation by which the influence of outliers (eg, values beyond a certain threshold) is mitigated by replacing the value of outliers with the value of the threshold. We used the 5^th and 95^th percentiles as thresholds for Winsorizing our point estimates.

Winsorized point estimates failing to meet the minimum sample size of 10 or nondecreasing monotonicity requirement were modified by one of the two following methods:

• Cost data were modeled using a generalized linear model assuming an exponential distribution. Covariates in the model included APR-DRG and SOI within APR-DRG as a continuous variable. Where applicable, Winsorized estimates of the mean were replaced with modeled estimates.
• Data from an APR-DRG SOI in question were combined with other SOIs within the same APR-DRG with the closest Winsorized mean value. Once data were combined, a common Winsorized value was re-computed and values across SOIs were checked to ensure that nondecreasing monotonicity was maintained. In some APR-DRGs with sparse data, this involved combining pairs of severity levels; in others, it involved combining three or four severity levels together.

For APR-DRGs in which no discharges at any SOI were recorded in the 2012 KID, we used the Winsorized mean of all encounters with a common major diagnostic category (MDC) as the missing APR-DRG as point estimate for all 4 SOI levels.

To calculate the CMI for a set of discharges (eg, discharges at a hospital in a year), RWs were assigned to each discharge based on APR-DRG SOI designation. Consequently, all discharges from a specific APR-DRG SOI were assigned the same RW. Once RWs were assigned, CMI was calculated as the mean RW across all discharges. To compare hospital types based on acute-care hospital stays which are usually considered with the realm of pediatric care, we excluded RWs for normal newborns, defined as APR-DRG 626 (neonate birthweight of 2000–2499 g, normal newborn or neonate with other problems) and 640 (neonate birthweight >2499 g, normal newborn or neonate with other problems), and maternal hospitalizations, defined as APR-DRG 540 (cesarean delivery) and 560 (vaginal delivery), from our CMI calculations.

Statistical Methodology

Categorical variables were summarized using frequencies and percentages; continuous variables were summarized using medians and interquartile ranges. Differences between hospital

types (eg, rural, urban nonteaching, urban teaching, and

free-standing) were assessed using a Chi-square test for association for categorical variables. Differences in continuous variables including comparisons of neonatal (MDC 15) and nonneonatal discharges, and medical versus procedural discharges as defined by the APR-DRG grouper were assessed using a Kruskal–Wallis test. All analyses were performed using SAS, Version 9.4 (SAS Institute, Cary, North Carolina); P values <.05 were considered statistically significant.

This study was considered nonhuman subjects research by the Institutional Review Board of Vanderbilt University Medical Center.

Patient Population

Table 1 summarizes the patient characteristics for all 4 hospital types. All comparisons of patient characteristics across the four hospital types are significant (P < .001). Of the 6,675,222 weighted discharges in HCUP KID 2012, almost two-thirds were less than one year old (4,269,984). Three-quarters of those infant discharges (3,733,760) were in-hospital births. The South was the Census region with the most number of discharges (38.8%), and over half of discharges (53.2%) included patients who lived in metro areas with more than 1 million residents. Patients disproportionately originated from lower-income areas with 30.9% living in zip codes with median incomes in the first quartile.

jhm013090602_t1.jpg

richardson0425e_t2.jpg

H-RISK Generation

richardson0425e_t3.jpg

The weighted Winsorized mean cost of all discharges was $6,135 per discharge. The majority of cost-based H-RISK were higher than 1, with 1,038 (82.5%) of APR-DRG SOIs incurring an estimated cost higher than $6,135. Solid organ and bone marrow transplantations represented 4 of the 10 highest cost-based RWs for procedural APR-DRG SOIs (Table 3). Neonatal APR-DRG SOIs accounted for 8 of the 10 highest medical RWs. A list of all APR-DRG SOIs and H-RISK can be found in Appendix A.

Hospital-Level Case-Mix Index for Acute Hospitalizations

After excluding normal newborn and maternal hospitalizations, median CMI of the 3117 hospitals with at least 20 unweighted discharges was 1.0 (interquartile range [IQR]: 0.8, 1.7). CMI varied significantly across hospital types (P < .001). Free-standing children’s hospitals exhibited the highest cost-based CMI (median: 2.7, IQR: 2.2–3.1), followed by urban teaching hospitals (median: 1.8, IQR: 1.3–2.6), urban nonteaching hospitals (median: 1.1, IQR: 0.9–1.5), and rural hospitals (median: 0.9, IQR: 0.7–0.9).

richardson0425e_f1.jpg

Currently, no widely available measures can compare the relative intensity of hospital care specific for inpatient pediatric populations. To meet this important need, we have developed a methodology to determine valid pediatric RWs (H-RISK) which can be used to estimate the intensity of care for applications across entire hospital patient populations and specific subpopulations. H-RISK allow calculation of CMIs for risk adjustment of various outcomes at the discharge- or hospital-level and for comparisons among hospitals and populations. Using this methodology, we demonstrated that the CMI for free-standing children’s hospitals was significantly higher than those of rural, urban, nonteaching and urban teaching hospitals for all discharges and medical or procedural subgroups.

CMS has used RWs based on DRGs since the inception of the prospective payment system in 1983. The sequence of DRGs used by CMS has purposely focused on older adult Medicare population, and CMS itself recommends applying Medicare-focused DRGs (MS-DRGs being the current iteration) only for the >65 year population.⁶ Nevertheless, many payers, both government and commercial, utilize MS-DRGs and their RWs for payment purposes when reimbursing children’s hospitals. The validity of using weights developed using this grouper in hospitals treating large numbers of pediatric patients and childhood illnesses has been called into question, particularly when such weights are used in reimbursement of children’s hospitals.⁷

Several factors contribute to the validity of a model for developing RWs. First, the system used to describe patient hospitalizations and illnesses should be appropriate to the population in question. As described above, the original DRG system and its subsequent iterations were designed to describe hospitalizations for adults >65 years of age.^{8, 9} Over the years, CMS DRGs incorporated rudimentary categories for neonatal and obstetrical hospitalizations. Still, the current MS-DRGs lack sufficient focus on common inpatient pediatric conditions to adequately describe pediatric hospitalizations, particularly those in free-standing children’s hospitals delivering tertiary and quaternary care. Thus, a more appropriate classification schema for developing RWs specific for pediatric hospitalization should include patients across the entire age spectrum. APR-DRGs represent one such classification system.

Once an appropriate patient classification system is selected, then the population of hospitalized patients to be used as the reference group becomes important. For a system targeting a pediatric inpatient population, a hospital discharge database representing a broad sample of pediatric hospitalizations offers the best basis for developing a system of weights applicable to different types of hospitals providing care for children. For this purpose, we selected the 2012 KID database, a nationally representative dataset containing data on newborn and pediatric discharges from the majority of states within the US. This choice assured that the RWs developed were based on and applicable to pediatric hospitalizations across the entire spectrum of SOI and resource intensity.

A number of measures of hospital performance and quality have been developed and are used by various entities, including individual hospitals, CMS, Leapfrog, Magnet, Joint Commission, and payers, for purposes ranging from benchmarking for improvement to payment models to reimbursement penalties. However, SOI of a hospital’s patient population influences not only the intensity of care that a hospital provides but also presents a potential impact on process and outcome measures. Thus, fair and appropriate measures must consider differences in SOI when comparing hospital performances. Using the weights derived in this paper, these adjustments can be possibly made at either the discharge- or hospital-level, depending on the application, and may include comparisons by hospital location, ownership, payer mix, or socioeconomic strata.

It is also common for hospitals to quantitatively express the uniqueness of services that they deliver to payers or the general public. A hospital-level CMI (derived as the average discharge weight for patients within a hospital) is one way that hospitals may differentiate themselves. This can be accomplished by considering the ratio of one hospital’s CMI to another hospital’s (or an average of a group of hospitals) as an expression of the relative intensity of services. For example, if hospital x has a CMI of 2.3, and hospital y has a CMI of 1.4, the population of children hospitalized at hospital x was 64.3% (1–2.3/1.4) more resource intensive than the children seen at hospital y.

This study should be considered in terms of several limitations. We used costs as the basis for determining intensity of service. Thus, the difference in cost structure among children’s hospitals and between children’s hospitals and other hospital types in the KID could have affected the final calculated weights. Also, the RWs calculated in this study rely on hospital discharge data. Thus, complications which were not “present on admission” and occurred during a hospitalization could have reflected poor quality of care yet still increase resource intensity as measured by total costs. Future studies should examine the potential impact of using present-on-admission diagnoses only for the APR-DRG grouping on the values of RWs. Significant variation may have existed among hospitals in resource utilization, and some hospitals may have exhibited significant overutilization of resources for the same conditions. However, as we used Winsorized means, the impact of potential outliers should have been reduced. Some APR-DRG-SOI combinations were seen mainly at children’s hospitals. Thus, cost structure and resource utilization practices of this subset of hospitals would have been the only contributors to weights for these patients. Given that the 2012 KID contained a broad representation of pediatric hospitalizations, with age 0–20 years, newborns accounted for the majority of total cases in the database. While providing a full range of pediatric weights, inclusion of these patients lowered the overall average RW. For this reason, we excluded normal newborn categories and maternal categories from analysis of CMI across hospital types and focused on acute-care hospitalizations. Lastly, as with any study relying on administrative data, there is always the possibility of coding errors or data entry errors in the reference dataset.

H-RISK can be used to risk adjust measures to account for severity differences across populations. These weights can also be averaged across hospitals’ patient populations to compare relative resource intensities of the patients served.

Disclosures

The authors have nothing to disclose.

Hospitals are increasingly assessed comparatively in terms of costs and quality for benchmarking purposes. These comparisons can be used by patients and families to determine where to seek care, to report compliance and grant certifications by oversight organizations (eg, Leapfrog, Magnet, Joint Commission), and by payers, to determine reimbursement models and/or to assess financial penalty or bonuses for underperforming or overperforming hospitals. As these efforts can cause substantial reputational and financial consequences for hospitals, these metrics must be contextualized within the population of patients that each hospital serves.

In adult Medicare patient populations, methods have been developed to assess the relative severity of a hospital’s full complement of patients.^1,2 These methods assume a relationship between severity and hospital resource intensity (ie, cost) and typically assume the form of relative weights (RWs), which are developed for clinically similar groups of patients (eg, Medicare Diagnosis Related Groups; MS-DRG) from a reference population. A RW for each MS-DRG is calculated as the average cost of patients within the group divided by the average cost for all patients in the reference population. These weights are then applied to a hospital’s discharges over a specific time period and averaged to obtain a hospital-level case-mix index (CMI). A value of 1 indicates that a hospital serves a mix of patients with similar severity (or resource intensity) to that of an “average” hospital discharge in the reference population, whereas a value of 1.2 indicates that a hospital serves a population of patients with 20% more severity than that of an “average” hospital discharge. Since 1983, the Centers for Medicare and Medicaid Services (CMS) has used RWs in their inpatient prospective payment system.³

Similar pediatric methods are less developed and necessitate special consideration as the use of existing weights may be inappropriate for a pediatric population. First, MS-DRGs were developed primarily for the Medicare population and lack sufficient granularity for pediatric populations, specifically newborns. Second, a severity stratification which incorporates important patient characteristics, such as age in pediatrics, does not exist in the MS-DRG system . Finally, although the reference populations that are used to develop MS-DRG weights do not explicitly exclude children, children typically account for approximately 15% of hospitalizations (6% excluding neonatal/maternal) and possibly feature different utilization patterns than adults with similar conditions. Thus, weights developed from a combined pediatric/adult reference population primarily reflect an adult population.

With valid pediatric RWs, stakeholders can assess a hospital’s severity mix of patients in a comparable fashion and contextualize outcome metrics. Additionally, these same weights can be used to estimate expected costs for hospitalizations or for risk adjusting various outcomes at the discharge- or hospital-level. Thus, we sought to develop hospitalization resource intensity scores for kids (H-RISK) using pediatric-specific weights and compare hospital-level CMIs across various hospital types and locations as an example of the application of this novel methodology.

Dataset

Data for this analysis were obtained from the 2012 Healthcare Cost and Utilization Project (HCUP) Kids’ Inpatient Database (KID).⁴ KID is the largest publicly available all-payer inpatient administrative database in the United States and is sponsored by the Agency for Healthcare Research and Quality as part of the HCUP. The 2012 KID included a sample of approximately 3.2 million discharge records of children <21 years old from 44 states and 4,179 community, nonrehabilitation hospitals weighted for national estimates.

Hospital discharge costs were estimated from charges using cost-to-charge ratios (CCR) provided by HCUP as a supplement to the 2012 KID.⁵ Cost estimates associated with a specific discharge were estimated by multiplying the total charges reported in the data by the appropriate hospital-specific CCR and then adjusted for price factors beyond a hospital’s control using the area wage index also provided by HCUP as a supplement.

H-RISK and Case-Mix Index Calculations

We calculated H-RISK as pediatric-specific RWs based on version 30 of 3M’s All Patient Refined DRG (APR-DRG; 3M Health Information Systems, Salt Lake City, Utah) system as a measure of resource intensity. The APR-DRG system classifies hospital discharges into over 300 base DRGs based on demographic, diagnostic, and therapeutic characteristics. Each APR-DRG is further sub-divided into 4 subclasses of severity of illness (SOI; eg, minor, moderate, major, and extreme) to indicate the intensity of resource utilization during hospitalization. However, SOI levels for differing APR-DRGs are not comparable.

For every APR-DRG SOI combinations available in the 2012 KID, calculation of RW was based on the ratio of the mean cost for patients assigned to a particular APR-DRG SOI compared with the mean cost for all patients in the database. Inpatient costs less than $0.50 were set to missing and removed from analysis. Mortalities and discharges with missing CCR and wage index values were also excluded from analysis. We required that estimates for RWs be based on a reasonable set of data (ie, 10 or more discharges) for each APR-DRG SOI, and that estimates across the 4 SOI levels within an APR-DRG be monotonically nondecreasing (ie, as SOI level increases, weights must either be the same or increasing). Winsorized means were used as point estimates for mean cost in both the numerator and denominator of RW computation. Winsorizing refers to an analytic transformation by which the influence of outliers (eg, values beyond a certain threshold) is mitigated by replacing the value of outliers with the value of the threshold. We used the 5^th and 95^th percentiles as thresholds for Winsorizing our point estimates.

Winsorized point estimates failing to meet the minimum sample size of 10 or nondecreasing monotonicity requirement were modified by one of the two following methods:

• Cost data were modeled using a generalized linear model assuming an exponential distribution. Covariates in the model included APR-DRG and SOI within APR-DRG as a continuous variable. Where applicable, Winsorized estimates of the mean were replaced with modeled estimates.
• Data from an APR-DRG SOI in question were combined with other SOIs within the same APR-DRG with the closest Winsorized mean value. Once data were combined, a common Winsorized value was re-computed and values across SOIs were checked to ensure that nondecreasing monotonicity was maintained. In some APR-DRGs with sparse data, this involved combining pairs of severity levels; in others, it involved combining three or four severity levels together.

For APR-DRGs in which no discharges at any SOI were recorded in the 2012 KID, we used the Winsorized mean of all encounters with a common major diagnostic category (MDC) as the missing APR-DRG as point estimate for all 4 SOI levels.

To calculate the CMI for a set of discharges (eg, discharges at a hospital in a year), RWs were assigned to each discharge based on APR-DRG SOI designation. Consequently, all discharges from a specific APR-DRG SOI were assigned the same RW. Once RWs were assigned, CMI was calculated as the mean RW across all discharges. To compare hospital types based on acute-care hospital stays which are usually considered with the realm of pediatric care, we excluded RWs for normal newborns, defined as APR-DRG 626 (neonate birthweight of 2000–2499 g, normal newborn or neonate with other problems) and 640 (neonate birthweight >2499 g, normal newborn or neonate with other problems), and maternal hospitalizations, defined as APR-DRG 540 (cesarean delivery) and 560 (vaginal delivery), from our CMI calculations.

Statistical Methodology

Categorical variables were summarized using frequencies and percentages; continuous variables were summarized using medians and interquartile ranges. Differences between hospital

types (eg, rural, urban nonteaching, urban teaching, and

free-standing) were assessed using a Chi-square test for association for categorical variables. Differences in continuous variables including comparisons of neonatal (MDC 15) and nonneonatal discharges, and medical versus procedural discharges as defined by the APR-DRG grouper were assessed using a Kruskal–Wallis test. All analyses were performed using SAS, Version 9.4 (SAS Institute, Cary, North Carolina); P values <.05 were considered statistically significant.

This study was considered nonhuman subjects research by the Institutional Review Board of Vanderbilt University Medical Center.

Patient Population

Table 1 summarizes the patient characteristics for all 4 hospital types. All comparisons of patient characteristics across the four hospital types are significant (P < .001). Of the 6,675,222 weighted discharges in HCUP KID 2012, almost two-thirds were less than one year old (4,269,984). Three-quarters of those infant discharges (3,733,760) were in-hospital births. The South was the Census region with the most number of discharges (38.8%), and over half of discharges (53.2%) included patients who lived in metro areas with more than 1 million residents. Patients disproportionately originated from lower-income areas with 30.9% living in zip codes with median incomes in the first quartile.

jhm013090602_t1.jpg

richardson0425e_t2.jpg

H-RISK Generation

richardson0425e_t3.jpg

The weighted Winsorized mean cost of all discharges was $6,135 per discharge. The majority of cost-based H-RISK were higher than 1, with 1,038 (82.5%) of APR-DRG SOIs incurring an estimated cost higher than $6,135. Solid organ and bone marrow transplantations represented 4 of the 10 highest cost-based RWs for procedural APR-DRG SOIs (Table 3). Neonatal APR-DRG SOIs accounted for 8 of the 10 highest medical RWs. A list of all APR-DRG SOIs and H-RISK can be found in Appendix A.

Hospital-Level Case-Mix Index for Acute Hospitalizations

After excluding normal newborn and maternal hospitalizations, median CMI of the 3117 hospitals with at least 20 unweighted discharges was 1.0 (interquartile range [IQR]: 0.8, 1.7). CMI varied significantly across hospital types (P < .001). Free-standing children’s hospitals exhibited the highest cost-based CMI (median: 2.7, IQR: 2.2–3.1), followed by urban teaching hospitals (median: 1.8, IQR: 1.3–2.6), urban nonteaching hospitals (median: 1.1, IQR: 0.9–1.5), and rural hospitals (median: 0.9, IQR: 0.7–0.9).

richardson0425e_f1.jpg

Currently, no widely available measures can compare the relative intensity of hospital care specific for inpatient pediatric populations. To meet this important need, we have developed a methodology to determine valid pediatric RWs (H-RISK) which can be used to estimate the intensity of care for applications across entire hospital patient populations and specific subpopulations. H-RISK allow calculation of CMIs for risk adjustment of various outcomes at the discharge- or hospital-level and for comparisons among hospitals and populations. Using this methodology, we demonstrated that the CMI for free-standing children’s hospitals was significantly higher than those of rural, urban, nonteaching and urban teaching hospitals for all discharges and medical or procedural subgroups.

CMS has used RWs based on DRGs since the inception of the prospective payment system in 1983. The sequence of DRGs used by CMS has purposely focused on older adult Medicare population, and CMS itself recommends applying Medicare-focused DRGs (MS-DRGs being the current iteration) only for the >65 year population.⁶ Nevertheless, many payers, both government and commercial, utilize MS-DRGs and their RWs for payment purposes when reimbursing children’s hospitals. The validity of using weights developed using this grouper in hospitals treating large numbers of pediatric patients and childhood illnesses has been called into question, particularly when such weights are used in reimbursement of children’s hospitals.⁷

Several factors contribute to the validity of a model for developing RWs. First, the system used to describe patient hospitalizations and illnesses should be appropriate to the population in question. As described above, the original DRG system and its subsequent iterations were designed to describe hospitalizations for adults >65 years of age.^{8, 9} Over the years, CMS DRGs incorporated rudimentary categories for neonatal and obstetrical hospitalizations. Still, the current MS-DRGs lack sufficient focus on common inpatient pediatric conditions to adequately describe pediatric hospitalizations, particularly those in free-standing children’s hospitals delivering tertiary and quaternary care. Thus, a more appropriate classification schema for developing RWs specific for pediatric hospitalization should include patients across the entire age spectrum. APR-DRGs represent one such classification system.

Once an appropriate patient classification system is selected, then the population of hospitalized patients to be used as the reference group becomes important. For a system targeting a pediatric inpatient population, a hospital discharge database representing a broad sample of pediatric hospitalizations offers the best basis for developing a system of weights applicable to different types of hospitals providing care for children. For this purpose, we selected the 2012 KID database, a nationally representative dataset containing data on newborn and pediatric discharges from the majority of states within the US. This choice assured that the RWs developed were based on and applicable to pediatric hospitalizations across the entire spectrum of SOI and resource intensity.

A number of measures of hospital performance and quality have been developed and are used by various entities, including individual hospitals, CMS, Leapfrog, Magnet, Joint Commission, and payers, for purposes ranging from benchmarking for improvement to payment models to reimbursement penalties. However, SOI of a hospital’s patient population influences not only the intensity of care that a hospital provides but also presents a potential impact on process and outcome measures. Thus, fair and appropriate measures must consider differences in SOI when comparing hospital performances. Using the weights derived in this paper, these adjustments can be possibly made at either the discharge- or hospital-level, depending on the application, and may include comparisons by hospital location, ownership, payer mix, or socioeconomic strata.

It is also common for hospitals to quantitatively express the uniqueness of services that they deliver to payers or the general public. A hospital-level CMI (derived as the average discharge weight for patients within a hospital) is one way that hospitals may differentiate themselves. This can be accomplished by considering the ratio of one hospital’s CMI to another hospital’s (or an average of a group of hospitals) as an expression of the relative intensity of services. For example, if hospital x has a CMI of 2.3, and hospital y has a CMI of 1.4, the population of children hospitalized at hospital x was 64.3% (1–2.3/1.4) more resource intensive than the children seen at hospital y.

This study should be considered in terms of several limitations. We used costs as the basis for determining intensity of service. Thus, the difference in cost structure among children’s hospitals and between children’s hospitals and other hospital types in the KID could have affected the final calculated weights. Also, the RWs calculated in this study rely on hospital discharge data. Thus, complications which were not “present on admission” and occurred during a hospitalization could have reflected poor quality of care yet still increase resource intensity as measured by total costs. Future studies should examine the potential impact of using present-on-admission diagnoses only for the APR-DRG grouping on the values of RWs. Significant variation may have existed among hospitals in resource utilization, and some hospitals may have exhibited significant overutilization of resources for the same conditions. However, as we used Winsorized means, the impact of potential outliers should have been reduced. Some APR-DRG-SOI combinations were seen mainly at children’s hospitals. Thus, cost structure and resource utilization practices of this subset of hospitals would have been the only contributors to weights for these patients. Given that the 2012 KID contained a broad representation of pediatric hospitalizations, with age 0–20 years, newborns accounted for the majority of total cases in the database. While providing a full range of pediatric weights, inclusion of these patients lowered the overall average RW. For this reason, we excluded normal newborn categories and maternal categories from analysis of CMI across hospital types and focused on acute-care hospitalizations. Lastly, as with any study relying on administrative data, there is always the possibility of coding errors or data entry errors in the reference dataset.

H-RISK can be used to risk adjust measures to account for severity differences across populations. These weights can also be averaged across hospitals’ patient populations to compare relative resource intensities of the patients served.

Disclosures

The authors have nothing to disclose.