Richard Lin, MD

METHODS

RESULTS

UNADJUSTED ANALYSIS OF CHANGES IN ALARM RATES

CONCLUSION

METHODS

RESULTS

UNADJUSTED ANALYSIS OF CHANGES IN ALARM RATES

CONCLUSION

METHODS

RESULTS

UNADJUSTED ANALYSIS OF CHANGES IN ALARM RATES

CONCLUSION

Hospital physiologic monitors can alert clinicians to early signs of physiologic deterioration, and thus have great potential to save lives. However, monitors generate frequent alarms,[1, 2, 3, 4, 5, 6, 7, 8] and most are not relevant to the patient's safety (over 90% of pediatric intensive care unit (PICU)[1, 2] and over 70% of adult intensive care alarms).[5, 6] In psychology experiments, humans rapidly learn to ignore or respond more slowly to alarms when exposed to high false‐alarm rates, exhibiting alarm fatigue.[9, 10] In 2013, The Joint Commission named alarm fatigue the most common contributing factor to alarm‐related sentinel events in hospitals.[11, 12]

Although alarm fatigue has been implicated as a major threat to patient safety, little empirical data support its existence in hospitals. In this study, we aimed to determine if there was an association between nurses' recent exposure to nonactionable physiologic monitor alarms and their response time to future alarms for the same patients. This exploratory work was designed to inform future research in this area, acknowledging that the sample size would be too small for multivariable modeling.

METHODS

Study Definitions

The alarm classification scheme is shown in Figure 1. Note that, for clarity, we have intentionally avoided using the terms true and false alarms because their interpretations vary across studies and can be misleading.

Figure 1

Potentially Critical Alarm

A potentially critical alarm is any alarm for a clinical condition for which a timely response is important to determine if the alarm requires intervention to save the patient's life. This is based on the alarm type alone, including alarms for life‐threatening arrhythmias such as asystole and ventricular tachycardia, as well as alarms for vital signs outside the set limits. Supporting Table 1 in the online version of this article lists the breakdown of alarm types that we defined a priori as potentially and not potentially critical.

Table 1.Characteristics of the 2,445 Alarms for Clinical Conditions

	PICU				Ward
Alarm type	No.	% of Total	% Valid	% Actionable	No.	% of Total	% Valid	% Actionable
NOTE: Abbreviations: N/A, not applicable; PICU, pediatric intensive care unit.
Oxygen saturation	197	19.4	82.7	38.6	590	41.2	24.4	1.9
Heart rate	194	19.1	95.4	1.0	266	18.6	87.2	0.0
Respiratory rate	229	22.6	80.8	13.5	316	22.1	48.1	1.0
Blood pressure	259	25.5	83.8	5.8	11	0.8	72.7	0.0
Critical arrhythmia	1	0.1	0.0	0.0	4	0.3	0.0	0.0
Noncritical arrhythmia	71	7.0	2.8	0.0	244	17.1	8.6	0.0
Central venous pressure	49	4.8	0.0	0.0	0	0.0	N/A	N/A
Exhaled carbon dioxide	14	1.4	92.9	50.0	0	0.0	N/A	N/A
Total	1014	100.0	75.6	12.9	1,431	100.0	38.9	1.0

Valid Alarm

A valid alarm is any alarm that correctly identifies the physiologic status of the patient. Validity was based on waveform quality, lead signal strength indicators, and artifact conditions, referencing each monitor's operator's manual.

Actionable Alarm

An actionable alarm is any valid alarm for a clinical condition that either: (1) leads to a clinical intervention; (2) leads to a consultation with another clinician at the bedside (and thus visible on camera); or (3) is a situation that should have led to intervention or consultation, but the alarm was unwitnessed or misinterpreted by the staff at the bedside.

Nonactionable Alarm

An unactionable alarm is any alarm that does not meet the actionable definition above, including invalid alarms such as those caused by motion artifact, equipment/technical alarms, and alarms that are valid but nonactionable (nuisance alarms).[13]

Response Time

The response time is the time elapsed from when the alarm fired at the bedside to when the nurse entered the room or peered through a window or door, measured in seconds.

RESULTS

We performed 40 sessions among 40 patients and 36 nurses over 210 hours. We performed 20 sessions in children with heart and/or lung failure in the PICU and 20 sessions in children on a general ward. Sessions took place on weekdays between 9:00 _am and 6:00 _pm. There were 3 occasions when we filmed 2 patients cared for by the same nurse at the same time.

Nurses were mostly female (94.4%) and had between 2 months and 28 years of experience (median, 4.8 years). Patients on the ward ranged from 5 days to 5.4 years old (median, 6 months). Patients in the PICU ranged from 5 months to 16 years old (median, 2.5 years). Among the PICU patients, 14 (70%) were receiving mechanical ventilation only, 3 (15%) were receiving vasopressors only, and 3 (15%) were receiving mechanical ventilation and vasopressors.

We observed 5070 alarms during the 40 sessions. We excluded 108 (2.1%) that occurred at the end of video recording sessions with the nurse absent from the room because the nurse's response could not be determined. Alarms per session ranged from 10 to 1430 (median, 75; interquartile range [IQR], 35138). We excluded the outlier PICU patient with 1430 alarms in 5 hours from the analysis to avoid the potential for biasing the results. Figure 2 depicts the data flow.

Figure 2

Following the 5 training sessions, research assistants independently reviewed and made preliminary assessments on 4674 alarms; these alarms were all secondarily reviewed by a physician. Using the physician reviewer as the gold standard, the research assistant's sensitivity (assess alarm as actionable when physician also assesses as actionable) was 96.8% and specificity (assess alarm as nonactionable when physician also assesses as nonactionable) was 96.9%. We had to review 54 of 4674 alarms (1.2%) with an additional physician or nurse investigator to achieve consensus.

Characteristics of the 2445 alarms for clinical conditions are shown in Table 1. Only 12.9% of alarms in heart‐ and/or lung‐failure patients in the PICU were actionable, and only 1.0% of alarms in medical patients on a general inpatient ward were actionable.

Response‐Time Association With Nonactionable Alarm Exposure

Next, we analyzed the association between response time to potentially critical alarms that occurred when the nurse was not in the patient's room and the number of nonactionable alarms occurring over the preceding 120‐minute window. This required excluding the alarms that occurred in the first 120 minutes of each session, leaving 647 alarms with eligible response times to evaluate the exposure between prior nonactionable alarm exposure and response time: 219 in the PICU and 428 on the ward. Kaplan‐Meier plots and tabulated response times demonstrated the incremental relationships between each nonactionable alarm exposure category in the observed data, with the effects most prominent as the Kaplan‐Meier plots diverged beyond the median (Figure 3 and Table 2). Excluding the extreme outlier patient had no effect on the results, because 1378 of the 1430 alarms occurred with the nurse present at the bedside, and only 2 of the remaining alarms were potentially critical.

Figure 3

Table 2.Association Between Nonactionable Alarm Exposure in Preceding 120 Minutes and Response Time to Potentially Critical Alarms Based on Observed Data and With Response Time Modeled Using Weibull Accelerated Failure‐Time Regression

	Observed Data					Accelerated Failure‐Time Model
	Number of Potentially Critical Alarms	Minutes Elapsed Until This Percentage of Alarms Was Responded to				Modeled Response Time, min	95% CI, min	P Value*
50% (Median)		75%	90%	95%
NOTE: Abbreviations: CI, confidence interval; PICU, pediatric intensive care unit. *The critical P value used as the cut point between significant and nonsignificant, accounting for multiple comparisons, is 0.0167.
PICU
029 nonactionable alarms	70	1.6	8.0	18.6	25.1	2.8	1.9‐3.8	Reference
3079 nonactionable alarms	122	6.3	17.8	22.5	26.0	5.3	4.06.7	0.001 (vs 029)
80+ nonactionable alarms	27	16.0	28.4	32.0	33.1	8.5	4.312.7	0.009 (vs 029), 0.15 (vs 3079)
Ward
029 nonactionable alarms	159	9.8	17.8	25.0	28.9	7.7	6.39.1	Reference
3079 nonactionable alarms	211	11.6	22.4	44.6	63.2	11.5	9.613.3	0.001 (vs 029)
80+ nonactionable alarms	58	8.3	57.6	63.8	69.5	15.6	11.020.1	0.001 (vs 029), 0.09 (vs 3079)

Accelerated failure‐time regressions revealed significant incremental increases in the modeled response time as the number of preceding nonactionable alarms increased in both the PICU and ward settings (Table 2).

DISCUSSION

Alarm fatigue has recently generated interest among nurses,[22] physicians,[23] regulatory bodies,[24] patient safety organizations,[25] and even attorneys,[26] despite a lack of prior evidence linking nonactionable alarm exposure to response time or other adverse patient‐relevant outcomes. This study's main findings were that (1) the vast majority of alarms were nonactionable, (2) response time to alarms occurring while the nurse was out of the room increased as the number of nonactionable alarms over the preceding 120 minutes increased. These findings may be explained by alarm fatigue.

Our results build upon the findings of other related studies. The nonactionable alarm proportions we found were similar to other pediatric studies, reporting greater than 90% nonactionable alarms.[1, 2] One other study has reported a relationship between alarm exposure and response time. In that study, Voepel‐Lewis and colleagues evaluated nurse responses to pulse oximetry desaturation alarms in adult orthopedic surgery patients using time‐stamp data from their monitor notification system.[27] They found that alarm response time was significantly longer for patients in the highest quartile of alarms compared to those in lower quartiles. Our study provides new data suggesting a relationship between nonactionable alarm exposure and nurse response time.

Our study has several limitations. First, as a preliminary study to investigate feasibility and possible association, the sample of patients and nurses was necessarily limited and did not permit adjustment for nurse‐ or patient‐level covariates. A multivariable analysis with a larger sample might provide insight into alternate explanations for these findings other than alarm fatigue, including measures of nurse workload and patient factors (such as age and illness severity). Additional factors that are not as easily measured can also contribute to the complex decision of when and how to respond to alarms.[28, 29] Second, nurses were aware that they were being video recorded as part of a study of nonactionable alarms, although they did not know the specific details of measurement. Although this lack of blinding might lead to a Hawthorne‐like effect, our positive results suggest that this effect, if present, did not fully obscure the association. Third, all sessions took place on weekdays during daytime hours, but effects of nonactionable alarms might vary by time and day. Finally, we suspect that when nurses experience critical alarms that require them to intervene and rescue a patient, their response times to that patient's alarms that occur later in their shift will be quicker due to a heightened concern for the alarm being actionable. We were unable to explore that relationship in this analysis because the number of critical alarms requiring intervention was very small. This is a topic of future study.

CONCLUSIONS

We identified an association between a nurse's prior exposure to nonactionable alarms and response time to future alarms. This finding is consistent with alarm fatigue, but requires further study to more clearly delineate other factors that might confound or modify that relationship.

Hospital physiologic monitors can alert clinicians to early signs of physiologic deterioration, and thus have great potential to save lives. However, monitors generate frequent alarms,[1, 2, 3, 4, 5, 6, 7, 8] and most are not relevant to the patient's safety (over 90% of pediatric intensive care unit (PICU)[1, 2] and over 70% of adult intensive care alarms).[5, 6] In psychology experiments, humans rapidly learn to ignore or respond more slowly to alarms when exposed to high false‐alarm rates, exhibiting alarm fatigue.[9, 10] In 2013, The Joint Commission named alarm fatigue the most common contributing factor to alarm‐related sentinel events in hospitals.[11, 12]

Although alarm fatigue has been implicated as a major threat to patient safety, little empirical data support its existence in hospitals. In this study, we aimed to determine if there was an association between nurses' recent exposure to nonactionable physiologic monitor alarms and their response time to future alarms for the same patients. This exploratory work was designed to inform future research in this area, acknowledging that the sample size would be too small for multivariable modeling.

METHODS

Study Definitions

The alarm classification scheme is shown in Figure 1. Note that, for clarity, we have intentionally avoided using the terms true and false alarms because their interpretations vary across studies and can be misleading.

Figure 1

Potentially Critical Alarm

A potentially critical alarm is any alarm for a clinical condition for which a timely response is important to determine if the alarm requires intervention to save the patient's life. This is based on the alarm type alone, including alarms for life‐threatening arrhythmias such as asystole and ventricular tachycardia, as well as alarms for vital signs outside the set limits. Supporting Table 1 in the online version of this article lists the breakdown of alarm types that we defined a priori as potentially and not potentially critical.

Table 1.Characteristics of the 2,445 Alarms for Clinical Conditions

	PICU				Ward
Alarm type	No.	% of Total	% Valid	% Actionable	No.	% of Total	% Valid	% Actionable
NOTE: Abbreviations: N/A, not applicable; PICU, pediatric intensive care unit.
Oxygen saturation	197	19.4	82.7	38.6	590	41.2	24.4	1.9
Heart rate	194	19.1	95.4	1.0	266	18.6	87.2	0.0
Respiratory rate	229	22.6	80.8	13.5	316	22.1	48.1	1.0
Blood pressure	259	25.5	83.8	5.8	11	0.8	72.7	0.0
Critical arrhythmia	1	0.1	0.0	0.0	4	0.3	0.0	0.0
Noncritical arrhythmia	71	7.0	2.8	0.0	244	17.1	8.6	0.0
Central venous pressure	49	4.8	0.0	0.0	0	0.0	N/A	N/A
Exhaled carbon dioxide	14	1.4	92.9	50.0	0	0.0	N/A	N/A
Total	1014	100.0	75.6	12.9	1,431	100.0	38.9	1.0

Valid Alarm

A valid alarm is any alarm that correctly identifies the physiologic status of the patient. Validity was based on waveform quality, lead signal strength indicators, and artifact conditions, referencing each monitor's operator's manual.

Actionable Alarm

An actionable alarm is any valid alarm for a clinical condition that either: (1) leads to a clinical intervention; (2) leads to a consultation with another clinician at the bedside (and thus visible on camera); or (3) is a situation that should have led to intervention or consultation, but the alarm was unwitnessed or misinterpreted by the staff at the bedside.

Nonactionable Alarm

An unactionable alarm is any alarm that does not meet the actionable definition above, including invalid alarms such as those caused by motion artifact, equipment/technical alarms, and alarms that are valid but nonactionable (nuisance alarms).[13]

Response Time

The response time is the time elapsed from when the alarm fired at the bedside to when the nurse entered the room or peered through a window or door, measured in seconds.

RESULTS

We performed 40 sessions among 40 patients and 36 nurses over 210 hours. We performed 20 sessions in children with heart and/or lung failure in the PICU and 20 sessions in children on a general ward. Sessions took place on weekdays between 9:00 _am and 6:00 _pm. There were 3 occasions when we filmed 2 patients cared for by the same nurse at the same time.

Nurses were mostly female (94.4%) and had between 2 months and 28 years of experience (median, 4.8 years). Patients on the ward ranged from 5 days to 5.4 years old (median, 6 months). Patients in the PICU ranged from 5 months to 16 years old (median, 2.5 years). Among the PICU patients, 14 (70%) were receiving mechanical ventilation only, 3 (15%) were receiving vasopressors only, and 3 (15%) were receiving mechanical ventilation and vasopressors.

We observed 5070 alarms during the 40 sessions. We excluded 108 (2.1%) that occurred at the end of video recording sessions with the nurse absent from the room because the nurse's response could not be determined. Alarms per session ranged from 10 to 1430 (median, 75; interquartile range [IQR], 35138). We excluded the outlier PICU patient with 1430 alarms in 5 hours from the analysis to avoid the potential for biasing the results. Figure 2 depicts the data flow.

Figure 2

Following the 5 training sessions, research assistants independently reviewed and made preliminary assessments on 4674 alarms; these alarms were all secondarily reviewed by a physician. Using the physician reviewer as the gold standard, the research assistant's sensitivity (assess alarm as actionable when physician also assesses as actionable) was 96.8% and specificity (assess alarm as nonactionable when physician also assesses as nonactionable) was 96.9%. We had to review 54 of 4674 alarms (1.2%) with an additional physician or nurse investigator to achieve consensus.

Characteristics of the 2445 alarms for clinical conditions are shown in Table 1. Only 12.9% of alarms in heart‐ and/or lung‐failure patients in the PICU were actionable, and only 1.0% of alarms in medical patients on a general inpatient ward were actionable.

Response‐Time Association With Nonactionable Alarm Exposure

Next, we analyzed the association between response time to potentially critical alarms that occurred when the nurse was not in the patient's room and the number of nonactionable alarms occurring over the preceding 120‐minute window. This required excluding the alarms that occurred in the first 120 minutes of each session, leaving 647 alarms with eligible response times to evaluate the exposure between prior nonactionable alarm exposure and response time: 219 in the PICU and 428 on the ward. Kaplan‐Meier plots and tabulated response times demonstrated the incremental relationships between each nonactionable alarm exposure category in the observed data, with the effects most prominent as the Kaplan‐Meier plots diverged beyond the median (Figure 3 and Table 2). Excluding the extreme outlier patient had no effect on the results, because 1378 of the 1430 alarms occurred with the nurse present at the bedside, and only 2 of the remaining alarms were potentially critical.

Figure 3

Table 2.Association Between Nonactionable Alarm Exposure in Preceding 120 Minutes and Response Time to Potentially Critical Alarms Based on Observed Data and With Response Time Modeled Using Weibull Accelerated Failure‐Time Regression

	Observed Data					Accelerated Failure‐Time Model
	Number of Potentially Critical Alarms	Minutes Elapsed Until This Percentage of Alarms Was Responded to				Modeled Response Time, min	95% CI, min	P Value*
50% (Median)		75%	90%	95%
NOTE: Abbreviations: CI, confidence interval; PICU, pediatric intensive care unit. *The critical P value used as the cut point between significant and nonsignificant, accounting for multiple comparisons, is 0.0167.
PICU
029 nonactionable alarms	70	1.6	8.0	18.6	25.1	2.8	1.9‐3.8	Reference
3079 nonactionable alarms	122	6.3	17.8	22.5	26.0	5.3	4.06.7	0.001 (vs 029)
80+ nonactionable alarms	27	16.0	28.4	32.0	33.1	8.5	4.312.7	0.009 (vs 029), 0.15 (vs 3079)
Ward
029 nonactionable alarms	159	9.8	17.8	25.0	28.9	7.7	6.39.1	Reference
3079 nonactionable alarms	211	11.6	22.4	44.6	63.2	11.5	9.613.3	0.001 (vs 029)
80+ nonactionable alarms	58	8.3	57.6	63.8	69.5	15.6	11.020.1	0.001 (vs 029), 0.09 (vs 3079)

Accelerated failure‐time regressions revealed significant incremental increases in the modeled response time as the number of preceding nonactionable alarms increased in both the PICU and ward settings (Table 2).

DISCUSSION

Alarm fatigue has recently generated interest among nurses,[22] physicians,[23] regulatory bodies,[24] patient safety organizations,[25] and even attorneys,[26] despite a lack of prior evidence linking nonactionable alarm exposure to response time or other adverse patient‐relevant outcomes. This study's main findings were that (1) the vast majority of alarms were nonactionable, (2) response time to alarms occurring while the nurse was out of the room increased as the number of nonactionable alarms over the preceding 120 minutes increased. These findings may be explained by alarm fatigue.

Our results build upon the findings of other related studies. The nonactionable alarm proportions we found were similar to other pediatric studies, reporting greater than 90% nonactionable alarms.[1, 2] One other study has reported a relationship between alarm exposure and response time. In that study, Voepel‐Lewis and colleagues evaluated nurse responses to pulse oximetry desaturation alarms in adult orthopedic surgery patients using time‐stamp data from their monitor notification system.[27] They found that alarm response time was significantly longer for patients in the highest quartile of alarms compared to those in lower quartiles. Our study provides new data suggesting a relationship between nonactionable alarm exposure and nurse response time.

Our study has several limitations. First, as a preliminary study to investigate feasibility and possible association, the sample of patients and nurses was necessarily limited and did not permit adjustment for nurse‐ or patient‐level covariates. A multivariable analysis with a larger sample might provide insight into alternate explanations for these findings other than alarm fatigue, including measures of nurse workload and patient factors (such as age and illness severity). Additional factors that are not as easily measured can also contribute to the complex decision of when and how to respond to alarms.[28, 29] Second, nurses were aware that they were being video recorded as part of a study of nonactionable alarms, although they did not know the specific details of measurement. Although this lack of blinding might lead to a Hawthorne‐like effect, our positive results suggest that this effect, if present, did not fully obscure the association. Third, all sessions took place on weekdays during daytime hours, but effects of nonactionable alarms might vary by time and day. Finally, we suspect that when nurses experience critical alarms that require them to intervene and rescue a patient, their response times to that patient's alarms that occur later in their shift will be quicker due to a heightened concern for the alarm being actionable. We were unable to explore that relationship in this analysis because the number of critical alarms requiring intervention was very small. This is a topic of future study.

CONCLUSIONS

We identified an association between a nurse's prior exposure to nonactionable alarms and response time to future alarms. This finding is consistent with alarm fatigue, but requires further study to more clearly delineate other factors that might confound or modify that relationship.

Hospital physiologic monitors can alert clinicians to early signs of physiologic deterioration, and thus have great potential to save lives. However, monitors generate frequent alarms,[1, 2, 3, 4, 5, 6, 7, 8] and most are not relevant to the patient's safety (over 90% of pediatric intensive care unit (PICU)[1, 2] and over 70% of adult intensive care alarms).[5, 6] In psychology experiments, humans rapidly learn to ignore or respond more slowly to alarms when exposed to high false‐alarm rates, exhibiting alarm fatigue.[9, 10] In 2013, The Joint Commission named alarm fatigue the most common contributing factor to alarm‐related sentinel events in hospitals.[11, 12]

Although alarm fatigue has been implicated as a major threat to patient safety, little empirical data support its existence in hospitals. In this study, we aimed to determine if there was an association between nurses' recent exposure to nonactionable physiologic monitor alarms and their response time to future alarms for the same patients. This exploratory work was designed to inform future research in this area, acknowledging that the sample size would be too small for multivariable modeling.

METHODS

Study Definitions

The alarm classification scheme is shown in Figure 1. Note that, for clarity, we have intentionally avoided using the terms true and false alarms because their interpretations vary across studies and can be misleading.

Figure 1

Potentially Critical Alarm

A potentially critical alarm is any alarm for a clinical condition for which a timely response is important to determine if the alarm requires intervention to save the patient's life. This is based on the alarm type alone, including alarms for life‐threatening arrhythmias such as asystole and ventricular tachycardia, as well as alarms for vital signs outside the set limits. Supporting Table 1 in the online version of this article lists the breakdown of alarm types that we defined a priori as potentially and not potentially critical.

Table 1.Characteristics of the 2,445 Alarms for Clinical Conditions

	PICU				Ward
Alarm type	No.	% of Total	% Valid	% Actionable	No.	% of Total	% Valid	% Actionable
NOTE: Abbreviations: N/A, not applicable; PICU, pediatric intensive care unit.
Oxygen saturation	197	19.4	82.7	38.6	590	41.2	24.4	1.9
Heart rate	194	19.1	95.4	1.0	266	18.6	87.2	0.0
Respiratory rate	229	22.6	80.8	13.5	316	22.1	48.1	1.0
Blood pressure	259	25.5	83.8	5.8	11	0.8	72.7	0.0
Critical arrhythmia	1	0.1	0.0	0.0	4	0.3	0.0	0.0
Noncritical arrhythmia	71	7.0	2.8	0.0	244	17.1	8.6	0.0
Central venous pressure	49	4.8	0.0	0.0	0	0.0	N/A	N/A
Exhaled carbon dioxide	14	1.4	92.9	50.0	0	0.0	N/A	N/A
Total	1014	100.0	75.6	12.9	1,431	100.0	38.9	1.0

Valid Alarm

A valid alarm is any alarm that correctly identifies the physiologic status of the patient. Validity was based on waveform quality, lead signal strength indicators, and artifact conditions, referencing each monitor's operator's manual.

Actionable Alarm

An actionable alarm is any valid alarm for a clinical condition that either: (1) leads to a clinical intervention; (2) leads to a consultation with another clinician at the bedside (and thus visible on camera); or (3) is a situation that should have led to intervention or consultation, but the alarm was unwitnessed or misinterpreted by the staff at the bedside.

Nonactionable Alarm

An unactionable alarm is any alarm that does not meet the actionable definition above, including invalid alarms such as those caused by motion artifact, equipment/technical alarms, and alarms that are valid but nonactionable (nuisance alarms).[13]

Response Time

The response time is the time elapsed from when the alarm fired at the bedside to when the nurse entered the room or peered through a window or door, measured in seconds.

RESULTS

We performed 40 sessions among 40 patients and 36 nurses over 210 hours. We performed 20 sessions in children with heart and/or lung failure in the PICU and 20 sessions in children on a general ward. Sessions took place on weekdays between 9:00 _am and 6:00 _pm. There were 3 occasions when we filmed 2 patients cared for by the same nurse at the same time.

Nurses were mostly female (94.4%) and had between 2 months and 28 years of experience (median, 4.8 years). Patients on the ward ranged from 5 days to 5.4 years old (median, 6 months). Patients in the PICU ranged from 5 months to 16 years old (median, 2.5 years). Among the PICU patients, 14 (70%) were receiving mechanical ventilation only, 3 (15%) were receiving vasopressors only, and 3 (15%) were receiving mechanical ventilation and vasopressors.

We observed 5070 alarms during the 40 sessions. We excluded 108 (2.1%) that occurred at the end of video recording sessions with the nurse absent from the room because the nurse's response could not be determined. Alarms per session ranged from 10 to 1430 (median, 75; interquartile range [IQR], 35138). We excluded the outlier PICU patient with 1430 alarms in 5 hours from the analysis to avoid the potential for biasing the results. Figure 2 depicts the data flow.

Figure 2

Following the 5 training sessions, research assistants independently reviewed and made preliminary assessments on 4674 alarms; these alarms were all secondarily reviewed by a physician. Using the physician reviewer as the gold standard, the research assistant's sensitivity (assess alarm as actionable when physician also assesses as actionable) was 96.8% and specificity (assess alarm as nonactionable when physician also assesses as nonactionable) was 96.9%. We had to review 54 of 4674 alarms (1.2%) with an additional physician or nurse investigator to achieve consensus.

Characteristics of the 2445 alarms for clinical conditions are shown in Table 1. Only 12.9% of alarms in heart‐ and/or lung‐failure patients in the PICU were actionable, and only 1.0% of alarms in medical patients on a general inpatient ward were actionable.

Response‐Time Association With Nonactionable Alarm Exposure

Next, we analyzed the association between response time to potentially critical alarms that occurred when the nurse was not in the patient's room and the number of nonactionable alarms occurring over the preceding 120‐minute window. This required excluding the alarms that occurred in the first 120 minutes of each session, leaving 647 alarms with eligible response times to evaluate the exposure between prior nonactionable alarm exposure and response time: 219 in the PICU and 428 on the ward. Kaplan‐Meier plots and tabulated response times demonstrated the incremental relationships between each nonactionable alarm exposure category in the observed data, with the effects most prominent as the Kaplan‐Meier plots diverged beyond the median (Figure 3 and Table 2). Excluding the extreme outlier patient had no effect on the results, because 1378 of the 1430 alarms occurred with the nurse present at the bedside, and only 2 of the remaining alarms were potentially critical.

Figure 3

Table 2.Association Between Nonactionable Alarm Exposure in Preceding 120 Minutes and Response Time to Potentially Critical Alarms Based on Observed Data and With Response Time Modeled Using Weibull Accelerated Failure‐Time Regression

	Observed Data					Accelerated Failure‐Time Model
	Number of Potentially Critical Alarms	Minutes Elapsed Until This Percentage of Alarms Was Responded to				Modeled Response Time, min	95% CI, min	P Value*
50% (Median)		75%	90%	95%
NOTE: Abbreviations: CI, confidence interval; PICU, pediatric intensive care unit. *The critical P value used as the cut point between significant and nonsignificant, accounting for multiple comparisons, is 0.0167.
PICU
029 nonactionable alarms	70	1.6	8.0	18.6	25.1	2.8	1.9‐3.8	Reference
3079 nonactionable alarms	122	6.3	17.8	22.5	26.0	5.3	4.06.7	0.001 (vs 029)
80+ nonactionable alarms	27	16.0	28.4	32.0	33.1	8.5	4.312.7	0.009 (vs 029), 0.15 (vs 3079)
Ward
029 nonactionable alarms	159	9.8	17.8	25.0	28.9	7.7	6.39.1	Reference
3079 nonactionable alarms	211	11.6	22.4	44.6	63.2	11.5	9.613.3	0.001 (vs 029)
80+ nonactionable alarms	58	8.3	57.6	63.8	69.5	15.6	11.020.1	0.001 (vs 029), 0.09 (vs 3079)

Accelerated failure‐time regressions revealed significant incremental increases in the modeled response time as the number of preceding nonactionable alarms increased in both the PICU and ward settings (Table 2).

DISCUSSION

Alarm fatigue has recently generated interest among nurses,[22] physicians,[23] regulatory bodies,[24] patient safety organizations,[25] and even attorneys,[26] despite a lack of prior evidence linking nonactionable alarm exposure to response time or other adverse patient‐relevant outcomes. This study's main findings were that (1) the vast majority of alarms were nonactionable, (2) response time to alarms occurring while the nurse was out of the room increased as the number of nonactionable alarms over the preceding 120 minutes increased. These findings may be explained by alarm fatigue.

Our results build upon the findings of other related studies. The nonactionable alarm proportions we found were similar to other pediatric studies, reporting greater than 90% nonactionable alarms.[1, 2] One other study has reported a relationship between alarm exposure and response time. In that study, Voepel‐Lewis and colleagues evaluated nurse responses to pulse oximetry desaturation alarms in adult orthopedic surgery patients using time‐stamp data from their monitor notification system.[27] They found that alarm response time was significantly longer for patients in the highest quartile of alarms compared to those in lower quartiles. Our study provides new data suggesting a relationship between nonactionable alarm exposure and nurse response time.

Our study has several limitations. First, as a preliminary study to investigate feasibility and possible association, the sample of patients and nurses was necessarily limited and did not permit adjustment for nurse‐ or patient‐level covariates. A multivariable analysis with a larger sample might provide insight into alternate explanations for these findings other than alarm fatigue, including measures of nurse workload and patient factors (such as age and illness severity). Additional factors that are not as easily measured can also contribute to the complex decision of when and how to respond to alarms.[28, 29] Second, nurses were aware that they were being video recorded as part of a study of nonactionable alarms, although they did not know the specific details of measurement. Although this lack of blinding might lead to a Hawthorne‐like effect, our positive results suggest that this effect, if present, did not fully obscure the association. Third, all sessions took place on weekdays during daytime hours, but effects of nonactionable alarms might vary by time and day. Finally, we suspect that when nurses experience critical alarms that require them to intervene and rescue a patient, their response times to that patient's alarms that occur later in their shift will be quicker due to a heightened concern for the alarm being actionable. We were unable to explore that relationship in this analysis because the number of critical alarms requiring intervention was very small. This is a topic of future study.

CONCLUSIONS

We identified an association between a nurse's prior exposure to nonactionable alarms and response time to future alarms. This finding is consistent with alarm fatigue, but requires further study to more clearly delineate other factors that might confound or modify that relationship.

Patient flow refers to the management and movement of patients in health care settings and is linked to quality, safety, and cost.16 The intensive care unit (ICU) is crucial in patient flow.7, 8 The limited number of beds and the resource‐intensive services and staffing associated with them require that hospitals optimize their utilization, as is increasingly true of all hospital resources. To maximize delivery of services to patients who need them and minimize real and opportunity losses (eg, postponed surgery, diverted transports, or inability to accept patients), patients in ICU beds should receive critical care medicine/nursing services while there and be transferred or discharged when appropriate.

The time between arrival and departure from any area of the hospital, including the ICU, is considered the time when a patient is receiving needed clinical carethe value‐added portion of health care operationsand time waiting to move on to the next step.911 This period includes both necessary logistics (eg, signing out a patient or waiting a reasonable amount of time for room cleaning) and nonvalue‐added time (eg, an excessively long amount of time for room cleaning). Operations management labels nonvalue‐added time as waste, and its reduction is vital for high‐quality health care.9, 12, 13 As in other industries, one important way to understand value versus waste is through direct observation.11, 14 Although operating rooms have been the subject of several published process improvement projects to improve efficiency,1518 inpatient beds have not been the subject of such scrutiny. The objectives of this study were to generate a direct observation method and use it to describe pediatric ICU (PICU) bed utilization from a value‐added perspective.

METHODS

An interdisciplinary work group of physicians, nurses, quality improvement specialists, and 1 operations management expert developed an Excel spreadsheet to categorize hour‐by‐hour status of PICU beds. The clinicians generated a list of 27 activities. A critical care nurse trained in quality improvement piloted the list for 3 separate 4‐hour blocks over 2 weeks adding 18 activities; 2 additional activities were added during the 5 weeks of observation (Table 1). (The recording tool is provided in the Supporting Information Appendix.) Three observers with knowledge of medical terminology (2 third‐year medical students and 1 premedical student with years of experience as an emergency medical technician) were trained over 12 hours to conduct the observations. Prior to the observations, the 3 observers also spent time in the PICU, and terminology used for recordings was reviewed. Interobserver reliability was checked during 3 sets of observation circuits by all 3 observers and the principal investigator, as well as by spot checks during the study.

The targeted area included 24 single‐patient rooms. The activity of each bed was recorded hourly. Real‐time recording in to the Excel spreadsheet on a dedicated laptop occurred from 8:00 _AM until 11:00 _PM. The most visible or critical event was recorded. Although some activities were not mutually exclusive (eg, a patient could be ventilated and on a continuous infusion simultaneously), the objective was to identify when a room was being used for any critical care service, not enumerate all of them. The observers noted overnight events that occurred from 11:00 _PM to 8:00 _AM in the morning by reviewing the bedside record and talking to the staff to complete each day's 24‐hour recording. The observers also recorded the hospital‐wide census and the census for the other half of the PICU every 4 hours. The observations occurred over 5 noncontiguous weeks between January 2009 and April 2009.

After all observations were complete, activities were classified as critical care services (CCS) or noncritical‐care services (NCCS). NCCSs were further divided into necessary logistics (defined for analysis purposes as the first hour of any NCCS activity) or nonvalue‐added (the second or greater hour of NCCS). A time limit of 1 hour was chosen to define necessary logistics based on a consensus that nonclinical activities optimally would not take more than 1 hour each. We also analyzed results with 2 hours as the cutoff for necessary logistics. Admission, discharge, and transfer records were reviewed to check for returns to the PICU or hospital within 48 hours of transfer or discharge from the PICU.

Analyses were conducted using Microsoft Excel (Microsoft, Redmond, WA) and Stata 10.0 (StataCorp, College Station, TX). The study was approved by the Children's Hospital of Philadelphia Institutional Review Board with waiver of consent.

RESULTS

A total of 824 hours of recordings included 19,887 bed‐hours with 219 unique patients; among them, 2 remained from the first day of recording in January to the last day in April (sample recording in Figure 1). A total of 50 patients (range, 812 per week) stayed for the entirety of each 1‐week observation period. Of the 47 possible activities, 45 of them were recorded for at least 1 hour in the 5 weeks. Overall, 14 activities accounted for 95% of the observed bed‐hours and 31 activities accounted for the remaining 5%. CCS accounted for 82% of observed bed‐hours, NCCS accounted for 10.4%, and empty unassigned accounted for 8% (Figure 2). Using the 1‐hour cutoff for necessary services, 77% of NCCS time was nonvalue‐added, whereas 23% of it was necessary logistics; using the 2‐hour cutoff, 54% was nonvalue‐added, and 46% was necessary logistics.

Figure 1

Figure 2

During the observation period, <1% of bed‐hours were used for CCS for overflow patients from the neonatal ICU (NICU), cardiac care unit (CCU), cardiac ICU (CICU), or progressive care unit (PCU; tracheostomy/ventilator unit). Although only 4 patients required transport to a rehabilitation facility, their wait time comprised 99 hours (<1%) of total recordings. Eight patients waited a mean of 2.6 hours for transportation home (maximum, 10 hours).

To demonstrate the cycle of room use, activities were divided into 4 categories: room preparation, critical care services, disposition pending, and postcritical care services (Figure 3). As an example of detailed data revealed by direct observation, we identified 102 instances totaling 919 hours when a patient was waiting for a bed assignment on another floor (5% of all bed‐hours). The mean wait time was 9 hours (range, 188 hours) and the median time was 5.5 hours. There were only 15 instances when floor bed assignment took 1 hour or less, and only 9 instances when it took 12 hours. Similarly, considerable time was spent on cleaning rooms between patients: only 66 of 146 instances of cleaning took 1 hour or less. The mean time for cleaning was 2.2 hours (range, 115), and the median was 2 hours. (There were 136 recorded instances of room cleaning and 10 additional episodes that were not recorded but had to be completed for the room to turnover from one patient to the next, yielding a total of 146 instances of cleaning.)

Figure 3

From the 824 hours of recording, we identified 200 hours (25% of time) when there were zero empty unassigned beds available in the section of the PICU being observed. Episodes of full occupancy occurred mostly on weekdays, with 23% of hours of full capacity on Thursdays, 21% on Mondays, and 21% on Wednesdays; only 8% were on Saturdays and <1% on Sundays. These 200 hours fell into 36 separate episodes of complete occupancy, each lasting 122 hours. Each patient, on average, received 3.1 hours of NCCS during each episode of full occupancy (range, 111 hours). Within these 200 hours at capacity, we identified only 15 hours (8%) when all 24 beds were used for CCS. For 72% of the time, there was at least 1 bed with NCCS, and for 37% at least 2 beds. A small portion of the time (7%), the lack of beds was affected by occupancy by patients who should have been in the NICU, CICU, CCU, or PCU.

Data collected through direct observation can be used to understand aggregated and averaged experiences, but also more specific time periods. For example, we identified 1 week with the highest consistent level of occupancy and turnover: March 915 had empty unassigned beds for only 4% of the week. Of the 168 hours in the week, 68 (40%) had full capacity. However, for 90% of the time, at least 1 bed was used for a NCCS. Other analytic options included varying the assumptions around time needed for logistics. Overall, NCCS time on necessary logistics changes from 23% to 46% using 1 hour versus 2 hours as the cutoff. For floor bed assignments, assuming that the first hour of this activity is necessary logistics and any hour thereafter is not, 817 hours were wasted. Even after assuming 2 hours of necessary logistical time (which may also include steps such as nursing and physician sign‐out to the receiving team, often not recorded in the observations), this left 715 hours of NCCS time in which patients waited to be placed elsewhere in the hospital. For room cleaning, because recordings were hourly, but room cleaning could take less time, we performed a sensitivity analysis, converting all 1‐hour recordings to half‐hour recordings to half‐hour recordings (an exaggerated shortening since industry‐standard cleaning may take longer).

Of the 219 patients directly observed, 15 were noted to be waiting for a transfer out of the PICU but experienced a change in disposition before the transfer. On average, these patients waited 8 hours for a floor bed assignment (range, 221) before reverting to a CCS, which then lasted an average of 16.5 hours (range, 149). (Included in this group are 2 patients who experienced this change in disposition twice.) In post hoc review across the 5 weeks, no patients were transferred back to the PICU within 48 hours after being transferred out. During the study period, 19 patients were discharged directly from the PICU (8 to home, 7 by transport to another facility, and 4 to rehabilitation). One patient returned to the hospital (but not the PICU) within 48 hours of being discharged home from the PICU.

During the study period, using the highest census value for recorded for each 24‐hour period and the number of beds available that day, median hospital‐wide occupancy was 93% (interquartile range, 90%96%). During the 35 days of observation, 71% of the days had occupancy >90%, 29% of days had occupancy >95%, and 3% of days had occupancy >100%.

DISCUSSION

In this direct observation of a PICU, we found high usage of beds for delivery of CCS. We identified many episodes in which the half of the PICU we observed was fully occupied (200 of 824 hours), but not necessarily delivering PICU‐level care to all patients. In fact, 75% of the full‐capacity hours had at least 1 patient receiving NCCS and 37% had at least 2. Patients waiting for a floor bed assignment represented nearly 5% of bed‐hours observed (mean 9 hours per patient). That full occupancy was not random, but rather clustered on weekdays, is consistent with other work showing that hospitals are at greater risk for midweek crowding due to the way in which scheduled admissions enter and leave.1925

Our methods provide the basis for operational analysis and improvement to patient flow, such as value stream mapping.9, 26 Process improvement work could be directed to areas of delay uncovered through this analysis and inform clinical and nonclinical management. For example, one of the key problems faced by the PICU was finding floor bed assignments for patients leaving the unit. Simply building more beds in the PICU will not solve this problemand at an estimated cost of $2 million to add a bed, it is likely not an efficient means of responding to poor flow. In these cases, the problem seems to lie downstream, and could suggest shortage of regular floor beds or inefficient bed assignment procedures within the hospital. The output also suggests that variation in nonclinical processes should be a target for improvement, such as time to clean rooms, because variation is known to be a source of nonvalue‐added time in many operations.9, 26 High occupancy on weekdays but low occupancy on weekends also emphasizes the potential for smoothing occupancy to reduce the risk of midweek crowding and to better manage bed utilization and staffing.24, 25

When seeking to reduce nonvalue‐added time, one must weigh the risks of increased efficiency against clinical outcomes. For example, if patients could be transferred out of the PICU faster, would the risk of returns to the PICU be higher? In this study, 15 patients (7%) had a change in disposition from awaiting transfer back to a CCS. The fact that transfers did not happen instantaneously may serve as a safety check to reduce rapid returns, but it is not possible for us to evaluate the reasons why patients did not actually complete the pending transfers. Specifically, we cannot determine whether the patient's clinical status objectively deteriorated, the ICU team made a judgment call to hold the patient, or the floor team refused to accept the transfer. Given this fact, although it appears in this study (and in the health care system more broadly) that there are opportunities to increase efficiency and reduce nonvalue‐added time, it is not realistic (nor advisable) that such time be reduced to zero. Along this line, one must consider separately purely nonclinical functions such as room cleaning and those that include some clinical element, such as time waiting for a patient to be transferred.

Beyond the direct findings of this study, the method should be replicable in other settings and can reveal important information about health care efficiency, capacity, and flexibility. The bottlenecks identified would have been difficult to identify through administrative record review. The exact amount of time to spend on observation may vary from place to place and would depend on the expected variation over time and the level of detail sought. In general, the more common the event and the less variation, the less time needed to observe it.

This study has several limitations that should be considered in terms of interpreting the results and in seeking to reproduce the approach. First, hourly recordings may not be discrete enough for events that took less than 1 hour. To assess the degree to which this would affect our results, we reanalyzed all NCCS by subtracting 30 minutes (0.5 hour) from all recordings, which increased total CCS from 82% to 87% and decreased NCCS by the same 5 percentage points. In a related fashion, our recordings were truncated at the start and end of each 1‐week period, so we could only observe a maximum of 168 hours for any given activity and did not record how long an activity was happening before or after the recordings started or stopped, respectively. Second, each recording could only be for 1 activity per hour. Separate from the level of granularity already noted, this also limits interpretation of critical care activities that may have been simultaneous. However, because the goal of the study was not to describe the provision of critical care services, but rather the times when they were not being delivered, this does not influence our conclusions. For movement of patients, however, we missed instances of physician and nursing calling sign‐out on patients to receiving units, as these events last less than 1 hour (and in the case of surgical patients, generally do not occur as the team provides continuous coverage). The time for such events is then included in other activities. To the extent that this may influence the results, it would increase the perceived time for nonvalue‐added services, but to a limited degree, and never by more than 59 minutes. Third, the overnight hours (11:00 _PM to 8:00 _AM) were not directly observed, but retrospectively recorded each morning by reviewing the records and discussing the overnight events with the clinical staff. For example, if a patient was intubated at 11:00 _PM and at 8:00 _AM, the observer would confirm this and record that status for the intervening hours. This is unlikely to result in a substantial impact on the findings, because the overnight hours have a relative degree of stability even for unstable patients in terms of their status of needing or not needing a CCS. Fourth, we did not evaluate the appropriateness of CCS delivered (eg, how long a patient was ventilated). Our definitions for CCS and NCSS were based on Children's Hospital of Philadelphia practices, which may not be the same as those of other facilities. The categorization of CCS was objective for activities such as ventilation or continuous infusion, but was less clear for the not otherwise specified recordings, which represented patients with a complex illness or projected organ, respiratory, cardiac, or neurological failure. These patients were not receiving a specific critical care intervention, but were deemed to need to be in the PICU as opposed to a regular floor (eg, for frequent monitoring of potential respiratory failure). It would also include patients receiving combinations of therapies more efficiently delivered in the PICU. For that, the observers relied on the judgment of clinicians (primarily nurses) to determine whether the patient needed to be in the PICU or not; if no specific reason could be provided, not otherwise specified was applied. These 192 instances accounted for 2982 aggregate bed‐hours (15% of total). It is difficult to judge the direction of bias, because overestimation of need to be in the PICU may be as likely to occur as underestimation. Fifth, the very presence of the observers may have changed behavior. Knowing that they were being observed staff may have acted with greater efficiency than otherwise. We expect that such a finding would lead to less time appearing as necessary logistics or NCCS. Finally, results may not be generalizable to other hospitals or hospital settings. There are clearly important contextual factors, not only for the location but also for the duration. For example, staffing was never an issue during the 5 weeks of observation, but there are locations where an empty bed is not necessarily usable due to lack of staffing. Nonetheless, we believe the results provide a generalizable approach and methodology for other settings (and staffing could be a reason for an empty bed).

In terms of the setting, as noted, we observed one discrete 24‐bed unit, which comprises half of the total PICU. Thus, statements that the PICU was at full capacity must be interpreted in the context that additional rooms may have been available on the other side. Patients are generally admitted alternately to each unit, so the occupancies should parallel each other. We recorded the census every 4 hours for both sides from the electronic system (Sunrise Clinical Manager [SCM]). However, this only accounts for patients physically in beds, not beds held for patients in other locations. Thus, we would expect a discrepancy between direct observation and the SCM value. Through analysis of the entire pediatric intensive care unit,* that part which observed directly, and that which we did not observe directly using census data, we think it reasonable to assert that both units of the total PICU had constrained capacity during the times we directly observed and recorded such constraint on one side.

This study demonstrates the use of direct observation for inpatient settings to learn about resource utilization and identification of value‐added services. PubMed searches for the terms efficiency, flow, process redesign, and time management bring up many more references for operating rooms than for ICUs or inpatient beds. Some examples of ICU‐directed work include videography of an ICU in Australia27 and human factor analysis in ICU nursing.5 Time‐motion studies have also been conducted on clinical staff, such as physicians.28, 29

In conclusion, we found that direct observation provided important insights into the utilization of patient rooms in an important inpatient setting. Data such as these are valuable for clinical and process improvement work, as well as understanding how best to match capacity to patient need. Finally, the methodology is reproducible for other settings and would be an additional tool to measuring and improving the efficiency and value of the health system. When appropriate, this approach can also evaluate the effectiveness of process improvement, help identify and reduce waste,13 and contribute to the growing field that merges operations management with hospital administration and clinical care: in other words, evidence‐based management.30

Patient flow refers to the management and movement of patients in health care settings and is linked to quality, safety, and cost.16 The intensive care unit (ICU) is crucial in patient flow.7, 8 The limited number of beds and the resource‐intensive services and staffing associated with them require that hospitals optimize their utilization, as is increasingly true of all hospital resources. To maximize delivery of services to patients who need them and minimize real and opportunity losses (eg, postponed surgery, diverted transports, or inability to accept patients), patients in ICU beds should receive critical care medicine/nursing services while there and be transferred or discharged when appropriate.

The time between arrival and departure from any area of the hospital, including the ICU, is considered the time when a patient is receiving needed clinical carethe value‐added portion of health care operationsand time waiting to move on to the next step.911 This period includes both necessary logistics (eg, signing out a patient or waiting a reasonable amount of time for room cleaning) and nonvalue‐added time (eg, an excessively long amount of time for room cleaning). Operations management labels nonvalue‐added time as waste, and its reduction is vital for high‐quality health care.9, 12, 13 As in other industries, one important way to understand value versus waste is through direct observation.11, 14 Although operating rooms have been the subject of several published process improvement projects to improve efficiency,1518 inpatient beds have not been the subject of such scrutiny. The objectives of this study were to generate a direct observation method and use it to describe pediatric ICU (PICU) bed utilization from a value‐added perspective.

METHODS

An interdisciplinary work group of physicians, nurses, quality improvement specialists, and 1 operations management expert developed an Excel spreadsheet to categorize hour‐by‐hour status of PICU beds. The clinicians generated a list of 27 activities. A critical care nurse trained in quality improvement piloted the list for 3 separate 4‐hour blocks over 2 weeks adding 18 activities; 2 additional activities were added during the 5 weeks of observation (Table 1). (The recording tool is provided in the Supporting Information Appendix.) Three observers with knowledge of medical terminology (2 third‐year medical students and 1 premedical student with years of experience as an emergency medical technician) were trained over 12 hours to conduct the observations. Prior to the observations, the 3 observers also spent time in the PICU, and terminology used for recordings was reviewed. Interobserver reliability was checked during 3 sets of observation circuits by all 3 observers and the principal investigator, as well as by spot checks during the study.

The targeted area included 24 single‐patient rooms. The activity of each bed was recorded hourly. Real‐time recording in to the Excel spreadsheet on a dedicated laptop occurred from 8:00 _AM until 11:00 _PM. The most visible or critical event was recorded. Although some activities were not mutually exclusive (eg, a patient could be ventilated and on a continuous infusion simultaneously), the objective was to identify when a room was being used for any critical care service, not enumerate all of them. The observers noted overnight events that occurred from 11:00 _PM to 8:00 _AM in the morning by reviewing the bedside record and talking to the staff to complete each day's 24‐hour recording. The observers also recorded the hospital‐wide census and the census for the other half of the PICU every 4 hours. The observations occurred over 5 noncontiguous weeks between January 2009 and April 2009.

After all observations were complete, activities were classified as critical care services (CCS) or noncritical‐care services (NCCS). NCCSs were further divided into necessary logistics (defined for analysis purposes as the first hour of any NCCS activity) or nonvalue‐added (the second or greater hour of NCCS). A time limit of 1 hour was chosen to define necessary logistics based on a consensus that nonclinical activities optimally would not take more than 1 hour each. We also analyzed results with 2 hours as the cutoff for necessary logistics. Admission, discharge, and transfer records were reviewed to check for returns to the PICU or hospital within 48 hours of transfer or discharge from the PICU.

Analyses were conducted using Microsoft Excel (Microsoft, Redmond, WA) and Stata 10.0 (StataCorp, College Station, TX). The study was approved by the Children's Hospital of Philadelphia Institutional Review Board with waiver of consent.

RESULTS

A total of 824 hours of recordings included 19,887 bed‐hours with 219 unique patients; among them, 2 remained from the first day of recording in January to the last day in April (sample recording in Figure 1). A total of 50 patients (range, 812 per week) stayed for the entirety of each 1‐week observation period. Of the 47 possible activities, 45 of them were recorded for at least 1 hour in the 5 weeks. Overall, 14 activities accounted for 95% of the observed bed‐hours and 31 activities accounted for the remaining 5%. CCS accounted for 82% of observed bed‐hours, NCCS accounted for 10.4%, and empty unassigned accounted for 8% (Figure 2). Using the 1‐hour cutoff for necessary services, 77% of NCCS time was nonvalue‐added, whereas 23% of it was necessary logistics; using the 2‐hour cutoff, 54% was nonvalue‐added, and 46% was necessary logistics.

Figure 1

Figure 2

During the observation period, <1% of bed‐hours were used for CCS for overflow patients from the neonatal ICU (NICU), cardiac care unit (CCU), cardiac ICU (CICU), or progressive care unit (PCU; tracheostomy/ventilator unit). Although only 4 patients required transport to a rehabilitation facility, their wait time comprised 99 hours (<1%) of total recordings. Eight patients waited a mean of 2.6 hours for transportation home (maximum, 10 hours).

To demonstrate the cycle of room use, activities were divided into 4 categories: room preparation, critical care services, disposition pending, and postcritical care services (Figure 3). As an example of detailed data revealed by direct observation, we identified 102 instances totaling 919 hours when a patient was waiting for a bed assignment on another floor (5% of all bed‐hours). The mean wait time was 9 hours (range, 188 hours) and the median time was 5.5 hours. There were only 15 instances when floor bed assignment took 1 hour or less, and only 9 instances when it took 12 hours. Similarly, considerable time was spent on cleaning rooms between patients: only 66 of 146 instances of cleaning took 1 hour or less. The mean time for cleaning was 2.2 hours (range, 115), and the median was 2 hours. (There were 136 recorded instances of room cleaning and 10 additional episodes that were not recorded but had to be completed for the room to turnover from one patient to the next, yielding a total of 146 instances of cleaning.)

Figure 3

From the 824 hours of recording, we identified 200 hours (25% of time) when there were zero empty unassigned beds available in the section of the PICU being observed. Episodes of full occupancy occurred mostly on weekdays, with 23% of hours of full capacity on Thursdays, 21% on Mondays, and 21% on Wednesdays; only 8% were on Saturdays and <1% on Sundays. These 200 hours fell into 36 separate episodes of complete occupancy, each lasting 122 hours. Each patient, on average, received 3.1 hours of NCCS during each episode of full occupancy (range, 111 hours). Within these 200 hours at capacity, we identified only 15 hours (8%) when all 24 beds were used for CCS. For 72% of the time, there was at least 1 bed with NCCS, and for 37% at least 2 beds. A small portion of the time (7%), the lack of beds was affected by occupancy by patients who should have been in the NICU, CICU, CCU, or PCU.

Data collected through direct observation can be used to understand aggregated and averaged experiences, but also more specific time periods. For example, we identified 1 week with the highest consistent level of occupancy and turnover: March 915 had empty unassigned beds for only 4% of the week. Of the 168 hours in the week, 68 (40%) had full capacity. However, for 90% of the time, at least 1 bed was used for a NCCS. Other analytic options included varying the assumptions around time needed for logistics. Overall, NCCS time on necessary logistics changes from 23% to 46% using 1 hour versus 2 hours as the cutoff. For floor bed assignments, assuming that the first hour of this activity is necessary logistics and any hour thereafter is not, 817 hours were wasted. Even after assuming 2 hours of necessary logistical time (which may also include steps such as nursing and physician sign‐out to the receiving team, often not recorded in the observations), this left 715 hours of NCCS time in which patients waited to be placed elsewhere in the hospital. For room cleaning, because recordings were hourly, but room cleaning could take less time, we performed a sensitivity analysis, converting all 1‐hour recordings to half‐hour recordings to half‐hour recordings (an exaggerated shortening since industry‐standard cleaning may take longer).

Of the 219 patients directly observed, 15 were noted to be waiting for a transfer out of the PICU but experienced a change in disposition before the transfer. On average, these patients waited 8 hours for a floor bed assignment (range, 221) before reverting to a CCS, which then lasted an average of 16.5 hours (range, 149). (Included in this group are 2 patients who experienced this change in disposition twice.) In post hoc review across the 5 weeks, no patients were transferred back to the PICU within 48 hours after being transferred out. During the study period, 19 patients were discharged directly from the PICU (8 to home, 7 by transport to another facility, and 4 to rehabilitation). One patient returned to the hospital (but not the PICU) within 48 hours of being discharged home from the PICU.

During the study period, using the highest census value for recorded for each 24‐hour period and the number of beds available that day, median hospital‐wide occupancy was 93% (interquartile range, 90%96%). During the 35 days of observation, 71% of the days had occupancy >90%, 29% of days had occupancy >95%, and 3% of days had occupancy >100%.

DISCUSSION

In this direct observation of a PICU, we found high usage of beds for delivery of CCS. We identified many episodes in which the half of the PICU we observed was fully occupied (200 of 824 hours), but not necessarily delivering PICU‐level care to all patients. In fact, 75% of the full‐capacity hours had at least 1 patient receiving NCCS and 37% had at least 2. Patients waiting for a floor bed assignment represented nearly 5% of bed‐hours observed (mean 9 hours per patient). That full occupancy was not random, but rather clustered on weekdays, is consistent with other work showing that hospitals are at greater risk for midweek crowding due to the way in which scheduled admissions enter and leave.1925

Our methods provide the basis for operational analysis and improvement to patient flow, such as value stream mapping.9, 26 Process improvement work could be directed to areas of delay uncovered through this analysis and inform clinical and nonclinical management. For example, one of the key problems faced by the PICU was finding floor bed assignments for patients leaving the unit. Simply building more beds in the PICU will not solve this problemand at an estimated cost of $2 million to add a bed, it is likely not an efficient means of responding to poor flow. In these cases, the problem seems to lie downstream, and could suggest shortage of regular floor beds or inefficient bed assignment procedures within the hospital. The output also suggests that variation in nonclinical processes should be a target for improvement, such as time to clean rooms, because variation is known to be a source of nonvalue‐added time in many operations.9, 26 High occupancy on weekdays but low occupancy on weekends also emphasizes the potential for smoothing occupancy to reduce the risk of midweek crowding and to better manage bed utilization and staffing.24, 25

When seeking to reduce nonvalue‐added time, one must weigh the risks of increased efficiency against clinical outcomes. For example, if patients could be transferred out of the PICU faster, would the risk of returns to the PICU be higher? In this study, 15 patients (7%) had a change in disposition from awaiting transfer back to a CCS. The fact that transfers did not happen instantaneously may serve as a safety check to reduce rapid returns, but it is not possible for us to evaluate the reasons why patients did not actually complete the pending transfers. Specifically, we cannot determine whether the patient's clinical status objectively deteriorated, the ICU team made a judgment call to hold the patient, or the floor team refused to accept the transfer. Given this fact, although it appears in this study (and in the health care system more broadly) that there are opportunities to increase efficiency and reduce nonvalue‐added time, it is not realistic (nor advisable) that such time be reduced to zero. Along this line, one must consider separately purely nonclinical functions such as room cleaning and those that include some clinical element, such as time waiting for a patient to be transferred.

Beyond the direct findings of this study, the method should be replicable in other settings and can reveal important information about health care efficiency, capacity, and flexibility. The bottlenecks identified would have been difficult to identify through administrative record review. The exact amount of time to spend on observation may vary from place to place and would depend on the expected variation over time and the level of detail sought. In general, the more common the event and the less variation, the less time needed to observe it.

This study has several limitations that should be considered in terms of interpreting the results and in seeking to reproduce the approach. First, hourly recordings may not be discrete enough for events that took less than 1 hour. To assess the degree to which this would affect our results, we reanalyzed all NCCS by subtracting 30 minutes (0.5 hour) from all recordings, which increased total CCS from 82% to 87% and decreased NCCS by the same 5 percentage points. In a related fashion, our recordings were truncated at the start and end of each 1‐week period, so we could only observe a maximum of 168 hours for any given activity and did not record how long an activity was happening before or after the recordings started or stopped, respectively. Second, each recording could only be for 1 activity per hour. Separate from the level of granularity already noted, this also limits interpretation of critical care activities that may have been simultaneous. However, because the goal of the study was not to describe the provision of critical care services, but rather the times when they were not being delivered, this does not influence our conclusions. For movement of patients, however, we missed instances of physician and nursing calling sign‐out on patients to receiving units, as these events last less than 1 hour (and in the case of surgical patients, generally do not occur as the team provides continuous coverage). The time for such events is then included in other activities. To the extent that this may influence the results, it would increase the perceived time for nonvalue‐added services, but to a limited degree, and never by more than 59 minutes. Third, the overnight hours (11:00 _PM to 8:00 _AM) were not directly observed, but retrospectively recorded each morning by reviewing the records and discussing the overnight events with the clinical staff. For example, if a patient was intubated at 11:00 _PM and at 8:00 _AM, the observer would confirm this and record that status for the intervening hours. This is unlikely to result in a substantial impact on the findings, because the overnight hours have a relative degree of stability even for unstable patients in terms of their status of needing or not needing a CCS. Fourth, we did not evaluate the appropriateness of CCS delivered (eg, how long a patient was ventilated). Our definitions for CCS and NCSS were based on Children's Hospital of Philadelphia practices, which may not be the same as those of other facilities. The categorization of CCS was objective for activities such as ventilation or continuous infusion, but was less clear for the not otherwise specified recordings, which represented patients with a complex illness or projected organ, respiratory, cardiac, or neurological failure. These patients were not receiving a specific critical care intervention, but were deemed to need to be in the PICU as opposed to a regular floor (eg, for frequent monitoring of potential respiratory failure). It would also include patients receiving combinations of therapies more efficiently delivered in the PICU. For that, the observers relied on the judgment of clinicians (primarily nurses) to determine whether the patient needed to be in the PICU or not; if no specific reason could be provided, not otherwise specified was applied. These 192 instances accounted for 2982 aggregate bed‐hours (15% of total). It is difficult to judge the direction of bias, because overestimation of need to be in the PICU may be as likely to occur as underestimation. Fifth, the very presence of the observers may have changed behavior. Knowing that they were being observed staff may have acted with greater efficiency than otherwise. We expect that such a finding would lead to less time appearing as necessary logistics or NCCS. Finally, results may not be generalizable to other hospitals or hospital settings. There are clearly important contextual factors, not only for the location but also for the duration. For example, staffing was never an issue during the 5 weeks of observation, but there are locations where an empty bed is not necessarily usable due to lack of staffing. Nonetheless, we believe the results provide a generalizable approach and methodology for other settings (and staffing could be a reason for an empty bed).

In terms of the setting, as noted, we observed one discrete 24‐bed unit, which comprises half of the total PICU. Thus, statements that the PICU was at full capacity must be interpreted in the context that additional rooms may have been available on the other side. Patients are generally admitted alternately to each unit, so the occupancies should parallel each other. We recorded the census every 4 hours for both sides from the electronic system (Sunrise Clinical Manager [SCM]). However, this only accounts for patients physically in beds, not beds held for patients in other locations. Thus, we would expect a discrepancy between direct observation and the SCM value. Through analysis of the entire pediatric intensive care unit,* that part which observed directly, and that which we did not observe directly using census data, we think it reasonable to assert that both units of the total PICU had constrained capacity during the times we directly observed and recorded such constraint on one side.

This study demonstrates the use of direct observation for inpatient settings to learn about resource utilization and identification of value‐added services. PubMed searches for the terms efficiency, flow, process redesign, and time management bring up many more references for operating rooms than for ICUs or inpatient beds. Some examples of ICU‐directed work include videography of an ICU in Australia27 and human factor analysis in ICU nursing.5 Time‐motion studies have also been conducted on clinical staff, such as physicians.28, 29

In conclusion, we found that direct observation provided important insights into the utilization of patient rooms in an important inpatient setting. Data such as these are valuable for clinical and process improvement work, as well as understanding how best to match capacity to patient need. Finally, the methodology is reproducible for other settings and would be an additional tool to measuring and improving the efficiency and value of the health system. When appropriate, this approach can also evaluate the effectiveness of process improvement, help identify and reduce waste,13 and contribute to the growing field that merges operations management with hospital administration and clinical care: in other words, evidence‐based management.30

Patient flow refers to the management and movement of patients in health care settings and is linked to quality, safety, and cost.16 The intensive care unit (ICU) is crucial in patient flow.7, 8 The limited number of beds and the resource‐intensive services and staffing associated with them require that hospitals optimize their utilization, as is increasingly true of all hospital resources. To maximize delivery of services to patients who need them and minimize real and opportunity losses (eg, postponed surgery, diverted transports, or inability to accept patients), patients in ICU beds should receive critical care medicine/nursing services while there and be transferred or discharged when appropriate.

The time between arrival and departure from any area of the hospital, including the ICU, is considered the time when a patient is receiving needed clinical carethe value‐added portion of health care operationsand time waiting to move on to the next step.911 This period includes both necessary logistics (eg, signing out a patient or waiting a reasonable amount of time for room cleaning) and nonvalue‐added time (eg, an excessively long amount of time for room cleaning). Operations management labels nonvalue‐added time as waste, and its reduction is vital for high‐quality health care.9, 12, 13 As in other industries, one important way to understand value versus waste is through direct observation.11, 14 Although operating rooms have been the subject of several published process improvement projects to improve efficiency,1518 inpatient beds have not been the subject of such scrutiny. The objectives of this study were to generate a direct observation method and use it to describe pediatric ICU (PICU) bed utilization from a value‐added perspective.

METHODS

An interdisciplinary work group of physicians, nurses, quality improvement specialists, and 1 operations management expert developed an Excel spreadsheet to categorize hour‐by‐hour status of PICU beds. The clinicians generated a list of 27 activities. A critical care nurse trained in quality improvement piloted the list for 3 separate 4‐hour blocks over 2 weeks adding 18 activities; 2 additional activities were added during the 5 weeks of observation (Table 1). (The recording tool is provided in the Supporting Information Appendix.) Three observers with knowledge of medical terminology (2 third‐year medical students and 1 premedical student with years of experience as an emergency medical technician) were trained over 12 hours to conduct the observations. Prior to the observations, the 3 observers also spent time in the PICU, and terminology used for recordings was reviewed. Interobserver reliability was checked during 3 sets of observation circuits by all 3 observers and the principal investigator, as well as by spot checks during the study.

The targeted area included 24 single‐patient rooms. The activity of each bed was recorded hourly. Real‐time recording in to the Excel spreadsheet on a dedicated laptop occurred from 8:00 _AM until 11:00 _PM. The most visible or critical event was recorded. Although some activities were not mutually exclusive (eg, a patient could be ventilated and on a continuous infusion simultaneously), the objective was to identify when a room was being used for any critical care service, not enumerate all of them. The observers noted overnight events that occurred from 11:00 _PM to 8:00 _AM in the morning by reviewing the bedside record and talking to the staff to complete each day's 24‐hour recording. The observers also recorded the hospital‐wide census and the census for the other half of the PICU every 4 hours. The observations occurred over 5 noncontiguous weeks between January 2009 and April 2009.

After all observations were complete, activities were classified as critical care services (CCS) or noncritical‐care services (NCCS). NCCSs were further divided into necessary logistics (defined for analysis purposes as the first hour of any NCCS activity) or nonvalue‐added (the second or greater hour of NCCS). A time limit of 1 hour was chosen to define necessary logistics based on a consensus that nonclinical activities optimally would not take more than 1 hour each. We also analyzed results with 2 hours as the cutoff for necessary logistics. Admission, discharge, and transfer records were reviewed to check for returns to the PICU or hospital within 48 hours of transfer or discharge from the PICU.

Analyses were conducted using Microsoft Excel (Microsoft, Redmond, WA) and Stata 10.0 (StataCorp, College Station, TX). The study was approved by the Children's Hospital of Philadelphia Institutional Review Board with waiver of consent.

RESULTS

A total of 824 hours of recordings included 19,887 bed‐hours with 219 unique patients; among them, 2 remained from the first day of recording in January to the last day in April (sample recording in Figure 1). A total of 50 patients (range, 812 per week) stayed for the entirety of each 1‐week observation period. Of the 47 possible activities, 45 of them were recorded for at least 1 hour in the 5 weeks. Overall, 14 activities accounted for 95% of the observed bed‐hours and 31 activities accounted for the remaining 5%. CCS accounted for 82% of observed bed‐hours, NCCS accounted for 10.4%, and empty unassigned accounted for 8% (Figure 2). Using the 1‐hour cutoff for necessary services, 77% of NCCS time was nonvalue‐added, whereas 23% of it was necessary logistics; using the 2‐hour cutoff, 54% was nonvalue‐added, and 46% was necessary logistics.

Figure 1

Figure 2

During the observation period, <1% of bed‐hours were used for CCS for overflow patients from the neonatal ICU (NICU), cardiac care unit (CCU), cardiac ICU (CICU), or progressive care unit (PCU; tracheostomy/ventilator unit). Although only 4 patients required transport to a rehabilitation facility, their wait time comprised 99 hours (<1%) of total recordings. Eight patients waited a mean of 2.6 hours for transportation home (maximum, 10 hours).

To demonstrate the cycle of room use, activities were divided into 4 categories: room preparation, critical care services, disposition pending, and postcritical care services (Figure 3). As an example of detailed data revealed by direct observation, we identified 102 instances totaling 919 hours when a patient was waiting for a bed assignment on another floor (5% of all bed‐hours). The mean wait time was 9 hours (range, 188 hours) and the median time was 5.5 hours. There were only 15 instances when floor bed assignment took 1 hour or less, and only 9 instances when it took 12 hours. Similarly, considerable time was spent on cleaning rooms between patients: only 66 of 146 instances of cleaning took 1 hour or less. The mean time for cleaning was 2.2 hours (range, 115), and the median was 2 hours. (There were 136 recorded instances of room cleaning and 10 additional episodes that were not recorded but had to be completed for the room to turnover from one patient to the next, yielding a total of 146 instances of cleaning.)

Figure 3

From the 824 hours of recording, we identified 200 hours (25% of time) when there were zero empty unassigned beds available in the section of the PICU being observed. Episodes of full occupancy occurred mostly on weekdays, with 23% of hours of full capacity on Thursdays, 21% on Mondays, and 21% on Wednesdays; only 8% were on Saturdays and <1% on Sundays. These 200 hours fell into 36 separate episodes of complete occupancy, each lasting 122 hours. Each patient, on average, received 3.1 hours of NCCS during each episode of full occupancy (range, 111 hours). Within these 200 hours at capacity, we identified only 15 hours (8%) when all 24 beds were used for CCS. For 72% of the time, there was at least 1 bed with NCCS, and for 37% at least 2 beds. A small portion of the time (7%), the lack of beds was affected by occupancy by patients who should have been in the NICU, CICU, CCU, or PCU.

Data collected through direct observation can be used to understand aggregated and averaged experiences, but also more specific time periods. For example, we identified 1 week with the highest consistent level of occupancy and turnover: March 915 had empty unassigned beds for only 4% of the week. Of the 168 hours in the week, 68 (40%) had full capacity. However, for 90% of the time, at least 1 bed was used for a NCCS. Other analytic options included varying the assumptions around time needed for logistics. Overall, NCCS time on necessary logistics changes from 23% to 46% using 1 hour versus 2 hours as the cutoff. For floor bed assignments, assuming that the first hour of this activity is necessary logistics and any hour thereafter is not, 817 hours were wasted. Even after assuming 2 hours of necessary logistical time (which may also include steps such as nursing and physician sign‐out to the receiving team, often not recorded in the observations), this left 715 hours of NCCS time in which patients waited to be placed elsewhere in the hospital. For room cleaning, because recordings were hourly, but room cleaning could take less time, we performed a sensitivity analysis, converting all 1‐hour recordings to half‐hour recordings to half‐hour recordings (an exaggerated shortening since industry‐standard cleaning may take longer).

Of the 219 patients directly observed, 15 were noted to be waiting for a transfer out of the PICU but experienced a change in disposition before the transfer. On average, these patients waited 8 hours for a floor bed assignment (range, 221) before reverting to a CCS, which then lasted an average of 16.5 hours (range, 149). (Included in this group are 2 patients who experienced this change in disposition twice.) In post hoc review across the 5 weeks, no patients were transferred back to the PICU within 48 hours after being transferred out. During the study period, 19 patients were discharged directly from the PICU (8 to home, 7 by transport to another facility, and 4 to rehabilitation). One patient returned to the hospital (but not the PICU) within 48 hours of being discharged home from the PICU.

During the study period, using the highest census value for recorded for each 24‐hour period and the number of beds available that day, median hospital‐wide occupancy was 93% (interquartile range, 90%96%). During the 35 days of observation, 71% of the days had occupancy >90%, 29% of days had occupancy >95%, and 3% of days had occupancy >100%.

DISCUSSION

In this direct observation of a PICU, we found high usage of beds for delivery of CCS. We identified many episodes in which the half of the PICU we observed was fully occupied (200 of 824 hours), but not necessarily delivering PICU‐level care to all patients. In fact, 75% of the full‐capacity hours had at least 1 patient receiving NCCS and 37% had at least 2. Patients waiting for a floor bed assignment represented nearly 5% of bed‐hours observed (mean 9 hours per patient). That full occupancy was not random, but rather clustered on weekdays, is consistent with other work showing that hospitals are at greater risk for midweek crowding due to the way in which scheduled admissions enter and leave.1925

Our methods provide the basis for operational analysis and improvement to patient flow, such as value stream mapping.9, 26 Process improvement work could be directed to areas of delay uncovered through this analysis and inform clinical and nonclinical management. For example, one of the key problems faced by the PICU was finding floor bed assignments for patients leaving the unit. Simply building more beds in the PICU will not solve this problemand at an estimated cost of $2 million to add a bed, it is likely not an efficient means of responding to poor flow. In these cases, the problem seems to lie downstream, and could suggest shortage of regular floor beds or inefficient bed assignment procedures within the hospital. The output also suggests that variation in nonclinical processes should be a target for improvement, such as time to clean rooms, because variation is known to be a source of nonvalue‐added time in many operations.9, 26 High occupancy on weekdays but low occupancy on weekends also emphasizes the potential for smoothing occupancy to reduce the risk of midweek crowding and to better manage bed utilization and staffing.24, 25

When seeking to reduce nonvalue‐added time, one must weigh the risks of increased efficiency against clinical outcomes. For example, if patients could be transferred out of the PICU faster, would the risk of returns to the PICU be higher? In this study, 15 patients (7%) had a change in disposition from awaiting transfer back to a CCS. The fact that transfers did not happen instantaneously may serve as a safety check to reduce rapid returns, but it is not possible for us to evaluate the reasons why patients did not actually complete the pending transfers. Specifically, we cannot determine whether the patient's clinical status objectively deteriorated, the ICU team made a judgment call to hold the patient, or the floor team refused to accept the transfer. Given this fact, although it appears in this study (and in the health care system more broadly) that there are opportunities to increase efficiency and reduce nonvalue‐added time, it is not realistic (nor advisable) that such time be reduced to zero. Along this line, one must consider separately purely nonclinical functions such as room cleaning and those that include some clinical element, such as time waiting for a patient to be transferred.

Beyond the direct findings of this study, the method should be replicable in other settings and can reveal important information about health care efficiency, capacity, and flexibility. The bottlenecks identified would have been difficult to identify through administrative record review. The exact amount of time to spend on observation may vary from place to place and would depend on the expected variation over time and the level of detail sought. In general, the more common the event and the less variation, the less time needed to observe it.

This study has several limitations that should be considered in terms of interpreting the results and in seeking to reproduce the approach. First, hourly recordings may not be discrete enough for events that took less than 1 hour. To assess the degree to which this would affect our results, we reanalyzed all NCCS by subtracting 30 minutes (0.5 hour) from all recordings, which increased total CCS from 82% to 87% and decreased NCCS by the same 5 percentage points. In a related fashion, our recordings were truncated at the start and end of each 1‐week period, so we could only observe a maximum of 168 hours for any given activity and did not record how long an activity was happening before or after the recordings started or stopped, respectively. Second, each recording could only be for 1 activity per hour. Separate from the level of granularity already noted, this also limits interpretation of critical care activities that may have been simultaneous. However, because the goal of the study was not to describe the provision of critical care services, but rather the times when they were not being delivered, this does not influence our conclusions. For movement of patients, however, we missed instances of physician and nursing calling sign‐out on patients to receiving units, as these events last less than 1 hour (and in the case of surgical patients, generally do not occur as the team provides continuous coverage). The time for such events is then included in other activities. To the extent that this may influence the results, it would increase the perceived time for nonvalue‐added services, but to a limited degree, and never by more than 59 minutes. Third, the overnight hours (11:00 _PM to 8:00 _AM) were not directly observed, but retrospectively recorded each morning by reviewing the records and discussing the overnight events with the clinical staff. For example, if a patient was intubated at 11:00 _PM and at 8:00 _AM, the observer would confirm this and record that status for the intervening hours. This is unlikely to result in a substantial impact on the findings, because the overnight hours have a relative degree of stability even for unstable patients in terms of their status of needing or not needing a CCS. Fourth, we did not evaluate the appropriateness of CCS delivered (eg, how long a patient was ventilated). Our definitions for CCS and NCSS were based on Children's Hospital of Philadelphia practices, which may not be the same as those of other facilities. The categorization of CCS was objective for activities such as ventilation or continuous infusion, but was less clear for the not otherwise specified recordings, which represented patients with a complex illness or projected organ, respiratory, cardiac, or neurological failure. These patients were not receiving a specific critical care intervention, but were deemed to need to be in the PICU as opposed to a regular floor (eg, for frequent monitoring of potential respiratory failure). It would also include patients receiving combinations of therapies more efficiently delivered in the PICU. For that, the observers relied on the judgment of clinicians (primarily nurses) to determine whether the patient needed to be in the PICU or not; if no specific reason could be provided, not otherwise specified was applied. These 192 instances accounted for 2982 aggregate bed‐hours (15% of total). It is difficult to judge the direction of bias, because overestimation of need to be in the PICU may be as likely to occur as underestimation. Fifth, the very presence of the observers may have changed behavior. Knowing that they were being observed staff may have acted with greater efficiency than otherwise. We expect that such a finding would lead to less time appearing as necessary logistics or NCCS. Finally, results may not be generalizable to other hospitals or hospital settings. There are clearly important contextual factors, not only for the location but also for the duration. For example, staffing was never an issue during the 5 weeks of observation, but there are locations where an empty bed is not necessarily usable due to lack of staffing. Nonetheless, we believe the results provide a generalizable approach and methodology for other settings (and staffing could be a reason for an empty bed).

In terms of the setting, as noted, we observed one discrete 24‐bed unit, which comprises half of the total PICU. Thus, statements that the PICU was at full capacity must be interpreted in the context that additional rooms may have been available on the other side. Patients are generally admitted alternately to each unit, so the occupancies should parallel each other. We recorded the census every 4 hours for both sides from the electronic system (Sunrise Clinical Manager [SCM]). However, this only accounts for patients physically in beds, not beds held for patients in other locations. Thus, we would expect a discrepancy between direct observation and the SCM value. Through analysis of the entire pediatric intensive care unit,* that part which observed directly, and that which we did not observe directly using census data, we think it reasonable to assert that both units of the total PICU had constrained capacity during the times we directly observed and recorded such constraint on one side.

This study demonstrates the use of direct observation for inpatient settings to learn about resource utilization and identification of value‐added services. PubMed searches for the terms efficiency, flow, process redesign, and time management bring up many more references for operating rooms than for ICUs or inpatient beds. Some examples of ICU‐directed work include videography of an ICU in Australia27 and human factor analysis in ICU nursing.5 Time‐motion studies have also been conducted on clinical staff, such as physicians.28, 29

In conclusion, we found that direct observation provided important insights into the utilization of patient rooms in an important inpatient setting. Data such as these are valuable for clinical and process improvement work, as well as understanding how best to match capacity to patient need. Finally, the methodology is reproducible for other settings and would be an additional tool to measuring and improving the efficiency and value of the health system. When appropriate, this approach can also evaluate the effectiveness of process improvement, help identify and reduce waste,13 and contribute to the growing field that merges operations management with hospital administration and clinical care: in other words, evidence‐based management.30

Thousands of hospitals have implemented rapid response systems in recent years in attempts to reduce mortality outside the intensive care unit (ICU).1 These systems have 2 components, a response arm and an identification arm. The response arm is usually comprised of a multidisciplinary critical care team that responds to calls for urgent assistance outside the ICU; this team is often called a rapid response team or a medical emergency team. The identification arm comes in 2 forms, predictive and detective. Predictive tools estimate a patient's risk of deterioration over time based on factors that are not rapidly changing, such as elements of the patient's history. In contrast, detective tools include highly time‐varying signs of active deterioration, such as vital sign abnormalities.2 To date, most pediatric studies have focused on developing detective tools, including several early warning scores.38

In this study, we sought to identify the characteristics that increase the probability that a hospitalized child will deteriorate, and combine these characteristics into a predictive score. Tools like this may be helpful in identifying and triaging the subset of high‐risk children who should be intensively monitored for early signs of deterioration at the time of admission, as well as in identifying very low‐risk children who, in the absence of other clinical concerns, may be monitored less intensively.

METHODS

Detailed methods, including the inclusion/exclusion criteria, the matching procedures, and a full description of the statistical analysis are provided as an appendix (see Supporting Online Appendix: Supplement to Methods Section in the online version of this article). An abbreviated version follows.

RESULTS

DISCUSSION

Despite the widespread adoption of rapid response systems, we know little about the optimal methods to identify patients whose clinical characteristics alone put them at increased risk of deterioration, and triage the care they receive based on this risk. Pediatric case series have suggested that younger children and those with chronic illnesses are more likely to require assistance from a medical emergency team,1112 but this is the first study to measure their association with this outcome in children.

Most studies with the objective of identifying patients at risk have focused on tools designed to detect symptoms of deterioration that have already begun, using single‐parameter medical emergency team calling criteria1316 or multi‐parameter early warning scores.38 Rather than create a tool to detect deterioration that has already begun, we developed a predictive score that incorporates patient characteristics independently associated with deterioration in hospitalized children, including age <1 year, epilepsy, congenital/genetic defects, history of transplant, enteral tube, hemoglobin <10 g/dL, and blood culture drawn in the preceding 72 hours. The score has the potential to help clinicians identify the children at highest risk of deterioration who might benefit most from the use of vital sign‐based methods to detect deterioration, as well as the children at lowest risk for whom monitoring may be unnecessary. For example, this score could be performed at the time of admission, and those at very low risk of deterioration and without other clinically concerning findings might be considered for a low‐intensity schedule of vital signs and monitoring (such as vital signs every 8 hours, no continuous cardiorespiratory monitoring or pulse oximetry, and early warning score calculation daily), while patients in the intermediate and high‐risk groups might be considered for a more intensive schedule of vital signs and monitoring (such as vital signs every 4 hours, continuous cardiorespiratory monitoring and pulse oximetry, and early warning score calculation every 4 hours). It should be noted, however, that 37 cases (26%) fell into the very low‐risk category, raising the importance of external validation at the point of admission from the emergency department, before the score can be implemented for the potential clinical use described above. If the score performs well in validation studies, then its use in tailoring monitoring parameters has the potential to reduce the amount of time nurses spend responding to false monitor alarms and calculating early warning scores on patients at very low risk of deterioration.

Of note, we excluded children hospitalized for fewer than 24 hours, resulting in the exclusion of 31% of the potentially eligible events. We also excluded 40% of the potentially eligible ICU transfers because they did not meet urgent criteria. These may be limitations because: (1) the first 24 hours of hospitalization may be a high‐risk period; and (2) patients who were on trajectories toward severe deterioration and received interventions that prevented further deterioration, but did not meet urgent criteria, were excluded. It may be that the children we included as cases were at increased risk of deterioration that is either more difficult to recognize early, or more difficult to treat effectively without ICU interventions. In addition, the population of patients meeting urgent criteria may vary across hospitals, limiting generalizability of this score.

In summary, we developed a predictive score and risk stratification tool that may be useful in triaging the intensity of monitoring and surveillance for deterioration that children receive when hospitalized on non‐ICU units. External validation using the setting and frequency of score measurement that would be most valuable clinically (for example, in the emergency department at the time of admission) is needed before clinical use can be recommended.

Thousands of hospitals have implemented rapid response systems in recent years in attempts to reduce mortality outside the intensive care unit (ICU).1 These systems have 2 components, a response arm and an identification arm. The response arm is usually comprised of a multidisciplinary critical care team that responds to calls for urgent assistance outside the ICU; this team is often called a rapid response team or a medical emergency team. The identification arm comes in 2 forms, predictive and detective. Predictive tools estimate a patient's risk of deterioration over time based on factors that are not rapidly changing, such as elements of the patient's history. In contrast, detective tools include highly time‐varying signs of active deterioration, such as vital sign abnormalities.2 To date, most pediatric studies have focused on developing detective tools, including several early warning scores.38

In this study, we sought to identify the characteristics that increase the probability that a hospitalized child will deteriorate, and combine these characteristics into a predictive score. Tools like this may be helpful in identifying and triaging the subset of high‐risk children who should be intensively monitored for early signs of deterioration at the time of admission, as well as in identifying very low‐risk children who, in the absence of other clinical concerns, may be monitored less intensively.

METHODS

Detailed methods, including the inclusion/exclusion criteria, the matching procedures, and a full description of the statistical analysis are provided as an appendix (see Supporting Online Appendix: Supplement to Methods Section in the online version of this article). An abbreviated version follows.

RESULTS

DISCUSSION

Despite the widespread adoption of rapid response systems, we know little about the optimal methods to identify patients whose clinical characteristics alone put them at increased risk of deterioration, and triage the care they receive based on this risk. Pediatric case series have suggested that younger children and those with chronic illnesses are more likely to require assistance from a medical emergency team,1112 but this is the first study to measure their association with this outcome in children.

Most studies with the objective of identifying patients at risk have focused on tools designed to detect symptoms of deterioration that have already begun, using single‐parameter medical emergency team calling criteria1316 or multi‐parameter early warning scores.38 Rather than create a tool to detect deterioration that has already begun, we developed a predictive score that incorporates patient characteristics independently associated with deterioration in hospitalized children, including age <1 year, epilepsy, congenital/genetic defects, history of transplant, enteral tube, hemoglobin <10 g/dL, and blood culture drawn in the preceding 72 hours. The score has the potential to help clinicians identify the children at highest risk of deterioration who might benefit most from the use of vital sign‐based methods to detect deterioration, as well as the children at lowest risk for whom monitoring may be unnecessary. For example, this score could be performed at the time of admission, and those at very low risk of deterioration and without other clinically concerning findings might be considered for a low‐intensity schedule of vital signs and monitoring (such as vital signs every 8 hours, no continuous cardiorespiratory monitoring or pulse oximetry, and early warning score calculation daily), while patients in the intermediate and high‐risk groups might be considered for a more intensive schedule of vital signs and monitoring (such as vital signs every 4 hours, continuous cardiorespiratory monitoring and pulse oximetry, and early warning score calculation every 4 hours). It should be noted, however, that 37 cases (26%) fell into the very low‐risk category, raising the importance of external validation at the point of admission from the emergency department, before the score can be implemented for the potential clinical use described above. If the score performs well in validation studies, then its use in tailoring monitoring parameters has the potential to reduce the amount of time nurses spend responding to false monitor alarms and calculating early warning scores on patients at very low risk of deterioration.

Of note, we excluded children hospitalized for fewer than 24 hours, resulting in the exclusion of 31% of the potentially eligible events. We also excluded 40% of the potentially eligible ICU transfers because they did not meet urgent criteria. These may be limitations because: (1) the first 24 hours of hospitalization may be a high‐risk period; and (2) patients who were on trajectories toward severe deterioration and received interventions that prevented further deterioration, but did not meet urgent criteria, were excluded. It may be that the children we included as cases were at increased risk of deterioration that is either more difficult to recognize early, or more difficult to treat effectively without ICU interventions. In addition, the population of patients meeting urgent criteria may vary across hospitals, limiting generalizability of this score.

In summary, we developed a predictive score and risk stratification tool that may be useful in triaging the intensity of monitoring and surveillance for deterioration that children receive when hospitalized on non‐ICU units. External validation using the setting and frequency of score measurement that would be most valuable clinically (for example, in the emergency department at the time of admission) is needed before clinical use can be recommended.

Thousands of hospitals have implemented rapid response systems in recent years in attempts to reduce mortality outside the intensive care unit (ICU).1 These systems have 2 components, a response arm and an identification arm. The response arm is usually comprised of a multidisciplinary critical care team that responds to calls for urgent assistance outside the ICU; this team is often called a rapid response team or a medical emergency team. The identification arm comes in 2 forms, predictive and detective. Predictive tools estimate a patient's risk of deterioration over time based on factors that are not rapidly changing, such as elements of the patient's history. In contrast, detective tools include highly time‐varying signs of active deterioration, such as vital sign abnormalities.2 To date, most pediatric studies have focused on developing detective tools, including several early warning scores.38

In this study, we sought to identify the characteristics that increase the probability that a hospitalized child will deteriorate, and combine these characteristics into a predictive score. Tools like this may be helpful in identifying and triaging the subset of high‐risk children who should be intensively monitored for early signs of deterioration at the time of admission, as well as in identifying very low‐risk children who, in the absence of other clinical concerns, may be monitored less intensively.

METHODS

Detailed methods, including the inclusion/exclusion criteria, the matching procedures, and a full description of the statistical analysis are provided as an appendix (see Supporting Online Appendix: Supplement to Methods Section in the online version of this article). An abbreviated version follows.

RESULTS

DISCUSSION

Despite the widespread adoption of rapid response systems, we know little about the optimal methods to identify patients whose clinical characteristics alone put them at increased risk of deterioration, and triage the care they receive based on this risk. Pediatric case series have suggested that younger children and those with chronic illnesses are more likely to require assistance from a medical emergency team,1112 but this is the first study to measure their association with this outcome in children.

Most studies with the objective of identifying patients at risk have focused on tools designed to detect symptoms of deterioration that have already begun, using single‐parameter medical emergency team calling criteria1316 or multi‐parameter early warning scores.38 Rather than create a tool to detect deterioration that has already begun, we developed a predictive score that incorporates patient characteristics independently associated with deterioration in hospitalized children, including age <1 year, epilepsy, congenital/genetic defects, history of transplant, enteral tube, hemoglobin <10 g/dL, and blood culture drawn in the preceding 72 hours. The score has the potential to help clinicians identify the children at highest risk of deterioration who might benefit most from the use of vital sign‐based methods to detect deterioration, as well as the children at lowest risk for whom monitoring may be unnecessary. For example, this score could be performed at the time of admission, and those at very low risk of deterioration and without other clinically concerning findings might be considered for a low‐intensity schedule of vital signs and monitoring (such as vital signs every 8 hours, no continuous cardiorespiratory monitoring or pulse oximetry, and early warning score calculation daily), while patients in the intermediate and high‐risk groups might be considered for a more intensive schedule of vital signs and monitoring (such as vital signs every 4 hours, continuous cardiorespiratory monitoring and pulse oximetry, and early warning score calculation every 4 hours). It should be noted, however, that 37 cases (26%) fell into the very low‐risk category, raising the importance of external validation at the point of admission from the emergency department, before the score can be implemented for the potential clinical use described above. If the score performs well in validation studies, then its use in tailoring monitoring parameters has the potential to reduce the amount of time nurses spend responding to false monitor alarms and calculating early warning scores on patients at very low risk of deterioration.

Of note, we excluded children hospitalized for fewer than 24 hours, resulting in the exclusion of 31% of the potentially eligible events. We also excluded 40% of the potentially eligible ICU transfers because they did not meet urgent criteria. These may be limitations because: (1) the first 24 hours of hospitalization may be a high‐risk period; and (2) patients who were on trajectories toward severe deterioration and received interventions that prevented further deterioration, but did not meet urgent criteria, were excluded. It may be that the children we included as cases were at increased risk of deterioration that is either more difficult to recognize early, or more difficult to treat effectively without ICU interventions. In addition, the population of patients meeting urgent criteria may vary across hospitals, limiting generalizability of this score.

In summary, we developed a predictive score and risk stratification tool that may be useful in triaging the intensity of monitoring and surveillance for deterioration that children receive when hospitalized on non‐ICU units. External validation using the setting and frequency of score measurement that would be most valuable clinically (for example, in the emergency department at the time of admission) is needed before clinical use can be recommended.