Comparing Collaborative and Toolkit QI

Continuous quality improvement (CQI) methodologies provide a framework for initiating and sustaining improvements in complex systems.1 By definition, CQI engages frontline staff in iterative problem solving using plandostudyact cycles of learning, with decision‐making based on real‐time process measurements.2 The Institute for Healthcare Improvement (IHI) has sponsored Breakthrough Series Collaboratives since 1996 to accelerate the uptake and impact of quality improvement (QI).3, 4 These collaboratives are typically guided by evidence‐based clinical practice guidelines, incorporate change methodologies, and rely on clinical and process improvement subject matter experts. Through the collaborative network, teams share knowledge and ideas about effective and ineffective interventions as well as strategies for overcoming barriers. The collaborative curriculum includes CQI methodology, multidisciplinary teamwork, leadership support, and tools for measurement. Participants are typically required to invest resources and send teams to face‐to‐face goal‐oriented meetings. It is costly for a large healthcare organization to incorporate travel to a learning session conference into its collaborative model. Thus, we attempted virtual learning sessions that make use of webcasts, a Web site, and teleconference calls for tools and networking.5

A recent derivative of collaboratives has been deployment of toolkits for QI. Intuition suggests that such toolkits may help to enable change, and thus some agencies advocate the simpler approach of disseminating toolkits as a change strategy.6 Toolkit dissemination is a passive approach in contrast to collaborative participation, and its effectiveness has not been critically examined in evidence‐based literature.

We sought to compare the virtual collaborative model with the toolkit model for improving care. Recommendations and guidelines for central lineassociated bloodstream infection (CLABSI) and ventilator‐associated pneumonia (VAP) prevention have not been implemented reliably, resulting in unnecessary intensive care unit (ICU) morbidity and mortality and fostering a national call for improvement.7 Our aim was to compare the effectiveness of the virtual collaborative and toolkit approaches on preventing CLABSI and VAP in the ICU.

Methods

This cluster randomized trial included medical centers within the Hospital Corporation of America (HCA), a network of hospitals located primarily in the southern United States. To minimize contamination bias between study groups within the same facility, the unit of randomization was the hospital and implementation was at the level of the ICU. The project received approval from the Vanderbilt University Institutional Review Board.

Leaders of all medical centers with at least 1 adult or pediatric ICU received an invitation from HCA leadership to participate in a QI initiative. Facility clinicians and managers completed baseline surveys (shown in the Supporting Information) on hospital characteristics, types of ICUs, patient safety climate, and QI resources between July and November 2005. Hospital‐level data were extracted from the enterprise‐wide data warehouse. Hospitals willing to participate were matched on geographic location and ICU volume and then randomized into either the Virtual Collaborative (n = 31) or Toolkit (n = 30) groups in December 20058; 1 of the hospitals was sold, yielding 29 hospitals in the Toolkit (n = 29) group. The study lasted 18 months from January 2006 through September 2007, with health careassociated infection data collected through December 2007, and follow‐up data collection through April 2008.

The QI initiative included educational opportunities, evidence‐based clinical prevention interventions, and processes and tools to implement and measure the impact of these interventions. Participants in both groups were offered interactive Web seminars during the study period; 5 of these seminars were on clinical subject matter, and 5 seminars were on patient safety, charting use of statistical process control and QI methods. The interventions were evidence‐based care bundles.9 The key interventions for preventing CLABSI were routine hand hygiene, use of chlorhexidine skin antisepsis, maximal barrier precautions during catheter insertion, catheter site and care, and avoidance of routine replacement of catheters. The key interventions to prevent VAP were routine elevation of head of the bed, regular oral care, daily sedation vacations, daily assessment of readiness to extubate, secretion cleaning, peptic ulcer disease prophylaxis, and deep vein thrombosis prophylaxis.

Results

Participating facilities included rural (11%), inner city (28%), and suburban (61%) medical centers. The 60 participating sites did not differ in administrative variables from the 113 nonparticipating HCA sites (results not shown). The median hospital size was 177 beds and the median ICU size was 16 beds. The hospitals did not differ between study groups (Table 1). At baseline, 45% of the facilities reported having a CLABSI program and 62% a VAP program.

The baseline and quarterly median and pooled infection rates for the Toolkit and Collaboration groups are shown in Table 2 for CLABSI and in Table 3 for VAP. There were no significant differences in the baseline rates for either CLABSI (P = 0.24) or VAP (P = 0.72) between the Collaborative and Toolkit groups. There was no significant change for either CLABSI or VAP outcomes at either 12 or 18 months of follow‐up. The median bloodstream infection rate for all participating hospitals was 2.27 at baseline, 1.18 at 12 months (P = 0.13), and 2.23 per 1000 catheter days 18 months later (P = 0.95). The median VAP rate for participating hospitals was 2.90 at baseline, 2.67 at 12 months (P = 0.44), and 2.52 per 1000 ventilator days 18 months later (P = 0.84). The hierarchical regression analysis found that neither the Collaborative nor Toolkit groups improved CLABSI (P = 0.75 and P = 0.83, respectively) or VAP (P = 0.61 and P = 0.37, respectively) rates over time, and there was no differential performance between the 2 groups for either outcome (bloodstream infection, P = 0.71; VAP, P = 0.80).

The poststudy survey was completed by 27 of 31 (87%) of Collaborative group hospitals and 19 of the 29 (66%) Toolkit hospitals. Both groups reported QI improvement efforts to prevent CLABSI (Collaborative 97% vs. Toolkit 88%, P = 0.29) and VAP (Collaborative 97% vs. Toolkit 96%, P = 0.99). Eighty‐three percent of the Collaborative group implemented all components of the bloodstream infection prevention interventions compared with 64% for the Toolkit group (P = 0.13; Figure 1). The Collaborative group implemented daily catheter review more often than the Toolkit group (P = 0.04) and began the process implementation sooner following study implementation (P = 0.006 vs. Toolkit; see Supporting Information Figure). Eighty‐six percent of the Collaborative group implemented the complete VAP prevention interventions vs. 64% of the Toolkit group (P = 0.06; Figure 1) and the Collaborative group conducted the sedation vacation intervention more often (P = 0.03).

Figure 1

The Collaborative group participated in 57% of the seminars, whereas the Toolkit group participated in 39% (P = 0.014). Members of both groups attended more than half the clinical topics (Collaborative 64% vs. Toolkit 56%, P = 0.37). The Collaborative group had greater participation in the data and method topics (Collaborative 50% vs. Toolkit 22%, P < 0.001). The proportion of hospitals finding the seminars useful to their QI efforts was 49% for the Collaborative and 30% for the Toolkit group (P = 0.017). When restricted to hospitals that participated in the seminars, the usefulness rating was higher for both clinical (91% for the Collaborative and 86% for Toolkit) and Data/Methods (79% for Collaborative and 55% for Toolkit) topics.

A set of 14 tools were produced during the study period (Table 4); 9 clinically related tools (eg, checklists, algorithms, protocols, and flowsheets) and 5 data monitoring and quality improvement tools (eg, easy‐to‐use statistical process control spreadsheet templates, quality improvement tools, and computer tools). The Collaborative group downloaded a median of 10 tools and the Toolkit group a median of 7 (P = 0.051). The groups did not differ in their access to the clinical tools (P = 0.23) but the Collaborative group accessed a greater proportion of the data/methods tools (P = 0.004).

Both groups relied primarily on implementation of protocols and informatics approaches (Table 4) without increasing staff levels. The predominant strategy was education; both groups provided written educational materials and classes to their providers. There was a trend for more Collaborative group members to implement QI teams (Table 4, P = 0.16 compared with the Toolkit group). Although the preponderance of both groups provided feedback reports to their hospital leaders and unit managers, Collaborative group hospitals showed a trend for providing feedback to front‐line providers (P = 0.11). With respect to self‐reported interventions, 78% of the Collaborative ICUs reported implementing a CLABSI checklist and 86% a VAP checklist, whereas only 60% of the Toolkit group reported implementation of a CLABSI checklist (P = 0.16) and 52% a VAP checklist (P = 0.007). Once a tool was implemented, both groups reported a high rate of sustaining the implementation (ranging from 86% to 100%). There also seemed to be a pattern of sequencing the interventions. Initial efforts tend to focus on provider education and evidence‐based protocols. Later efforts include more formal formation of QI teams followed by implementation of checklists. The evidence for sequencing of interventions is qualitative; we lacked subgroup sample size to substantiate these results with statistical analysis.

Discussion

In our investigation of Virtual Collaborative and Toolkit strategies for spreading the implementation of safe practices for CLABSI and VAP, ICUs in the Collaborative group had more complete implementation of the processes for prevention of hospital‐associated infections. Although both groups accessed clinical resources consistent with surveillance and clinical education, the Virtual Collaborative group attended to data and implementation methods more likely to lead to systemic CQI and organizational changes. ICUs that engaged these resources believed them useful in implementing QI, and more than 85% of the practices were sustained once integrated into routine care. Although the Collaborative ICUs were about 50% more likely to implement improvement strategies, these differences in implementation and process of care did not translate into group differences or longitudinal changes in infection rates.

In contrast to the context of our investigation, most published QI studies on health careassociated infection prevention report high baseline rates followed by a significant decline in infection rates.1419 The baseline infection rates in our study hospitals were actually below the endpoint found in many prior studies, suggesting that any marginal effects from our intervention would be more difficult to detect. Our study was implemented during the IHI's 100,000 Lives Campaign,20 a trend that may have brought about these lower baseline rates and thus a tighter margin for improvement.

The median CLABSI baseline rate in the well‐publicized Michigan hospital study was 2.7 per 1000 catheter days.21, 22 Although our baseline rate was similar (2.27 per 1000 catheter days), their reported postintervention rate was near zero, inferring nearly total elimination of the risk for CLABSI within 3‐18 months of study implementation. Several other studies using a collaborative approach have similarly reported high‐performance near‐zero results in reducing VAP23, 24 and CLABSI2528 rates. The difference between the present and previously published near‐zero result outcomes raises questions about collaboration‐based studies. We noticed 2 phenomena. First, there was slow uptake of data‐driven QI, and second, there was a differential uptake between general knowledge (clinical evidence and education) and QI implementation knowledge.29, 30

Lack of infrastructure to support data‐driven QI remains a significant barrier throughout the health care system, and teams in collaboratives often must work intensively toward improving their information systems' capability for the purpose of data‐driven decision support.1, 15, 31, 32 Systematic, standardized collection of CLABSI and VAP outcomes was initially lacking in many of our study hospitals,10 and our project expended early effort to deploy a system‐wide standardized infection control database registry.

Both of our study groups gravitated toward educational training and evidence‐based protocol decision‐support strategies. A focus only on established surveillance and education‐based fixes (eg, asking clinicians to follow a protocol within their existing care processes) have produced 32% to 57% reductions in health careacquired infections.3335 These early gains, however, are unlikely to produce the sustained near‐zero results that some collaborative teams have reported.22, 25

The ability to achieve sustained high‐performance results depends on organizational context and requires time.31 A potential benefit of collaboratives might be the return on investment attained by organizational change in quality and safety climate and its influence across the whole organization.19, 31, 36 Participants requiring systems training in the CQI process may not gain these benefits until well into their collaborative.31 For example, accumulating evidence demonstrates that the use of checklists can reduce errors of omission. Although a checklist seems a simple intervention, its effective implementation into routine care processes actually requires time for system redesign that addresses changes in multidisciplinary roles and responsibilities, frontline clinician and mid‐level management buy‐in, new methods of data collection and feedback, unanticipated involvement of ancillary services (eg, medical records, housekeeping), as well as changes to organizational policies, expectations, and priorities that connect silos of care and integrate hierarchical operations. Wall et al.37 and Pronovost and colleagues19, 21, 22, 25 highlighted the strategic effectiveness of embedding a checklist as a behavioral and data collection tool into frontline care process, leading to a redefined role of nursing, as well as new data for further cycles of improvement that collectively reduced infection rates. In our study, the Virtual Collaborative group did not have greater use of CLABSI and VAP checklists until the QI teams had been formed months into the project, consistent with the hypothesis that beneficial translation of desired changes in process of care to observed improvements in patient outcomes may take longer than 18 months to achieve19, 25, 27, 38 as opposed to the remarkable 3 months reported in the Keystone ICU project.21

Our study has several limitations. Our intervention did not mandate fixed specific components of intervention or QI methods. Each medical center was free to tailor its use of tools and change ideas, producing site variation in implementation methods and investment in support of QI. Like other multicomponent, multidimensional intervention studies, we were not able to test the effectiveness of particular QI components or the thoroughness of surveillance for CLABSI and VAP related to efforts to standardize the approach, and we did not have the resources to monitor the intensity with which participants approached QI. Furthermore, our data were dependent on self‐reports and were not verified by independent assessment of the fidelity with which the interventions were implemented, a checklist was embedded into usual care, or practices were enforced by nurses. In addition, the virtual collaborative circumvents the face‐to‐face learning sessions that might play a role in collaborative social networking, peer pressure, and acculturation.31, 36

Despite these limitations, we found that the Virtual Collaborative performed just like a Breakthrough Collaborative with a gradual uptake of implementation science using QI methods, team management, and statistical process control tools. The Toolkit condition had an even slower uptake. From an organization's perspective, the bottom‐line decision is whether a greater and meaningful proportion of collaborative participants will be successful to justify the investment of effort compared to a toolkit‐only approach. Our findings suggest that organizations engaged in change but lacking expertise in implementation science can potentially benefit from the acculturation, experiential learning, and uptake of QI provided by a collaborative.

In summary, although our Virtual Collaborative intervention was more likely to produce changes in ICU processes of care, there were no improvements in patient outcomes over this 18‐month study. The current popularity of evidence‐based guidelines, care protocols, prevention awareness, and surveillance may have produced a background of secular trend, making it difficult to ascertain effects of our QI intervention. Nonetheless, important lessons can be gleaned from this randomized controlled trial. Our study supports the proposition that as long as organizations vary in their capacity for and commitment to the science of QI and systems engineering, we should anticipate variation, uncertainty, and mixed results from short‐term, rapid cycle initiatives.27, 28, 31, 32, 39, 40 The untested, longer‐term benefit produced by a collaborative may be its stimulation of enduring systems engineering that optimizes an environment for QI of health care processes focused on desired outcomes.

Continuous quality improvement (CQI) methodologies provide a framework for initiating and sustaining improvements in complex systems.1 By definition, CQI engages frontline staff in iterative problem solving using plandostudyact cycles of learning, with decision‐making based on real‐time process measurements.2 The Institute for Healthcare Improvement (IHI) has sponsored Breakthrough Series Collaboratives since 1996 to accelerate the uptake and impact of quality improvement (QI).3, 4 These collaboratives are typically guided by evidence‐based clinical practice guidelines, incorporate change methodologies, and rely on clinical and process improvement subject matter experts. Through the collaborative network, teams share knowledge and ideas about effective and ineffective interventions as well as strategies for overcoming barriers. The collaborative curriculum includes CQI methodology, multidisciplinary teamwork, leadership support, and tools for measurement. Participants are typically required to invest resources and send teams to face‐to‐face goal‐oriented meetings. It is costly for a large healthcare organization to incorporate travel to a learning session conference into its collaborative model. Thus, we attempted virtual learning sessions that make use of webcasts, a Web site, and teleconference calls for tools and networking.5

A recent derivative of collaboratives has been deployment of toolkits for QI. Intuition suggests that such toolkits may help to enable change, and thus some agencies advocate the simpler approach of disseminating toolkits as a change strategy.6 Toolkit dissemination is a passive approach in contrast to collaborative participation, and its effectiveness has not been critically examined in evidence‐based literature.

We sought to compare the virtual collaborative model with the toolkit model for improving care. Recommendations and guidelines for central lineassociated bloodstream infection (CLABSI) and ventilator‐associated pneumonia (VAP) prevention have not been implemented reliably, resulting in unnecessary intensive care unit (ICU) morbidity and mortality and fostering a national call for improvement.7 Our aim was to compare the effectiveness of the virtual collaborative and toolkit approaches on preventing CLABSI and VAP in the ICU.

Methods

This cluster randomized trial included medical centers within the Hospital Corporation of America (HCA), a network of hospitals located primarily in the southern United States. To minimize contamination bias between study groups within the same facility, the unit of randomization was the hospital and implementation was at the level of the ICU. The project received approval from the Vanderbilt University Institutional Review Board.

Leaders of all medical centers with at least 1 adult or pediatric ICU received an invitation from HCA leadership to participate in a QI initiative. Facility clinicians and managers completed baseline surveys (shown in the Supporting Information) on hospital characteristics, types of ICUs, patient safety climate, and QI resources between July and November 2005. Hospital‐level data were extracted from the enterprise‐wide data warehouse. Hospitals willing to participate were matched on geographic location and ICU volume and then randomized into either the Virtual Collaborative (n = 31) or Toolkit (n = 30) groups in December 20058; 1 of the hospitals was sold, yielding 29 hospitals in the Toolkit (n = 29) group. The study lasted 18 months from January 2006 through September 2007, with health careassociated infection data collected through December 2007, and follow‐up data collection through April 2008.

The QI initiative included educational opportunities, evidence‐based clinical prevention interventions, and processes and tools to implement and measure the impact of these interventions. Participants in both groups were offered interactive Web seminars during the study period; 5 of these seminars were on clinical subject matter, and 5 seminars were on patient safety, charting use of statistical process control and QI methods. The interventions were evidence‐based care bundles.9 The key interventions for preventing CLABSI were routine hand hygiene, use of chlorhexidine skin antisepsis, maximal barrier precautions during catheter insertion, catheter site and care, and avoidance of routine replacement of catheters. The key interventions to prevent VAP were routine elevation of head of the bed, regular oral care, daily sedation vacations, daily assessment of readiness to extubate, secretion cleaning, peptic ulcer disease prophylaxis, and deep vein thrombosis prophylaxis.

Results

Participating facilities included rural (11%), inner city (28%), and suburban (61%) medical centers. The 60 participating sites did not differ in administrative variables from the 113 nonparticipating HCA sites (results not shown). The median hospital size was 177 beds and the median ICU size was 16 beds. The hospitals did not differ between study groups (Table 1). At baseline, 45% of the facilities reported having a CLABSI program and 62% a VAP program.

The baseline and quarterly median and pooled infection rates for the Toolkit and Collaboration groups are shown in Table 2 for CLABSI and in Table 3 for VAP. There were no significant differences in the baseline rates for either CLABSI (P = 0.24) or VAP (P = 0.72) between the Collaborative and Toolkit groups. There was no significant change for either CLABSI or VAP outcomes at either 12 or 18 months of follow‐up. The median bloodstream infection rate for all participating hospitals was 2.27 at baseline, 1.18 at 12 months (P = 0.13), and 2.23 per 1000 catheter days 18 months later (P = 0.95). The median VAP rate for participating hospitals was 2.90 at baseline, 2.67 at 12 months (P = 0.44), and 2.52 per 1000 ventilator days 18 months later (P = 0.84). The hierarchical regression analysis found that neither the Collaborative nor Toolkit groups improved CLABSI (P = 0.75 and P = 0.83, respectively) or VAP (P = 0.61 and P = 0.37, respectively) rates over time, and there was no differential performance between the 2 groups for either outcome (bloodstream infection, P = 0.71; VAP, P = 0.80).

The poststudy survey was completed by 27 of 31 (87%) of Collaborative group hospitals and 19 of the 29 (66%) Toolkit hospitals. Both groups reported QI improvement efforts to prevent CLABSI (Collaborative 97% vs. Toolkit 88%, P = 0.29) and VAP (Collaborative 97% vs. Toolkit 96%, P = 0.99). Eighty‐three percent of the Collaborative group implemented all components of the bloodstream infection prevention interventions compared with 64% for the Toolkit group (P = 0.13; Figure 1). The Collaborative group implemented daily catheter review more often than the Toolkit group (P = 0.04) and began the process implementation sooner following study implementation (P = 0.006 vs. Toolkit; see Supporting Information Figure). Eighty‐six percent of the Collaborative group implemented the complete VAP prevention interventions vs. 64% of the Toolkit group (P = 0.06; Figure 1) and the Collaborative group conducted the sedation vacation intervention more often (P = 0.03).

Figure 1

The Collaborative group participated in 57% of the seminars, whereas the Toolkit group participated in 39% (P = 0.014). Members of both groups attended more than half the clinical topics (Collaborative 64% vs. Toolkit 56%, P = 0.37). The Collaborative group had greater participation in the data and method topics (Collaborative 50% vs. Toolkit 22%, P < 0.001). The proportion of hospitals finding the seminars useful to their QI efforts was 49% for the Collaborative and 30% for the Toolkit group (P = 0.017). When restricted to hospitals that participated in the seminars, the usefulness rating was higher for both clinical (91% for the Collaborative and 86% for Toolkit) and Data/Methods (79% for Collaborative and 55% for Toolkit) topics.

A set of 14 tools were produced during the study period (Table 4); 9 clinically related tools (eg, checklists, algorithms, protocols, and flowsheets) and 5 data monitoring and quality improvement tools (eg, easy‐to‐use statistical process control spreadsheet templates, quality improvement tools, and computer tools). The Collaborative group downloaded a median of 10 tools and the Toolkit group a median of 7 (P = 0.051). The groups did not differ in their access to the clinical tools (P = 0.23) but the Collaborative group accessed a greater proportion of the data/methods tools (P = 0.004).

Both groups relied primarily on implementation of protocols and informatics approaches (Table 4) without increasing staff levels. The predominant strategy was education; both groups provided written educational materials and classes to their providers. There was a trend for more Collaborative group members to implement QI teams (Table 4, P = 0.16 compared with the Toolkit group). Although the preponderance of both groups provided feedback reports to their hospital leaders and unit managers, Collaborative group hospitals showed a trend for providing feedback to front‐line providers (P = 0.11). With respect to self‐reported interventions, 78% of the Collaborative ICUs reported implementing a CLABSI checklist and 86% a VAP checklist, whereas only 60% of the Toolkit group reported implementation of a CLABSI checklist (P = 0.16) and 52% a VAP checklist (P = 0.007). Once a tool was implemented, both groups reported a high rate of sustaining the implementation (ranging from 86% to 100%). There also seemed to be a pattern of sequencing the interventions. Initial efforts tend to focus on provider education and evidence‐based protocols. Later efforts include more formal formation of QI teams followed by implementation of checklists. The evidence for sequencing of interventions is qualitative; we lacked subgroup sample size to substantiate these results with statistical analysis.

Discussion

In our investigation of Virtual Collaborative and Toolkit strategies for spreading the implementation of safe practices for CLABSI and VAP, ICUs in the Collaborative group had more complete implementation of the processes for prevention of hospital‐associated infections. Although both groups accessed clinical resources consistent with surveillance and clinical education, the Virtual Collaborative group attended to data and implementation methods more likely to lead to systemic CQI and organizational changes. ICUs that engaged these resources believed them useful in implementing QI, and more than 85% of the practices were sustained once integrated into routine care. Although the Collaborative ICUs were about 50% more likely to implement improvement strategies, these differences in implementation and process of care did not translate into group differences or longitudinal changes in infection rates.

In contrast to the context of our investigation, most published QI studies on health careassociated infection prevention report high baseline rates followed by a significant decline in infection rates.1419 The baseline infection rates in our study hospitals were actually below the endpoint found in many prior studies, suggesting that any marginal effects from our intervention would be more difficult to detect. Our study was implemented during the IHI's 100,000 Lives Campaign,20 a trend that may have brought about these lower baseline rates and thus a tighter margin for improvement.

The median CLABSI baseline rate in the well‐publicized Michigan hospital study was 2.7 per 1000 catheter days.21, 22 Although our baseline rate was similar (2.27 per 1000 catheter days), their reported postintervention rate was near zero, inferring nearly total elimination of the risk for CLABSI within 3‐18 months of study implementation. Several other studies using a collaborative approach have similarly reported high‐performance near‐zero results in reducing VAP23, 24 and CLABSI2528 rates. The difference between the present and previously published near‐zero result outcomes raises questions about collaboration‐based studies. We noticed 2 phenomena. First, there was slow uptake of data‐driven QI, and second, there was a differential uptake between general knowledge (clinical evidence and education) and QI implementation knowledge.29, 30

Lack of infrastructure to support data‐driven QI remains a significant barrier throughout the health care system, and teams in collaboratives often must work intensively toward improving their information systems' capability for the purpose of data‐driven decision support.1, 15, 31, 32 Systematic, standardized collection of CLABSI and VAP outcomes was initially lacking in many of our study hospitals,10 and our project expended early effort to deploy a system‐wide standardized infection control database registry.

Both of our study groups gravitated toward educational training and evidence‐based protocol decision‐support strategies. A focus only on established surveillance and education‐based fixes (eg, asking clinicians to follow a protocol within their existing care processes) have produced 32% to 57% reductions in health careacquired infections.3335 These early gains, however, are unlikely to produce the sustained near‐zero results that some collaborative teams have reported.22, 25

The ability to achieve sustained high‐performance results depends on organizational context and requires time.31 A potential benefit of collaboratives might be the return on investment attained by organizational change in quality and safety climate and its influence across the whole organization.19, 31, 36 Participants requiring systems training in the CQI process may not gain these benefits until well into their collaborative.31 For example, accumulating evidence demonstrates that the use of checklists can reduce errors of omission. Although a checklist seems a simple intervention, its effective implementation into routine care processes actually requires time for system redesign that addresses changes in multidisciplinary roles and responsibilities, frontline clinician and mid‐level management buy‐in, new methods of data collection and feedback, unanticipated involvement of ancillary services (eg, medical records, housekeeping), as well as changes to organizational policies, expectations, and priorities that connect silos of care and integrate hierarchical operations. Wall et al.37 and Pronovost and colleagues19, 21, 22, 25 highlighted the strategic effectiveness of embedding a checklist as a behavioral and data collection tool into frontline care process, leading to a redefined role of nursing, as well as new data for further cycles of improvement that collectively reduced infection rates. In our study, the Virtual Collaborative group did not have greater use of CLABSI and VAP checklists until the QI teams had been formed months into the project, consistent with the hypothesis that beneficial translation of desired changes in process of care to observed improvements in patient outcomes may take longer than 18 months to achieve19, 25, 27, 38 as opposed to the remarkable 3 months reported in the Keystone ICU project.21

Our study has several limitations. Our intervention did not mandate fixed specific components of intervention or QI methods. Each medical center was free to tailor its use of tools and change ideas, producing site variation in implementation methods and investment in support of QI. Like other multicomponent, multidimensional intervention studies, we were not able to test the effectiveness of particular QI components or the thoroughness of surveillance for CLABSI and VAP related to efforts to standardize the approach, and we did not have the resources to monitor the intensity with which participants approached QI. Furthermore, our data were dependent on self‐reports and were not verified by independent assessment of the fidelity with which the interventions were implemented, a checklist was embedded into usual care, or practices were enforced by nurses. In addition, the virtual collaborative circumvents the face‐to‐face learning sessions that might play a role in collaborative social networking, peer pressure, and acculturation.31, 36

Despite these limitations, we found that the Virtual Collaborative performed just like a Breakthrough Collaborative with a gradual uptake of implementation science using QI methods, team management, and statistical process control tools. The Toolkit condition had an even slower uptake. From an organization's perspective, the bottom‐line decision is whether a greater and meaningful proportion of collaborative participants will be successful to justify the investment of effort compared to a toolkit‐only approach. Our findings suggest that organizations engaged in change but lacking expertise in implementation science can potentially benefit from the acculturation, experiential learning, and uptake of QI provided by a collaborative.

In summary, although our Virtual Collaborative intervention was more likely to produce changes in ICU processes of care, there were no improvements in patient outcomes over this 18‐month study. The current popularity of evidence‐based guidelines, care protocols, prevention awareness, and surveillance may have produced a background of secular trend, making it difficult to ascertain effects of our QI intervention. Nonetheless, important lessons can be gleaned from this randomized controlled trial. Our study supports the proposition that as long as organizations vary in their capacity for and commitment to the science of QI and systems engineering, we should anticipate variation, uncertainty, and mixed results from short‐term, rapid cycle initiatives.27, 28, 31, 32, 39, 40 The untested, longer‐term benefit produced by a collaborative may be its stimulation of enduring systems engineering that optimizes an environment for QI of health care processes focused on desired outcomes.

Continuous quality improvement (CQI) methodologies provide a framework for initiating and sustaining improvements in complex systems.1 By definition, CQI engages frontline staff in iterative problem solving using plandostudyact cycles of learning, with decision‐making based on real‐time process measurements.2 The Institute for Healthcare Improvement (IHI) has sponsored Breakthrough Series Collaboratives since 1996 to accelerate the uptake and impact of quality improvement (QI).3, 4 These collaboratives are typically guided by evidence‐based clinical practice guidelines, incorporate change methodologies, and rely on clinical and process improvement subject matter experts. Through the collaborative network, teams share knowledge and ideas about effective and ineffective interventions as well as strategies for overcoming barriers. The collaborative curriculum includes CQI methodology, multidisciplinary teamwork, leadership support, and tools for measurement. Participants are typically required to invest resources and send teams to face‐to‐face goal‐oriented meetings. It is costly for a large healthcare organization to incorporate travel to a learning session conference into its collaborative model. Thus, we attempted virtual learning sessions that make use of webcasts, a Web site, and teleconference calls for tools and networking.5

A recent derivative of collaboratives has been deployment of toolkits for QI. Intuition suggests that such toolkits may help to enable change, and thus some agencies advocate the simpler approach of disseminating toolkits as a change strategy.6 Toolkit dissemination is a passive approach in contrast to collaborative participation, and its effectiveness has not been critically examined in evidence‐based literature.

We sought to compare the virtual collaborative model with the toolkit model for improving care. Recommendations and guidelines for central lineassociated bloodstream infection (CLABSI) and ventilator‐associated pneumonia (VAP) prevention have not been implemented reliably, resulting in unnecessary intensive care unit (ICU) morbidity and mortality and fostering a national call for improvement.7 Our aim was to compare the effectiveness of the virtual collaborative and toolkit approaches on preventing CLABSI and VAP in the ICU.

Methods

This cluster randomized trial included medical centers within the Hospital Corporation of America (HCA), a network of hospitals located primarily in the southern United States. To minimize contamination bias between study groups within the same facility, the unit of randomization was the hospital and implementation was at the level of the ICU. The project received approval from the Vanderbilt University Institutional Review Board.

Leaders of all medical centers with at least 1 adult or pediatric ICU received an invitation from HCA leadership to participate in a QI initiative. Facility clinicians and managers completed baseline surveys (shown in the Supporting Information) on hospital characteristics, types of ICUs, patient safety climate, and QI resources between July and November 2005. Hospital‐level data were extracted from the enterprise‐wide data warehouse. Hospitals willing to participate were matched on geographic location and ICU volume and then randomized into either the Virtual Collaborative (n = 31) or Toolkit (n = 30) groups in December 20058; 1 of the hospitals was sold, yielding 29 hospitals in the Toolkit (n = 29) group. The study lasted 18 months from January 2006 through September 2007, with health careassociated infection data collected through December 2007, and follow‐up data collection through April 2008.

The QI initiative included educational opportunities, evidence‐based clinical prevention interventions, and processes and tools to implement and measure the impact of these interventions. Participants in both groups were offered interactive Web seminars during the study period; 5 of these seminars were on clinical subject matter, and 5 seminars were on patient safety, charting use of statistical process control and QI methods. The interventions were evidence‐based care bundles.9 The key interventions for preventing CLABSI were routine hand hygiene, use of chlorhexidine skin antisepsis, maximal barrier precautions during catheter insertion, catheter site and care, and avoidance of routine replacement of catheters. The key interventions to prevent VAP were routine elevation of head of the bed, regular oral care, daily sedation vacations, daily assessment of readiness to extubate, secretion cleaning, peptic ulcer disease prophylaxis, and deep vein thrombosis prophylaxis.

Results

Participating facilities included rural (11%), inner city (28%), and suburban (61%) medical centers. The 60 participating sites did not differ in administrative variables from the 113 nonparticipating HCA sites (results not shown). The median hospital size was 177 beds and the median ICU size was 16 beds. The hospitals did not differ between study groups (Table 1). At baseline, 45% of the facilities reported having a CLABSI program and 62% a VAP program.

The baseline and quarterly median and pooled infection rates for the Toolkit and Collaboration groups are shown in Table 2 for CLABSI and in Table 3 for VAP. There were no significant differences in the baseline rates for either CLABSI (P = 0.24) or VAP (P = 0.72) between the Collaborative and Toolkit groups. There was no significant change for either CLABSI or VAP outcomes at either 12 or 18 months of follow‐up. The median bloodstream infection rate for all participating hospitals was 2.27 at baseline, 1.18 at 12 months (P = 0.13), and 2.23 per 1000 catheter days 18 months later (P = 0.95). The median VAP rate for participating hospitals was 2.90 at baseline, 2.67 at 12 months (P = 0.44), and 2.52 per 1000 ventilator days 18 months later (P = 0.84). The hierarchical regression analysis found that neither the Collaborative nor Toolkit groups improved CLABSI (P = 0.75 and P = 0.83, respectively) or VAP (P = 0.61 and P = 0.37, respectively) rates over time, and there was no differential performance between the 2 groups for either outcome (bloodstream infection, P = 0.71; VAP, P = 0.80).

The poststudy survey was completed by 27 of 31 (87%) of Collaborative group hospitals and 19 of the 29 (66%) Toolkit hospitals. Both groups reported QI improvement efforts to prevent CLABSI (Collaborative 97% vs. Toolkit 88%, P = 0.29) and VAP (Collaborative 97% vs. Toolkit 96%, P = 0.99). Eighty‐three percent of the Collaborative group implemented all components of the bloodstream infection prevention interventions compared with 64% for the Toolkit group (P = 0.13; Figure 1). The Collaborative group implemented daily catheter review more often than the Toolkit group (P = 0.04) and began the process implementation sooner following study implementation (P = 0.006 vs. Toolkit; see Supporting Information Figure). Eighty‐six percent of the Collaborative group implemented the complete VAP prevention interventions vs. 64% of the Toolkit group (P = 0.06; Figure 1) and the Collaborative group conducted the sedation vacation intervention more often (P = 0.03).

Figure 1

The Collaborative group participated in 57% of the seminars, whereas the Toolkit group participated in 39% (P = 0.014). Members of both groups attended more than half the clinical topics (Collaborative 64% vs. Toolkit 56%, P = 0.37). The Collaborative group had greater participation in the data and method topics (Collaborative 50% vs. Toolkit 22%, P < 0.001). The proportion of hospitals finding the seminars useful to their QI efforts was 49% for the Collaborative and 30% for the Toolkit group (P = 0.017). When restricted to hospitals that participated in the seminars, the usefulness rating was higher for both clinical (91% for the Collaborative and 86% for Toolkit) and Data/Methods (79% for Collaborative and 55% for Toolkit) topics.

A set of 14 tools were produced during the study period (Table 4); 9 clinically related tools (eg, checklists, algorithms, protocols, and flowsheets) and 5 data monitoring and quality improvement tools (eg, easy‐to‐use statistical process control spreadsheet templates, quality improvement tools, and computer tools). The Collaborative group downloaded a median of 10 tools and the Toolkit group a median of 7 (P = 0.051). The groups did not differ in their access to the clinical tools (P = 0.23) but the Collaborative group accessed a greater proportion of the data/methods tools (P = 0.004).

Both groups relied primarily on implementation of protocols and informatics approaches (Table 4) without increasing staff levels. The predominant strategy was education; both groups provided written educational materials and classes to their providers. There was a trend for more Collaborative group members to implement QI teams (Table 4, P = 0.16 compared with the Toolkit group). Although the preponderance of both groups provided feedback reports to their hospital leaders and unit managers, Collaborative group hospitals showed a trend for providing feedback to front‐line providers (P = 0.11). With respect to self‐reported interventions, 78% of the Collaborative ICUs reported implementing a CLABSI checklist and 86% a VAP checklist, whereas only 60% of the Toolkit group reported implementation of a CLABSI checklist (P = 0.16) and 52% a VAP checklist (P = 0.007). Once a tool was implemented, both groups reported a high rate of sustaining the implementation (ranging from 86% to 100%). There also seemed to be a pattern of sequencing the interventions. Initial efforts tend to focus on provider education and evidence‐based protocols. Later efforts include more formal formation of QI teams followed by implementation of checklists. The evidence for sequencing of interventions is qualitative; we lacked subgroup sample size to substantiate these results with statistical analysis.

Discussion

In our investigation of Virtual Collaborative and Toolkit strategies for spreading the implementation of safe practices for CLABSI and VAP, ICUs in the Collaborative group had more complete implementation of the processes for prevention of hospital‐associated infections. Although both groups accessed clinical resources consistent with surveillance and clinical education, the Virtual Collaborative group attended to data and implementation methods more likely to lead to systemic CQI and organizational changes. ICUs that engaged these resources believed them useful in implementing QI, and more than 85% of the practices were sustained once integrated into routine care. Although the Collaborative ICUs were about 50% more likely to implement improvement strategies, these differences in implementation and process of care did not translate into group differences or longitudinal changes in infection rates.

In contrast to the context of our investigation, most published QI studies on health careassociated infection prevention report high baseline rates followed by a significant decline in infection rates.1419 The baseline infection rates in our study hospitals were actually below the endpoint found in many prior studies, suggesting that any marginal effects from our intervention would be more difficult to detect. Our study was implemented during the IHI's 100,000 Lives Campaign,20 a trend that may have brought about these lower baseline rates and thus a tighter margin for improvement.

The median CLABSI baseline rate in the well‐publicized Michigan hospital study was 2.7 per 1000 catheter days.21, 22 Although our baseline rate was similar (2.27 per 1000 catheter days), their reported postintervention rate was near zero, inferring nearly total elimination of the risk for CLABSI within 3‐18 months of study implementation. Several other studies using a collaborative approach have similarly reported high‐performance near‐zero results in reducing VAP23, 24 and CLABSI2528 rates. The difference between the present and previously published near‐zero result outcomes raises questions about collaboration‐based studies. We noticed 2 phenomena. First, there was slow uptake of data‐driven QI, and second, there was a differential uptake between general knowledge (clinical evidence and education) and QI implementation knowledge.29, 30

Lack of infrastructure to support data‐driven QI remains a significant barrier throughout the health care system, and teams in collaboratives often must work intensively toward improving their information systems' capability for the purpose of data‐driven decision support.1, 15, 31, 32 Systematic, standardized collection of CLABSI and VAP outcomes was initially lacking in many of our study hospitals,10 and our project expended early effort to deploy a system‐wide standardized infection control database registry.

Both of our study groups gravitated toward educational training and evidence‐based protocol decision‐support strategies. A focus only on established surveillance and education‐based fixes (eg, asking clinicians to follow a protocol within their existing care processes) have produced 32% to 57% reductions in health careacquired infections.3335 These early gains, however, are unlikely to produce the sustained near‐zero results that some collaborative teams have reported.22, 25

The ability to achieve sustained high‐performance results depends on organizational context and requires time.31 A potential benefit of collaboratives might be the return on investment attained by organizational change in quality and safety climate and its influence across the whole organization.19, 31, 36 Participants requiring systems training in the CQI process may not gain these benefits until well into their collaborative.31 For example, accumulating evidence demonstrates that the use of checklists can reduce errors of omission. Although a checklist seems a simple intervention, its effective implementation into routine care processes actually requires time for system redesign that addresses changes in multidisciplinary roles and responsibilities, frontline clinician and mid‐level management buy‐in, new methods of data collection and feedback, unanticipated involvement of ancillary services (eg, medical records, housekeeping), as well as changes to organizational policies, expectations, and priorities that connect silos of care and integrate hierarchical operations. Wall et al.37 and Pronovost and colleagues19, 21, 22, 25 highlighted the strategic effectiveness of embedding a checklist as a behavioral and data collection tool into frontline care process, leading to a redefined role of nursing, as well as new data for further cycles of improvement that collectively reduced infection rates. In our study, the Virtual Collaborative group did not have greater use of CLABSI and VAP checklists until the QI teams had been formed months into the project, consistent with the hypothesis that beneficial translation of desired changes in process of care to observed improvements in patient outcomes may take longer than 18 months to achieve19, 25, 27, 38 as opposed to the remarkable 3 months reported in the Keystone ICU project.21

Our study has several limitations. Our intervention did not mandate fixed specific components of intervention or QI methods. Each medical center was free to tailor its use of tools and change ideas, producing site variation in implementation methods and investment in support of QI. Like other multicomponent, multidimensional intervention studies, we were not able to test the effectiveness of particular QI components or the thoroughness of surveillance for CLABSI and VAP related to efforts to standardize the approach, and we did not have the resources to monitor the intensity with which participants approached QI. Furthermore, our data were dependent on self‐reports and were not verified by independent assessment of the fidelity with which the interventions were implemented, a checklist was embedded into usual care, or practices were enforced by nurses. In addition, the virtual collaborative circumvents the face‐to‐face learning sessions that might play a role in collaborative social networking, peer pressure, and acculturation.31, 36

Despite these limitations, we found that the Virtual Collaborative performed just like a Breakthrough Collaborative with a gradual uptake of implementation science using QI methods, team management, and statistical process control tools. The Toolkit condition had an even slower uptake. From an organization's perspective, the bottom‐line decision is whether a greater and meaningful proportion of collaborative participants will be successful to justify the investment of effort compared to a toolkit‐only approach. Our findings suggest that organizations engaged in change but lacking expertise in implementation science can potentially benefit from the acculturation, experiential learning, and uptake of QI provided by a collaborative.

In summary, although our Virtual Collaborative intervention was more likely to produce changes in ICU processes of care, there were no improvements in patient outcomes over this 18‐month study. The current popularity of evidence‐based guidelines, care protocols, prevention awareness, and surveillance may have produced a background of secular trend, making it difficult to ascertain effects of our QI intervention. Nonetheless, important lessons can be gleaned from this randomized controlled trial. Our study supports the proposition that as long as organizations vary in their capacity for and commitment to the science of QI and systems engineering, we should anticipate variation, uncertainty, and mixed results from short‐term, rapid cycle initiatives.27, 28, 31, 32, 39, 40 The untested, longer‐term benefit produced by a collaborative may be its stimulation of enduring systems engineering that optimizes an environment for QI of health care processes focused on desired outcomes.