Reports From the Field

From Health Center, New York University Abu Dhabi, Abu Dhabi, United Arab Emirates (Dr. Virji and Aisha Al Hamiz), Public Health, Abu Dhabi Public Health Center, Abu Dhabi, United Arab Emirates (Drs. Al Hajeri, Al Shehhi, Al Memari, and Ahlam Al Maskari), College of Medicine and Health Sciences, Khalifa University, Abu Dhabi, United Arab Emirates, Department of Medicine, Sheikh Shakhbout Medical City, Abu Dhabi, United Arab Emirates (Dr. Alhajri), Public Health Research Center, New York University Abu Dhabi, Abu Dhabi, United Arab Emirates, Oxford University Hospitals NHS Foundation Trust, Oxford, England, and the MRC Epidemiology Unit, University of Cambridge, Cambridge, England (Dr. Ali).

Objective: The pandemic has forced closures of primary schools, resulting in loss of learning time on a global scale. In addition to face coverings, social distancing, and hand hygiene, an efficient testing method is important to mitigate the spread of COVID-19 in schools. We evaluated the feasibility of a saliva-based SARS-CoV-2 polymerase chain reaction testing program among 18 primary schools in the Emirate of Abu Dhabi, United Arab Emirates. Qualitative results show that children 4 to 5 years old had difficulty producing an adequate saliva specimen compared to those 6 to 12 years old.

Methods: A short training video on saliva collection beforehand helps demystify the process for students and parents alike. Informed consent was challenging yet should be done beforehand by school health nurses or other medical professionals to reassure parents and maximize participation.

Results: Telephone interviews with school administrators resulted in an 83% response rate. Overall, 93% of school administrators had a positive experience with saliva testing and felt the program improved the safety of their schools. The ongoing use of saliva testing for SARS-CoV-2 was supported by 73% of respondents.

Conclusion: On-campus saliva testing is a feasible option for primary schools to screen for COVID-19 in their student population to help keep their campuses safe and open for learning.

Keywords: COVID-19; saliva testing; mitigation; primary school.

The COVID-19 pandemic is a leading cause of morbidity and mortality worldwide and continues to exhaust health care resources on a large scale.¹ Efficient testing is critical to identify cases early and to help mitigate the deleterious effects of the pandemic.² Saliva polymerase chain reaction (PCR) nucleic acid amplification testing (NAAT) is more comfortable than nasopharyngeal (NP) NAAT and has been validated as a test for SARS-CoV-2.¹ Although children are less susceptible to severe disease, primary schools are considered a vector for transmission and community spread.³ Efficient and scalable methods of routine testing are needed globally to help keep schools open. Saliva testing has proven a useful resource for this population.^4,5

Abu Dhabi is the largest Emirate in the United Arab Emirates (UAE), with an estimated population of 2.5 million.⁶ The first case of COVID-19 was discovered in the UAE on January 29, 2020.⁷ The UAE has been recognized worldwide for its robust pandemic response. Along with the coordinated and swift application of public health measures, the country has one of the highest COVID-19 testing rates per capita and one of the highest vaccination rates worldwide.^8,9 The Abu Dhabi Public Health Center (ADPHC) works alongside the Ministry of Education (MOE) to establish testing, quarantine, and general safety guidelines for primary schools. In December 2020, the ADPHC partnered with a local, accredited diagnostic laboratory to test the feasibility of a saliva-based screening program for COVID-19 directly on school campuses for 18 primary schools in the Emirate.

Saliva-based PCR testing for COVID-19 was approved for use in schools in the UAE on January 24, 2021.¹⁰ As part of a greater mitigation strategy to reduce both school-based transmission and, hence, community spread, the ADPHC focused its on-site testing program on children aged 4 to 12 years. The program required collaboration among medical professionals, school administrators and teachers, students, and parents. Our study evaluates the feasibility of implementing a saliva-based COVID-19 screening program directly on primary school campuses involving children as young as 4 years of age.

Methods

The ADPHC, in collaboration with G42 Biogenix Labs, conducted a saliva SARS-CoV-2 NAAT testing program in 18 primary schools in the Emirate. Schools were selected based on outbreak prevalence at the time and focused on “hot spot” areas. The school on-site saliva testing program included children aged 4 to 12 years old in a “bubble” attendance model during the school day. This model involved children being assigned to groups or “pods.” This allowed us to limit a potential outbreak to a single pod, as opposed to risk exposing the entire school, should a single student test positive. The well-established SalivaDirect protocol developed at Yale University was used for testing and included an RNA extraction-free, RT-qPCR method for SARS-CoV-2 detection.¹¹

We conducted a qualitative study involving telephone interviews of school administrators to evaluate their experience with the ADPHC testing program at their schools. In addition, we interviewed the G42 Biogenix Lab providers to understand the logistics that supported on-campus collection of saliva specimens for this age group. We also gathered the attitudes of school children before and after testing. This study was reviewed and approved by the Abu Dhabi Health Research and Technology Committee and the Institutional Review Board (IRB), New York University Abu Dhabi (NYUAD).

Sample and recruitment

The original sample collection of saliva specimens was performed by the ADPHC in collaboration with G42 Biogenix Lab providers on school campuses between December 6 and December 10, 2020. During this time, schools operated in a hybrid teaching model, where learning took place both online and in person. Infection control measures were deployed based on ADPHC standards and guidelines. Nurses utilized appropriate patient protective equipment, frequent hand hygiene, and social distancing during the collection process. Inclusion criteria included asymptomatic students aged 4 to 12 years attending in-person classes on campus. Students with respiratory symptoms who were asked to stay home or those not attending in-person classes were excluded.

Data with regard to school children’s attitudes before and after testing were compiled through an online survey sent randomly to participants postintervention. Data from school administrators were collected through video and telephone interviews between April 14 and April 29, 2021. We first interviewed G42 Biogenix Lab providers to obtain previously acquired qualitative and quantitative data, which were collected during the intervention itself. After obtaining this information, we designed a questionnaire and proceeded with a structured interview process for school officials.

We interviewed school principals and administrators to collect their overall experiences with the saliva testing program. Before starting each interview, we established the interviewees preferred language, either English or Arabic. We then introduced the meeting attendees and provided study details, aims, and objectives, and described collaborating entities. We obtained verbal informed consent from a script approved by the NYUAD IRB and then proceeded with the interview, which included 4 questions. The first 3 questions were answered on a 5-point Likert scale model that consisted of 5 answer options: 5 being completely agree, 4 agree, 3 somewhat agree, 2 somewhat disagree, and 1 completely disagree. The fourth question invited open-ended feedback and comments on the following statements:

1. I believe the COVID-19 saliva testing program improved the safety for my school campus.
2. Our community had an overall positive experience with the COVID saliva testing.
3. We would like to continue a saliva-based COVID testing program on our school campus.
4. Please provide any additional comments you feel important about the program.

During the interview, we transcribed the answers as the interviewee was answering. We then translated those in Arabic into English and collected the data in 1 Excel spreadsheet. School interviewees and school names were de-identified in the collection and storage process.

Results

A total of 2011 saliva samples were collected from 18 different primary school campuses. Samples were sent the same day to G42 Biogenix Labs in Abu Dhabi for COVID PCR testing. A team consisting of 5 doctors providing general oversight, along with 2 to 6 nurses per site, were able to manage the collection process for all 18 school campuses. Samples were collected between 8 am and 2 pm and required variation among sites depending on factors such as staff availability and class schedules. Additional scheduling challenges included compliance with public safety mandates involving the maintenance of defined “safety bubbles” that forbid certain personnel from moving between floors, and the avoidance of mixing students from different classes.

Sample stations were set up in either the school auditorium or gymnasium to ensure appropriate crowd control and ventilation. Teachers and other school staff, including public safety, were able to manage lines and the shuttling of students back and forth from classes to testing stations, which allowed medical staff to focus on sample collection.

Informed consent was obtained by prior electronic communication to parents from school staff, asking them to agree to allow their child to participate in the testing program. Informed consent was identified as a challenge: Getting parents to understand that saliva testing was more comfortable than NP testing, and that the results were only being used to help keep the school safe, took time. School staff are used to obtaining consent from parents for field trips, but this was clearly more challenging for them.

The saliva collection process per child took more time than expected. Children fasted for 45 minutes before saliva collection. We used an active drool technique, which required children to pool saliva in their mouth then express it into a collection tube. Adults can generally do this on command, but we found it took 10 to 12 minutes per child. Saliva production was cued by asking the children to think about food, and by showing them pictures and TV commercials depicting food. Children 4 to 5 years old had more difficulty with the process despite active cueing, while those 6 to 12 years old had an easier time with the process. We collected data on a cohort of 80 children regarding their attitudes pre (Figure 1) and post collection (Figure 2). Children felt happier, less nervous, and less scared after collection than before collection. This trend reassured us that future collections would be easier for students.

A total of 15 of 18 school principals completed the telephone interview, yielding a response rate of 83%. Overall, 93% of the school principals agreed or completely agreed that the COVID-19 saliva testing program improved school safety; 93% agreed or completely agreed that they had an overall positive experience with the program; and 73% supported the ongoing use of saliva testing in their schools (Table 1). Administrators’ open-ended comments on their experience were positive overall (Table 2).

Discussion

By March 2020, many kindergarten to grade 12 public and private schools suspended in-person classes due to the pandemic and turned to online learning platforms. The negative impact of school closures on academic achievement is projected to be significant.^7,12,13 Ensuring schools can stay open and run operations safely will require routine SARS-CoV-2 testing. Our study investigated the feasibility of routine saliva testing on children aged 4 to 12 years on their school campuses. The ADPHC school on-site saliva testing program involved bringing lab providers onto 18 primary school campuses and required cooperation among parents, students, school administrators, and health care professionals.

Children younger than 6 years had difficulty producing an adequate saliva specimen, whereas those 6 to 12 years did so with relative ease when cued by thoughts or pictures of food while waiting in line for collection. Schools considering on-site testing programs should consider the age range of 6 to 12 years as a viable age range for saliva screening. Children should fast for a minimum of 45 minutes prior to saliva collection and should be cued by thoughts of food, food pictures, or food commercials. Setting up a sampling station close to the cafeteria where students can smell meal preparation may also help.^14,15 Sampling before breakfast or lunch, when children are potentially at their hungriest, should also be considered.

The greatest challenge was obtaining informed consent from parents who were not yet familiar with the reliability of saliva testing as a tool for SARS-CoV-2 screening or with the saliva collection process as a whole. Informed consent was initially done electronically, lacking direct human interaction to answer parents’ questions. Parents who refused had a follow-up call from the school nurse to further explain the logistics and rationale for saliva screening. Having medical professionals directly answer parents’ questions was helpful. Parents were reassured that the process was painless, confidential, and only to be used for school safety purposes. Despite school administrators being experienced in obtaining consent from parents for field trips, obtaining informed consent for a medical testing procedure is more complicated, and parents aren’t accustomed to providing such consent in a school environment. Schools considering on-site testing should ensure that their school nurse or other health care providers are on the front line obtaining informed consent and allaying parents’ fears.

School staff were able to effectively provide crowd control for testing, and children felt at ease being in a familiar environment. Teachers and public safety officers are well-equipped at managing the shuttling of students to class, to lunch, to physical education, and, finally, to dismissal. They were equally equipped at handling the logistics of students to and from testing, including minimizing crowds and helping students feel at ease during the process. This effective collaboration allowed the lab personnel to focus on sample collection and storage, while school staff managed all other aspects of the children’s safety and care.

Conclusion

Overall, school administrators had a positive experience with the testing program, felt the program improved the safety of their schools, and supported the ongoing use of saliva testing for SARS-CoV-2 on their school campuses. Children aged 6 years and older were able to provide adequate saliva samples, and children felt happier and less nervous after the process, indicating repeatability. Our findings highlight the feasibility of an integrated on-site saliva testing model for primary school campuses. Further research is needed to determine the scalability of such a model and whether the added compliance and safety of on-site testing compensates for the potential loss of learning time that testing during school hours would require.

Corresponding author: Ayaz Virji, MD, New York University Abu Dhabi, PO Box 129188, Abu Dhabi, United Arab Emirates; av102@nyu.edu.

Financial disclosures: None.

From Health Center, New York University Abu Dhabi, Abu Dhabi, United Arab Emirates (Dr. Virji and Aisha Al Hamiz), Public Health, Abu Dhabi Public Health Center, Abu Dhabi, United Arab Emirates (Drs. Al Hajeri, Al Shehhi, Al Memari, and Ahlam Al Maskari), College of Medicine and Health Sciences, Khalifa University, Abu Dhabi, United Arab Emirates, Department of Medicine, Sheikh Shakhbout Medical City, Abu Dhabi, United Arab Emirates (Dr. Alhajri), Public Health Research Center, New York University Abu Dhabi, Abu Dhabi, United Arab Emirates, Oxford University Hospitals NHS Foundation Trust, Oxford, England, and the MRC Epidemiology Unit, University of Cambridge, Cambridge, England (Dr. Ali).

Objective: The pandemic has forced closures of primary schools, resulting in loss of learning time on a global scale. In addition to face coverings, social distancing, and hand hygiene, an efficient testing method is important to mitigate the spread of COVID-19 in schools. We evaluated the feasibility of a saliva-based SARS-CoV-2 polymerase chain reaction testing program among 18 primary schools in the Emirate of Abu Dhabi, United Arab Emirates. Qualitative results show that children 4 to 5 years old had difficulty producing an adequate saliva specimen compared to those 6 to 12 years old.

Methods: A short training video on saliva collection beforehand helps demystify the process for students and parents alike. Informed consent was challenging yet should be done beforehand by school health nurses or other medical professionals to reassure parents and maximize participation.

Results: Telephone interviews with school administrators resulted in an 83% response rate. Overall, 93% of school administrators had a positive experience with saliva testing and felt the program improved the safety of their schools. The ongoing use of saliva testing for SARS-CoV-2 was supported by 73% of respondents.

Conclusion: On-campus saliva testing is a feasible option for primary schools to screen for COVID-19 in their student population to help keep their campuses safe and open for learning.

Keywords: COVID-19; saliva testing; mitigation; primary school.

The COVID-19 pandemic is a leading cause of morbidity and mortality worldwide and continues to exhaust health care resources on a large scale.¹ Efficient testing is critical to identify cases early and to help mitigate the deleterious effects of the pandemic.² Saliva polymerase chain reaction (PCR) nucleic acid amplification testing (NAAT) is more comfortable than nasopharyngeal (NP) NAAT and has been validated as a test for SARS-CoV-2.¹ Although children are less susceptible to severe disease, primary schools are considered a vector for transmission and community spread.³ Efficient and scalable methods of routine testing are needed globally to help keep schools open. Saliva testing has proven a useful resource for this population.^4,5

Abu Dhabi is the largest Emirate in the United Arab Emirates (UAE), with an estimated population of 2.5 million.⁶ The first case of COVID-19 was discovered in the UAE on January 29, 2020.⁷ The UAE has been recognized worldwide for its robust pandemic response. Along with the coordinated and swift application of public health measures, the country has one of the highest COVID-19 testing rates per capita and one of the highest vaccination rates worldwide.^8,9 The Abu Dhabi Public Health Center (ADPHC) works alongside the Ministry of Education (MOE) to establish testing, quarantine, and general safety guidelines for primary schools. In December 2020, the ADPHC partnered with a local, accredited diagnostic laboratory to test the feasibility of a saliva-based screening program for COVID-19 directly on school campuses for 18 primary schools in the Emirate.

Saliva-based PCR testing for COVID-19 was approved for use in schools in the UAE on January 24, 2021.¹⁰ As part of a greater mitigation strategy to reduce both school-based transmission and, hence, community spread, the ADPHC focused its on-site testing program on children aged 4 to 12 years. The program required collaboration among medical professionals, school administrators and teachers, students, and parents. Our study evaluates the feasibility of implementing a saliva-based COVID-19 screening program directly on primary school campuses involving children as young as 4 years of age.

Methods

The ADPHC, in collaboration with G42 Biogenix Labs, conducted a saliva SARS-CoV-2 NAAT testing program in 18 primary schools in the Emirate. Schools were selected based on outbreak prevalence at the time and focused on “hot spot” areas. The school on-site saliva testing program included children aged 4 to 12 years old in a “bubble” attendance model during the school day. This model involved children being assigned to groups or “pods.” This allowed us to limit a potential outbreak to a single pod, as opposed to risk exposing the entire school, should a single student test positive. The well-established SalivaDirect protocol developed at Yale University was used for testing and included an RNA extraction-free, RT-qPCR method for SARS-CoV-2 detection.¹¹

We conducted a qualitative study involving telephone interviews of school administrators to evaluate their experience with the ADPHC testing program at their schools. In addition, we interviewed the G42 Biogenix Lab providers to understand the logistics that supported on-campus collection of saliva specimens for this age group. We also gathered the attitudes of school children before and after testing. This study was reviewed and approved by the Abu Dhabi Health Research and Technology Committee and the Institutional Review Board (IRB), New York University Abu Dhabi (NYUAD).

Sample and recruitment

The original sample collection of saliva specimens was performed by the ADPHC in collaboration with G42 Biogenix Lab providers on school campuses between December 6 and December 10, 2020. During this time, schools operated in a hybrid teaching model, where learning took place both online and in person. Infection control measures were deployed based on ADPHC standards and guidelines. Nurses utilized appropriate patient protective equipment, frequent hand hygiene, and social distancing during the collection process. Inclusion criteria included asymptomatic students aged 4 to 12 years attending in-person classes on campus. Students with respiratory symptoms who were asked to stay home or those not attending in-person classes were excluded.

Data with regard to school children’s attitudes before and after testing were compiled through an online survey sent randomly to participants postintervention. Data from school administrators were collected through video and telephone interviews between April 14 and April 29, 2021. We first interviewed G42 Biogenix Lab providers to obtain previously acquired qualitative and quantitative data, which were collected during the intervention itself. After obtaining this information, we designed a questionnaire and proceeded with a structured interview process for school officials.

We interviewed school principals and administrators to collect their overall experiences with the saliva testing program. Before starting each interview, we established the interviewees preferred language, either English or Arabic. We then introduced the meeting attendees and provided study details, aims, and objectives, and described collaborating entities. We obtained verbal informed consent from a script approved by the NYUAD IRB and then proceeded with the interview, which included 4 questions. The first 3 questions were answered on a 5-point Likert scale model that consisted of 5 answer options: 5 being completely agree, 4 agree, 3 somewhat agree, 2 somewhat disagree, and 1 completely disagree. The fourth question invited open-ended feedback and comments on the following statements:

1. I believe the COVID-19 saliva testing program improved the safety for my school campus.
2. Our community had an overall positive experience with the COVID saliva testing.
3. We would like to continue a saliva-based COVID testing program on our school campus.
4. Please provide any additional comments you feel important about the program.

During the interview, we transcribed the answers as the interviewee was answering. We then translated those in Arabic into English and collected the data in 1 Excel spreadsheet. School interviewees and school names were de-identified in the collection and storage process.

Results

A total of 2011 saliva samples were collected from 18 different primary school campuses. Samples were sent the same day to G42 Biogenix Labs in Abu Dhabi for COVID PCR testing. A team consisting of 5 doctors providing general oversight, along with 2 to 6 nurses per site, were able to manage the collection process for all 18 school campuses. Samples were collected between 8 am and 2 pm and required variation among sites depending on factors such as staff availability and class schedules. Additional scheduling challenges included compliance with public safety mandates involving the maintenance of defined “safety bubbles” that forbid certain personnel from moving between floors, and the avoidance of mixing students from different classes.

Sample stations were set up in either the school auditorium or gymnasium to ensure appropriate crowd control and ventilation. Teachers and other school staff, including public safety, were able to manage lines and the shuttling of students back and forth from classes to testing stations, which allowed medical staff to focus on sample collection.

Informed consent was obtained by prior electronic communication to parents from school staff, asking them to agree to allow their child to participate in the testing program. Informed consent was identified as a challenge: Getting parents to understand that saliva testing was more comfortable than NP testing, and that the results were only being used to help keep the school safe, took time. School staff are used to obtaining consent from parents for field trips, but this was clearly more challenging for them.

The saliva collection process per child took more time than expected. Children fasted for 45 minutes before saliva collection. We used an active drool technique, which required children to pool saliva in their mouth then express it into a collection tube. Adults can generally do this on command, but we found it took 10 to 12 minutes per child. Saliva production was cued by asking the children to think about food, and by showing them pictures and TV commercials depicting food. Children 4 to 5 years old had more difficulty with the process despite active cueing, while those 6 to 12 years old had an easier time with the process. We collected data on a cohort of 80 children regarding their attitudes pre (Figure 1) and post collection (Figure 2). Children felt happier, less nervous, and less scared after collection than before collection. This trend reassured us that future collections would be easier for students.

A total of 15 of 18 school principals completed the telephone interview, yielding a response rate of 83%. Overall, 93% of the school principals agreed or completely agreed that the COVID-19 saliva testing program improved school safety; 93% agreed or completely agreed that they had an overall positive experience with the program; and 73% supported the ongoing use of saliva testing in their schools (Table 1). Administrators’ open-ended comments on their experience were positive overall (Table 2).

Discussion

By March 2020, many kindergarten to grade 12 public and private schools suspended in-person classes due to the pandemic and turned to online learning platforms. The negative impact of school closures on academic achievement is projected to be significant.^7,12,13 Ensuring schools can stay open and run operations safely will require routine SARS-CoV-2 testing. Our study investigated the feasibility of routine saliva testing on children aged 4 to 12 years on their school campuses. The ADPHC school on-site saliva testing program involved bringing lab providers onto 18 primary school campuses and required cooperation among parents, students, school administrators, and health care professionals.

Children younger than 6 years had difficulty producing an adequate saliva specimen, whereas those 6 to 12 years did so with relative ease when cued by thoughts or pictures of food while waiting in line for collection. Schools considering on-site testing programs should consider the age range of 6 to 12 years as a viable age range for saliva screening. Children should fast for a minimum of 45 minutes prior to saliva collection and should be cued by thoughts of food, food pictures, or food commercials. Setting up a sampling station close to the cafeteria where students can smell meal preparation may also help.^14,15 Sampling before breakfast or lunch, when children are potentially at their hungriest, should also be considered.

The greatest challenge was obtaining informed consent from parents who were not yet familiar with the reliability of saliva testing as a tool for SARS-CoV-2 screening or with the saliva collection process as a whole. Informed consent was initially done electronically, lacking direct human interaction to answer parents’ questions. Parents who refused had a follow-up call from the school nurse to further explain the logistics and rationale for saliva screening. Having medical professionals directly answer parents’ questions was helpful. Parents were reassured that the process was painless, confidential, and only to be used for school safety purposes. Despite school administrators being experienced in obtaining consent from parents for field trips, obtaining informed consent for a medical testing procedure is more complicated, and parents aren’t accustomed to providing such consent in a school environment. Schools considering on-site testing should ensure that their school nurse or other health care providers are on the front line obtaining informed consent and allaying parents’ fears.

School staff were able to effectively provide crowd control for testing, and children felt at ease being in a familiar environment. Teachers and public safety officers are well-equipped at managing the shuttling of students to class, to lunch, to physical education, and, finally, to dismissal. They were equally equipped at handling the logistics of students to and from testing, including minimizing crowds and helping students feel at ease during the process. This effective collaboration allowed the lab personnel to focus on sample collection and storage, while school staff managed all other aspects of the children’s safety and care.

Conclusion

Overall, school administrators had a positive experience with the testing program, felt the program improved the safety of their schools, and supported the ongoing use of saliva testing for SARS-CoV-2 on their school campuses. Children aged 6 years and older were able to provide adequate saliva samples, and children felt happier and less nervous after the process, indicating repeatability. Our findings highlight the feasibility of an integrated on-site saliva testing model for primary school campuses. Further research is needed to determine the scalability of such a model and whether the added compliance and safety of on-site testing compensates for the potential loss of learning time that testing during school hours would require.

Corresponding author: Ayaz Virji, MD, New York University Abu Dhabi, PO Box 129188, Abu Dhabi, United Arab Emirates; av102@nyu.edu.

Financial disclosures: None.

From Health Center, New York University Abu Dhabi, Abu Dhabi, United Arab Emirates (Dr. Virji and Aisha Al Hamiz), Public Health, Abu Dhabi Public Health Center, Abu Dhabi, United Arab Emirates (Drs. Al Hajeri, Al Shehhi, Al Memari, and Ahlam Al Maskari), College of Medicine and Health Sciences, Khalifa University, Abu Dhabi, United Arab Emirates, Department of Medicine, Sheikh Shakhbout Medical City, Abu Dhabi, United Arab Emirates (Dr. Alhajri), Public Health Research Center, New York University Abu Dhabi, Abu Dhabi, United Arab Emirates, Oxford University Hospitals NHS Foundation Trust, Oxford, England, and the MRC Epidemiology Unit, University of Cambridge, Cambridge, England (Dr. Ali).

Objective: The pandemic has forced closures of primary schools, resulting in loss of learning time on a global scale. In addition to face coverings, social distancing, and hand hygiene, an efficient testing method is important to mitigate the spread of COVID-19 in schools. We evaluated the feasibility of a saliva-based SARS-CoV-2 polymerase chain reaction testing program among 18 primary schools in the Emirate of Abu Dhabi, United Arab Emirates. Qualitative results show that children 4 to 5 years old had difficulty producing an adequate saliva specimen compared to those 6 to 12 years old.

Methods: A short training video on saliva collection beforehand helps demystify the process for students and parents alike. Informed consent was challenging yet should be done beforehand by school health nurses or other medical professionals to reassure parents and maximize participation.

Results: Telephone interviews with school administrators resulted in an 83% response rate. Overall, 93% of school administrators had a positive experience with saliva testing and felt the program improved the safety of their schools. The ongoing use of saliva testing for SARS-CoV-2 was supported by 73% of respondents.

Conclusion: On-campus saliva testing is a feasible option for primary schools to screen for COVID-19 in their student population to help keep their campuses safe and open for learning.

Keywords: COVID-19; saliva testing; mitigation; primary school.

The COVID-19 pandemic is a leading cause of morbidity and mortality worldwide and continues to exhaust health care resources on a large scale.¹ Efficient testing is critical to identify cases early and to help mitigate the deleterious effects of the pandemic.² Saliva polymerase chain reaction (PCR) nucleic acid amplification testing (NAAT) is more comfortable than nasopharyngeal (NP) NAAT and has been validated as a test for SARS-CoV-2.¹ Although children are less susceptible to severe disease, primary schools are considered a vector for transmission and community spread.³ Efficient and scalable methods of routine testing are needed globally to help keep schools open. Saliva testing has proven a useful resource for this population.^4,5

Abu Dhabi is the largest Emirate in the United Arab Emirates (UAE), with an estimated population of 2.5 million.⁶ The first case of COVID-19 was discovered in the UAE on January 29, 2020.⁷ The UAE has been recognized worldwide for its robust pandemic response. Along with the coordinated and swift application of public health measures, the country has one of the highest COVID-19 testing rates per capita and one of the highest vaccination rates worldwide.^8,9 The Abu Dhabi Public Health Center (ADPHC) works alongside the Ministry of Education (MOE) to establish testing, quarantine, and general safety guidelines for primary schools. In December 2020, the ADPHC partnered with a local, accredited diagnostic laboratory to test the feasibility of a saliva-based screening program for COVID-19 directly on school campuses for 18 primary schools in the Emirate.

Saliva-based PCR testing for COVID-19 was approved for use in schools in the UAE on January 24, 2021.¹⁰ As part of a greater mitigation strategy to reduce both school-based transmission and, hence, community spread, the ADPHC focused its on-site testing program on children aged 4 to 12 years. The program required collaboration among medical professionals, school administrators and teachers, students, and parents. Our study evaluates the feasibility of implementing a saliva-based COVID-19 screening program directly on primary school campuses involving children as young as 4 years of age.

Methods

The ADPHC, in collaboration with G42 Biogenix Labs, conducted a saliva SARS-CoV-2 NAAT testing program in 18 primary schools in the Emirate. Schools were selected based on outbreak prevalence at the time and focused on “hot spot” areas. The school on-site saliva testing program included children aged 4 to 12 years old in a “bubble” attendance model during the school day. This model involved children being assigned to groups or “pods.” This allowed us to limit a potential outbreak to a single pod, as opposed to risk exposing the entire school, should a single student test positive. The well-established SalivaDirect protocol developed at Yale University was used for testing and included an RNA extraction-free, RT-qPCR method for SARS-CoV-2 detection.¹¹

We conducted a qualitative study involving telephone interviews of school administrators to evaluate their experience with the ADPHC testing program at their schools. In addition, we interviewed the G42 Biogenix Lab providers to understand the logistics that supported on-campus collection of saliva specimens for this age group. We also gathered the attitudes of school children before and after testing. This study was reviewed and approved by the Abu Dhabi Health Research and Technology Committee and the Institutional Review Board (IRB), New York University Abu Dhabi (NYUAD).

Sample and recruitment

The original sample collection of saliva specimens was performed by the ADPHC in collaboration with G42 Biogenix Lab providers on school campuses between December 6 and December 10, 2020. During this time, schools operated in a hybrid teaching model, where learning took place both online and in person. Infection control measures were deployed based on ADPHC standards and guidelines. Nurses utilized appropriate patient protective equipment, frequent hand hygiene, and social distancing during the collection process. Inclusion criteria included asymptomatic students aged 4 to 12 years attending in-person classes on campus. Students with respiratory symptoms who were asked to stay home or those not attending in-person classes were excluded.

Data with regard to school children’s attitudes before and after testing were compiled through an online survey sent randomly to participants postintervention. Data from school administrators were collected through video and telephone interviews between April 14 and April 29, 2021. We first interviewed G42 Biogenix Lab providers to obtain previously acquired qualitative and quantitative data, which were collected during the intervention itself. After obtaining this information, we designed a questionnaire and proceeded with a structured interview process for school officials.

We interviewed school principals and administrators to collect their overall experiences with the saliva testing program. Before starting each interview, we established the interviewees preferred language, either English or Arabic. We then introduced the meeting attendees and provided study details, aims, and objectives, and described collaborating entities. We obtained verbal informed consent from a script approved by the NYUAD IRB and then proceeded with the interview, which included 4 questions. The first 3 questions were answered on a 5-point Likert scale model that consisted of 5 answer options: 5 being completely agree, 4 agree, 3 somewhat agree, 2 somewhat disagree, and 1 completely disagree. The fourth question invited open-ended feedback and comments on the following statements:

1. I believe the COVID-19 saliva testing program improved the safety for my school campus.
2. Our community had an overall positive experience with the COVID saliva testing.
3. We would like to continue a saliva-based COVID testing program on our school campus.
4. Please provide any additional comments you feel important about the program.

During the interview, we transcribed the answers as the interviewee was answering. We then translated those in Arabic into English and collected the data in 1 Excel spreadsheet. School interviewees and school names were de-identified in the collection and storage process.

Results

A total of 2011 saliva samples were collected from 18 different primary school campuses. Samples were sent the same day to G42 Biogenix Labs in Abu Dhabi for COVID PCR testing. A team consisting of 5 doctors providing general oversight, along with 2 to 6 nurses per site, were able to manage the collection process for all 18 school campuses. Samples were collected between 8 am and 2 pm and required variation among sites depending on factors such as staff availability and class schedules. Additional scheduling challenges included compliance with public safety mandates involving the maintenance of defined “safety bubbles” that forbid certain personnel from moving between floors, and the avoidance of mixing students from different classes.

Sample stations were set up in either the school auditorium or gymnasium to ensure appropriate crowd control and ventilation. Teachers and other school staff, including public safety, were able to manage lines and the shuttling of students back and forth from classes to testing stations, which allowed medical staff to focus on sample collection.

Informed consent was obtained by prior electronic communication to parents from school staff, asking them to agree to allow their child to participate in the testing program. Informed consent was identified as a challenge: Getting parents to understand that saliva testing was more comfortable than NP testing, and that the results were only being used to help keep the school safe, took time. School staff are used to obtaining consent from parents for field trips, but this was clearly more challenging for them.

The saliva collection process per child took more time than expected. Children fasted for 45 minutes before saliva collection. We used an active drool technique, which required children to pool saliva in their mouth then express it into a collection tube. Adults can generally do this on command, but we found it took 10 to 12 minutes per child. Saliva production was cued by asking the children to think about food, and by showing them pictures and TV commercials depicting food. Children 4 to 5 years old had more difficulty with the process despite active cueing, while those 6 to 12 years old had an easier time with the process. We collected data on a cohort of 80 children regarding their attitudes pre (Figure 1) and post collection (Figure 2). Children felt happier, less nervous, and less scared after collection than before collection. This trend reassured us that future collections would be easier for students.

A total of 15 of 18 school principals completed the telephone interview, yielding a response rate of 83%. Overall, 93% of the school principals agreed or completely agreed that the COVID-19 saliva testing program improved school safety; 93% agreed or completely agreed that they had an overall positive experience with the program; and 73% supported the ongoing use of saliva testing in their schools (Table 1). Administrators’ open-ended comments on their experience were positive overall (Table 2).

Discussion

By March 2020, many kindergarten to grade 12 public and private schools suspended in-person classes due to the pandemic and turned to online learning platforms. The negative impact of school closures on academic achievement is projected to be significant.^7,12,13 Ensuring schools can stay open and run operations safely will require routine SARS-CoV-2 testing. Our study investigated the feasibility of routine saliva testing on children aged 4 to 12 years on their school campuses. The ADPHC school on-site saliva testing program involved bringing lab providers onto 18 primary school campuses and required cooperation among parents, students, school administrators, and health care professionals.

Children younger than 6 years had difficulty producing an adequate saliva specimen, whereas those 6 to 12 years did so with relative ease when cued by thoughts or pictures of food while waiting in line for collection. Schools considering on-site testing programs should consider the age range of 6 to 12 years as a viable age range for saliva screening. Children should fast for a minimum of 45 minutes prior to saliva collection and should be cued by thoughts of food, food pictures, or food commercials. Setting up a sampling station close to the cafeteria where students can smell meal preparation may also help.^14,15 Sampling before breakfast or lunch, when children are potentially at their hungriest, should also be considered.

The greatest challenge was obtaining informed consent from parents who were not yet familiar with the reliability of saliva testing as a tool for SARS-CoV-2 screening or with the saliva collection process as a whole. Informed consent was initially done electronically, lacking direct human interaction to answer parents’ questions. Parents who refused had a follow-up call from the school nurse to further explain the logistics and rationale for saliva screening. Having medical professionals directly answer parents’ questions was helpful. Parents were reassured that the process was painless, confidential, and only to be used for school safety purposes. Despite school administrators being experienced in obtaining consent from parents for field trips, obtaining informed consent for a medical testing procedure is more complicated, and parents aren’t accustomed to providing such consent in a school environment. Schools considering on-site testing should ensure that their school nurse or other health care providers are on the front line obtaining informed consent and allaying parents’ fears.

School staff were able to effectively provide crowd control for testing, and children felt at ease being in a familiar environment. Teachers and public safety officers are well-equipped at managing the shuttling of students to class, to lunch, to physical education, and, finally, to dismissal. They were equally equipped at handling the logistics of students to and from testing, including minimizing crowds and helping students feel at ease during the process. This effective collaboration allowed the lab personnel to focus on sample collection and storage, while school staff managed all other aspects of the children’s safety and care.

Conclusion

Overall, school administrators had a positive experience with the testing program, felt the program improved the safety of their schools, and supported the ongoing use of saliva testing for SARS-CoV-2 on their school campuses. Children aged 6 years and older were able to provide adequate saliva samples, and children felt happier and less nervous after the process, indicating repeatability. Our findings highlight the feasibility of an integrated on-site saliva testing model for primary school campuses. Further research is needed to determine the scalability of such a model and whether the added compliance and safety of on-site testing compensates for the potential loss of learning time that testing during school hours would require.

Corresponding author: Ayaz Virji, MD, New York University Abu Dhabi, PO Box 129188, Abu Dhabi, United Arab Emirates; av102@nyu.edu.

Financial disclosures: None.

From Children’s Health System of Texas, Division of Pediatric Anesthesiology, Dallas, TX (Drs. Sakhai, Bocanegra, Chandran, Kimatian, and Kiss), UT Southwestern Medical Center, Department of Anesthesiology and Pain Management, Dallas, TX (Drs. Bocanegra, Chandran, Kimatian, and Kiss), and UT Southwestern Medical Center, Department of Population and Data Sciences, Dallas, TX (Dr. Reisch).

Objective: Policy-driven changes in medical practice have long been the norm. Seldom are changes in clinical practice sought to be brought about by a person’s tendency toward growth or self‐actualization. Many hospitals have instituted hypothermia bundles to help reduce the incidence of unanticipated postoperative hypothermia. Although successful in the short-term, sustained changes are difficult to maintain. We implemented a quality-improvement project focused on addressing the affective components of self-determination theory (SDT) to create sustainable behavioral change while satisfying providers’ basic psychological needs for autonomy, competence, and relatedness.

Methods: A total of 3 Plan-Do-Study-Act (PDSA) cycles were enacted over the span of 14 months at a major tertiary care pediatric hospital to recruit and motivate anesthesia providers and perioperative team members to reduce the percentage of hypothermic postsurgical patients by 50%. As an optional initial incentive for participation, anesthesiologists would qualify for American Board of Anesthesiology Maintenance of Certification in Anesthesiology (MOCA) Part 4 Quality Improvement credits for monitoring their own temperature data and participating in project-related meetings. Providers were given autonomy to develop a personal plan for achieving the desired goals.

Results: The median rate of hypothermia was reduced from 6.9% to 1.6% in July 2019 and was reduced again in July 2020 to 1.3%, an 81% reduction overall. A low hypothermia rate was successfully maintained for at least 21 subsequent months after participants received their MOCA credits in July 2019.

Conclusions: Using an approach that focused on the elements of competency, autonomy, and relatedness central to the principles of SDT, we observed the development of a new culture of vigilance for prevention of hypothermia that successfully endured beyond the project end date.

Keywords: postoperative hypothermia; self-determination theory; motivation; quality improvement.

Perioperative hypothermia, generally accepted as a core temperature less than 36 °C in clinical practice, is a common complication in the pediatric surgical population and is associated with poor postoperative outcomes.¹ Hypothermic patients may develop respiratory depression, hypoglycemia, and metabolic acidosis that may lead to decreased oxygen delivery and end organ tissue hypoxia.^2-4 Other potential detrimental effects of failing to maintain normal body temperature are impaired clotting factor enzyme function and platelet dysfunction, increasing the risk for postoperative bleeding.^5,6 In addition, there are financial implications when hypothermic patients require care and resources postoperatively because of delayed emergence or shivering.⁷

The American Society of Anesthesiologists recommends intraoperative temperature monitoring for procedures when clinically significant changes in body temperature are anticipated.⁸ Maintenance of normothermia in the pediatric population is especially challenging owing to a larger skin-surface area compared with body mass ratio and less subcutaneous fat content than in adults. Preventing postoperative hypothermia starts preoperatively with parental education and can be as simple as covering the child with a blanket and setting the preoperative room to an acceptably warm temperature.^9,10 Intraoperatively, maintaining operating room (OR) temperatures at or above 21.1 °C and using active warming devices and radiant warmers when appropriate are important techniques to preserve the child’s body temperature.^11,12

Despite the knowledge of these risks and vigilant avoidance of hypothermia, unplanned perioperative hypothermia can occur in up to 70% of surgical patients.¹ Beyond the clinical benefits, as health care marches toward a value-based payment methodology, quality indicators such as avoiding hypothermia may be linked directly to payment.

Self-determination theory (SDT) was first developed in 1980 by Deci and Ryan.¹³ The central premise of the theory states that people develop their full potential if circumstances allow them to satisfy their basic psychological needs: autonomy, competence, and relatedness. Under these conditions, people’s natural inclination toward growth can be realized, and they are more likely to internalize external goals. Under an extrinsic reward system, motivation can waver, as people may perceive rewards as controlling.

Many institutions have implemented hypothermia bundles to help decrease the rate of hypothermic patients, but while initially successful, the effectiveness of these interventions tends to fade over time as participants settle into old, comfortable routines.¹⁴ With SDT in mind, we designed our quality-improvement (QI) project with interventions to allow clinicians autonomy without instituting rigid guidelines or punitive actions. We aimed to directly address the affective components central to motivation and engagement so that we could bring about long-term meaningful changes in our practice.

Methods

Setting

The hypothermia QI intervention was instituted at a major tertiary care children’s hospital that performs more than 40 000 pediatric general anesthetics annually. Our division of pediatric anesthesiology consists of 66 fellowship-trained pediatric anesthesiologists, 15 or more rotating trainees per month, 13 anesthesiology assistants, 15 anesthesia technicians, and more than 50 perioperative nurses.

The most frequent pediatric surgeries include, but are not limited to, general surgery, otolaryngology, urology, gastroenterology, plastic surgery, neurosurgery, and dentistry. The surgeries are conducted in the hospital’s main operative floor, which consists of 15 ORs and 2 gastroenterology procedure rooms. Although the implementation of the QI project included several operating sites, we focused on collecting temperature data from surgical patients at our main campus recovery unit. We obtained the patients’ initial temperatures upon arrival to the recovery unit from a retrospective electronic health record review of all patients who underwent anesthesia from January 2016 through April 2021.

Postoperative hypothermia was identified as an area of potential improvement after several patients were reported to be hypothermic upon arrival to the recovery unit in the later part of 2018. Further review revealed significant heterogeneity of practices and lack of standardization of patient-warming methods. By comparing the temperatures pre- and postintervention, we could measure the effectiveness of the QI initiative. Prior to the start of our project, the hypothermia rate in our patient population was not actively tracked, and the effectiveness of our variable practice was not measured.

The cutoff for hypothermia for our QI project was defined as body temperature below 36 °C, since this value has been previously used in the literature and is commonly accepted in anesthesia practice as the delineation for hypothermia in patients undergoing general anesthesia.¹

Interventions

This QI project was designed and modeled after the Institute for Healthcare Improvement Model for Improvement.¹⁵ Three cycles of Plan-Do-Study-Act (PDSA) were developed and instituted over a 14-month period until December 2019 (Table 1).

A retrospective review was conducted to determine the percentage of surgical patients arriving to our recovery units with an initial temperature reading of less than 36 °C. A project key driver diagram and smart aim were created and approved by the hospital’s continuing medical education (CME) committee for credit via the American Board of Medical Specialties (ABMS) Multi-Specialty Portfolio Program, Maintenance of Certification in Anesthesiology (MOCA) Part 4.

The first PDSA cycle involved introducing the QI project and sharing the aims of the project at a department grand rounds in the latter part of October 2018. Enrollment to participate in the project was open to all anesthesiologists in the division, and participants could earn up to 20 hours of MOCA Part 4 credits. A spreadsheet was developed and maintained to track each anesthesiologist’s monthly percentage of hypothermic patients. The de-identified patient data were shared with the division via monthly emails. In addition, individual providers with a hypothermic patient in the recovery room received a notification email.

The anesthesiologists participated in the QI project by reviewing their personal percentage of hypothermic patients on an ongoing basis to earn the credit. There was no explicit requirement to decrease their own rate of patients with body temperature less than 36 °C or expectation to achieve a predetermined goal, so the participants could not “fail.”

Because of the large interest in this project, a hypothermia committee was formed that consisted of 36 anesthesiologists. This group reviewed the data and exchanged ideas for improvement in November 2018 as part of the first PDSA cycle. The committee met monthly and was responsible for actively engaging other members of the department and perioperative staff to help in this multidisciplinary effort of combating hypothermia in our surgical pediatric population.

PDSA cycle 2 involved several major initiatives, including direct incorporation of the rest of the perioperative team. The perioperative nursing team was educated on the risks of hypothermia and engaged to take an active role by maintaining the operating suite temperature at 21.1 °C and turning on the Bair Hugger (3M) blanket to 43 °C on the OR bed prior to patient arrival to the OR. Additionally, anesthesia technicians (ATs) were tasked with ensuring an adequate supply of Bair Hugger drapes for all cases of the day. The facility’s engineering team was engaged to move the preoperative room temperature controls away from families (who frequently made the rooms cold) and instead set it at a consistent temperature of 23.9 °C. ATs were also asked to place axillary and nasal temperature probes on the anesthesia workstations as a visual reminder to facilitate temperature monitoring closer to the start of anesthesia (instead of the anesthesia provider having to remember to retrieve a temperature probe out of a drawer and place it on the patient). Furthermore, anesthesiologists were instructed via the aforementioned monthly emails and at monthly department meetings to place the temperature probes as early as possible in order to recognize and respond to intraoperative hypothermia in a timelier manner. Finally, supply chain leaders were informed of our expected increase in the use of the blankets and probes and proportionally increased ordering of these supplies to make sure availability would not present an obstacle.

In PDSA cycle 3, trainees (anesthesia assistant students, anesthesia residents and fellows) and advanced practice providers (APPs) (certified registered nurse-anesthetists [CRNAs] and certified anesthesia assistants [C-AAs]) were informed of the QI project. This initiative was guided toward improving vigilance for hypothermia in the rest of the anesthesia team members. The trainees and APPs usually set up the anesthesia area prior to patient arrival, so their recruitment in support of this effort would ensure appropriate OR temperature, active warming device deployment, and the availability and early placement of the correct temperature probe for the case. To facilitate personal accountability, the trainees and APPs were also emailed their own patients’ rate of hypothermia.

Along the course of the project, quarterly committee meetings and departmental monthly meetings served as venues to express concerns and look for areas of improvement, such as specific patterns or trends leading to hypothermic patients. One specific example was the identification of the gastrointestinal endoscopic patients having a rate of hypothermia that was 2% higher than average. Directed education on the importance of Bair Hugger blankets and using warm intravenous fluids worked well to decrease the rate of hypothermia in these patients. This collection of data was shared at regular intervals during monthly department meetings as well and more frequently using departmental emails. The hospital’s secure intranet SharePoint (Microsoft) site was used to share the data among providers.

Study of the interventions and measures

To study the effectiveness and impact of the project to motivate our anesthesiologists and other team members, we compared the first temperatures obtained in the recovery unit prior to the start of the intervention with those collected after the start of the QI project in November 2018. Because of the variability of temperature monitoring intraoperatively (nasal, axillary, rectal), we decided to use the temperature obtained by the nurse in the recovery room upon the patient’s arrival. Over the years analyzed, the nurse’s technique of measuring the temperature remained consistent. All patient temperature measurements were performed using the TAT-5000 (Exergen Corporation). This temporal artery thermometer has been previously shown to correlate well with bladder temperatures (70% of measurements differ by no more than 0.5 °C, as reported by Langham et al¹⁶).

Admittedly, we could not measure the degree of motivation or internalization of the project goals by our cohort, but we could measure the reduction in the rate of hypothermia and subjectively gauge engagement in the project by the various groups of participants and the sustainability of the results. In addition, all participating anesthesiologists received MOCA Part 4 credits in July 2019. We continued our data collection until April 2021 to determine if our project had brought about sustainable changes in practice that would continue past the initial motivator of obtaining CME credit.

Analysis

Data analysis was performed using Excel (Microsoft) and SAS, version 9.4 (SAS Institute).

The median of the monthly percentage of patients with a temperature of less than 36.0 °C was also determined for the preintervention time frame. This served as our baseline hypothermia rate, and we aimed to lower it by 50%. Run charts, a well-described methodology to gauge the effectiveness of the QI project, were constructed with the collected data.¹⁷

We performed additional analysis to adjust for different time periods throughout the year. The time period between January 2016 and October 2018 was considered preintervention. We considered November 2018 the start of our intervention, or more specifically, the start of our PDSA cycles. October 2018 was analyzed as part of the preintervention data. To account for seasonal temperature variations, the statistical analysis focused on the comparisons of the same calendar quarters for before and after starting intervention using Wilcoxon Mann-Whitney U tests. To reach an overall conclusion, the probabilities for the 4 quarters were combined for each criterion separately utilizing the Fisher χ² combined probability method.

The hypothermia QI project was reviewed by the institutional review board and determined to be exempt.

Results

The temperatures of 40 875 patients were available for analysis for the preintervention period between January 2016 and October 2018. The median percentage of patients with temperatures less than 36.0 °C was 6.9% (interquartile range [IQR], 5.8%-8.4%). The highest percentage was in February 2016 (9.9%), and the lowest was in March 2018 (3.4%). Following the start of the first PDSA cycle, the next 6 consecutive rates of hypothermia were below the median preintervention value, and a new median for these percentages was calculated at 3.4% (IQR, 2.6%-4.3%). In July 2019, the proportion of hypothermic patients decreased once more for 6 consecutive months, yielding a new median of 1.6% (IQR, 1.2%-1.8%) and again in July 2020, to yield a median of 1.3% (IQR, 1.2%-1.5%) (Figure). In all, 33 799 patients were analyzed after the start of the project from November 2018 to the end of the data collection period through April 2021.

The preintervention monthly rates of hypothermia were compared, quarter to quarter, with those starting in November 2018 using the Wilcoxon Mann-Whitney U test. The decrease in proportion of hypothermic patients after the start of the intervention was statistically significant (P < .001). In addition, the percentage of patients with temperatures greater than 38 °C was not significantly different between the pre- and postintervention time periods (P < .25) (Table 2). The decrease in the number of patients available for analysis from March 2020 to May 2020 was due to the COVID-19 pandemic.

Subjectively, we did not experience any notable resistance to our efforts, and the experience was largely positive for everyone involved. Clinicians identified as having high monthly rates of hypothermia (5% or higher) corrected their numbers the following month after being notified via email or in person.

To achieve changes in practice, the health care industry has relied on instituting guidelines, regulations, and policies, often with punitive consequences. We call into question this long-standing framework and propose a novel approach to help evolve the field of QI. Studies in human psychology have long demonstrated the demotivation power of a reward system and the negative response to attempts by authority to use incentives to control or coerce. In our QI project, we instituted 3 PDSA cycles and applied elements from SDT to motivate people’s behaviors. We demonstrate how a new culture focused on maintaining intraoperative normothermia was developed and brought about a measurable and significant decrease in the rate of hypothermia. The relevance of SDT, a widely accepted unifying theory that bridges and links social and personality psychology, should not be understated in health care. Authorities wishing to have long-standing influence should consider a person’s right to make their own decisions and, if possible, a unique way of doing things.

Positively reinforcing behavior has been shown to have a paradoxical effect by dampening an individual’s intrinsic motivation or desire to perform certain tasks.¹⁸ Deadlines, surveillance, and authoritative commands are also deterrents.^19,20 We focused on providing the tools and information to the clinicians and relied on their innate need for autonomy, growth, and self-actualization to bring about change in clinical practice.²¹ Group meetings served as a construct for exchanging ideas and to encourage participation, but without the implementation of rigid guidelines or policies. Intraoperative active warming devices and temperature probes were made available, but their use was not mandated. The use of these devices was intentionally not audited to avoid any overbearing control. Providers were, however, given monthly temperature data to help individually assess the effectiveness of their interventions. We did not impose any negative or punitive actions for those clinicians who had high rates of hypothermic patients, and we did not reward those who had low rates of hypothermia. We wanted the participants to feel that the inner self was the source of their behavior, and this was in parallel with their own interests and values. If providers could feel their need for competency could be realized, we hoped they would continue to adhere to the measures we provided to maintain a low rate of hypothermia.

The effectiveness of our efforts was demonstrated by a decrease in the prevalence of postoperative hypothermia in our surgical patients. The initial decrease of the median rate of hypothermia from 6.9% to 3.4% occurred shortly into the start of the first PDSA cycle. The second PDSA cycle started in January 2019 with a multimodal approach and included almost all parties involved in the perioperative care of our surgical patients. Not only was this intervention responsible for a continued downward trend in the percentage of hypothermic patients, but it set the stage for the third and final PDSA cycle, which started in July 2019. The architecture was in place to integrate trainees and APPs to reinforce our initiative. Subsequently, the new median percentage of hypothermic patients was further decreased to an all-time low of 1.6% per month, satisfying and surpassing the goal of the QI project of decreasing the rate of hypothermia by only 50%. Our organization thereafter maintained a monthly hypothermia rate below 2%, except for April 2020, when it reached 2.5%. Our lowest median percentage was obtained after July 2020, reaching 1.3%.

To account for seasonal variations in temperatures and types of surgeries performed, we compared the percentage of hypothermic patients before and after the start of intervention, quarter by quarter. The decrease in the proportion of hypothermic patients after the start of intervention was statistically significant (P < .001). In addition, the data failed to prove any statistical difference for temperatures above 38 °C between the 2 periods, indicating that our interventions did not result in significant overwarming of patients. The clinical implications of decreasing the percentage of hypothermic patients from 6.9% to 1.3% is likely clinically important when considering the large number of patients who undergo surgery at large tertiary care pediatric centers. Even if simple interventions reduce hypothermia in only a handful of patients, routine applications of simple measures to keep patients normothermic is likely best clinical practice.

Anesthesiologists who participated in the hypothermia QI project by tracking the incidence of hypothermia in their patients were able to collect MOCA Part 4 credits in July 2019. There was no requirement for the individual anesthesiologist to reduce the rate of hypothermia or apply any of the encouraged strategies to obtain credit. As previously stated, there were also no rewards for obtaining low hypothermia rates for the providers. The temperature data continued to be collected through April 2021, 21 months after the credits were distributed, to demonstrate a continued, meaningful change, at least in the short-term. While the MOCA Part 4 credits likely served as an initial motivating factor to encourage participation in the QI project, they certainly were not responsible for the sustained low hypothermia rate after July 2019. We showed that the low rate of hypothermia was successfully maintained, indicating that the change in providers’ behavior was independent of the external motivator of obtaining the credit hours. Mere participation in the project by reviewing one’s temperature data was all that was required to obtain the credit. The Organismic Integration Theory, a mini-theory within SDT, best explains this phenomenon by describing any motivated behavior on a continuum ranging from controlled to autonomous.²² Do people perform the task resentfully, on their own volition because they believe it is the correct action, or somewhere in between? We explain the sustained low rates of hypothermia after the MOCA credits were distributed due to a shift to the autonomous end of the continuum with the clinician’s active willingness to meet the challenges and apply intrinsically motivated behaviors to lower the rate of hypothermia. The internalization of external motivators is difficult to prove, but the evidence supports that the methods we used to motivate individuals were effective and have resulted in a significant downward trend in our hypothermia rate.

There are several limitations to our QI project. The first involves the measuring of postoperative temperature in the recovery units. The temperatures were obtained using the same medical-grade infrared thermometer for all the patients, but other variables, such as timing and techniques, were not standardized. Secondly, overall surgical outcomes related to hypothermia were not tracked because we were unable to control for other confounding variables in our large cohort of patients, so we cannot say if the drop in the hypothermia rate had a clinically significant outcome. Thirdly, we propose that SDT offers a compellingly fitting explanation of the psychology of motivation in our efforts, but it may be possible that other theories may offer equally fitting explanations. The ability to measure the degree of motivation is lacking, and we did not explicitly ask participants what their specific source of motivation was. Aside from SDT, the reduction in hypothermia rate could also be attributed to the ease and availability of warming equipment that was made in each OR. This QI project was successfully applied to only 1 institution, so its ability to be widely applicable remains uncertain. In addition, data collection continued during the COVID-19 pandemic when case volumes decreased. However, by June 2020, the number of surgical cases at our institution had largely returned to prepandemic levels. Additional data collection beyond April 2021 would be helpful to determine if the reduction in hypothermia rates is truly sustained.

Conclusion

Overall, the importance of maintaining perioperative normothermia was well disseminated and agreed upon by all departments involved. Despite the limitations of the project, there was a significant reduction in rates of hypothermia, and sustainability of outcomes was consistently demonstrated in the poststudy period.

Using 3 cycles of the PDSA method, we successfully decreased the median rate of postoperative hypothermia in our pediatric surgical population from a preintervention value of 6.9% to 1.3%—a reduction of more than 81.2%. We provided motivation for members of our anesthesiology staff to participate by offering MOCA 2.0 Part 4 credits, but the lower rate of hypothermic patients was maintained for 15 months after the credits were distributed. Over the course of the project, there was a shift in culture, and extra vigilance was given to temperature monitoring and assessment. We attribute this sustained cultural change to the deliberate incorporation of the principles of competency, autonomy, and relatedness central to SDT to the structure of the interventions, avoiding rigid guidelines and pathways in favor of affective engagement to establish intrinsic motivation.

Acknowledgements: The authors thank Joan Reisch, PhD, for her assistance with the statistical analysis.

Corresponding author: Edgar Erold Kiss, MD, 1935 Medical District Dr, Dallas, TX 75235; edgar.kiss@UTSouthwestern.edu.

Financial disclosures: None.

From Children’s Health System of Texas, Division of Pediatric Anesthesiology, Dallas, TX (Drs. Sakhai, Bocanegra, Chandran, Kimatian, and Kiss), UT Southwestern Medical Center, Department of Anesthesiology and Pain Management, Dallas, TX (Drs. Bocanegra, Chandran, Kimatian, and Kiss), and UT Southwestern Medical Center, Department of Population and Data Sciences, Dallas, TX (Dr. Reisch).

Objective: Policy-driven changes in medical practice have long been the norm. Seldom are changes in clinical practice sought to be brought about by a person’s tendency toward growth or self‐actualization. Many hospitals have instituted hypothermia bundles to help reduce the incidence of unanticipated postoperative hypothermia. Although successful in the short-term, sustained changes are difficult to maintain. We implemented a quality-improvement project focused on addressing the affective components of self-determination theory (SDT) to create sustainable behavioral change while satisfying providers’ basic psychological needs for autonomy, competence, and relatedness.

Methods: A total of 3 Plan-Do-Study-Act (PDSA) cycles were enacted over the span of 14 months at a major tertiary care pediatric hospital to recruit and motivate anesthesia providers and perioperative team members to reduce the percentage of hypothermic postsurgical patients by 50%. As an optional initial incentive for participation, anesthesiologists would qualify for American Board of Anesthesiology Maintenance of Certification in Anesthesiology (MOCA) Part 4 Quality Improvement credits for monitoring their own temperature data and participating in project-related meetings. Providers were given autonomy to develop a personal plan for achieving the desired goals.

Results: The median rate of hypothermia was reduced from 6.9% to 1.6% in July 2019 and was reduced again in July 2020 to 1.3%, an 81% reduction overall. A low hypothermia rate was successfully maintained for at least 21 subsequent months after participants received their MOCA credits in July 2019.

Conclusions: Using an approach that focused on the elements of competency, autonomy, and relatedness central to the principles of SDT, we observed the development of a new culture of vigilance for prevention of hypothermia that successfully endured beyond the project end date.

Keywords: postoperative hypothermia; self-determination theory; motivation; quality improvement.

Perioperative hypothermia, generally accepted as a core temperature less than 36 °C in clinical practice, is a common complication in the pediatric surgical population and is associated with poor postoperative outcomes.¹ Hypothermic patients may develop respiratory depression, hypoglycemia, and metabolic acidosis that may lead to decreased oxygen delivery and end organ tissue hypoxia.^2-4 Other potential detrimental effects of failing to maintain normal body temperature are impaired clotting factor enzyme function and platelet dysfunction, increasing the risk for postoperative bleeding.^5,6 In addition, there are financial implications when hypothermic patients require care and resources postoperatively because of delayed emergence or shivering.⁷

The American Society of Anesthesiologists recommends intraoperative temperature monitoring for procedures when clinically significant changes in body temperature are anticipated.⁸ Maintenance of normothermia in the pediatric population is especially challenging owing to a larger skin-surface area compared with body mass ratio and less subcutaneous fat content than in adults. Preventing postoperative hypothermia starts preoperatively with parental education and can be as simple as covering the child with a blanket and setting the preoperative room to an acceptably warm temperature.^9,10 Intraoperatively, maintaining operating room (OR) temperatures at or above 21.1 °C and using active warming devices and radiant warmers when appropriate are important techniques to preserve the child’s body temperature.^11,12

Despite the knowledge of these risks and vigilant avoidance of hypothermia, unplanned perioperative hypothermia can occur in up to 70% of surgical patients.¹ Beyond the clinical benefits, as health care marches toward a value-based payment methodology, quality indicators such as avoiding hypothermia may be linked directly to payment.

Self-determination theory (SDT) was first developed in 1980 by Deci and Ryan.¹³ The central premise of the theory states that people develop their full potential if circumstances allow them to satisfy their basic psychological needs: autonomy, competence, and relatedness. Under these conditions, people’s natural inclination toward growth can be realized, and they are more likely to internalize external goals. Under an extrinsic reward system, motivation can waver, as people may perceive rewards as controlling.

Many institutions have implemented hypothermia bundles to help decrease the rate of hypothermic patients, but while initially successful, the effectiveness of these interventions tends to fade over time as participants settle into old, comfortable routines.¹⁴ With SDT in mind, we designed our quality-improvement (QI) project with interventions to allow clinicians autonomy without instituting rigid guidelines or punitive actions. We aimed to directly address the affective components central to motivation and engagement so that we could bring about long-term meaningful changes in our practice.

Methods

Setting

The hypothermia QI intervention was instituted at a major tertiary care children’s hospital that performs more than 40 000 pediatric general anesthetics annually. Our division of pediatric anesthesiology consists of 66 fellowship-trained pediatric anesthesiologists, 15 or more rotating trainees per month, 13 anesthesiology assistants, 15 anesthesia technicians, and more than 50 perioperative nurses.

The most frequent pediatric surgeries include, but are not limited to, general surgery, otolaryngology, urology, gastroenterology, plastic surgery, neurosurgery, and dentistry. The surgeries are conducted in the hospital’s main operative floor, which consists of 15 ORs and 2 gastroenterology procedure rooms. Although the implementation of the QI project included several operating sites, we focused on collecting temperature data from surgical patients at our main campus recovery unit. We obtained the patients’ initial temperatures upon arrival to the recovery unit from a retrospective electronic health record review of all patients who underwent anesthesia from January 2016 through April 2021.

Postoperative hypothermia was identified as an area of potential improvement after several patients were reported to be hypothermic upon arrival to the recovery unit in the later part of 2018. Further review revealed significant heterogeneity of practices and lack of standardization of patient-warming methods. By comparing the temperatures pre- and postintervention, we could measure the effectiveness of the QI initiative. Prior to the start of our project, the hypothermia rate in our patient population was not actively tracked, and the effectiveness of our variable practice was not measured.

The cutoff for hypothermia for our QI project was defined as body temperature below 36 °C, since this value has been previously used in the literature and is commonly accepted in anesthesia practice as the delineation for hypothermia in patients undergoing general anesthesia.¹

Interventions

This QI project was designed and modeled after the Institute for Healthcare Improvement Model for Improvement.¹⁵ Three cycles of Plan-Do-Study-Act (PDSA) were developed and instituted over a 14-month period until December 2019 (Table 1).

A retrospective review was conducted to determine the percentage of surgical patients arriving to our recovery units with an initial temperature reading of less than 36 °C. A project key driver diagram and smart aim were created and approved by the hospital’s continuing medical education (CME) committee for credit via the American Board of Medical Specialties (ABMS) Multi-Specialty Portfolio Program, Maintenance of Certification in Anesthesiology (MOCA) Part 4.

The first PDSA cycle involved introducing the QI project and sharing the aims of the project at a department grand rounds in the latter part of October 2018. Enrollment to participate in the project was open to all anesthesiologists in the division, and participants could earn up to 20 hours of MOCA Part 4 credits. A spreadsheet was developed and maintained to track each anesthesiologist’s monthly percentage of hypothermic patients. The de-identified patient data were shared with the division via monthly emails. In addition, individual providers with a hypothermic patient in the recovery room received a notification email.

The anesthesiologists participated in the QI project by reviewing their personal percentage of hypothermic patients on an ongoing basis to earn the credit. There was no explicit requirement to decrease their own rate of patients with body temperature less than 36 °C or expectation to achieve a predetermined goal, so the participants could not “fail.”

Because of the large interest in this project, a hypothermia committee was formed that consisted of 36 anesthesiologists. This group reviewed the data and exchanged ideas for improvement in November 2018 as part of the first PDSA cycle. The committee met monthly and was responsible for actively engaging other members of the department and perioperative staff to help in this multidisciplinary effort of combating hypothermia in our surgical pediatric population.

PDSA cycle 2 involved several major initiatives, including direct incorporation of the rest of the perioperative team. The perioperative nursing team was educated on the risks of hypothermia and engaged to take an active role by maintaining the operating suite temperature at 21.1 °C and turning on the Bair Hugger (3M) blanket to 43 °C on the OR bed prior to patient arrival to the OR. Additionally, anesthesia technicians (ATs) were tasked with ensuring an adequate supply of Bair Hugger drapes for all cases of the day. The facility’s engineering team was engaged to move the preoperative room temperature controls away from families (who frequently made the rooms cold) and instead set it at a consistent temperature of 23.9 °C. ATs were also asked to place axillary and nasal temperature probes on the anesthesia workstations as a visual reminder to facilitate temperature monitoring closer to the start of anesthesia (instead of the anesthesia provider having to remember to retrieve a temperature probe out of a drawer and place it on the patient). Furthermore, anesthesiologists were instructed via the aforementioned monthly emails and at monthly department meetings to place the temperature probes as early as possible in order to recognize and respond to intraoperative hypothermia in a timelier manner. Finally, supply chain leaders were informed of our expected increase in the use of the blankets and probes and proportionally increased ordering of these supplies to make sure availability would not present an obstacle.

In PDSA cycle 3, trainees (anesthesia assistant students, anesthesia residents and fellows) and advanced practice providers (APPs) (certified registered nurse-anesthetists [CRNAs] and certified anesthesia assistants [C-AAs]) were informed of the QI project. This initiative was guided toward improving vigilance for hypothermia in the rest of the anesthesia team members. The trainees and APPs usually set up the anesthesia area prior to patient arrival, so their recruitment in support of this effort would ensure appropriate OR temperature, active warming device deployment, and the availability and early placement of the correct temperature probe for the case. To facilitate personal accountability, the trainees and APPs were also emailed their own patients’ rate of hypothermia.

Along the course of the project, quarterly committee meetings and departmental monthly meetings served as venues to express concerns and look for areas of improvement, such as specific patterns or trends leading to hypothermic patients. One specific example was the identification of the gastrointestinal endoscopic patients having a rate of hypothermia that was 2% higher than average. Directed education on the importance of Bair Hugger blankets and using warm intravenous fluids worked well to decrease the rate of hypothermia in these patients. This collection of data was shared at regular intervals during monthly department meetings as well and more frequently using departmental emails. The hospital’s secure intranet SharePoint (Microsoft) site was used to share the data among providers.

Study of the interventions and measures

To study the effectiveness and impact of the project to motivate our anesthesiologists and other team members, we compared the first temperatures obtained in the recovery unit prior to the start of the intervention with those collected after the start of the QI project in November 2018. Because of the variability of temperature monitoring intraoperatively (nasal, axillary, rectal), we decided to use the temperature obtained by the nurse in the recovery room upon the patient’s arrival. Over the years analyzed, the nurse’s technique of measuring the temperature remained consistent. All patient temperature measurements were performed using the TAT-5000 (Exergen Corporation). This temporal artery thermometer has been previously shown to correlate well with bladder temperatures (70% of measurements differ by no more than 0.5 °C, as reported by Langham et al¹⁶).

Admittedly, we could not measure the degree of motivation or internalization of the project goals by our cohort, but we could measure the reduction in the rate of hypothermia and subjectively gauge engagement in the project by the various groups of participants and the sustainability of the results. In addition, all participating anesthesiologists received MOCA Part 4 credits in July 2019. We continued our data collection until April 2021 to determine if our project had brought about sustainable changes in practice that would continue past the initial motivator of obtaining CME credit.

Analysis

Data analysis was performed using Excel (Microsoft) and SAS, version 9.4 (SAS Institute).

The median of the monthly percentage of patients with a temperature of less than 36.0 °C was also determined for the preintervention time frame. This served as our baseline hypothermia rate, and we aimed to lower it by 50%. Run charts, a well-described methodology to gauge the effectiveness of the QI project, were constructed with the collected data.¹⁷

We performed additional analysis to adjust for different time periods throughout the year. The time period between January 2016 and October 2018 was considered preintervention. We considered November 2018 the start of our intervention, or more specifically, the start of our PDSA cycles. October 2018 was analyzed as part of the preintervention data. To account for seasonal temperature variations, the statistical analysis focused on the comparisons of the same calendar quarters for before and after starting intervention using Wilcoxon Mann-Whitney U tests. To reach an overall conclusion, the probabilities for the 4 quarters were combined for each criterion separately utilizing the Fisher χ² combined probability method.

The hypothermia QI project was reviewed by the institutional review board and determined to be exempt.

Results

The temperatures of 40 875 patients were available for analysis for the preintervention period between January 2016 and October 2018. The median percentage of patients with temperatures less than 36.0 °C was 6.9% (interquartile range [IQR], 5.8%-8.4%). The highest percentage was in February 2016 (9.9%), and the lowest was in March 2018 (3.4%). Following the start of the first PDSA cycle, the next 6 consecutive rates of hypothermia were below the median preintervention value, and a new median for these percentages was calculated at 3.4% (IQR, 2.6%-4.3%). In July 2019, the proportion of hypothermic patients decreased once more for 6 consecutive months, yielding a new median of 1.6% (IQR, 1.2%-1.8%) and again in July 2020, to yield a median of 1.3% (IQR, 1.2%-1.5%) (Figure). In all, 33 799 patients were analyzed after the start of the project from November 2018 to the end of the data collection period through April 2021.

The preintervention monthly rates of hypothermia were compared, quarter to quarter, with those starting in November 2018 using the Wilcoxon Mann-Whitney U test. The decrease in proportion of hypothermic patients after the start of the intervention was statistically significant (P < .001). In addition, the percentage of patients with temperatures greater than 38 °C was not significantly different between the pre- and postintervention time periods (P < .25) (Table 2). The decrease in the number of patients available for analysis from March 2020 to May 2020 was due to the COVID-19 pandemic.

Subjectively, we did not experience any notable resistance to our efforts, and the experience was largely positive for everyone involved. Clinicians identified as having high monthly rates of hypothermia (5% or higher) corrected their numbers the following month after being notified via email or in person.

To achieve changes in practice, the health care industry has relied on instituting guidelines, regulations, and policies, often with punitive consequences. We call into question this long-standing framework and propose a novel approach to help evolve the field of QI. Studies in human psychology have long demonstrated the demotivation power of a reward system and the negative response to attempts by authority to use incentives to control or coerce. In our QI project, we instituted 3 PDSA cycles and applied elements from SDT to motivate people’s behaviors. We demonstrate how a new culture focused on maintaining intraoperative normothermia was developed and brought about a measurable and significant decrease in the rate of hypothermia. The relevance of SDT, a widely accepted unifying theory that bridges and links social and personality psychology, should not be understated in health care. Authorities wishing to have long-standing influence should consider a person’s right to make their own decisions and, if possible, a unique way of doing things.

Positively reinforcing behavior has been shown to have a paradoxical effect by dampening an individual’s intrinsic motivation or desire to perform certain tasks.¹⁸ Deadlines, surveillance, and authoritative commands are also deterrents.^19,20 We focused on providing the tools and information to the clinicians and relied on their innate need for autonomy, growth, and self-actualization to bring about change in clinical practice.²¹ Group meetings served as a construct for exchanging ideas and to encourage participation, but without the implementation of rigid guidelines or policies. Intraoperative active warming devices and temperature probes were made available, but their use was not mandated. The use of these devices was intentionally not audited to avoid any overbearing control. Providers were, however, given monthly temperature data to help individually assess the effectiveness of their interventions. We did not impose any negative or punitive actions for those clinicians who had high rates of hypothermic patients, and we did not reward those who had low rates of hypothermia. We wanted the participants to feel that the inner self was the source of their behavior, and this was in parallel with their own interests and values. If providers could feel their need for competency could be realized, we hoped they would continue to adhere to the measures we provided to maintain a low rate of hypothermia.

The effectiveness of our efforts was demonstrated by a decrease in the prevalence of postoperative hypothermia in our surgical patients. The initial decrease of the median rate of hypothermia from 6.9% to 3.4% occurred shortly into the start of the first PDSA cycle. The second PDSA cycle started in January 2019 with a multimodal approach and included almost all parties involved in the perioperative care of our surgical patients. Not only was this intervention responsible for a continued downward trend in the percentage of hypothermic patients, but it set the stage for the third and final PDSA cycle, which started in July 2019. The architecture was in place to integrate trainees and APPs to reinforce our initiative. Subsequently, the new median percentage of hypothermic patients was further decreased to an all-time low of 1.6% per month, satisfying and surpassing the goal of the QI project of decreasing the rate of hypothermia by only 50%. Our organization thereafter maintained a monthly hypothermia rate below 2%, except for April 2020, when it reached 2.5%. Our lowest median percentage was obtained after July 2020, reaching 1.3%.

To account for seasonal variations in temperatures and types of surgeries performed, we compared the percentage of hypothermic patients before and after the start of intervention, quarter by quarter. The decrease in the proportion of hypothermic patients after the start of intervention was statistically significant (P < .001). In addition, the data failed to prove any statistical difference for temperatures above 38 °C between the 2 periods, indicating that our interventions did not result in significant overwarming of patients. The clinical implications of decreasing the percentage of hypothermic patients from 6.9% to 1.3% is likely clinically important when considering the large number of patients who undergo surgery at large tertiary care pediatric centers. Even if simple interventions reduce hypothermia in only a handful of patients, routine applications of simple measures to keep patients normothermic is likely best clinical practice.

Anesthesiologists who participated in the hypothermia QI project by tracking the incidence of hypothermia in their patients were able to collect MOCA Part 4 credits in July 2019. There was no requirement for the individual anesthesiologist to reduce the rate of hypothermia or apply any of the encouraged strategies to obtain credit. As previously stated, there were also no rewards for obtaining low hypothermia rates for the providers. The temperature data continued to be collected through April 2021, 21 months after the credits were distributed, to demonstrate a continued, meaningful change, at least in the short-term. While the MOCA Part 4 credits likely served as an initial motivating factor to encourage participation in the QI project, they certainly were not responsible for the sustained low hypothermia rate after July 2019. We showed that the low rate of hypothermia was successfully maintained, indicating that the change in providers’ behavior was independent of the external motivator of obtaining the credit hours. Mere participation in the project by reviewing one’s temperature data was all that was required to obtain the credit. The Organismic Integration Theory, a mini-theory within SDT, best explains this phenomenon by describing any motivated behavior on a continuum ranging from controlled to autonomous.²² Do people perform the task resentfully, on their own volition because they believe it is the correct action, or somewhere in between? We explain the sustained low rates of hypothermia after the MOCA credits were distributed due to a shift to the autonomous end of the continuum with the clinician’s active willingness to meet the challenges and apply intrinsically motivated behaviors to lower the rate of hypothermia. The internalization of external motivators is difficult to prove, but the evidence supports that the methods we used to motivate individuals were effective and have resulted in a significant downward trend in our hypothermia rate.

There are several limitations to our QI project. The first involves the measuring of postoperative temperature in the recovery units. The temperatures were obtained using the same medical-grade infrared thermometer for all the patients, but other variables, such as timing and techniques, were not standardized. Secondly, overall surgical outcomes related to hypothermia were not tracked because we were unable to control for other confounding variables in our large cohort of patients, so we cannot say if the drop in the hypothermia rate had a clinically significant outcome. Thirdly, we propose that SDT offers a compellingly fitting explanation of the psychology of motivation in our efforts, but it may be possible that other theories may offer equally fitting explanations. The ability to measure the degree of motivation is lacking, and we did not explicitly ask participants what their specific source of motivation was. Aside from SDT, the reduction in hypothermia rate could also be attributed to the ease and availability of warming equipment that was made in each OR. This QI project was successfully applied to only 1 institution, so its ability to be widely applicable remains uncertain. In addition, data collection continued during the COVID-19 pandemic when case volumes decreased. However, by June 2020, the number of surgical cases at our institution had largely returned to prepandemic levels. Additional data collection beyond April 2021 would be helpful to determine if the reduction in hypothermia rates is truly sustained.

Conclusion

Overall, the importance of maintaining perioperative normothermia was well disseminated and agreed upon by all departments involved. Despite the limitations of the project, there was a significant reduction in rates of hypothermia, and sustainability of outcomes was consistently demonstrated in the poststudy period.

Using 3 cycles of the PDSA method, we successfully decreased the median rate of postoperative hypothermia in our pediatric surgical population from a preintervention value of 6.9% to 1.3%—a reduction of more than 81.2%. We provided motivation for members of our anesthesiology staff to participate by offering MOCA 2.0 Part 4 credits, but the lower rate of hypothermic patients was maintained for 15 months after the credits were distributed. Over the course of the project, there was a shift in culture, and extra vigilance was given to temperature monitoring and assessment. We attribute this sustained cultural change to the deliberate incorporation of the principles of competency, autonomy, and relatedness central to SDT to the structure of the interventions, avoiding rigid guidelines and pathways in favor of affective engagement to establish intrinsic motivation.

Acknowledgements: The authors thank Joan Reisch, PhD, for her assistance with the statistical analysis.

Corresponding author: Edgar Erold Kiss, MD, 1935 Medical District Dr, Dallas, TX 75235; edgar.kiss@UTSouthwestern.edu.

Financial disclosures: None.

From Children’s Health System of Texas, Division of Pediatric Anesthesiology, Dallas, TX (Drs. Sakhai, Bocanegra, Chandran, Kimatian, and Kiss), UT Southwestern Medical Center, Department of Anesthesiology and Pain Management, Dallas, TX (Drs. Bocanegra, Chandran, Kimatian, and Kiss), and UT Southwestern Medical Center, Department of Population and Data Sciences, Dallas, TX (Dr. Reisch).

Objective: Policy-driven changes in medical practice have long been the norm. Seldom are changes in clinical practice sought to be brought about by a person’s tendency toward growth or self‐actualization. Many hospitals have instituted hypothermia bundles to help reduce the incidence of unanticipated postoperative hypothermia. Although successful in the short-term, sustained changes are difficult to maintain. We implemented a quality-improvement project focused on addressing the affective components of self-determination theory (SDT) to create sustainable behavioral change while satisfying providers’ basic psychological needs for autonomy, competence, and relatedness.

Methods: A total of 3 Plan-Do-Study-Act (PDSA) cycles were enacted over the span of 14 months at a major tertiary care pediatric hospital to recruit and motivate anesthesia providers and perioperative team members to reduce the percentage of hypothermic postsurgical patients by 50%. As an optional initial incentive for participation, anesthesiologists would qualify for American Board of Anesthesiology Maintenance of Certification in Anesthesiology (MOCA) Part 4 Quality Improvement credits for monitoring their own temperature data and participating in project-related meetings. Providers were given autonomy to develop a personal plan for achieving the desired goals.

Results: The median rate of hypothermia was reduced from 6.9% to 1.6% in July 2019 and was reduced again in July 2020 to 1.3%, an 81% reduction overall. A low hypothermia rate was successfully maintained for at least 21 subsequent months after participants received their MOCA credits in July 2019.

Conclusions: Using an approach that focused on the elements of competency, autonomy, and relatedness central to the principles of SDT, we observed the development of a new culture of vigilance for prevention of hypothermia that successfully endured beyond the project end date.

Keywords: postoperative hypothermia; self-determination theory; motivation; quality improvement.

Perioperative hypothermia, generally accepted as a core temperature less than 36 °C in clinical practice, is a common complication in the pediatric surgical population and is associated with poor postoperative outcomes.¹ Hypothermic patients may develop respiratory depression, hypoglycemia, and metabolic acidosis that may lead to decreased oxygen delivery and end organ tissue hypoxia.^2-4 Other potential detrimental effects of failing to maintain normal body temperature are impaired clotting factor enzyme function and platelet dysfunction, increasing the risk for postoperative bleeding.^5,6 In addition, there are financial implications when hypothermic patients require care and resources postoperatively because of delayed emergence or shivering.⁷

The American Society of Anesthesiologists recommends intraoperative temperature monitoring for procedures when clinically significant changes in body temperature are anticipated.⁸ Maintenance of normothermia in the pediatric population is especially challenging owing to a larger skin-surface area compared with body mass ratio and less subcutaneous fat content than in adults. Preventing postoperative hypothermia starts preoperatively with parental education and can be as simple as covering the child with a blanket and setting the preoperative room to an acceptably warm temperature.^9,10 Intraoperatively, maintaining operating room (OR) temperatures at or above 21.1 °C and using active warming devices and radiant warmers when appropriate are important techniques to preserve the child’s body temperature.^11,12

Despite the knowledge of these risks and vigilant avoidance of hypothermia, unplanned perioperative hypothermia can occur in up to 70% of surgical patients.¹ Beyond the clinical benefits, as health care marches toward a value-based payment methodology, quality indicators such as avoiding hypothermia may be linked directly to payment.

Self-determination theory (SDT) was first developed in 1980 by Deci and Ryan.¹³ The central premise of the theory states that people develop their full potential if circumstances allow them to satisfy their basic psychological needs: autonomy, competence, and relatedness. Under these conditions, people’s natural inclination toward growth can be realized, and they are more likely to internalize external goals. Under an extrinsic reward system, motivation can waver, as people may perceive rewards as controlling.

Many institutions have implemented hypothermia bundles to help decrease the rate of hypothermic patients, but while initially successful, the effectiveness of these interventions tends to fade over time as participants settle into old, comfortable routines.¹⁴ With SDT in mind, we designed our quality-improvement (QI) project with interventions to allow clinicians autonomy without instituting rigid guidelines or punitive actions. We aimed to directly address the affective components central to motivation and engagement so that we could bring about long-term meaningful changes in our practice.

Methods

Setting

The hypothermia QI intervention was instituted at a major tertiary care children’s hospital that performs more than 40 000 pediatric general anesthetics annually. Our division of pediatric anesthesiology consists of 66 fellowship-trained pediatric anesthesiologists, 15 or more rotating trainees per month, 13 anesthesiology assistants, 15 anesthesia technicians, and more than 50 perioperative nurses.

The most frequent pediatric surgeries include, but are not limited to, general surgery, otolaryngology, urology, gastroenterology, plastic surgery, neurosurgery, and dentistry. The surgeries are conducted in the hospital’s main operative floor, which consists of 15 ORs and 2 gastroenterology procedure rooms. Although the implementation of the QI project included several operating sites, we focused on collecting temperature data from surgical patients at our main campus recovery unit. We obtained the patients’ initial temperatures upon arrival to the recovery unit from a retrospective electronic health record review of all patients who underwent anesthesia from January 2016 through April 2021.

Postoperative hypothermia was identified as an area of potential improvement after several patients were reported to be hypothermic upon arrival to the recovery unit in the later part of 2018. Further review revealed significant heterogeneity of practices and lack of standardization of patient-warming methods. By comparing the temperatures pre- and postintervention, we could measure the effectiveness of the QI initiative. Prior to the start of our project, the hypothermia rate in our patient population was not actively tracked, and the effectiveness of our variable practice was not measured.

The cutoff for hypothermia for our QI project was defined as body temperature below 36 °C, since this value has been previously used in the literature and is commonly accepted in anesthesia practice as the delineation for hypothermia in patients undergoing general anesthesia.¹

Interventions

This QI project was designed and modeled after the Institute for Healthcare Improvement Model for Improvement.¹⁵ Three cycles of Plan-Do-Study-Act (PDSA) were developed and instituted over a 14-month period until December 2019 (Table 1).

A retrospective review was conducted to determine the percentage of surgical patients arriving to our recovery units with an initial temperature reading of less than 36 °C. A project key driver diagram and smart aim were created and approved by the hospital’s continuing medical education (CME) committee for credit via the American Board of Medical Specialties (ABMS) Multi-Specialty Portfolio Program, Maintenance of Certification in Anesthesiology (MOCA) Part 4.

The first PDSA cycle involved introducing the QI project and sharing the aims of the project at a department grand rounds in the latter part of October 2018. Enrollment to participate in the project was open to all anesthesiologists in the division, and participants could earn up to 20 hours of MOCA Part 4 credits. A spreadsheet was developed and maintained to track each anesthesiologist’s monthly percentage of hypothermic patients. The de-identified patient data were shared with the division via monthly emails. In addition, individual providers with a hypothermic patient in the recovery room received a notification email.

The anesthesiologists participated in the QI project by reviewing their personal percentage of hypothermic patients on an ongoing basis to earn the credit. There was no explicit requirement to decrease their own rate of patients with body temperature less than 36 °C or expectation to achieve a predetermined goal, so the participants could not “fail.”

Because of the large interest in this project, a hypothermia committee was formed that consisted of 36 anesthesiologists. This group reviewed the data and exchanged ideas for improvement in November 2018 as part of the first PDSA cycle. The committee met monthly and was responsible for actively engaging other members of the department and perioperative staff to help in this multidisciplinary effort of combating hypothermia in our surgical pediatric population.

PDSA cycle 2 involved several major initiatives, including direct incorporation of the rest of the perioperative team. The perioperative nursing team was educated on the risks of hypothermia and engaged to take an active role by maintaining the operating suite temperature at 21.1 °C and turning on the Bair Hugger (3M) blanket to 43 °C on the OR bed prior to patient arrival to the OR. Additionally, anesthesia technicians (ATs) were tasked with ensuring an adequate supply of Bair Hugger drapes for all cases of the day. The facility’s engineering team was engaged to move the preoperative room temperature controls away from families (who frequently made the rooms cold) and instead set it at a consistent temperature of 23.9 °C. ATs were also asked to place axillary and nasal temperature probes on the anesthesia workstations as a visual reminder to facilitate temperature monitoring closer to the start of anesthesia (instead of the anesthesia provider having to remember to retrieve a temperature probe out of a drawer and place it on the patient). Furthermore, anesthesiologists were instructed via the aforementioned monthly emails and at monthly department meetings to place the temperature probes as early as possible in order to recognize and respond to intraoperative hypothermia in a timelier manner. Finally, supply chain leaders were informed of our expected increase in the use of the blankets and probes and proportionally increased ordering of these supplies to make sure availability would not present an obstacle.

In PDSA cycle 3, trainees (anesthesia assistant students, anesthesia residents and fellows) and advanced practice providers (APPs) (certified registered nurse-anesthetists [CRNAs] and certified anesthesia assistants [C-AAs]) were informed of the QI project. This initiative was guided toward improving vigilance for hypothermia in the rest of the anesthesia team members. The trainees and APPs usually set up the anesthesia area prior to patient arrival, so their recruitment in support of this effort would ensure appropriate OR temperature, active warming device deployment, and the availability and early placement of the correct temperature probe for the case. To facilitate personal accountability, the trainees and APPs were also emailed their own patients’ rate of hypothermia.

Along the course of the project, quarterly committee meetings and departmental monthly meetings served as venues to express concerns and look for areas of improvement, such as specific patterns or trends leading to hypothermic patients. One specific example was the identification of the gastrointestinal endoscopic patients having a rate of hypothermia that was 2% higher than average. Directed education on the importance of Bair Hugger blankets and using warm intravenous fluids worked well to decrease the rate of hypothermia in these patients. This collection of data was shared at regular intervals during monthly department meetings as well and more frequently using departmental emails. The hospital’s secure intranet SharePoint (Microsoft) site was used to share the data among providers.

Study of the interventions and measures

To study the effectiveness and impact of the project to motivate our anesthesiologists and other team members, we compared the first temperatures obtained in the recovery unit prior to the start of the intervention with those collected after the start of the QI project in November 2018. Because of the variability of temperature monitoring intraoperatively (nasal, axillary, rectal), we decided to use the temperature obtained by the nurse in the recovery room upon the patient’s arrival. Over the years analyzed, the nurse’s technique of measuring the temperature remained consistent. All patient temperature measurements were performed using the TAT-5000 (Exergen Corporation). This temporal artery thermometer has been previously shown to correlate well with bladder temperatures (70% of measurements differ by no more than 0.5 °C, as reported by Langham et al¹⁶).

Admittedly, we could not measure the degree of motivation or internalization of the project goals by our cohort, but we could measure the reduction in the rate of hypothermia and subjectively gauge engagement in the project by the various groups of participants and the sustainability of the results. In addition, all participating anesthesiologists received MOCA Part 4 credits in July 2019. We continued our data collection until April 2021 to determine if our project had brought about sustainable changes in practice that would continue past the initial motivator of obtaining CME credit.

Analysis

Data analysis was performed using Excel (Microsoft) and SAS, version 9.4 (SAS Institute).

The median of the monthly percentage of patients with a temperature of less than 36.0 °C was also determined for the preintervention time frame. This served as our baseline hypothermia rate, and we aimed to lower it by 50%. Run charts, a well-described methodology to gauge the effectiveness of the QI project, were constructed with the collected data.¹⁷

We performed additional analysis to adjust for different time periods throughout the year. The time period between January 2016 and October 2018 was considered preintervention. We considered November 2018 the start of our intervention, or more specifically, the start of our PDSA cycles. October 2018 was analyzed as part of the preintervention data. To account for seasonal temperature variations, the statistical analysis focused on the comparisons of the same calendar quarters for before and after starting intervention using Wilcoxon Mann-Whitney U tests. To reach an overall conclusion, the probabilities for the 4 quarters were combined for each criterion separately utilizing the Fisher χ² combined probability method.

The hypothermia QI project was reviewed by the institutional review board and determined to be exempt.

Results

The temperatures of 40 875 patients were available for analysis for the preintervention period between January 2016 and October 2018. The median percentage of patients with temperatures less than 36.0 °C was 6.9% (interquartile range [IQR], 5.8%-8.4%). The highest percentage was in February 2016 (9.9%), and the lowest was in March 2018 (3.4%). Following the start of the first PDSA cycle, the next 6 consecutive rates of hypothermia were below the median preintervention value, and a new median for these percentages was calculated at 3.4% (IQR, 2.6%-4.3%). In July 2019, the proportion of hypothermic patients decreased once more for 6 consecutive months, yielding a new median of 1.6% (IQR, 1.2%-1.8%) and again in July 2020, to yield a median of 1.3% (IQR, 1.2%-1.5%) (Figure). In all, 33 799 patients were analyzed after the start of the project from November 2018 to the end of the data collection period through April 2021.

The preintervention monthly rates of hypothermia were compared, quarter to quarter, with those starting in November 2018 using the Wilcoxon Mann-Whitney U test. The decrease in proportion of hypothermic patients after the start of the intervention was statistically significant (P < .001). In addition, the percentage of patients with temperatures greater than 38 °C was not significantly different between the pre- and postintervention time periods (P < .25) (Table 2). The decrease in the number of patients available for analysis from March 2020 to May 2020 was due to the COVID-19 pandemic.

Subjectively, we did not experience any notable resistance to our efforts, and the experience was largely positive for everyone involved. Clinicians identified as having high monthly rates of hypothermia (5% or higher) corrected their numbers the following month after being notified via email or in person.

To achieve changes in practice, the health care industry has relied on instituting guidelines, regulations, and policies, often with punitive consequences. We call into question this long-standing framework and propose a novel approach to help evolve the field of QI. Studies in human psychology have long demonstrated the demotivation power of a reward system and the negative response to attempts by authority to use incentives to control or coerce. In our QI project, we instituted 3 PDSA cycles and applied elements from SDT to motivate people’s behaviors. We demonstrate how a new culture focused on maintaining intraoperative normothermia was developed and brought about a measurable and significant decrease in the rate of hypothermia. The relevance of SDT, a widely accepted unifying theory that bridges and links social and personality psychology, should not be understated in health care. Authorities wishing to have long-standing influence should consider a person’s right to make their own decisions and, if possible, a unique way of doing things.

Positively reinforcing behavior has been shown to have a paradoxical effect by dampening an individual’s intrinsic motivation or desire to perform certain tasks.¹⁸ Deadlines, surveillance, and authoritative commands are also deterrents.^19,20 We focused on providing the tools and information to the clinicians and relied on their innate need for autonomy, growth, and self-actualization to bring about change in clinical practice.²¹ Group meetings served as a construct for exchanging ideas and to encourage participation, but without the implementation of rigid guidelines or policies. Intraoperative active warming devices and temperature probes were made available, but their use was not mandated. The use of these devices was intentionally not audited to avoid any overbearing control. Providers were, however, given monthly temperature data to help individually assess the effectiveness of their interventions. We did not impose any negative or punitive actions for those clinicians who had high rates of hypothermic patients, and we did not reward those who had low rates of hypothermia. We wanted the participants to feel that the inner self was the source of their behavior, and this was in parallel with their own interests and values. If providers could feel their need for competency could be realized, we hoped they would continue to adhere to the measures we provided to maintain a low rate of hypothermia.

The effectiveness of our efforts was demonstrated by a decrease in the prevalence of postoperative hypothermia in our surgical patients. The initial decrease of the median rate of hypothermia from 6.9% to 3.4% occurred shortly into the start of the first PDSA cycle. The second PDSA cycle started in January 2019 with a multimodal approach and included almost all parties involved in the perioperative care of our surgical patients. Not only was this intervention responsible for a continued downward trend in the percentage of hypothermic patients, but it set the stage for the third and final PDSA cycle, which started in July 2019. The architecture was in place to integrate trainees and APPs to reinforce our initiative. Subsequently, the new median percentage of hypothermic patients was further decreased to an all-time low of 1.6% per month, satisfying and surpassing the goal of the QI project of decreasing the rate of hypothermia by only 50%. Our organization thereafter maintained a monthly hypothermia rate below 2%, except for April 2020, when it reached 2.5%. Our lowest median percentage was obtained after July 2020, reaching 1.3%.

To account for seasonal variations in temperatures and types of surgeries performed, we compared the percentage of hypothermic patients before and after the start of intervention, quarter by quarter. The decrease in the proportion of hypothermic patients after the start of intervention was statistically significant (P < .001). In addition, the data failed to prove any statistical difference for temperatures above 38 °C between the 2 periods, indicating that our interventions did not result in significant overwarming of patients. The clinical implications of decreasing the percentage of hypothermic patients from 6.9% to 1.3% is likely clinically important when considering the large number of patients who undergo surgery at large tertiary care pediatric centers. Even if simple interventions reduce hypothermia in only a handful of patients, routine applications of simple measures to keep patients normothermic is likely best clinical practice.

Anesthesiologists who participated in the hypothermia QI project by tracking the incidence of hypothermia in their patients were able to collect MOCA Part 4 credits in July 2019. There was no requirement for the individual anesthesiologist to reduce the rate of hypothermia or apply any of the encouraged strategies to obtain credit. As previously stated, there were also no rewards for obtaining low hypothermia rates for the providers. The temperature data continued to be collected through April 2021, 21 months after the credits were distributed, to demonstrate a continued, meaningful change, at least in the short-term. While the MOCA Part 4 credits likely served as an initial motivating factor to encourage participation in the QI project, they certainly were not responsible for the sustained low hypothermia rate after July 2019. We showed that the low rate of hypothermia was successfully maintained, indicating that the change in providers’ behavior was independent of the external motivator of obtaining the credit hours. Mere participation in the project by reviewing one’s temperature data was all that was required to obtain the credit. The Organismic Integration Theory, a mini-theory within SDT, best explains this phenomenon by describing any motivated behavior on a continuum ranging from controlled to autonomous.²² Do people perform the task resentfully, on their own volition because they believe it is the correct action, or somewhere in between? We explain the sustained low rates of hypothermia after the MOCA credits were distributed due to a shift to the autonomous end of the continuum with the clinician’s active willingness to meet the challenges and apply intrinsically motivated behaviors to lower the rate of hypothermia. The internalization of external motivators is difficult to prove, but the evidence supports that the methods we used to motivate individuals were effective and have resulted in a significant downward trend in our hypothermia rate.

There are several limitations to our QI project. The first involves the measuring of postoperative temperature in the recovery units. The temperatures were obtained using the same medical-grade infrared thermometer for all the patients, but other variables, such as timing and techniques, were not standardized. Secondly, overall surgical outcomes related to hypothermia were not tracked because we were unable to control for other confounding variables in our large cohort of patients, so we cannot say if the drop in the hypothermia rate had a clinically significant outcome. Thirdly, we propose that SDT offers a compellingly fitting explanation of the psychology of motivation in our efforts, but it may be possible that other theories may offer equally fitting explanations. The ability to measure the degree of motivation is lacking, and we did not explicitly ask participants what their specific source of motivation was. Aside from SDT, the reduction in hypothermia rate could also be attributed to the ease and availability of warming equipment that was made in each OR. This QI project was successfully applied to only 1 institution, so its ability to be widely applicable remains uncertain. In addition, data collection continued during the COVID-19 pandemic when case volumes decreased. However, by June 2020, the number of surgical cases at our institution had largely returned to prepandemic levels. Additional data collection beyond April 2021 would be helpful to determine if the reduction in hypothermia rates is truly sustained.

Conclusion

Overall, the importance of maintaining perioperative normothermia was well disseminated and agreed upon by all departments involved. Despite the limitations of the project, there was a significant reduction in rates of hypothermia, and sustainability of outcomes was consistently demonstrated in the poststudy period.

Using 3 cycles of the PDSA method, we successfully decreased the median rate of postoperative hypothermia in our pediatric surgical population from a preintervention value of 6.9% to 1.3%—a reduction of more than 81.2%. We provided motivation for members of our anesthesiology staff to participate by offering MOCA 2.0 Part 4 credits, but the lower rate of hypothermic patients was maintained for 15 months after the credits were distributed. Over the course of the project, there was a shift in culture, and extra vigilance was given to temperature monitoring and assessment. We attribute this sustained cultural change to the deliberate incorporation of the principles of competency, autonomy, and relatedness central to SDT to the structure of the interventions, avoiding rigid guidelines and pathways in favor of affective engagement to establish intrinsic motivation.

Acknowledgements: The authors thank Joan Reisch, PhD, for her assistance with the statistical analysis.

Corresponding author: Edgar Erold Kiss, MD, 1935 Medical District Dr, Dallas, TX 75235; edgar.kiss@UTSouthwestern.edu.

Financial disclosures: None.

From Western Sussex Hospitals NHS Foundation Trust, Physiotherapy Department, Chichester, UK (Richard J. Holmes), and Western Sussex Hospitals NHS Foundation Trust, Department of Occupational Therapy, Chichester, UK (Sophie Stratford).

Objective: Despite the benefits of early and frequent input from a neurologist, there is wide variation in the availability of this service, especially in district general hospitals, with many patients managed on clinically inappropriate wards. The purpose of this service evaluation was to explore the impact this had on patient care.

Methods: A retrospective service evaluation was undertaken at a National Health Service hospital by reviewing patient records over a 6-month period. Data related to demographics, processes within the patient’s care, and secondary complications were recorded. Findings were compared with those of stroke patients managed on a specialist stroke ward.

Results: A total of 63 patients were identified, with a mean age of 72 years. The mean length of stay was 25.9 days, with a readmission rate of 16.7%. Only 15.9% of patients were reviewed by a neurologist. There was a high rate of secondary complications, with a number of patients experiencing falls (11.1%), pressure ulcers (14.3%), and health care–acquired infections (33.3%) during their admission.

Conclusions: The lack of specialist input from a neurologist and the management of patients on clinically inappropriate wards may have negatively impacted length of stay, readmission rates, and the frequency of secondary complications.

Keywords: evaluation; clinical safety; neurology; patient-centered care; clinical outcomes; length of stay.

It is estimated that 10% of acute admissions to district general hospitals (DGHs) of the National Health Service (NHS) in the United Kingdom are due to a neurological problem other than stroke.¹ In 2011, a joint report from the Royal College of Physicians and the Association of British Neurologists (ABN) recommended that all of these patients should be admitted under the care of a neurologist and be regularly reviewed by a neurologist during their admission.² The rationale for this recommendation is clear. The involvement of a neurologist has been shown to improve accuracy of the diagnosis³ and significantly reduce length of stay.^4,5 Studies have also shown that the involvement of a neurologist has led to a change in the management plan in as high as 79%⁶ to 89%³ of cases, suggesting that a high proportion of neurological patients not seen by a neurologist are being managed suboptimally.

Despite this, a recent ABN survey of acute neurology services found ongoing wide variations in the availability of this specialist care, with a large proportion of DGHs having limited or no access to a neurologist and very few having dedicated neurology beds.⁷ While it is recognized that services have been structured in response to the reduced numbers of neurologists within the United Kingdom,⁸ it is prudent to assess the impact that such services have on patient care.

With this in mind, we planned to evaluate the current provision of care provided to neurological patients in a real-world setting. This was conducted in the context of a neurology liaison service at a DGH with no dedicated neurology beds.

Methods

A retrospective service evaluation was undertaken at a DGH in the southeast of England. The NHS hospital has neurologists on site who provide diagnostic and therapeutic consultations on the wards, but there are no dedicated beds for patients with neurological conditions. Patients requiring neurosurgical input are referred to a tertiary neurosciences center.

Patients were selected from the neurotherapy database if they were referred into the service between August 1, 2019, and January 31, 2020. The neurotherapy database was used as this was the only source that held thorough data on this patient group and allowed for the identification of patients who were not referred into the neurologist’s service. Patients were included if they had a new neurological condition as their primary diagnosis or if they had an exacerbation of an already established neurological condition. If a patient was admitted with more than 1 neurological diagnosis then the primary diagnosis for the admission was to be used in the analysis, though this did not occur during this evaluation. Patients with a primary diagnosis of a stroke were included if they were not managed on the acute stroke ward. Those managed on the stroke ward were excluded so that an analysis of patients managed on wards that were deemed clinically inappropriate could be undertaken. Patients were not included if they had a pre-existing neurological condition (ie, dementia, multiple sclerosis) but were admitted due to a non-neurological cause such as a fall or infection. All patients who met the criteria were included.

A team member independently reviewed each set of patient notes. Demographic data extracted from the medical notes included the patient’s age (on admission), gender, and diagnosis. Medical, nursing, and therapy notes were reviewed to identify secondary complications that arose during the patient’s admission. The secondary complications reviewed were falls (defined as the patient unexpectedly coming to the ground or other lower level), health care–acquired infections (HAIs) (defined as any infection acquired during the hospital admission), and pressure ulcers (defined as injuries to the skin or underlying tissue during the hospital admission). Other details, obtained from the patient administration system, included the length of stay (days), the number of ward moves the patient experienced, the speciality of the consultant responsible for the patient’s care, the discharge destination, and whether the patient was readmitted for any cause within 30 days. All data collected were stored on a password-protected computer and no patient-identifiable data were included.

The results were collated using descriptive statistics. The χ² test was used to compare categorical data between those patients who were and were not reviewed by a neurologist, and the Mann-Whitney U test was used to compare differences in the length of stay between these 2 groups.

No national data relating to this specific patient group were available within the literature. Therefore, to provide a comparator of neurological patients within the same hospital, data were collected on stroke patients managed on the stroke ward. This group was deemed most appropriate for comparison as they present with similar neurological symptoms but are cared for on a specialist ward. During the evaluation period, 284 stroke patients were admitted to the stroke ward. A sample of 75 patients was randomly selected using a random number generator, and the procedure for data collection was repeated. It was not appropriate to make direct comparative analysis on these 2 groups due to the inherent differences, but it was felt important to provide context with regards to what usual care was like on a specialist ward within the same hospital.

Ethical approval was not required as this was a service evaluation of routinely collected data within a single hospital site.

Results

In total, 63 patients were identified: 26 females and 37 males. The median age of patients was 74 years (range, 39-92 years). These demographic details and comparisons to stroke patients managed on a specialist ward can be seen in Table 1. To quantify the range of diagnoses, the condition groups defined by GIRFT Neurology Methodology⁹ were used. The most common diagnoses were tumors of the nervous system (25.4%) and traumatic brain and spine injury (23.8%). The other conditions included in the analysis can be seen in Table 2.

Despite having a neurological condition as their primary diagnosis, only 15.9% of patients were reviewed by a neurologist during their hospital admission. Patients were most commonly under the care of a geriatrician (60.3%), but they were also managed by orthopedics (12.6%), acute medicine (7.9%), respiratory (6.3%), cardiology (4.8%), gastroenterology (3.2%), and surgery (3.2%). One patient (1.6%) was managed by intensivists.

The average length of stay was 25.9 days (range, 2-78 days). This was more than double the average length of stay on the stroke ward (11.4 days) (Table 1) and the national average for patients with neurological conditions (9.78 days).¹⁰ During their stay, 33% had 2 or more ward moves, with 1 patient moving wards a total of 6 times. Just over half (52.4%) of the patients returned to their usual residence on discharge. The remainder were discharged to rehabilitation units (15.9%), nursing homes (14.3%), residential homes (6.3%), tertiary centers (4.8%), and hospice (1.6%). Unfortunately, 3 patients (4.8%) passed away. Of those still alive (n = 60), 16.7% were readmitted to the hospital within 30 days, compared to a readmission rate of 11% on the stroke ward. None of the patients who were readmitted were seen by a neurologist during their initial admission.

The frequency of secondary complications was reviewed as a measure of the multidisciplinary management of this patient group. It was noted that 11.1% had a fall on the ward, which was similar to a rate of 10.7% on the stroke ward. More striking was the fact that 14.3% of patients developed a pressure ulcer and 33.3% developed an HAI during their admission, compared with rates of 1.3% and 10.7%, respectively, on the stroke ward (Table 1).

There were no significant differences found in length of stay between those who were and were not reviewed by a neurologist (P = .73). This was also true for categorical data, whereby readmission rate (P = .13), frequency of falls (P = .22), frequency of pressure ulcers (P = .67), and HAIs (P = .81) all failed to show a significant difference between groups.

Discussion

The findings of this service evaluation show markedly poorer outcomes for neurological patients compared to stroke patients managed on a specialist stroke ward. It is suggested that these results are in part due to the lack of specialist input from a neurologist in the majority of cases and the fact that all were managed on clinically inappropriate wards. Only 15.9% of neurological patients were seen by a neurologist. This is a slight improvement compared to previous studies in DGHs that showed rates of 10%¹ and 11%,¹¹ but it is still a far cry from the goal of 100% set out in recommendations.² In addition, the increased readmission rate may be suggestive of suboptimal management, especially given that none of those readmitted had been reviewed by a neurologist. There are undoubtedly other factors that may influence readmissions, such as comorbidities, the severity/complexity of the condition, and the strength of community services. However, the impact of a lack of input from a specialist should not be underestimated, and further evaluation of this factor (with confounding factors controlled) would be beneficial.

The result of an extended length of stay was also a predictable outcome based on previous evidence.^4,5 With the potential for suboptimal management plans and inaccurate diagnoses, it is inevitable that the patient’s movement through the hospital system will be impeded. In our example, it is possible that the extended length of stay was influenced by the fact that patients included in the evaluation were managed on nonspecialist wards and a large proportion had multiple ward changes.

Given that the evidence clearly shows that stroke patients are most effectively managed by a multidisciplinary team (MDT) with specialist skills,¹² it is likely that other neurological patients, who have similar multifactorial needs, would also benefit. The patients in our evaluation were cared for by nursing staff who lacked specific skills and experience in neurology. The allied health professionals involved were specialists in neurotherapy but were not based on the ward and not directly linked to the ward MDT. A review by Epstein found that the benefits of having a MDT, in any speciality, working together on a ward included improved communication, reduced adverse events, and a reduced length of stay.¹³ This lack of an effective MDT approach may provide some explanation as to why the average length of stay and the rates of some secondary complications were at such elevated levels.

A systematic review exploring the impact of patients admitted to clinically inappropriate wards in a range of specialities found that these patients were associated with worse outcomes.¹⁴ This is supported by our findings, in which a higher rate of pressure ulcers and HAIs were observed when compared to rates in the specialist stroke ward. Again, a potential explanation for this is the impact of patients being managed by clinicians who lack the specialist knowledge of the patient group and the risks they face. Another explanation could be due to the high number of ward moves the patients experienced. Blay et al found that ward moves increased length of stay and carried an associated clinical risk, with the odds of falls and HAIs increasing with each move.¹⁵ A case example of this is apparent within our analysis in that the patient who experienced 6 ward moves not only had the longest length of stay (78 days), but also developed a pressure ulcer and 2 HAIs during their admission.

This service evaluation had a number of limitations that should be considered when interpreting the results. First, despite including all patients who met the criteria within the stipulated time frame, the sample size was relatively small, making it difficult to identify consistent patterns of behavior within the data.

Furthermore, caution should be applied when interpreting the comparators used, as the patient groups are not equivalent. The use of comparison against a standard is not a prerequisite in a service evaluation of this nature, but comparators were included to help frame the context for the reader. As such, they should only be used in this way rather than to make any firm conclusions.

Finally, as the evaluation was limited to the use of routinely collected data, there are several variables, other than those reported, which may have influenced the results. For example, it was not possible to ascertain certain demographic details, such as body mass index and socioeconomic factors, nor lifestyle factors such as smoking status, alcohol consumption, and exercise levels, all of which could impact negatively on the outcomes of interest. Furthermore, data were not collected on follow-up services after discharge to evaluate whether these had any impact on readmission rates.

Conclusion

This service evaluation highlights the potential impact of managing neurological patients on clinically inappropriate wards with limited input from a neurologist. There is the potential to ameliorate these impacts by cohorting these patients in neurologist-led beds with a specialist MDT. While there are limitations in the design of our study, including the lack of a controlled comparison, the small sample size, and the fact that this is an evaluation of a single service, the negative impacts to patients are concerning and warrant further investigation.

Corresponding author: Richard J. Holmes, MSc, Physiotherapy Department, St. Richard’s Hospital, Chichester, West Sussex, PO19 6SE; richard.holmes8@nhs.net.

Financial disclosures: None.

From Western Sussex Hospitals NHS Foundation Trust, Physiotherapy Department, Chichester, UK (Richard J. Holmes), and Western Sussex Hospitals NHS Foundation Trust, Department of Occupational Therapy, Chichester, UK (Sophie Stratford).

Objective: Despite the benefits of early and frequent input from a neurologist, there is wide variation in the availability of this service, especially in district general hospitals, with many patients managed on clinically inappropriate wards. The purpose of this service evaluation was to explore the impact this had on patient care.

Methods: A retrospective service evaluation was undertaken at a National Health Service hospital by reviewing patient records over a 6-month period. Data related to demographics, processes within the patient’s care, and secondary complications were recorded. Findings were compared with those of stroke patients managed on a specialist stroke ward.

Results: A total of 63 patients were identified, with a mean age of 72 years. The mean length of stay was 25.9 days, with a readmission rate of 16.7%. Only 15.9% of patients were reviewed by a neurologist. There was a high rate of secondary complications, with a number of patients experiencing falls (11.1%), pressure ulcers (14.3%), and health care–acquired infections (33.3%) during their admission.

Conclusions: The lack of specialist input from a neurologist and the management of patients on clinically inappropriate wards may have negatively impacted length of stay, readmission rates, and the frequency of secondary complications.

Keywords: evaluation; clinical safety; neurology; patient-centered care; clinical outcomes; length of stay.

It is estimated that 10% of acute admissions to district general hospitals (DGHs) of the National Health Service (NHS) in the United Kingdom are due to a neurological problem other than stroke.¹ In 2011, a joint report from the Royal College of Physicians and the Association of British Neurologists (ABN) recommended that all of these patients should be admitted under the care of a neurologist and be regularly reviewed by a neurologist during their admission.² The rationale for this recommendation is clear. The involvement of a neurologist has been shown to improve accuracy of the diagnosis³ and significantly reduce length of stay.^4,5 Studies have also shown that the involvement of a neurologist has led to a change in the management plan in as high as 79%⁶ to 89%³ of cases, suggesting that a high proportion of neurological patients not seen by a neurologist are being managed suboptimally.

Despite this, a recent ABN survey of acute neurology services found ongoing wide variations in the availability of this specialist care, with a large proportion of DGHs having limited or no access to a neurologist and very few having dedicated neurology beds.⁷ While it is recognized that services have been structured in response to the reduced numbers of neurologists within the United Kingdom,⁸ it is prudent to assess the impact that such services have on patient care.

With this in mind, we planned to evaluate the current provision of care provided to neurological patients in a real-world setting. This was conducted in the context of a neurology liaison service at a DGH with no dedicated neurology beds.

Methods

A retrospective service evaluation was undertaken at a DGH in the southeast of England. The NHS hospital has neurologists on site who provide diagnostic and therapeutic consultations on the wards, but there are no dedicated beds for patients with neurological conditions. Patients requiring neurosurgical input are referred to a tertiary neurosciences center.

Patients were selected from the neurotherapy database if they were referred into the service between August 1, 2019, and January 31, 2020. The neurotherapy database was used as this was the only source that held thorough data on this patient group and allowed for the identification of patients who were not referred into the neurologist’s service. Patients were included if they had a new neurological condition as their primary diagnosis or if they had an exacerbation of an already established neurological condition. If a patient was admitted with more than 1 neurological diagnosis then the primary diagnosis for the admission was to be used in the analysis, though this did not occur during this evaluation. Patients with a primary diagnosis of a stroke were included if they were not managed on the acute stroke ward. Those managed on the stroke ward were excluded so that an analysis of patients managed on wards that were deemed clinically inappropriate could be undertaken. Patients were not included if they had a pre-existing neurological condition (ie, dementia, multiple sclerosis) but were admitted due to a non-neurological cause such as a fall or infection. All patients who met the criteria were included.

A team member independently reviewed each set of patient notes. Demographic data extracted from the medical notes included the patient’s age (on admission), gender, and diagnosis. Medical, nursing, and therapy notes were reviewed to identify secondary complications that arose during the patient’s admission. The secondary complications reviewed were falls (defined as the patient unexpectedly coming to the ground or other lower level), health care–acquired infections (HAIs) (defined as any infection acquired during the hospital admission), and pressure ulcers (defined as injuries to the skin or underlying tissue during the hospital admission). Other details, obtained from the patient administration system, included the length of stay (days), the number of ward moves the patient experienced, the speciality of the consultant responsible for the patient’s care, the discharge destination, and whether the patient was readmitted for any cause within 30 days. All data collected were stored on a password-protected computer and no patient-identifiable data were included.

The results were collated using descriptive statistics. The χ² test was used to compare categorical data between those patients who were and were not reviewed by a neurologist, and the Mann-Whitney U test was used to compare differences in the length of stay between these 2 groups.

No national data relating to this specific patient group were available within the literature. Therefore, to provide a comparator of neurological patients within the same hospital, data were collected on stroke patients managed on the stroke ward. This group was deemed most appropriate for comparison as they present with similar neurological symptoms but are cared for on a specialist ward. During the evaluation period, 284 stroke patients were admitted to the stroke ward. A sample of 75 patients was randomly selected using a random number generator, and the procedure for data collection was repeated. It was not appropriate to make direct comparative analysis on these 2 groups due to the inherent differences, but it was felt important to provide context with regards to what usual care was like on a specialist ward within the same hospital.

Ethical approval was not required as this was a service evaluation of routinely collected data within a single hospital site.

Results

In total, 63 patients were identified: 26 females and 37 males. The median age of patients was 74 years (range, 39-92 years). These demographic details and comparisons to stroke patients managed on a specialist ward can be seen in Table 1. To quantify the range of diagnoses, the condition groups defined by GIRFT Neurology Methodology⁹ were used. The most common diagnoses were tumors of the nervous system (25.4%) and traumatic brain and spine injury (23.8%). The other conditions included in the analysis can be seen in Table 2.

Despite having a neurological condition as their primary diagnosis, only 15.9% of patients were reviewed by a neurologist during their hospital admission. Patients were most commonly under the care of a geriatrician (60.3%), but they were also managed by orthopedics (12.6%), acute medicine (7.9%), respiratory (6.3%), cardiology (4.8%), gastroenterology (3.2%), and surgery (3.2%). One patient (1.6%) was managed by intensivists.

The average length of stay was 25.9 days (range, 2-78 days). This was more than double the average length of stay on the stroke ward (11.4 days) (Table 1) and the national average for patients with neurological conditions (9.78 days).¹⁰ During their stay, 33% had 2 or more ward moves, with 1 patient moving wards a total of 6 times. Just over half (52.4%) of the patients returned to their usual residence on discharge. The remainder were discharged to rehabilitation units (15.9%), nursing homes (14.3%), residential homes (6.3%), tertiary centers (4.8%), and hospice (1.6%). Unfortunately, 3 patients (4.8%) passed away. Of those still alive (n = 60), 16.7% were readmitted to the hospital within 30 days, compared to a readmission rate of 11% on the stroke ward. None of the patients who were readmitted were seen by a neurologist during their initial admission.

The frequency of secondary complications was reviewed as a measure of the multidisciplinary management of this patient group. It was noted that 11.1% had a fall on the ward, which was similar to a rate of 10.7% on the stroke ward. More striking was the fact that 14.3% of patients developed a pressure ulcer and 33.3% developed an HAI during their admission, compared with rates of 1.3% and 10.7%, respectively, on the stroke ward (Table 1).

There were no significant differences found in length of stay between those who were and were not reviewed by a neurologist (P = .73). This was also true for categorical data, whereby readmission rate (P = .13), frequency of falls (P = .22), frequency of pressure ulcers (P = .67), and HAIs (P = .81) all failed to show a significant difference between groups.

Discussion

The findings of this service evaluation show markedly poorer outcomes for neurological patients compared to stroke patients managed on a specialist stroke ward. It is suggested that these results are in part due to the lack of specialist input from a neurologist in the majority of cases and the fact that all were managed on clinically inappropriate wards. Only 15.9% of neurological patients were seen by a neurologist. This is a slight improvement compared to previous studies in DGHs that showed rates of 10%¹ and 11%,¹¹ but it is still a far cry from the goal of 100% set out in recommendations.² In addition, the increased readmission rate may be suggestive of suboptimal management, especially given that none of those readmitted had been reviewed by a neurologist. There are undoubtedly other factors that may influence readmissions, such as comorbidities, the severity/complexity of the condition, and the strength of community services. However, the impact of a lack of input from a specialist should not be underestimated, and further evaluation of this factor (with confounding factors controlled) would be beneficial.

The result of an extended length of stay was also a predictable outcome based on previous evidence.^4,5 With the potential for suboptimal management plans and inaccurate diagnoses, it is inevitable that the patient’s movement through the hospital system will be impeded. In our example, it is possible that the extended length of stay was influenced by the fact that patients included in the evaluation were managed on nonspecialist wards and a large proportion had multiple ward changes.

Given that the evidence clearly shows that stroke patients are most effectively managed by a multidisciplinary team (MDT) with specialist skills,¹² it is likely that other neurological patients, who have similar multifactorial needs, would also benefit. The patients in our evaluation were cared for by nursing staff who lacked specific skills and experience in neurology. The allied health professionals involved were specialists in neurotherapy but were not based on the ward and not directly linked to the ward MDT. A review by Epstein found that the benefits of having a MDT, in any speciality, working together on a ward included improved communication, reduced adverse events, and a reduced length of stay.¹³ This lack of an effective MDT approach may provide some explanation as to why the average length of stay and the rates of some secondary complications were at such elevated levels.

A systematic review exploring the impact of patients admitted to clinically inappropriate wards in a range of specialities found that these patients were associated with worse outcomes.¹⁴ This is supported by our findings, in which a higher rate of pressure ulcers and HAIs were observed when compared to rates in the specialist stroke ward. Again, a potential explanation for this is the impact of patients being managed by clinicians who lack the specialist knowledge of the patient group and the risks they face. Another explanation could be due to the high number of ward moves the patients experienced. Blay et al found that ward moves increased length of stay and carried an associated clinical risk, with the odds of falls and HAIs increasing with each move.¹⁵ A case example of this is apparent within our analysis in that the patient who experienced 6 ward moves not only had the longest length of stay (78 days), but also developed a pressure ulcer and 2 HAIs during their admission.

This service evaluation had a number of limitations that should be considered when interpreting the results. First, despite including all patients who met the criteria within the stipulated time frame, the sample size was relatively small, making it difficult to identify consistent patterns of behavior within the data.

Furthermore, caution should be applied when interpreting the comparators used, as the patient groups are not equivalent. The use of comparison against a standard is not a prerequisite in a service evaluation of this nature, but comparators were included to help frame the context for the reader. As such, they should only be used in this way rather than to make any firm conclusions.

Finally, as the evaluation was limited to the use of routinely collected data, there are several variables, other than those reported, which may have influenced the results. For example, it was not possible to ascertain certain demographic details, such as body mass index and socioeconomic factors, nor lifestyle factors such as smoking status, alcohol consumption, and exercise levels, all of which could impact negatively on the outcomes of interest. Furthermore, data were not collected on follow-up services after discharge to evaluate whether these had any impact on readmission rates.

Conclusion

This service evaluation highlights the potential impact of managing neurological patients on clinically inappropriate wards with limited input from a neurologist. There is the potential to ameliorate these impacts by cohorting these patients in neurologist-led beds with a specialist MDT. While there are limitations in the design of our study, including the lack of a controlled comparison, the small sample size, and the fact that this is an evaluation of a single service, the negative impacts to patients are concerning and warrant further investigation.

Corresponding author: Richard J. Holmes, MSc, Physiotherapy Department, St. Richard’s Hospital, Chichester, West Sussex, PO19 6SE; richard.holmes8@nhs.net.

Financial disclosures: None.

From Western Sussex Hospitals NHS Foundation Trust, Physiotherapy Department, Chichester, UK (Richard J. Holmes), and Western Sussex Hospitals NHS Foundation Trust, Department of Occupational Therapy, Chichester, UK (Sophie Stratford).

Objective: Despite the benefits of early and frequent input from a neurologist, there is wide variation in the availability of this service, especially in district general hospitals, with many patients managed on clinically inappropriate wards. The purpose of this service evaluation was to explore the impact this had on patient care.

Methods: A retrospective service evaluation was undertaken at a National Health Service hospital by reviewing patient records over a 6-month period. Data related to demographics, processes within the patient’s care, and secondary complications were recorded. Findings were compared with those of stroke patients managed on a specialist stroke ward.

Results: A total of 63 patients were identified, with a mean age of 72 years. The mean length of stay was 25.9 days, with a readmission rate of 16.7%. Only 15.9% of patients were reviewed by a neurologist. There was a high rate of secondary complications, with a number of patients experiencing falls (11.1%), pressure ulcers (14.3%), and health care–acquired infections (33.3%) during their admission.

Conclusions: The lack of specialist input from a neurologist and the management of patients on clinically inappropriate wards may have negatively impacted length of stay, readmission rates, and the frequency of secondary complications.

Keywords: evaluation; clinical safety; neurology; patient-centered care; clinical outcomes; length of stay.

It is estimated that 10% of acute admissions to district general hospitals (DGHs) of the National Health Service (NHS) in the United Kingdom are due to a neurological problem other than stroke.¹ In 2011, a joint report from the Royal College of Physicians and the Association of British Neurologists (ABN) recommended that all of these patients should be admitted under the care of a neurologist and be regularly reviewed by a neurologist during their admission.² The rationale for this recommendation is clear. The involvement of a neurologist has been shown to improve accuracy of the diagnosis³ and significantly reduce length of stay.^4,5 Studies have also shown that the involvement of a neurologist has led to a change in the management plan in as high as 79%⁶ to 89%³ of cases, suggesting that a high proportion of neurological patients not seen by a neurologist are being managed suboptimally.

Despite this, a recent ABN survey of acute neurology services found ongoing wide variations in the availability of this specialist care, with a large proportion of DGHs having limited or no access to a neurologist and very few having dedicated neurology beds.⁷ While it is recognized that services have been structured in response to the reduced numbers of neurologists within the United Kingdom,⁸ it is prudent to assess the impact that such services have on patient care.

With this in mind, we planned to evaluate the current provision of care provided to neurological patients in a real-world setting. This was conducted in the context of a neurology liaison service at a DGH with no dedicated neurology beds.

Methods

A retrospective service evaluation was undertaken at a DGH in the southeast of England. The NHS hospital has neurologists on site who provide diagnostic and therapeutic consultations on the wards, but there are no dedicated beds for patients with neurological conditions. Patients requiring neurosurgical input are referred to a tertiary neurosciences center.

Patients were selected from the neurotherapy database if they were referred into the service between August 1, 2019, and January 31, 2020. The neurotherapy database was used as this was the only source that held thorough data on this patient group and allowed for the identification of patients who were not referred into the neurologist’s service. Patients were included if they had a new neurological condition as their primary diagnosis or if they had an exacerbation of an already established neurological condition. If a patient was admitted with more than 1 neurological diagnosis then the primary diagnosis for the admission was to be used in the analysis, though this did not occur during this evaluation. Patients with a primary diagnosis of a stroke were included if they were not managed on the acute stroke ward. Those managed on the stroke ward were excluded so that an analysis of patients managed on wards that were deemed clinically inappropriate could be undertaken. Patients were not included if they had a pre-existing neurological condition (ie, dementia, multiple sclerosis) but were admitted due to a non-neurological cause such as a fall or infection. All patients who met the criteria were included.

A team member independently reviewed each set of patient notes. Demographic data extracted from the medical notes included the patient’s age (on admission), gender, and diagnosis. Medical, nursing, and therapy notes were reviewed to identify secondary complications that arose during the patient’s admission. The secondary complications reviewed were falls (defined as the patient unexpectedly coming to the ground or other lower level), health care–acquired infections (HAIs) (defined as any infection acquired during the hospital admission), and pressure ulcers (defined as injuries to the skin or underlying tissue during the hospital admission). Other details, obtained from the patient administration system, included the length of stay (days), the number of ward moves the patient experienced, the speciality of the consultant responsible for the patient’s care, the discharge destination, and whether the patient was readmitted for any cause within 30 days. All data collected were stored on a password-protected computer and no patient-identifiable data were included.

The results were collated using descriptive statistics. The χ² test was used to compare categorical data between those patients who were and were not reviewed by a neurologist, and the Mann-Whitney U test was used to compare differences in the length of stay between these 2 groups.

No national data relating to this specific patient group were available within the literature. Therefore, to provide a comparator of neurological patients within the same hospital, data were collected on stroke patients managed on the stroke ward. This group was deemed most appropriate for comparison as they present with similar neurological symptoms but are cared for on a specialist ward. During the evaluation period, 284 stroke patients were admitted to the stroke ward. A sample of 75 patients was randomly selected using a random number generator, and the procedure for data collection was repeated. It was not appropriate to make direct comparative analysis on these 2 groups due to the inherent differences, but it was felt important to provide context with regards to what usual care was like on a specialist ward within the same hospital.

Ethical approval was not required as this was a service evaluation of routinely collected data within a single hospital site.

Results

In total, 63 patients were identified: 26 females and 37 males. The median age of patients was 74 years (range, 39-92 years). These demographic details and comparisons to stroke patients managed on a specialist ward can be seen in Table 1. To quantify the range of diagnoses, the condition groups defined by GIRFT Neurology Methodology⁹ were used. The most common diagnoses were tumors of the nervous system (25.4%) and traumatic brain and spine injury (23.8%). The other conditions included in the analysis can be seen in Table 2.

Despite having a neurological condition as their primary diagnosis, only 15.9% of patients were reviewed by a neurologist during their hospital admission. Patients were most commonly under the care of a geriatrician (60.3%), but they were also managed by orthopedics (12.6%), acute medicine (7.9%), respiratory (6.3%), cardiology (4.8%), gastroenterology (3.2%), and surgery (3.2%). One patient (1.6%) was managed by intensivists.

The average length of stay was 25.9 days (range, 2-78 days). This was more than double the average length of stay on the stroke ward (11.4 days) (Table 1) and the national average for patients with neurological conditions (9.78 days).¹⁰ During their stay, 33% had 2 or more ward moves, with 1 patient moving wards a total of 6 times. Just over half (52.4%) of the patients returned to their usual residence on discharge. The remainder were discharged to rehabilitation units (15.9%), nursing homes (14.3%), residential homes (6.3%), tertiary centers (4.8%), and hospice (1.6%). Unfortunately, 3 patients (4.8%) passed away. Of those still alive (n = 60), 16.7% were readmitted to the hospital within 30 days, compared to a readmission rate of 11% on the stroke ward. None of the patients who were readmitted were seen by a neurologist during their initial admission.

The frequency of secondary complications was reviewed as a measure of the multidisciplinary management of this patient group. It was noted that 11.1% had a fall on the ward, which was similar to a rate of 10.7% on the stroke ward. More striking was the fact that 14.3% of patients developed a pressure ulcer and 33.3% developed an HAI during their admission, compared with rates of 1.3% and 10.7%, respectively, on the stroke ward (Table 1).

There were no significant differences found in length of stay between those who were and were not reviewed by a neurologist (P = .73). This was also true for categorical data, whereby readmission rate (P = .13), frequency of falls (P = .22), frequency of pressure ulcers (P = .67), and HAIs (P = .81) all failed to show a significant difference between groups.

Discussion

The findings of this service evaluation show markedly poorer outcomes for neurological patients compared to stroke patients managed on a specialist stroke ward. It is suggested that these results are in part due to the lack of specialist input from a neurologist in the majority of cases and the fact that all were managed on clinically inappropriate wards. Only 15.9% of neurological patients were seen by a neurologist. This is a slight improvement compared to previous studies in DGHs that showed rates of 10%¹ and 11%,¹¹ but it is still a far cry from the goal of 100% set out in recommendations.² In addition, the increased readmission rate may be suggestive of suboptimal management, especially given that none of those readmitted had been reviewed by a neurologist. There are undoubtedly other factors that may influence readmissions, such as comorbidities, the severity/complexity of the condition, and the strength of community services. However, the impact of a lack of input from a specialist should not be underestimated, and further evaluation of this factor (with confounding factors controlled) would be beneficial.

The result of an extended length of stay was also a predictable outcome based on previous evidence.^4,5 With the potential for suboptimal management plans and inaccurate diagnoses, it is inevitable that the patient’s movement through the hospital system will be impeded. In our example, it is possible that the extended length of stay was influenced by the fact that patients included in the evaluation were managed on nonspecialist wards and a large proportion had multiple ward changes.

Given that the evidence clearly shows that stroke patients are most effectively managed by a multidisciplinary team (MDT) with specialist skills,¹² it is likely that other neurological patients, who have similar multifactorial needs, would also benefit. The patients in our evaluation were cared for by nursing staff who lacked specific skills and experience in neurology. The allied health professionals involved were specialists in neurotherapy but were not based on the ward and not directly linked to the ward MDT. A review by Epstein found that the benefits of having a MDT, in any speciality, working together on a ward included improved communication, reduced adverse events, and a reduced length of stay.¹³ This lack of an effective MDT approach may provide some explanation as to why the average length of stay and the rates of some secondary complications were at such elevated levels.

A systematic review exploring the impact of patients admitted to clinically inappropriate wards in a range of specialities found that these patients were associated with worse outcomes.¹⁴ This is supported by our findings, in which a higher rate of pressure ulcers and HAIs were observed when compared to rates in the specialist stroke ward. Again, a potential explanation for this is the impact of patients being managed by clinicians who lack the specialist knowledge of the patient group and the risks they face. Another explanation could be due to the high number of ward moves the patients experienced. Blay et al found that ward moves increased length of stay and carried an associated clinical risk, with the odds of falls and HAIs increasing with each move.¹⁵ A case example of this is apparent within our analysis in that the patient who experienced 6 ward moves not only had the longest length of stay (78 days), but also developed a pressure ulcer and 2 HAIs during their admission.

This service evaluation had a number of limitations that should be considered when interpreting the results. First, despite including all patients who met the criteria within the stipulated time frame, the sample size was relatively small, making it difficult to identify consistent patterns of behavior within the data.

Furthermore, caution should be applied when interpreting the comparators used, as the patient groups are not equivalent. The use of comparison against a standard is not a prerequisite in a service evaluation of this nature, but comparators were included to help frame the context for the reader. As such, they should only be used in this way rather than to make any firm conclusions.

Finally, as the evaluation was limited to the use of routinely collected data, there are several variables, other than those reported, which may have influenced the results. For example, it was not possible to ascertain certain demographic details, such as body mass index and socioeconomic factors, nor lifestyle factors such as smoking status, alcohol consumption, and exercise levels, all of which could impact negatively on the outcomes of interest. Furthermore, data were not collected on follow-up services after discharge to evaluate whether these had any impact on readmission rates.

Conclusion

This service evaluation highlights the potential impact of managing neurological patients on clinically inappropriate wards with limited input from a neurologist. There is the potential to ameliorate these impacts by cohorting these patients in neurologist-led beds with a specialist MDT. While there are limitations in the design of our study, including the lack of a controlled comparison, the small sample size, and the fact that this is an evaluation of a single service, the negative impacts to patients are concerning and warrant further investigation.

Corresponding author: Richard J. Holmes, MSc, Physiotherapy Department, St. Richard’s Hospital, Chichester, West Sussex, PO19 6SE; richard.holmes8@nhs.net.

Financial disclosures: None.

From Mount Sinai Medical Center, Miami Beach, FL.

Abstract

Objective: To develop and implement a process for administering COVID-19 monoclonal antibody infusions for outpatients with mild or moderate COVID-19 at high risk for hospitalization, using multidisciplinary collaboration, US Food and Drug Administration (FDA) guidance, and infection prevention standards.

Methods: When monoclonal antibody therapy became available for mild or moderate COVID-19 outpatients via Emergency Use Authorization (EUA), our institution sought to provide this therapy option to our patients. We describe the process for planning, implementing, and maintaining a successful program for administering novel therapies based on FDA guidance and infection prevention standards. Key components of our implementation process were multidisciplinary planning involving decision makers and stakeholders; setting realistic goals in the process; team communication; and measuring and reporting quality improvement on a regular basis.

Results: A total of 790 COVID-19 monoclonal antibody infusions were administered from November 20, 2020 to March 5, 2021. Steps to minimize the likelihood of adverse drug reactions were implemented and a low incidence (< 1%) has occurred. There has been no concern from staff regarding infection during the process. Rarely, patients have raised cost-related concerns, typically due to incomplete communication regarding billing prior to the infusion. Patients, families, nursing staff, physicians, pharmacy, and hospital administration have expressed satisfaction with the program.

Conclusion: This process can provide a template for other hospitals or health care delivery facilities to provide novel therapies to patients with mild or moderate COVID-19 in a safe and effective manner.

Keywords: COVID-19; monoclonal antibody; infusion; emergency use authorization.

SARS-CoV-2 and the disease it causes, COVID-19, have transformed from scientific vernacular to common household terms. It began with a cluster of pneumonia cases of unknown etiology in December 2019 in Wuhan, China, with physicians there reporting a novel coronavirus strain (2019-nCoV), now referred to as SARS-CoV-2. Rapid spread of this virus resulted in the World Health Organization (WHO) declaring an international public health emergency. Since this time, the virus has evolved into a worldwide pandemic. COVID-19 has dramatically impacted our society, resulting in more than 2.63 million global deaths as of this writing, of which more than 527,000 deaths have occurred in the United States.¹ This novel virus has resulted in a flurry of literature, research, therapies, and collaboration across multiple disciplines in an effort to prevent, treat, and mitigate cases and complications of this disease.

On November 9, 2020, and November 21, 2020, the US Food and Drug Administration (FDA) issued Emergency Use Authorizations (EUA) for 2 novel COVID-19 monoclonal therapies, bamlanivimab^2-3 and casirivimab/imdevimab,^3-4 respectively. The EUAs granted permission for these therapies to be administered for the treatment of mild to moderate COVID-19 in adult and pediatric patients (≥ 12 years and weighing at least 40 kg) with positive results of direct SARS-CoV-2 viral testing and who are at high risk for progressing to severe COVID-19 and/or hospitalization. The therapies work by targeting the SARS-CoV-2 spike protein and subsequent attachment to human angiotensin-converting enzyme 2 receptors. Clinical trial data leading to the EUA demonstrated a reduction in viral load, safe outcome, and most importantly, fewer hospitalization and emergency room visits, as compared to the placebo group.^5-7 The use of monoclonal antibodies is not new and gained recognition during the Ebola crisis, when the monoclonal antibody to the Ebola virus showed a significant survival benefit.⁸ Providing monoclonal antibody therapy soon after symptom onset aligns with a shift from the onset of the pandemic to the current focus on the administration of pharmaceutical therapy early in the disease course. This shift prevents progression to severe COVID-19, with the goal of reducing patient mortality, hospitalizations, and strain on health care systems.

The availability of novel neutralizing monoclonal antibodies for COVID-19 led to discussions of how to incorporate these therapies as new options for patients. Our institution networked with colleagues from multiple disciplines to discuss processes and policies for the safe administration of the monoclonal antibody infusion therapies. Federal health leaders urge more use of monoclonal antibodies, but many hospitals have been unable to successfully implement infusions due to staff and logistical challenges.⁹ This article presents a viable process that hospitals can use to provide these novel therapies to outpatients with mild to moderate COVID-19.

The Mount Sinai Medical Center, Florida Experience

Mount Sinai Medical Center in Miami Beach, Florida, is the largest private, independent, not-for-profit teaching hospital in South Florida, comprising 672 licensed beds and supporting 150,000 emergency department (ED) visits annually. Per the EUA criteria for use, COVID-19 monoclonal antibody therapies are not authorized for patients who are hospitalized or who require oxygen therapy due to COVID-19. Therefore, options for outpatient administration needed to be evaluated. Directly following the first EUA press release, a task force of key stakeholders was assembled to brainstorm and develop a process to offer this therapy to the community. A multidisciplinary task force with representation from the ED, nursing, primary care, hospital medicine, pharmacy, risk management, billing, information technology, infection prevention, and senior level leadership participated (Table).

The task force reviewed institutional outpatient locations to determine whether offering this service would be feasible (eg, ED, ambulatory care facilities, cancer center). The ED was selected because it would offer the largest array of appointment times to meet the community needs with around-the-clock availability. While Mount Sinai Medical Center offers care in 3 emergency center locations in Aventura, Hialeah, and Miami Beach, it was determined to initiate the infusions at the main campus center in Miami Beach only. The main campus affords an onsite pharmacy with suitable staffing to prepare the anticipated volume of infusions in a timely manner, as both therapies have short stabilities following preparation. Thus, it was decided that patients from freestanding emergency centers in Aventura and Hialeah would be moved to the Miami Beach ED location to receive therapy. Operating at a single site also allowed for more rapid implementation, monitoring, and ability to make modifications more easily. Discussions for the possible expansion of COVID-19 monoclonal antibody infusions at satellite locations are underway.

On November 20, 2020, 11 days after the formation of the multidisciplinary task force, the first COVID-19 monoclonal infusion was successfully administered. Figure 1 depicts the timeline from assessment to program implementation. Critical to implementation was the involvement of decision makers from all necessary departments early in the planning process to ensure that standard operating procedures were followed and that the patients, community, and organization had a positive experience. This allowed for simultaneous planning of electronic health record (Epic; EHR) builds, departmental workflows, and staff education, as described in the following section. Figure 2 shows the patient safety activities included in the implementation process.

On the day of bamlanivimab EUA release, email communication was shared among hospital leadership with details of the press release. Departments were quickly involved to initiate a task force to assess if and how this therapy could be offered at Mount Sinai Medical Center. The following sections explain the role of each stakeholder and their essential role to operationalize these novel EUA treatment options. The task force was organized and led by our chief medical officer and chief nursing officer.

Information Technology

Medication Ordering and Documentation EHR and Smart Pumps. Early in the pandemic, the antimicrobial stewardship (ASP) clinical coordinator became the designated point person for pharmacy assessment of novel COVID-19 therapies. As such, this pharmacist began reviewing the bamlanivimab and, later, the casirivimab/imdevimab EUA Fact Sheet for Health Care Providers. All necessary elements for the complete and safe ordering and dispensing of the medication were developed and reviewed by pharmacy administration and ED nursing leadership for input, prior to submitting to the information technology team for implementation. Building the COVID-19 monoclonal medication records into the EHR allowed for detailed direction (ie, administration and preparation instructions) to be consistently applied. The medication records were also built into hospital smart pumps so that nurses could access prepopulated, accurate volumes and infusion rates to minimize errors.

Order Set Development. The pharmacy medication build was added to a comprehensive order set (Figure 3), which was then developed to guide prescribers and standardize the process around ordering of COVID-19 monoclonal therapies. While these therapies are new, oncology monoclonal therapies are regularly administered to outpatients at Mount Sinai Cancer Center. The cancer center was therefore consulted on their process surrounding best practices in administration of monoclonal antibody therapies. This included protocols for medications used in pretreatment and management of hypersensitivity reactions and potential adverse drug reactions of both COVID-19 monoclonal therapies. These medication orders were selected by default in the order set to ensure that all patients received premedications aimed at minimizing the risk of hypersensitivity reaction, and had as-needed medication orders, in the event a hypersensitivity reaction occurred. Reducing hypersensitivity reaction risk is important as well to increase the likelihood that the patient would receive full therapy, as management of this adverse drug reactions involves possible cessation of therapy depending on the level of severity. The pharmacy department also ensured these medications were stocked in ED automated dispensing cabinets to promote quick access. In addition to the aforementioned nursing orders, we added EUA criteria for use and hyperlinks to the Fact Sheets for Patients and Caregivers and Health Care Providers for each monoclonal therapy, and restricted ordering to ED physicians, nurse practitioners, and physician assistants.

The order set underwent multidisciplinary review by pharmacy administration, the chair of emergency medicine, physicians, and ED nursing leadership prior to presentation and approval by the Pharmacy and Therapeutics Committee. Lastly, at time of implementation, the order set was added to the ED preference list, preventing inpatient access. Additionally, as a patient safety action, free- standing orders of COVID-19 monoclonal therapies were disabled, so providers could only order therapies via the approved, comprehensive order set.

Preliminary Assessment Tool. A provider assessment tool was developed to document patient-specific EUA criteria for use during initial assessment (Figure 4). This tool serves as a checklist and is visible to the full multidisciplinary team in the patient’s EHR. It is used as a resource at the time of pharmacist verification and ED physician assessment to ensure criteria for use are met.

Patient Referral. Patients with symptoms or concerns of COVID-19 exposure can make physician appointments via telemedicine or in person at Mount Sinai Medical Center’s primary care and specialty offices. At the time of patient encounter, physicians suspecting a COVID-19 diagnosis will refer patients for outpatient COVID-19 polymerase chain reaction (PCR) laboratory testing, which has an approximate 24-hour turnaround to results. Physicians also assess whether the patient meets EUA criteria for use, pending results of testing. In the event a patient meets EUA criteria for use, the physician provides patient counseling and requests verbal consent. Following this, the physician enters a note in the EHR describing the patient’s condition, criteria for use evaluation, and the patient’s verbal agreement to therapy. This preliminary screening is beneficial to begin planning with both the patient and ED to minimize delays. Patients are notified of the results of their test once available. If the COVID-19 PCR test returns positive, the physician will call the ED at the main campus and schedule the patient for COVID-19 monoclonal therapy. As the desired timeframe for administering COVID-19 monoclonal therapies is within less than 10 days of symptom onset, timely scheduling of appointments is crucial. Infusion appointments are typically provided the same or next day. The patients are informed that they must bring documentation of their positive COVID-19 PCR test to their ED visit. Lastly, because patients are pretreated with medication that may potentially impair driving, they are instructed that they cannot drive themselves home; ride shares also are not allowed in order to limit the spread of infection.

Emergency Department

Patient Arrival and Screening. A COVID-19 patient can be evaluated in the ED 1 of 2 ways. The first option is via outpatient office referral, as described previously. Upon arrival to the ED, a second screening is performed to ensure the patient still meets EUA criteria for use and the positive COVID-19 PCR test result is confirmed. If the patient no longer meets criteria, the patient is triaged accordingly, including evaluation for higher-level care (eg, supplemental oxygen, hospital admission). The second optoion is via new patient walk-ins without outpatient physician referral (Figure 4). In these cases, an initial screening is performed, documenting EUA criteria for use in the preliminary assessment (Figure 5). Physicians will consider an outside COVID-19 test as valid, so long as documentation is readily available confirming a positive PCR result. Otherwise, an in-house COVID-19 PCR test will be performed, which has a 2-hour turnaround time.

Infusion Schedule. The ED offers a total of 16 COVID-19 monoclonal infusions slots daily. These are broken up into 4 infusion time blocks (eg, 8 am, 12 pm, 4 pm, 8 pm), with each infusion time block consisting of 4 available patient appointments. A list of scheduled infusions for the day is emailed to the pharmacy department every morning, and patients are instructed to arrive 1 hour prior to their appointment time. This allows time for patient registration, assessment, and pharmacy notification in advance of order entry. For logistical purposes, and as a patient safety initiative to reduce the likelihood of medication errors, each of the available COVID-19 monoclonal antibodies is offered on a designated day. Bamlanivimab is offered on Tuesday, Thursday, Saturday, and Sunday, while casirivimab/imdevimab is offered Monday, Wednesday, and Friday. This provides flexibility to adjust should supply deviate based on Department of Health allocation or should new therapy options within this class of medication become available.

Patient Education. Prior to administration of the monoclonal therapy, physician and nursing staff obtain a formal, written patient consent for therapy and provide patients with the option of participating in the institutional review board (IRB) approved study. Details of this are discussed in the risk management and IRB sections of the article. Nursing staff also provides the medication-specific Fact Sheet for Patients and Caregivers in either Spanish or English, which is also included as a hyperlink on the COVID-19 Monoclonal Antibody Order Set for ease of access. Interpreter services are available for patients who speak other languages. An ED decentralized pharmacist is also available onsite Monday through Friday from 12 pm to 8:30 pm to supplement education and serve as a resource for any questions.

Infusion Ordering. Once the patient is ready to begin therapy, the he/she is brought to a dedicated overflow area of the ED. There are few, if any, patients in this location, and it is adjacent to the main emergency center for easy access by the patients, nurses, pharmacists, and physicians. The physician then enters orders in the EHR using the COVID-19 Monoclonal Antibody Order Set (Figure 3). Three discrete questions were built into the medication order: (1) Was patient consent obtained? (2) Was the Fact Sheet for Patient/Caregiver provided to the patient? (3) Is the patient COVID-19 PCR-positive? These questions were built as hard stops so that the medication orders cannot be placed without a response. This serves as another double-check to ensure processes are followed and helps facilitate timely verification by the pharmacist.

Medication Administration. One nurse is dedicated to administering the monoclonal therapies scheduled at 8 am and 12 pm and another at 4 pm and 8 pm. Each appointment block is 4 hours in duration to allow adequate time for patient registration, infusion, and postinfusion observation. The nurse administers the premedications and COVID-19 monoclonal therapy, and observes the patient for the required 1-hour postadministration observation period. Nursing orders detailing monitoring parameters for mild, moderate, and severe reactions, along with associated medication orders to administer in the event they occur, are detailed in the nursing orders of the COVID-19 Monoclonal Antibody Order Set (Figure 3). Prior to administration, the nurse scans each medication and the patient’s wrist identification band, and documents the time of administration within the EHR medication administration report.

Pharmacy Department

Medication Receipt Process. Inventory is currently allocated biweekly from the state department of health and will soon be transitioning to a direct order system. The pharmacy technician in charge of deliveries notifies the pharmacy Antimicrobial Stewardship Program (ASP) clinical coordinator upon receipt of the monoclonal therapies. Bamlanivimab is supplied as 1 vial per dose, whereas casirivimab/imdevimab is supplied as 4 vials or 8 vials per dose, depending how it is shipped. To reduce the likelihood of medication errors, the ASP clinical coordinator assembles each of the casirivimab/imdevimab vials into kits, where 1 kit equals 1 dose. Labels are then affixed to each kit indicating the medication name, number of vials which equal a full dose, and pharmacist signature. The kits are stored in a dedicated refrigerator, and inventory logs are affixed to the outside of the refrigerator and updated daily. This inventory is also communicated daily to ED physician, nursing, and pharmacy leadership, as well as the director of patient safety, who reports weekly usage to the state Department of Health and Human Services. These weekly reports are used to determine allocation amounts.

Medication Verification and Delivery. The Mount Sinai Medical Center pharmacist staffing model consists of centralized order entry and specialized, decentralized positions. All orders are verified by the ED pharmacist when scheduled (not a 24/7 service) and by the designated pharmacist for all other times. At the time of medication verification, the pharmacist documents patient-specific EUA criteria for use and confirms that consent was obtained and the Fact Sheet for Patients/Caregivers was provided. A pharmacist intervention was developed to assist with this documentation. Pharmacists input smart text “.COVIDmonoclonal” and a drop-down menu of EUA criteria for use appears. The pharmacist reviews the patient care notes and medication order question responses to ascertain this information, contacting the ED prescriber if further clarification is required. This verification serves as another check to ensure processes put in place are followed. Lastly, intravenous preparation and delivery are electronically recorded in the EHR, and the medications require nursing signature at the time of delivery to ensure a formal chain of custody.

Risk Management

At Mount Sinai Medical Center, all EUA and investigational therapies require patient consent. Consistent with this requirement, a COVID-19 monoclonal specific consent was developed by risk management. This is provided to every patient receiving a COVID-19 monoclonal infusion, in addition to the FDA EUA Fact Sheet for Patients and Caregivers, and documented as part of their EHR. The questions providers must answer are built into the order set to ensure this process is followed and these patient safety checks are incorporated into the workflow.

Billing and Finance Department

In alignment with Mount Sinai Medical Center’s mission to provide high-quality health care to its diverse community through teaching, research, charity care, and financial responsibility, it was determined that this therapy would be provided to all patients regardless of insurance type, including those who are uninsured. The billing and finance department was consulted prior to this service being offered, to provide patients with accurate and pertinent information. The billing and finance department provided guidance on how to document patient encounters at time of registration to facilitate appropriate billing. At this time, the medication is free of charge, but nonmedication-related ED fees apply. This is explained to patients so there is a clear understanding prior to booking their appointment.

Infection Prevention

As patients receiving COVID-19 monoclonal therapies can transmit the virus to others, measures to ensure protection for other patients and staff are vital. To minimize exposure, specific nursing and physician staff from the ED are assigned to the treatment of these patients, and patients receive infusions and postobservation monitoring in a designated wing of the ED. Additionally, all staff who interact with these patients are required to don full personal protective equipment. This includes not only physicians and nurses but all specialties such as physician assistants, nurse practitioners, pharmacists, and laboratory technicians. Moreover, patients are not permitted to go home in a ride share and are counseled on Centers for Disease Control and Prevention quarantining following infusion.

Measurement of Process and Outcomes and Reporting

IRB approval was sought and obtained early during initiation of this service, allowing study consent to be offered to patients at the time general consent was obtained, which maximized patient recruitment and streamlined workflow. The study is a prospective observational research study to determine the impact of administration of COVID-19 monoclonal antibody therapy on length of symptoms, chronic illness, and rate of hospitalization. Most patients were eager to participate and offer their assistance to the scientific community during this pandemic.

Staff Education

In order to successfully implement this multidisciplinary EUA treatment option, comprehensive staff education was paramount after the workflow was developed. Prior to the first day of infusions, nurses and pharmacists were provided education during multiple huddle announcements. The pharmacy team also provided screen captures via email to the pharmacists so they could become familiar with the order set, intervention documentation, and location of the preliminary assessment of EUA criteria for use at the time of order verification. The emergency medicine department chair and chief medical officer also provided education via several virtual meetings and email to referring physicians (specialists and primary care) and residents in the emergency centers involved in COVID-19 monoclonal therapy-related patient care.

Factors Contributing to Success

We believe the reasons for continued success of this process are multifactorial and include the following key elements. Multidisciplinary planning, which included decision makers and all stakeholders, began at the time the idea was conceived. This allowed quick implementation of this service by efficiently navigating barriers to engaging impacted staff early on. Throughout this process, the authors set realistic step-wise goals. While navigating through the many details to implementation described, we also kept in mind the big picture, which was to provide this potentially lifesaving therapy to as many qualifying members of our community as possible. This included being flexible with the process and adapting when needed to achieve this ultimate goal. A focus on safety remained a priority to minimize possible errors and enhance patient and staff satisfaction. The optimization of the EHR streamlined workflow, provided point-of-care resources, and enhanced patient safety. Additionally, the target date set for implementation allowed staff and department leads adequate time to plan for and anticipate the changes. Serving only 1 patient on the first day allowed time for staff to experience this new process hands-on and provided opportunity for focused education. This team communication was essential to implementing this project, including staff training of processes and procedures prior to go-live. Early incorporation of IRB approval allowed the experience to be assessed and considered for contribution to the scientific literature to tackle this novel virus that has impacted our communities locally, nationally, and abroad. Moreover, continued measurement and reporting on a regular basis leads to performance improvement. The process outlined here can be adapted to incorporate other new therapies in the future, such as the recent February 9, 2021, EUA of the COVID-19 monoclonal antibody combination bamlanivimab and etesevimab.¹⁰

Conclusion

We administered 790 COVID-19 monoclonal antibody infusions between November 20, 2020 and March 5, 2021. Steps to minimize the likelihood of hypersensitivity reactions were implemented, and a low incidence (< 1%) has been observed. There has been no incidence of infection, concern from staff about infection prevention, or risk of infection during the processes. There have been very infrequent cost-related concerns raised by patients, typically due to incomplete communication regarding billing prior to the infusion. To address these issues, staff education has been provided to enhance patient instruction on this topic. The program has provided patient and family satisfaction, as well nursing, physician, pharmacist, clinical staff, and hospital administration pride and gratification. Setting up a new program to provide a 4-hour patient encounter to infuse therapy to high-risk patients with COVID-19 requires commitment and effort. This article describes the experience, ideas, and formula others may consider using to set up such a program. Through networking and formal phone calls and meetings about monoclonal antibody therapy, we have heard about other institutions who have not been able to institute this program due to various barriers to implementation. We hope our experience serves as a resource for others to provide this therapy to their patients and expand access in an effort to mitigate COVID-19 consequences and cases affecting our communities.

Corresponding author: Kathleen Jodoin, PharmD, BCPS, Mount Sinai Medical Center, 4300 Alton Rd, Miami Beach, FL 33140; kathleen.jodoin@msmc.com.

Financial disclosures: None.

From Mount Sinai Medical Center, Miami Beach, FL.

Abstract

Objective: To develop and implement a process for administering COVID-19 monoclonal antibody infusions for outpatients with mild or moderate COVID-19 at high risk for hospitalization, using multidisciplinary collaboration, US Food and Drug Administration (FDA) guidance, and infection prevention standards.

Methods: When monoclonal antibody therapy became available for mild or moderate COVID-19 outpatients via Emergency Use Authorization (EUA), our institution sought to provide this therapy option to our patients. We describe the process for planning, implementing, and maintaining a successful program for administering novel therapies based on FDA guidance and infection prevention standards. Key components of our implementation process were multidisciplinary planning involving decision makers and stakeholders; setting realistic goals in the process; team communication; and measuring and reporting quality improvement on a regular basis.

Results: A total of 790 COVID-19 monoclonal antibody infusions were administered from November 20, 2020 to March 5, 2021. Steps to minimize the likelihood of adverse drug reactions were implemented and a low incidence (< 1%) has occurred. There has been no concern from staff regarding infection during the process. Rarely, patients have raised cost-related concerns, typically due to incomplete communication regarding billing prior to the infusion. Patients, families, nursing staff, physicians, pharmacy, and hospital administration have expressed satisfaction with the program.

Conclusion: This process can provide a template for other hospitals or health care delivery facilities to provide novel therapies to patients with mild or moderate COVID-19 in a safe and effective manner.

Keywords: COVID-19; monoclonal antibody; infusion; emergency use authorization.

SARS-CoV-2 and the disease it causes, COVID-19, have transformed from scientific vernacular to common household terms. It began with a cluster of pneumonia cases of unknown etiology in December 2019 in Wuhan, China, with physicians there reporting a novel coronavirus strain (2019-nCoV), now referred to as SARS-CoV-2. Rapid spread of this virus resulted in the World Health Organization (WHO) declaring an international public health emergency. Since this time, the virus has evolved into a worldwide pandemic. COVID-19 has dramatically impacted our society, resulting in more than 2.63 million global deaths as of this writing, of which more than 527,000 deaths have occurred in the United States.¹ This novel virus has resulted in a flurry of literature, research, therapies, and collaboration across multiple disciplines in an effort to prevent, treat, and mitigate cases and complications of this disease.

On November 9, 2020, and November 21, 2020, the US Food and Drug Administration (FDA) issued Emergency Use Authorizations (EUA) for 2 novel COVID-19 monoclonal therapies, bamlanivimab^2-3 and casirivimab/imdevimab,^3-4 respectively. The EUAs granted permission for these therapies to be administered for the treatment of mild to moderate COVID-19 in adult and pediatric patients (≥ 12 years and weighing at least 40 kg) with positive results of direct SARS-CoV-2 viral testing and who are at high risk for progressing to severe COVID-19 and/or hospitalization. The therapies work by targeting the SARS-CoV-2 spike protein and subsequent attachment to human angiotensin-converting enzyme 2 receptors. Clinical trial data leading to the EUA demonstrated a reduction in viral load, safe outcome, and most importantly, fewer hospitalization and emergency room visits, as compared to the placebo group.^5-7 The use of monoclonal antibodies is not new and gained recognition during the Ebola crisis, when the monoclonal antibody to the Ebola virus showed a significant survival benefit.⁸ Providing monoclonal antibody therapy soon after symptom onset aligns with a shift from the onset of the pandemic to the current focus on the administration of pharmaceutical therapy early in the disease course. This shift prevents progression to severe COVID-19, with the goal of reducing patient mortality, hospitalizations, and strain on health care systems.

The availability of novel neutralizing monoclonal antibodies for COVID-19 led to discussions of how to incorporate these therapies as new options for patients. Our institution networked with colleagues from multiple disciplines to discuss processes and policies for the safe administration of the monoclonal antibody infusion therapies. Federal health leaders urge more use of monoclonal antibodies, but many hospitals have been unable to successfully implement infusions due to staff and logistical challenges.⁹ This article presents a viable process that hospitals can use to provide these novel therapies to outpatients with mild to moderate COVID-19.

The Mount Sinai Medical Center, Florida Experience

Mount Sinai Medical Center in Miami Beach, Florida, is the largest private, independent, not-for-profit teaching hospital in South Florida, comprising 672 licensed beds and supporting 150,000 emergency department (ED) visits annually. Per the EUA criteria for use, COVID-19 monoclonal antibody therapies are not authorized for patients who are hospitalized or who require oxygen therapy due to COVID-19. Therefore, options for outpatient administration needed to be evaluated. Directly following the first EUA press release, a task force of key stakeholders was assembled to brainstorm and develop a process to offer this therapy to the community. A multidisciplinary task force with representation from the ED, nursing, primary care, hospital medicine, pharmacy, risk management, billing, information technology, infection prevention, and senior level leadership participated (Table).

The task force reviewed institutional outpatient locations to determine whether offering this service would be feasible (eg, ED, ambulatory care facilities, cancer center). The ED was selected because it would offer the largest array of appointment times to meet the community needs with around-the-clock availability. While Mount Sinai Medical Center offers care in 3 emergency center locations in Aventura, Hialeah, and Miami Beach, it was determined to initiate the infusions at the main campus center in Miami Beach only. The main campus affords an onsite pharmacy with suitable staffing to prepare the anticipated volume of infusions in a timely manner, as both therapies have short stabilities following preparation. Thus, it was decided that patients from freestanding emergency centers in Aventura and Hialeah would be moved to the Miami Beach ED location to receive therapy. Operating at a single site also allowed for more rapid implementation, monitoring, and ability to make modifications more easily. Discussions for the possible expansion of COVID-19 monoclonal antibody infusions at satellite locations are underway.

On November 20, 2020, 11 days after the formation of the multidisciplinary task force, the first COVID-19 monoclonal infusion was successfully administered. Figure 1 depicts the timeline from assessment to program implementation. Critical to implementation was the involvement of decision makers from all necessary departments early in the planning process to ensure that standard operating procedures were followed and that the patients, community, and organization had a positive experience. This allowed for simultaneous planning of electronic health record (Epic; EHR) builds, departmental workflows, and staff education, as described in the following section. Figure 2 shows the patient safety activities included in the implementation process.

On the day of bamlanivimab EUA release, email communication was shared among hospital leadership with details of the press release. Departments were quickly involved to initiate a task force to assess if and how this therapy could be offered at Mount Sinai Medical Center. The following sections explain the role of each stakeholder and their essential role to operationalize these novel EUA treatment options. The task force was organized and led by our chief medical officer and chief nursing officer.

Information Technology

Medication Ordering and Documentation EHR and Smart Pumps. Early in the pandemic, the antimicrobial stewardship (ASP) clinical coordinator became the designated point person for pharmacy assessment of novel COVID-19 therapies. As such, this pharmacist began reviewing the bamlanivimab and, later, the casirivimab/imdevimab EUA Fact Sheet for Health Care Providers. All necessary elements for the complete and safe ordering and dispensing of the medication were developed and reviewed by pharmacy administration and ED nursing leadership for input, prior to submitting to the information technology team for implementation. Building the COVID-19 monoclonal medication records into the EHR allowed for detailed direction (ie, administration and preparation instructions) to be consistently applied. The medication records were also built into hospital smart pumps so that nurses could access prepopulated, accurate volumes and infusion rates to minimize errors.

Order Set Development. The pharmacy medication build was added to a comprehensive order set (Figure 3), which was then developed to guide prescribers and standardize the process around ordering of COVID-19 monoclonal therapies. While these therapies are new, oncology monoclonal therapies are regularly administered to outpatients at Mount Sinai Cancer Center. The cancer center was therefore consulted on their process surrounding best practices in administration of monoclonal antibody therapies. This included protocols for medications used in pretreatment and management of hypersensitivity reactions and potential adverse drug reactions of both COVID-19 monoclonal therapies. These medication orders were selected by default in the order set to ensure that all patients received premedications aimed at minimizing the risk of hypersensitivity reaction, and had as-needed medication orders, in the event a hypersensitivity reaction occurred. Reducing hypersensitivity reaction risk is important as well to increase the likelihood that the patient would receive full therapy, as management of this adverse drug reactions involves possible cessation of therapy depending on the level of severity. The pharmacy department also ensured these medications were stocked in ED automated dispensing cabinets to promote quick access. In addition to the aforementioned nursing orders, we added EUA criteria for use and hyperlinks to the Fact Sheets for Patients and Caregivers and Health Care Providers for each monoclonal therapy, and restricted ordering to ED physicians, nurse practitioners, and physician assistants.

The order set underwent multidisciplinary review by pharmacy administration, the chair of emergency medicine, physicians, and ED nursing leadership prior to presentation and approval by the Pharmacy and Therapeutics Committee. Lastly, at time of implementation, the order set was added to the ED preference list, preventing inpatient access. Additionally, as a patient safety action, free- standing orders of COVID-19 monoclonal therapies were disabled, so providers could only order therapies via the approved, comprehensive order set.

Preliminary Assessment Tool. A provider assessment tool was developed to document patient-specific EUA criteria for use during initial assessment (Figure 4). This tool serves as a checklist and is visible to the full multidisciplinary team in the patient’s EHR. It is used as a resource at the time of pharmacist verification and ED physician assessment to ensure criteria for use are met.

Patient Referral. Patients with symptoms or concerns of COVID-19 exposure can make physician appointments via telemedicine or in person at Mount Sinai Medical Center’s primary care and specialty offices. At the time of patient encounter, physicians suspecting a COVID-19 diagnosis will refer patients for outpatient COVID-19 polymerase chain reaction (PCR) laboratory testing, which has an approximate 24-hour turnaround to results. Physicians also assess whether the patient meets EUA criteria for use, pending results of testing. In the event a patient meets EUA criteria for use, the physician provides patient counseling and requests verbal consent. Following this, the physician enters a note in the EHR describing the patient’s condition, criteria for use evaluation, and the patient’s verbal agreement to therapy. This preliminary screening is beneficial to begin planning with both the patient and ED to minimize delays. Patients are notified of the results of their test once available. If the COVID-19 PCR test returns positive, the physician will call the ED at the main campus and schedule the patient for COVID-19 monoclonal therapy. As the desired timeframe for administering COVID-19 monoclonal therapies is within less than 10 days of symptom onset, timely scheduling of appointments is crucial. Infusion appointments are typically provided the same or next day. The patients are informed that they must bring documentation of their positive COVID-19 PCR test to their ED visit. Lastly, because patients are pretreated with medication that may potentially impair driving, they are instructed that they cannot drive themselves home; ride shares also are not allowed in order to limit the spread of infection.

Emergency Department

Patient Arrival and Screening. A COVID-19 patient can be evaluated in the ED 1 of 2 ways. The first option is via outpatient office referral, as described previously. Upon arrival to the ED, a second screening is performed to ensure the patient still meets EUA criteria for use and the positive COVID-19 PCR test result is confirmed. If the patient no longer meets criteria, the patient is triaged accordingly, including evaluation for higher-level care (eg, supplemental oxygen, hospital admission). The second optoion is via new patient walk-ins without outpatient physician referral (Figure 4). In these cases, an initial screening is performed, documenting EUA criteria for use in the preliminary assessment (Figure 5). Physicians will consider an outside COVID-19 test as valid, so long as documentation is readily available confirming a positive PCR result. Otherwise, an in-house COVID-19 PCR test will be performed, which has a 2-hour turnaround time.

Infusion Schedule. The ED offers a total of 16 COVID-19 monoclonal infusions slots daily. These are broken up into 4 infusion time blocks (eg, 8 am, 12 pm, 4 pm, 8 pm), with each infusion time block consisting of 4 available patient appointments. A list of scheduled infusions for the day is emailed to the pharmacy department every morning, and patients are instructed to arrive 1 hour prior to their appointment time. This allows time for patient registration, assessment, and pharmacy notification in advance of order entry. For logistical purposes, and as a patient safety initiative to reduce the likelihood of medication errors, each of the available COVID-19 monoclonal antibodies is offered on a designated day. Bamlanivimab is offered on Tuesday, Thursday, Saturday, and Sunday, while casirivimab/imdevimab is offered Monday, Wednesday, and Friday. This provides flexibility to adjust should supply deviate based on Department of Health allocation or should new therapy options within this class of medication become available.

Patient Education. Prior to administration of the monoclonal therapy, physician and nursing staff obtain a formal, written patient consent for therapy and provide patients with the option of participating in the institutional review board (IRB) approved study. Details of this are discussed in the risk management and IRB sections of the article. Nursing staff also provides the medication-specific Fact Sheet for Patients and Caregivers in either Spanish or English, which is also included as a hyperlink on the COVID-19 Monoclonal Antibody Order Set for ease of access. Interpreter services are available for patients who speak other languages. An ED decentralized pharmacist is also available onsite Monday through Friday from 12 pm to 8:30 pm to supplement education and serve as a resource for any questions.

Infusion Ordering. Once the patient is ready to begin therapy, the he/she is brought to a dedicated overflow area of the ED. There are few, if any, patients in this location, and it is adjacent to the main emergency center for easy access by the patients, nurses, pharmacists, and physicians. The physician then enters orders in the EHR using the COVID-19 Monoclonal Antibody Order Set (Figure 3). Three discrete questions were built into the medication order: (1) Was patient consent obtained? (2) Was the Fact Sheet for Patient/Caregiver provided to the patient? (3) Is the patient COVID-19 PCR-positive? These questions were built as hard stops so that the medication orders cannot be placed without a response. This serves as another double-check to ensure processes are followed and helps facilitate timely verification by the pharmacist.

Medication Administration. One nurse is dedicated to administering the monoclonal therapies scheduled at 8 am and 12 pm and another at 4 pm and 8 pm. Each appointment block is 4 hours in duration to allow adequate time for patient registration, infusion, and postinfusion observation. The nurse administers the premedications and COVID-19 monoclonal therapy, and observes the patient for the required 1-hour postadministration observation period. Nursing orders detailing monitoring parameters for mild, moderate, and severe reactions, along with associated medication orders to administer in the event they occur, are detailed in the nursing orders of the COVID-19 Monoclonal Antibody Order Set (Figure 3). Prior to administration, the nurse scans each medication and the patient’s wrist identification band, and documents the time of administration within the EHR medication administration report.

Pharmacy Department

Medication Receipt Process. Inventory is currently allocated biweekly from the state department of health and will soon be transitioning to a direct order system. The pharmacy technician in charge of deliveries notifies the pharmacy Antimicrobial Stewardship Program (ASP) clinical coordinator upon receipt of the monoclonal therapies. Bamlanivimab is supplied as 1 vial per dose, whereas casirivimab/imdevimab is supplied as 4 vials or 8 vials per dose, depending how it is shipped. To reduce the likelihood of medication errors, the ASP clinical coordinator assembles each of the casirivimab/imdevimab vials into kits, where 1 kit equals 1 dose. Labels are then affixed to each kit indicating the medication name, number of vials which equal a full dose, and pharmacist signature. The kits are stored in a dedicated refrigerator, and inventory logs are affixed to the outside of the refrigerator and updated daily. This inventory is also communicated daily to ED physician, nursing, and pharmacy leadership, as well as the director of patient safety, who reports weekly usage to the state Department of Health and Human Services. These weekly reports are used to determine allocation amounts.

Medication Verification and Delivery. The Mount Sinai Medical Center pharmacist staffing model consists of centralized order entry and specialized, decentralized positions. All orders are verified by the ED pharmacist when scheduled (not a 24/7 service) and by the designated pharmacist for all other times. At the time of medication verification, the pharmacist documents patient-specific EUA criteria for use and confirms that consent was obtained and the Fact Sheet for Patients/Caregivers was provided. A pharmacist intervention was developed to assist with this documentation. Pharmacists input smart text “.COVIDmonoclonal” and a drop-down menu of EUA criteria for use appears. The pharmacist reviews the patient care notes and medication order question responses to ascertain this information, contacting the ED prescriber if further clarification is required. This verification serves as another check to ensure processes put in place are followed. Lastly, intravenous preparation and delivery are electronically recorded in the EHR, and the medications require nursing signature at the time of delivery to ensure a formal chain of custody.

Risk Management

At Mount Sinai Medical Center, all EUA and investigational therapies require patient consent. Consistent with this requirement, a COVID-19 monoclonal specific consent was developed by risk management. This is provided to every patient receiving a COVID-19 monoclonal infusion, in addition to the FDA EUA Fact Sheet for Patients and Caregivers, and documented as part of their EHR. The questions providers must answer are built into the order set to ensure this process is followed and these patient safety checks are incorporated into the workflow.

Billing and Finance Department

In alignment with Mount Sinai Medical Center’s mission to provide high-quality health care to its diverse community through teaching, research, charity care, and financial responsibility, it was determined that this therapy would be provided to all patients regardless of insurance type, including those who are uninsured. The billing and finance department was consulted prior to this service being offered, to provide patients with accurate and pertinent information. The billing and finance department provided guidance on how to document patient encounters at time of registration to facilitate appropriate billing. At this time, the medication is free of charge, but nonmedication-related ED fees apply. This is explained to patients so there is a clear understanding prior to booking their appointment.

Infection Prevention

As patients receiving COVID-19 monoclonal therapies can transmit the virus to others, measures to ensure protection for other patients and staff are vital. To minimize exposure, specific nursing and physician staff from the ED are assigned to the treatment of these patients, and patients receive infusions and postobservation monitoring in a designated wing of the ED. Additionally, all staff who interact with these patients are required to don full personal protective equipment. This includes not only physicians and nurses but all specialties such as physician assistants, nurse practitioners, pharmacists, and laboratory technicians. Moreover, patients are not permitted to go home in a ride share and are counseled on Centers for Disease Control and Prevention quarantining following infusion.

Measurement of Process and Outcomes and Reporting

IRB approval was sought and obtained early during initiation of this service, allowing study consent to be offered to patients at the time general consent was obtained, which maximized patient recruitment and streamlined workflow. The study is a prospective observational research study to determine the impact of administration of COVID-19 monoclonal antibody therapy on length of symptoms, chronic illness, and rate of hospitalization. Most patients were eager to participate and offer their assistance to the scientific community during this pandemic.

Staff Education

In order to successfully implement this multidisciplinary EUA treatment option, comprehensive staff education was paramount after the workflow was developed. Prior to the first day of infusions, nurses and pharmacists were provided education during multiple huddle announcements. The pharmacy team also provided screen captures via email to the pharmacists so they could become familiar with the order set, intervention documentation, and location of the preliminary assessment of EUA criteria for use at the time of order verification. The emergency medicine department chair and chief medical officer also provided education via several virtual meetings and email to referring physicians (specialists and primary care) and residents in the emergency centers involved in COVID-19 monoclonal therapy-related patient care.

Factors Contributing to Success

We believe the reasons for continued success of this process are multifactorial and include the following key elements. Multidisciplinary planning, which included decision makers and all stakeholders, began at the time the idea was conceived. This allowed quick implementation of this service by efficiently navigating barriers to engaging impacted staff early on. Throughout this process, the authors set realistic step-wise goals. While navigating through the many details to implementation described, we also kept in mind the big picture, which was to provide this potentially lifesaving therapy to as many qualifying members of our community as possible. This included being flexible with the process and adapting when needed to achieve this ultimate goal. A focus on safety remained a priority to minimize possible errors and enhance patient and staff satisfaction. The optimization of the EHR streamlined workflow, provided point-of-care resources, and enhanced patient safety. Additionally, the target date set for implementation allowed staff and department leads adequate time to plan for and anticipate the changes. Serving only 1 patient on the first day allowed time for staff to experience this new process hands-on and provided opportunity for focused education. This team communication was essential to implementing this project, including staff training of processes and procedures prior to go-live. Early incorporation of IRB approval allowed the experience to be assessed and considered for contribution to the scientific literature to tackle this novel virus that has impacted our communities locally, nationally, and abroad. Moreover, continued measurement and reporting on a regular basis leads to performance improvement. The process outlined here can be adapted to incorporate other new therapies in the future, such as the recent February 9, 2021, EUA of the COVID-19 monoclonal antibody combination bamlanivimab and etesevimab.¹⁰

Conclusion

We administered 790 COVID-19 monoclonal antibody infusions between November 20, 2020 and March 5, 2021. Steps to minimize the likelihood of hypersensitivity reactions were implemented, and a low incidence (< 1%) has been observed. There has been no incidence of infection, concern from staff about infection prevention, or risk of infection during the processes. There have been very infrequent cost-related concerns raised by patients, typically due to incomplete communication regarding billing prior to the infusion. To address these issues, staff education has been provided to enhance patient instruction on this topic. The program has provided patient and family satisfaction, as well nursing, physician, pharmacist, clinical staff, and hospital administration pride and gratification. Setting up a new program to provide a 4-hour patient encounter to infuse therapy to high-risk patients with COVID-19 requires commitment and effort. This article describes the experience, ideas, and formula others may consider using to set up such a program. Through networking and formal phone calls and meetings about monoclonal antibody therapy, we have heard about other institutions who have not been able to institute this program due to various barriers to implementation. We hope our experience serves as a resource for others to provide this therapy to their patients and expand access in an effort to mitigate COVID-19 consequences and cases affecting our communities.

Corresponding author: Kathleen Jodoin, PharmD, BCPS, Mount Sinai Medical Center, 4300 Alton Rd, Miami Beach, FL 33140; kathleen.jodoin@msmc.com.

Financial disclosures: None.

From Mount Sinai Medical Center, Miami Beach, FL.

Abstract

Objective: To develop and implement a process for administering COVID-19 monoclonal antibody infusions for outpatients with mild or moderate COVID-19 at high risk for hospitalization, using multidisciplinary collaboration, US Food and Drug Administration (FDA) guidance, and infection prevention standards.

Methods: When monoclonal antibody therapy became available for mild or moderate COVID-19 outpatients via Emergency Use Authorization (EUA), our institution sought to provide this therapy option to our patients. We describe the process for planning, implementing, and maintaining a successful program for administering novel therapies based on FDA guidance and infection prevention standards. Key components of our implementation process were multidisciplinary planning involving decision makers and stakeholders; setting realistic goals in the process; team communication; and measuring and reporting quality improvement on a regular basis.

Results: A total of 790 COVID-19 monoclonal antibody infusions were administered from November 20, 2020 to March 5, 2021. Steps to minimize the likelihood of adverse drug reactions were implemented and a low incidence (< 1%) has occurred. There has been no concern from staff regarding infection during the process. Rarely, patients have raised cost-related concerns, typically due to incomplete communication regarding billing prior to the infusion. Patients, families, nursing staff, physicians, pharmacy, and hospital administration have expressed satisfaction with the program.

Conclusion: This process can provide a template for other hospitals or health care delivery facilities to provide novel therapies to patients with mild or moderate COVID-19 in a safe and effective manner.

Keywords: COVID-19; monoclonal antibody; infusion; emergency use authorization.

SARS-CoV-2 and the disease it causes, COVID-19, have transformed from scientific vernacular to common household terms. It began with a cluster of pneumonia cases of unknown etiology in December 2019 in Wuhan, China, with physicians there reporting a novel coronavirus strain (2019-nCoV), now referred to as SARS-CoV-2. Rapid spread of this virus resulted in the World Health Organization (WHO) declaring an international public health emergency. Since this time, the virus has evolved into a worldwide pandemic. COVID-19 has dramatically impacted our society, resulting in more than 2.63 million global deaths as of this writing, of which more than 527,000 deaths have occurred in the United States.¹ This novel virus has resulted in a flurry of literature, research, therapies, and collaboration across multiple disciplines in an effort to prevent, treat, and mitigate cases and complications of this disease.

On November 9, 2020, and November 21, 2020, the US Food and Drug Administration (FDA) issued Emergency Use Authorizations (EUA) for 2 novel COVID-19 monoclonal therapies, bamlanivimab^2-3 and casirivimab/imdevimab,^3-4 respectively. The EUAs granted permission for these therapies to be administered for the treatment of mild to moderate COVID-19 in adult and pediatric patients (≥ 12 years and weighing at least 40 kg) with positive results of direct SARS-CoV-2 viral testing and who are at high risk for progressing to severe COVID-19 and/or hospitalization. The therapies work by targeting the SARS-CoV-2 spike protein and subsequent attachment to human angiotensin-converting enzyme 2 receptors. Clinical trial data leading to the EUA demonstrated a reduction in viral load, safe outcome, and most importantly, fewer hospitalization and emergency room visits, as compared to the placebo group.^5-7 The use of monoclonal antibodies is not new and gained recognition during the Ebola crisis, when the monoclonal antibody to the Ebola virus showed a significant survival benefit.⁸ Providing monoclonal antibody therapy soon after symptom onset aligns with a shift from the onset of the pandemic to the current focus on the administration of pharmaceutical therapy early in the disease course. This shift prevents progression to severe COVID-19, with the goal of reducing patient mortality, hospitalizations, and strain on health care systems.

The availability of novel neutralizing monoclonal antibodies for COVID-19 led to discussions of how to incorporate these therapies as new options for patients. Our institution networked with colleagues from multiple disciplines to discuss processes and policies for the safe administration of the monoclonal antibody infusion therapies. Federal health leaders urge more use of monoclonal antibodies, but many hospitals have been unable to successfully implement infusions due to staff and logistical challenges.⁹ This article presents a viable process that hospitals can use to provide these novel therapies to outpatients with mild to moderate COVID-19.

The Mount Sinai Medical Center, Florida Experience

Mount Sinai Medical Center in Miami Beach, Florida, is the largest private, independent, not-for-profit teaching hospital in South Florida, comprising 672 licensed beds and supporting 150,000 emergency department (ED) visits annually. Per the EUA criteria for use, COVID-19 monoclonal antibody therapies are not authorized for patients who are hospitalized or who require oxygen therapy due to COVID-19. Therefore, options for outpatient administration needed to be evaluated. Directly following the first EUA press release, a task force of key stakeholders was assembled to brainstorm and develop a process to offer this therapy to the community. A multidisciplinary task force with representation from the ED, nursing, primary care, hospital medicine, pharmacy, risk management, billing, information technology, infection prevention, and senior level leadership participated (Table).

The task force reviewed institutional outpatient locations to determine whether offering this service would be feasible (eg, ED, ambulatory care facilities, cancer center). The ED was selected because it would offer the largest array of appointment times to meet the community needs with around-the-clock availability. While Mount Sinai Medical Center offers care in 3 emergency center locations in Aventura, Hialeah, and Miami Beach, it was determined to initiate the infusions at the main campus center in Miami Beach only. The main campus affords an onsite pharmacy with suitable staffing to prepare the anticipated volume of infusions in a timely manner, as both therapies have short stabilities following preparation. Thus, it was decided that patients from freestanding emergency centers in Aventura and Hialeah would be moved to the Miami Beach ED location to receive therapy. Operating at a single site also allowed for more rapid implementation, monitoring, and ability to make modifications more easily. Discussions for the possible expansion of COVID-19 monoclonal antibody infusions at satellite locations are underway.

On November 20, 2020, 11 days after the formation of the multidisciplinary task force, the first COVID-19 monoclonal infusion was successfully administered. Figure 1 depicts the timeline from assessment to program implementation. Critical to implementation was the involvement of decision makers from all necessary departments early in the planning process to ensure that standard operating procedures were followed and that the patients, community, and organization had a positive experience. This allowed for simultaneous planning of electronic health record (Epic; EHR) builds, departmental workflows, and staff education, as described in the following section. Figure 2 shows the patient safety activities included in the implementation process.

On the day of bamlanivimab EUA release, email communication was shared among hospital leadership with details of the press release. Departments were quickly involved to initiate a task force to assess if and how this therapy could be offered at Mount Sinai Medical Center. The following sections explain the role of each stakeholder and their essential role to operationalize these novel EUA treatment options. The task force was organized and led by our chief medical officer and chief nursing officer.

Information Technology

Medication Ordering and Documentation EHR and Smart Pumps. Early in the pandemic, the antimicrobial stewardship (ASP) clinical coordinator became the designated point person for pharmacy assessment of novel COVID-19 therapies. As such, this pharmacist began reviewing the bamlanivimab and, later, the casirivimab/imdevimab EUA Fact Sheet for Health Care Providers. All necessary elements for the complete and safe ordering and dispensing of the medication were developed and reviewed by pharmacy administration and ED nursing leadership for input, prior to submitting to the information technology team for implementation. Building the COVID-19 monoclonal medication records into the EHR allowed for detailed direction (ie, administration and preparation instructions) to be consistently applied. The medication records were also built into hospital smart pumps so that nurses could access prepopulated, accurate volumes and infusion rates to minimize errors.

Order Set Development. The pharmacy medication build was added to a comprehensive order set (Figure 3), which was then developed to guide prescribers and standardize the process around ordering of COVID-19 monoclonal therapies. While these therapies are new, oncology monoclonal therapies are regularly administered to outpatients at Mount Sinai Cancer Center. The cancer center was therefore consulted on their process surrounding best practices in administration of monoclonal antibody therapies. This included protocols for medications used in pretreatment and management of hypersensitivity reactions and potential adverse drug reactions of both COVID-19 monoclonal therapies. These medication orders were selected by default in the order set to ensure that all patients received premedications aimed at minimizing the risk of hypersensitivity reaction, and had as-needed medication orders, in the event a hypersensitivity reaction occurred. Reducing hypersensitivity reaction risk is important as well to increase the likelihood that the patient would receive full therapy, as management of this adverse drug reactions involves possible cessation of therapy depending on the level of severity. The pharmacy department also ensured these medications were stocked in ED automated dispensing cabinets to promote quick access. In addition to the aforementioned nursing orders, we added EUA criteria for use and hyperlinks to the Fact Sheets for Patients and Caregivers and Health Care Providers for each monoclonal therapy, and restricted ordering to ED physicians, nurse practitioners, and physician assistants.

The order set underwent multidisciplinary review by pharmacy administration, the chair of emergency medicine, physicians, and ED nursing leadership prior to presentation and approval by the Pharmacy and Therapeutics Committee. Lastly, at time of implementation, the order set was added to the ED preference list, preventing inpatient access. Additionally, as a patient safety action, free- standing orders of COVID-19 monoclonal therapies were disabled, so providers could only order therapies via the approved, comprehensive order set.

Preliminary Assessment Tool. A provider assessment tool was developed to document patient-specific EUA criteria for use during initial assessment (Figure 4). This tool serves as a checklist and is visible to the full multidisciplinary team in the patient’s EHR. It is used as a resource at the time of pharmacist verification and ED physician assessment to ensure criteria for use are met.

Patient Referral. Patients with symptoms or concerns of COVID-19 exposure can make physician appointments via telemedicine or in person at Mount Sinai Medical Center’s primary care and specialty offices. At the time of patient encounter, physicians suspecting a COVID-19 diagnosis will refer patients for outpatient COVID-19 polymerase chain reaction (PCR) laboratory testing, which has an approximate 24-hour turnaround to results. Physicians also assess whether the patient meets EUA criteria for use, pending results of testing. In the event a patient meets EUA criteria for use, the physician provides patient counseling and requests verbal consent. Following this, the physician enters a note in the EHR describing the patient’s condition, criteria for use evaluation, and the patient’s verbal agreement to therapy. This preliminary screening is beneficial to begin planning with both the patient and ED to minimize delays. Patients are notified of the results of their test once available. If the COVID-19 PCR test returns positive, the physician will call the ED at the main campus and schedule the patient for COVID-19 monoclonal therapy. As the desired timeframe for administering COVID-19 monoclonal therapies is within less than 10 days of symptom onset, timely scheduling of appointments is crucial. Infusion appointments are typically provided the same or next day. The patients are informed that they must bring documentation of their positive COVID-19 PCR test to their ED visit. Lastly, because patients are pretreated with medication that may potentially impair driving, they are instructed that they cannot drive themselves home; ride shares also are not allowed in order to limit the spread of infection.

Emergency Department

Patient Arrival and Screening. A COVID-19 patient can be evaluated in the ED 1 of 2 ways. The first option is via outpatient office referral, as described previously. Upon arrival to the ED, a second screening is performed to ensure the patient still meets EUA criteria for use and the positive COVID-19 PCR test result is confirmed. If the patient no longer meets criteria, the patient is triaged accordingly, including evaluation for higher-level care (eg, supplemental oxygen, hospital admission). The second optoion is via new patient walk-ins without outpatient physician referral (Figure 4). In these cases, an initial screening is performed, documenting EUA criteria for use in the preliminary assessment (Figure 5). Physicians will consider an outside COVID-19 test as valid, so long as documentation is readily available confirming a positive PCR result. Otherwise, an in-house COVID-19 PCR test will be performed, which has a 2-hour turnaround time.

Infusion Schedule. The ED offers a total of 16 COVID-19 monoclonal infusions slots daily. These are broken up into 4 infusion time blocks (eg, 8 am, 12 pm, 4 pm, 8 pm), with each infusion time block consisting of 4 available patient appointments. A list of scheduled infusions for the day is emailed to the pharmacy department every morning, and patients are instructed to arrive 1 hour prior to their appointment time. This allows time for patient registration, assessment, and pharmacy notification in advance of order entry. For logistical purposes, and as a patient safety initiative to reduce the likelihood of medication errors, each of the available COVID-19 monoclonal antibodies is offered on a designated day. Bamlanivimab is offered on Tuesday, Thursday, Saturday, and Sunday, while casirivimab/imdevimab is offered Monday, Wednesday, and Friday. This provides flexibility to adjust should supply deviate based on Department of Health allocation or should new therapy options within this class of medication become available.

Patient Education. Prior to administration of the monoclonal therapy, physician and nursing staff obtain a formal, written patient consent for therapy and provide patients with the option of participating in the institutional review board (IRB) approved study. Details of this are discussed in the risk management and IRB sections of the article. Nursing staff also provides the medication-specific Fact Sheet for Patients and Caregivers in either Spanish or English, which is also included as a hyperlink on the COVID-19 Monoclonal Antibody Order Set for ease of access. Interpreter services are available for patients who speak other languages. An ED decentralized pharmacist is also available onsite Monday through Friday from 12 pm to 8:30 pm to supplement education and serve as a resource for any questions.

Infusion Ordering. Once the patient is ready to begin therapy, the he/she is brought to a dedicated overflow area of the ED. There are few, if any, patients in this location, and it is adjacent to the main emergency center for easy access by the patients, nurses, pharmacists, and physicians. The physician then enters orders in the EHR using the COVID-19 Monoclonal Antibody Order Set (Figure 3). Three discrete questions were built into the medication order: (1) Was patient consent obtained? (2) Was the Fact Sheet for Patient/Caregiver provided to the patient? (3) Is the patient COVID-19 PCR-positive? These questions were built as hard stops so that the medication orders cannot be placed without a response. This serves as another double-check to ensure processes are followed and helps facilitate timely verification by the pharmacist.

Medication Administration. One nurse is dedicated to administering the monoclonal therapies scheduled at 8 am and 12 pm and another at 4 pm and 8 pm. Each appointment block is 4 hours in duration to allow adequate time for patient registration, infusion, and postinfusion observation. The nurse administers the premedications and COVID-19 monoclonal therapy, and observes the patient for the required 1-hour postadministration observation period. Nursing orders detailing monitoring parameters for mild, moderate, and severe reactions, along with associated medication orders to administer in the event they occur, are detailed in the nursing orders of the COVID-19 Monoclonal Antibody Order Set (Figure 3). Prior to administration, the nurse scans each medication and the patient’s wrist identification band, and documents the time of administration within the EHR medication administration report.

Pharmacy Department

Medication Receipt Process. Inventory is currently allocated biweekly from the state department of health and will soon be transitioning to a direct order system. The pharmacy technician in charge of deliveries notifies the pharmacy Antimicrobial Stewardship Program (ASP) clinical coordinator upon receipt of the monoclonal therapies. Bamlanivimab is supplied as 1 vial per dose, whereas casirivimab/imdevimab is supplied as 4 vials or 8 vials per dose, depending how it is shipped. To reduce the likelihood of medication errors, the ASP clinical coordinator assembles each of the casirivimab/imdevimab vials into kits, where 1 kit equals 1 dose. Labels are then affixed to each kit indicating the medication name, number of vials which equal a full dose, and pharmacist signature. The kits are stored in a dedicated refrigerator, and inventory logs are affixed to the outside of the refrigerator and updated daily. This inventory is also communicated daily to ED physician, nursing, and pharmacy leadership, as well as the director of patient safety, who reports weekly usage to the state Department of Health and Human Services. These weekly reports are used to determine allocation amounts.

Medication Verification and Delivery. The Mount Sinai Medical Center pharmacist staffing model consists of centralized order entry and specialized, decentralized positions. All orders are verified by the ED pharmacist when scheduled (not a 24/7 service) and by the designated pharmacist for all other times. At the time of medication verification, the pharmacist documents patient-specific EUA criteria for use and confirms that consent was obtained and the Fact Sheet for Patients/Caregivers was provided. A pharmacist intervention was developed to assist with this documentation. Pharmacists input smart text “.COVIDmonoclonal” and a drop-down menu of EUA criteria for use appears. The pharmacist reviews the patient care notes and medication order question responses to ascertain this information, contacting the ED prescriber if further clarification is required. This verification serves as another check to ensure processes put in place are followed. Lastly, intravenous preparation and delivery are electronically recorded in the EHR, and the medications require nursing signature at the time of delivery to ensure a formal chain of custody.

Risk Management

At Mount Sinai Medical Center, all EUA and investigational therapies require patient consent. Consistent with this requirement, a COVID-19 monoclonal specific consent was developed by risk management. This is provided to every patient receiving a COVID-19 monoclonal infusion, in addition to the FDA EUA Fact Sheet for Patients and Caregivers, and documented as part of their EHR. The questions providers must answer are built into the order set to ensure this process is followed and these patient safety checks are incorporated into the workflow.

Billing and Finance Department

In alignment with Mount Sinai Medical Center’s mission to provide high-quality health care to its diverse community through teaching, research, charity care, and financial responsibility, it was determined that this therapy would be provided to all patients regardless of insurance type, including those who are uninsured. The billing and finance department was consulted prior to this service being offered, to provide patients with accurate and pertinent information. The billing and finance department provided guidance on how to document patient encounters at time of registration to facilitate appropriate billing. At this time, the medication is free of charge, but nonmedication-related ED fees apply. This is explained to patients so there is a clear understanding prior to booking their appointment.

Infection Prevention

As patients receiving COVID-19 monoclonal therapies can transmit the virus to others, measures to ensure protection for other patients and staff are vital. To minimize exposure, specific nursing and physician staff from the ED are assigned to the treatment of these patients, and patients receive infusions and postobservation monitoring in a designated wing of the ED. Additionally, all staff who interact with these patients are required to don full personal protective equipment. This includes not only physicians and nurses but all specialties such as physician assistants, nurse practitioners, pharmacists, and laboratory technicians. Moreover, patients are not permitted to go home in a ride share and are counseled on Centers for Disease Control and Prevention quarantining following infusion.

Measurement of Process and Outcomes and Reporting

IRB approval was sought and obtained early during initiation of this service, allowing study consent to be offered to patients at the time general consent was obtained, which maximized patient recruitment and streamlined workflow. The study is a prospective observational research study to determine the impact of administration of COVID-19 monoclonal antibody therapy on length of symptoms, chronic illness, and rate of hospitalization. Most patients were eager to participate and offer their assistance to the scientific community during this pandemic.

Staff Education

In order to successfully implement this multidisciplinary EUA treatment option, comprehensive staff education was paramount after the workflow was developed. Prior to the first day of infusions, nurses and pharmacists were provided education during multiple huddle announcements. The pharmacy team also provided screen captures via email to the pharmacists so they could become familiar with the order set, intervention documentation, and location of the preliminary assessment of EUA criteria for use at the time of order verification. The emergency medicine department chair and chief medical officer also provided education via several virtual meetings and email to referring physicians (specialists and primary care) and residents in the emergency centers involved in COVID-19 monoclonal therapy-related patient care.

Factors Contributing to Success

We believe the reasons for continued success of this process are multifactorial and include the following key elements. Multidisciplinary planning, which included decision makers and all stakeholders, began at the time the idea was conceived. This allowed quick implementation of this service by efficiently navigating barriers to engaging impacted staff early on. Throughout this process, the authors set realistic step-wise goals. While navigating through the many details to implementation described, we also kept in mind the big picture, which was to provide this potentially lifesaving therapy to as many qualifying members of our community as possible. This included being flexible with the process and adapting when needed to achieve this ultimate goal. A focus on safety remained a priority to minimize possible errors and enhance patient and staff satisfaction. The optimization of the EHR streamlined workflow, provided point-of-care resources, and enhanced patient safety. Additionally, the target date set for implementation allowed staff and department leads adequate time to plan for and anticipate the changes. Serving only 1 patient on the first day allowed time for staff to experience this new process hands-on and provided opportunity for focused education. This team communication was essential to implementing this project, including staff training of processes and procedures prior to go-live. Early incorporation of IRB approval allowed the experience to be assessed and considered for contribution to the scientific literature to tackle this novel virus that has impacted our communities locally, nationally, and abroad. Moreover, continued measurement and reporting on a regular basis leads to performance improvement. The process outlined here can be adapted to incorporate other new therapies in the future, such as the recent February 9, 2021, EUA of the COVID-19 monoclonal antibody combination bamlanivimab and etesevimab.¹⁰

Conclusion

We administered 790 COVID-19 monoclonal antibody infusions between November 20, 2020 and March 5, 2021. Steps to minimize the likelihood of hypersensitivity reactions were implemented, and a low incidence (< 1%) has been observed. There has been no incidence of infection, concern from staff about infection prevention, or risk of infection during the processes. There have been very infrequent cost-related concerns raised by patients, typically due to incomplete communication regarding billing prior to the infusion. To address these issues, staff education has been provided to enhance patient instruction on this topic. The program has provided patient and family satisfaction, as well nursing, physician, pharmacist, clinical staff, and hospital administration pride and gratification. Setting up a new program to provide a 4-hour patient encounter to infuse therapy to high-risk patients with COVID-19 requires commitment and effort. This article describes the experience, ideas, and formula others may consider using to set up such a program. Through networking and formal phone calls and meetings about monoclonal antibody therapy, we have heard about other institutions who have not been able to institute this program due to various barriers to implementation. We hope our experience serves as a resource for others to provide this therapy to their patients and expand access in an effort to mitigate COVID-19 consequences and cases affecting our communities.

Corresponding author: Kathleen Jodoin, PharmD, BCPS, Mount Sinai Medical Center, 4300 Alton Rd, Miami Beach, FL 33140; kathleen.jodoin@msmc.com.

Financial disclosures: None.

From Baylor College of Medicine, Houston, TX (Drs. Spezia-Lindner, Montealegre, Muldrew, and Suarez) and Harris Health System, Houston, TX (Shanna L. Harris, Maria Daheri, and Drs. Muldrew and Suarez).

Abstract

Objective: To characterize and analyze the prevalence, indications for, and outcomes of fecal immunochemical testing (FIT) in acute patient care within a safety net health care system’s emergency departments (EDs) and inpatient settings.

Design: Retrospective cohort study derived from administrative data.

Setting: A large, urban, safety net health care delivery system in Texas. The data gathered were from the health care system’s 2 primary hospitals and their associated EDs. This health care system utilizes FIT exclusively for fecal occult blood testing.

Participants: Adults ≥18 years who underwent FIT in the ED or inpatient setting between August 2016 and March 2017. Chart review abstractions were performed on a sample (n = 382) from the larger subset.

Measurements: Primary data points included total FITs performed in acute patient care during the study period, basic demographic data, FIT indications, FIT result, receipt of invasive diagnostic follow-up, and result of invasive diagnostic follow-up. Multivariable log-binomial regression was used to calculate risk ratios (RRs) to assess the association between FIT result and receipt of diagnostic follow-up. Chi-square analysis was used to compare the proportion of abnormal findings on diagnostic follow-up by FIT result.

Results: During the 8-month study period, 2718 FITs were performed in the ED and inpatient setting, comprising 5.7% of system-wide FITs. Of the 382 patients included in the chart review who underwent acute care FIT, a majority had their test performed in the ED (304, 79.6%), 133 of which were positive (34.8%). The most common indication for FIT was evidence of overt gastrointestinal (GI) bleed (207, 54.2%), followed by anemia (84, 22.0%). While a positive FIT result was significantly associated with obtaining a diagnostic exam in multivariate analysis (RR, 1.72; P < 0.001), having signs of overt GI bleeding was a stronger predictor of diagnostic follow-up (RR, 2.00; P = 0.003). Of patients who underwent FIT and received diagnostic follow-up (n = 110), 48.2% were FIT negative. These patients were just as likely to have an abnormal finding as FIT-positive patients (90.6% vs 91.2%; P = 0.86). Of the 382 patients in the study, 4 (1.0%) were subsequently diagnosed with colorectal cancer (CRC). Of those 4 patients, 1 (25%) was FIT positive.

Conclusion: FIT is being utilized in acute patient care outside of its established indication for CRC screening in asymptomatic, average-risk adults. Our study demonstrates that FIT is not useful in acute patient care.

Keywords: FOBT; FIT; fecal immunochemical testing; inpatient.

Despite the exclusive validation of FOBTs for use in CRC screening, studies have demonstrated that they are commonly used for a multitude of additional indications in emergency department (ED) and inpatient settings, most aimed at detecting or confirming GI blood loss. This may lead to inappropriate patient management, including the receipt of unnecessary follow-up procedures, which can incur significant costs to the patient and the health system.^13-19 These costs may be particularly burdensome in safety net health systems (ie, those that offer access to care regardless of the patient’s ability to pay), which serve a large proportion of socioeconomically disadvantaged individuals in the United States.^20,21 To our knowledge, no published study to date has specifically investigated the role of FIT in acute patient management.

This study characterizes the use of FIT in acute patient care within a large, urban, safety net health care system. Through a retrospective review of administrative data and patient charts, we evaluated FIT use prevalence, indications, and patient outcomes in the ED and inpatient settings.

Methods

Setting

This study was conducted in a large, urban, county-based integrated delivery system in Houston, Texas, that provides health care services to one of the largest uninsured and underinsured populations in the country.²² The health system includes 2 main hospitals and more than 20 ambulatory care clinics. Within its ambulatory care clinics, the health system implements a population-based screening strategy using stool-based testing. All adults aged 50 years or older who are due for FIT are identified through the health-maintenance module of the electronic medical record (EMR) and offered a take-home FIT. The health system utilizes FIT exclusively (OC-Light S FIT, Polymedco, Cortlandt Manor, NY); no guaiac-based assays are available.

Design and Data Collection

We began by using administrative records to determine the proportion of FITs conducted health system-wide that were ordered and completed in the acute care setting over the study period (August 2016-March 2017). Specifically, we used aggregate quality metric reports, which quantify the number of FITs conducted at each health system clinic and hospital each month, to calculate the proportion of FITs done in the ED and inpatient hospital setting.

We then conducted a retrospective cohort study of 382 adult patients who received FIT in the EDs and inpatient wards in both of the health system’s hospitals over the study period. All data were collected by retrospective chart review in Epic (Madison, WI) EMRs. Sampling was performed by selecting the medical record numbers corresponding to the first 50 completed FITs chronologically each month over the 8-month period, with a total of 400 charts reviewed.

Data collected included basic patient demographics, location of FIT ordering (ED vs inpatient), primary service ordering FIT, FIT indication, FIT result, and receipt and results of invasive diagnostic follow-up. Demographics collected included age, biological sex, race (self-selected), and insurance coverage.

FIT indication was determined based on resident or attending physician notes. The history of present illness, physical exam, and assessment and plan section of notes were reviewed by the lead author for a specific statement of indication for FIT or for evidence of clinical presentation for which FIT could reasonably be ordered. Indications were iteratively reviewed and collapsed into 6 different categories: anemia, iron deficiency with or without anemia, overt GIB, suspected GIB/miscellaneous, non-bloody diarrhea, and no indication identified. Overt GIB was defined as reported or witnessed hematemesis, coffee-ground emesis, hematochezia, bright red blood per rectum, or melena irrespective of time frame (current or remote) or chronicity (acute, subacute, or chronic). In cases where signs of overt bleed were not witnessed by medical professionals, determination of conditions such as melena or coffee-ground emesis were made based on health care providers’ assessment of patient history as documented in his or her notes. Suspected GIB/miscellaneous was defined with the following parameters: any new drop in hemoglobin, abdominal pain, anorectal pain, non-bloody vomiting, hemoptysis, isolated rising blood urea nitrogen, or patient noticing blood on self, clothing, or in the commode without an identified source. Patients who were anemic and found to have iron deficiency on recent lab studies (within 6 months) were reflexively categorized into iron deficiency with or without anemia as opposed to the “anemia” category, which was comprised of any anemia without recent iron studies or non-iron deficient anemia. FIT result was determined by test result entry in Epic, with results either reading positive or negative.

Diagnostic follow-up, for our purposes, was defined as receipt of an invasive procedure or surgery, including esophagogastroduodenoscopy (EGD), colonoscopy, flexible sigmoidoscopy, diagnostic and/or therapeutic abdominal surgical intervention, or any combination of these. Results of diagnostic follow-up were coded as normal or abnormal. A normal result was determined if all procedures performed were listed as normal or as “no pathological findings” on the operative or endoscopic report. Any reported pathologic findings on the operative/endoscopic report were coded as abnormal.

Statistical Analysis

Proportions were used to describe demographic characteristics of patients who received a FIT in acute hospital settings. Bivariable tables and Chi-square tests were used to compare indications and outcomes for FIT-positive and FIT-negative patients. The association between receipt of an invasive diagnostic follow-up (outcome) and the results of an inpatient FIT (predictor) was assessed using multivariable log-binomial regression to calculate risk ratios (RRs) and corresponding 95% confidence intervals. Log-binomial regression was used over logistic regression given that adjusted odds ratios generated by logistic regression often overestimate the association between the risk factor and the outcome when the outcome is common,²³ as in the case of diagnostic follow-up. The model was adjusted for variables selected a priori, specifically, age, gender, and FIT indication. Chi-square analysis was used to compare the proportion of abnormal findings on diagnostic follow-up by FIT result (negative vs positive).

Results

During the 8-month study period, there were 2718 FITs ordered and completed in the acute care setting, compared to 44,662 FITs ordered and completed in the outpatient setting (5.7% performed during acute care).

Among the 400 charts reviewed, 7 were excluded from the analysis because they were duplicates from the same patient, and 11 were excluded due to insufficient information in the patient’s medical record, resulting in 382 patients included in the analysis. Patient demographic characteristics are described in Table 1. Patients were predominantly Hispanic/Latino or Black/African American (51.0% and 32.5%, respectively), a majority had insurance through the county health system (50.5%), and most were male (58.1%). The average age of those receiving FIT was 52 years (standard deviation, 14.8 years), with 40.8% being under the age of 50. For a majority of patients, FIT was ordered in the ED by emergency medicine providers (79.8%). The remaining FITs were ordered by providers in 12 different inpatient departments. Of the FITs ordered, 35.1% were positive.

Indications for ordering FIT are listed in Table 2. The largest proportion of FITs were ordered for overt signs of GIB (54.2%), followed by anemia (22.0%), suspected GIB/miscellaneous reasons (12.3%), iron deficiency with or without anemia (7.6%), and non-bloody diarrhea (2.1%). In 1.8% of cases, no indication for FIT was found in the EMR. No FITs were ordered for the indication of CRC detection. Of these indication categories, overt GIB yielded the highest percentage of FIT positive results (44.0%), and non-bloody diarrhea yielded the lowest (0%).

A total of 110 patients (28.7%) underwent FIT and received invasive diagnostic follow-up. Of these 110 patients, 57 (51.8%) underwent EGD (2 of whom had further surgical intervention), 21 (19.1%) underwent colonoscopy (1 of whom had further surgical intervention), 25 (22.7%) underwent dual EGD and colonoscopy, 1 (0.9%) underwent flexible sigmoidoscopy, and 6 (5.5%) directly underwent abdominal surgical intervention. There was a significantly higher rate of diagnostic follow-up for FIT-positive vs FIT-negative patients (42.9% vs 21.3%; P < 0.001). However, of the 110 patients who underwent subsequent diagnostic follow-up, 48.2% were FIT negative. FIT-negative patients who received diagnostic follow-up were just as likely to have an abnormal finding as FIT-positive patients (90.6% vs 91.2%; P = 0.86).

Of the 382 patients in the study, 4 were diagnosed with CRC through diagnostic follow-up (1.0%). Of those 4 patients, 1 was FIT positive.

The results of the multivariable analyses to evaluate predictors of diagnostic colonoscopy are described in Table 3. Variables in the final model were FITresult, age, and FIT indication. After adjusting for other variables in the model, receipt of diagnostic follow-up was significantly associated with having a positive FIT (adjusted RR, 1.72; P < 0.001) and an overt GIB as an indication (adjusted RR, 2.00; P < 0.01).

Discussion

During the time frame of our study, 5.7% of all FITs ordered within our health system were ordered in the acute patient care setting at our hospitals. The most common indication was overt GIB, which was the indication for 54.2% of patients. Of note, none of the FITs ordered in the acute patient care setting were ordered for CRC screening. These findings support the evidence in the literature that stool-based screening tests, including FIT, are commonly used in US health care systems for diagnostic purposes and risk stratification in acute patient care to detect GIBs.^13-18

Our data suggest that FIT was not a clinically useful test in determining a patient’s need for diagnostic follow-up. While having a positive FIT was significantly associated with obtaining a diagnostic exam in multivariate analysis (RR, 1.72), having signs of overt GI bleeding was a stronger predictor of diagnostic follow-up (RR, 2.00). This salient finding is evidence that a thorough clinical history and physical exam may more strongly predict whether a patient will undergo endoscopy or other follow-up than a FIT result. These findings support other studies in the literature that have called into question the utility of FOBTs in these acute settings.^13-19 Under such circumstances, FOBTs have been shown to rarely influence patient management and thus represent an unnecessary expense.^13–17 Additionally, in some cases, FOBT use in these settings may negatively affect patient outcomes. Such adverse effects include delaying treatment until results are returned or obfuscating indicated management with the results (eg, a patient with indications for colonoscopy not being referred due to a negative FOBT).^13,14,17

We found that, for patients who subsequently went on to have diagnostic follow-up (most commonly endoscopy), there was no difference in the likelihood of FIT-positive and FIT-negative patients to have an abnormality discovered (91.2% vs 90.6%; P = 0.86). This analysis demonstrates no post-hoc support for FIT positivity as a predictor of presence of pathology in patients who were discriminately selected for diagnostic follow-up on clinical grounds by gastroenterologists and surgeons. It does, however, further support that clinical judgment about the need for diagnostic follow-up—irrespective of FIT result—has a very high yield for discovery of pathology in the acute setting.

There are multiple reasons why FOBTs, and specifically FIT, contribute little in management decisions for patients with suspected GI blood loss. Use of FIT raises concern for both false-negatives and false-positives when used outside of its indication. Regarding false- negatives, FIT is an unreliable test for detection of blood loss from the upper GI tract. As FITs utilize antibodies to detect the presence of globin, a byproduct of red blood cell breakdown, it is expected that FIT would fail to detect many cases of upper GI bleeding, as globin is broken down in the upper GI tract.²⁴ This fact is part of what has made FIT a more effective CRC screening test than its guaiac-based counterparts—it has greater specificity for lower GI tract blood loss compared to tests relying on detection of heme.⁸ While guaiac-based assays like Hemoccult have also been shown to be poor tests in acute patient care, they may more frequently, though still unreliably, detect blood of upper GI origin. We believe that part of the ongoing use of FIT in patients with a suspected upper GIB may be from lack of understanding among providers on the mechanistic difference between gFOBTs and FITs, even though gFOBTs also yield highly unreliable results.

FIT does not have the same risk of false-positive results that guaiac-based tests have, which can yield positive results with extra-intestinal blood ingestion, aspirin, or alcohol use; insignificant GI bleeding; and consumption of peroxidase-containing foods.^13,17,25 However, from a clinical standpoint, there are several scenarios of insignificant bleeding that would yield a positive FIT result, such as hemorrhoids, which are common in the US population.^26,27 Additionally, in the ED, where most FITs were performed in our study, it is possible that samples for FITs are being obtained via digital rectal exam (DRE) given patients’ acuity of medical conditions and time constraints. However, FIT has been validated when using a formed stool sample. Obtaining FIT via DRE may lead to microtrauma to the rectum, which could hypothetically yield a positive FIT.

Strengths of this study include its use of in-depth chart data on a large number of FIT-positive patients, which allowed us to discern indications, outcomes, and other clinical data that may have influenced clinical decision-making. Additionally, whereas other studies that address FOBT use in acute patient care have focused on guaiac-based assays, our findings regarding the lack of utility of FIT are novel and have particular relevance as FITs continue to grow in popularity. Nonetheless, there are certain limitations future research should seek to address. In this study, the diagnostic follow-up result was coded by presence or absence of pathologic findings but did not qualify findings by severity or attempt to determine whether the pathology noted on diagnostic follow-up was the definitive source of the suspected GI bleed. These variables could help determine whether there was a difference in severity of bleeding between FIT-positive and FIT-negative patients and could potentially be studied with a prospective research design. Our own study was not designed to address the question of whether FIT result informs patient management decisions. To answer this directly, interviews would have to be conducted with those making the follow-up decision (ie, endoscopists and surgeons). Additionally, this study was not adequately powered to make determinations on the efficacy of FIT in the acute care setting for detection of CRC. As mentioned, only 1 of the 4 patients (25%) who went on to be diagnosed with CRC on follow-up was initially FIT-positive. This would require further investigation.

Conclusion

FIT is being utilized for diagnostic purposes in the acute care of symptomatic patients, which is a misuse of an established screening test for CRC. While our study was not designed to answer whether and how often a FIT result informs subsequent patient management, our results indicate that FIT is an ineffective diagnostic and risk-stratification tool when used in the acute care setting. Our findings add to existing evidence that indicates FOBTs should not be used in acute patient care.

Taken as a whole, the results of our study add to a growing body of evidence demonstrating no role for FOBTs, and specifically FIT, in acute patient care. In light of this evidence, some health care systems have already demonstrated success with system-wide disinvestment from the test in acute patient care settings, with one group publishing about their disinvestment process.²⁸ After completion of our study, our preliminary data were presented to leadership from the internal medicine, emergency medicine, and laboratory divisions within our health care delivery system to galvanize complete disinvestment of FIT from acute care at our hospitals, a policy that was put into effect in July 2019.

Corresponding author: Nathaniel J. Spezia-Lindner, MD, Baylor College of Medicine, 7200 Cambridge St, BCM 903, Ste A10.197, Houston, TX 77030; speziali@bcm.edu.

Financial disclosures: None.

Funding: Cancer Prevention and Research Institute of Texas, CPRIT (PP170094, PDs: ML Jibaja-Weiss and JR Montealegre).

From Baylor College of Medicine, Houston, TX (Drs. Spezia-Lindner, Montealegre, Muldrew, and Suarez) and Harris Health System, Houston, TX (Shanna L. Harris, Maria Daheri, and Drs. Muldrew and Suarez).

Abstract

Objective: To characterize and analyze the prevalence, indications for, and outcomes of fecal immunochemical testing (FIT) in acute patient care within a safety net health care system’s emergency departments (EDs) and inpatient settings.

Design: Retrospective cohort study derived from administrative data.

Setting: A large, urban, safety net health care delivery system in Texas. The data gathered were from the health care system’s 2 primary hospitals and their associated EDs. This health care system utilizes FIT exclusively for fecal occult blood testing.

Participants: Adults ≥18 years who underwent FIT in the ED or inpatient setting between August 2016 and March 2017. Chart review abstractions were performed on a sample (n = 382) from the larger subset.

Measurements: Primary data points included total FITs performed in acute patient care during the study period, basic demographic data, FIT indications, FIT result, receipt of invasive diagnostic follow-up, and result of invasive diagnostic follow-up. Multivariable log-binomial regression was used to calculate risk ratios (RRs) to assess the association between FIT result and receipt of diagnostic follow-up. Chi-square analysis was used to compare the proportion of abnormal findings on diagnostic follow-up by FIT result.

Results: During the 8-month study period, 2718 FITs were performed in the ED and inpatient setting, comprising 5.7% of system-wide FITs. Of the 382 patients included in the chart review who underwent acute care FIT, a majority had their test performed in the ED (304, 79.6%), 133 of which were positive (34.8%). The most common indication for FIT was evidence of overt gastrointestinal (GI) bleed (207, 54.2%), followed by anemia (84, 22.0%). While a positive FIT result was significantly associated with obtaining a diagnostic exam in multivariate analysis (RR, 1.72; P < 0.001), having signs of overt GI bleeding was a stronger predictor of diagnostic follow-up (RR, 2.00; P = 0.003). Of patients who underwent FIT and received diagnostic follow-up (n = 110), 48.2% were FIT negative. These patients were just as likely to have an abnormal finding as FIT-positive patients (90.6% vs 91.2%; P = 0.86). Of the 382 patients in the study, 4 (1.0%) were subsequently diagnosed with colorectal cancer (CRC). Of those 4 patients, 1 (25%) was FIT positive.

Conclusion: FIT is being utilized in acute patient care outside of its established indication for CRC screening in asymptomatic, average-risk adults. Our study demonstrates that FIT is not useful in acute patient care.

Keywords: FOBT; FIT; fecal immunochemical testing; inpatient.

Despite the exclusive validation of FOBTs for use in CRC screening, studies have demonstrated that they are commonly used for a multitude of additional indications in emergency department (ED) and inpatient settings, most aimed at detecting or confirming GI blood loss. This may lead to inappropriate patient management, including the receipt of unnecessary follow-up procedures, which can incur significant costs to the patient and the health system.^13-19 These costs may be particularly burdensome in safety net health systems (ie, those that offer access to care regardless of the patient’s ability to pay), which serve a large proportion of socioeconomically disadvantaged individuals in the United States.^20,21 To our knowledge, no published study to date has specifically investigated the role of FIT in acute patient management.

This study characterizes the use of FIT in acute patient care within a large, urban, safety net health care system. Through a retrospective review of administrative data and patient charts, we evaluated FIT use prevalence, indications, and patient outcomes in the ED and inpatient settings.

Methods

Setting

This study was conducted in a large, urban, county-based integrated delivery system in Houston, Texas, that provides health care services to one of the largest uninsured and underinsured populations in the country.²² The health system includes 2 main hospitals and more than 20 ambulatory care clinics. Within its ambulatory care clinics, the health system implements a population-based screening strategy using stool-based testing. All adults aged 50 years or older who are due for FIT are identified through the health-maintenance module of the electronic medical record (EMR) and offered a take-home FIT. The health system utilizes FIT exclusively (OC-Light S FIT, Polymedco, Cortlandt Manor, NY); no guaiac-based assays are available.

Design and Data Collection

We began by using administrative records to determine the proportion of FITs conducted health system-wide that were ordered and completed in the acute care setting over the study period (August 2016-March 2017). Specifically, we used aggregate quality metric reports, which quantify the number of FITs conducted at each health system clinic and hospital each month, to calculate the proportion of FITs done in the ED and inpatient hospital setting.

We then conducted a retrospective cohort study of 382 adult patients who received FIT in the EDs and inpatient wards in both of the health system’s hospitals over the study period. All data were collected by retrospective chart review in Epic (Madison, WI) EMRs. Sampling was performed by selecting the medical record numbers corresponding to the first 50 completed FITs chronologically each month over the 8-month period, with a total of 400 charts reviewed.

Data collected included basic patient demographics, location of FIT ordering (ED vs inpatient), primary service ordering FIT, FIT indication, FIT result, and receipt and results of invasive diagnostic follow-up. Demographics collected included age, biological sex, race (self-selected), and insurance coverage.

FIT indication was determined based on resident or attending physician notes. The history of present illness, physical exam, and assessment and plan section of notes were reviewed by the lead author for a specific statement of indication for FIT or for evidence of clinical presentation for which FIT could reasonably be ordered. Indications were iteratively reviewed and collapsed into 6 different categories: anemia, iron deficiency with or without anemia, overt GIB, suspected GIB/miscellaneous, non-bloody diarrhea, and no indication identified. Overt GIB was defined as reported or witnessed hematemesis, coffee-ground emesis, hematochezia, bright red blood per rectum, or melena irrespective of time frame (current or remote) or chronicity (acute, subacute, or chronic). In cases where signs of overt bleed were not witnessed by medical professionals, determination of conditions such as melena or coffee-ground emesis were made based on health care providers’ assessment of patient history as documented in his or her notes. Suspected GIB/miscellaneous was defined with the following parameters: any new drop in hemoglobin, abdominal pain, anorectal pain, non-bloody vomiting, hemoptysis, isolated rising blood urea nitrogen, or patient noticing blood on self, clothing, or in the commode without an identified source. Patients who were anemic and found to have iron deficiency on recent lab studies (within 6 months) were reflexively categorized into iron deficiency with or without anemia as opposed to the “anemia” category, which was comprised of any anemia without recent iron studies or non-iron deficient anemia. FIT result was determined by test result entry in Epic, with results either reading positive or negative.

Diagnostic follow-up, for our purposes, was defined as receipt of an invasive procedure or surgery, including esophagogastroduodenoscopy (EGD), colonoscopy, flexible sigmoidoscopy, diagnostic and/or therapeutic abdominal surgical intervention, or any combination of these. Results of diagnostic follow-up were coded as normal or abnormal. A normal result was determined if all procedures performed were listed as normal or as “no pathological findings” on the operative or endoscopic report. Any reported pathologic findings on the operative/endoscopic report were coded as abnormal.

Statistical Analysis

Proportions were used to describe demographic characteristics of patients who received a FIT in acute hospital settings. Bivariable tables and Chi-square tests were used to compare indications and outcomes for FIT-positive and FIT-negative patients. The association between receipt of an invasive diagnostic follow-up (outcome) and the results of an inpatient FIT (predictor) was assessed using multivariable log-binomial regression to calculate risk ratios (RRs) and corresponding 95% confidence intervals. Log-binomial regression was used over logistic regression given that adjusted odds ratios generated by logistic regression often overestimate the association between the risk factor and the outcome when the outcome is common,²³ as in the case of diagnostic follow-up. The model was adjusted for variables selected a priori, specifically, age, gender, and FIT indication. Chi-square analysis was used to compare the proportion of abnormal findings on diagnostic follow-up by FIT result (negative vs positive).

Results

During the 8-month study period, there were 2718 FITs ordered and completed in the acute care setting, compared to 44,662 FITs ordered and completed in the outpatient setting (5.7% performed during acute care).

Among the 400 charts reviewed, 7 were excluded from the analysis because they were duplicates from the same patient, and 11 were excluded due to insufficient information in the patient’s medical record, resulting in 382 patients included in the analysis. Patient demographic characteristics are described in Table 1. Patients were predominantly Hispanic/Latino or Black/African American (51.0% and 32.5%, respectively), a majority had insurance through the county health system (50.5%), and most were male (58.1%). The average age of those receiving FIT was 52 years (standard deviation, 14.8 years), with 40.8% being under the age of 50. For a majority of patients, FIT was ordered in the ED by emergency medicine providers (79.8%). The remaining FITs were ordered by providers in 12 different inpatient departments. Of the FITs ordered, 35.1% were positive.

Indications for ordering FIT are listed in Table 2. The largest proportion of FITs were ordered for overt signs of GIB (54.2%), followed by anemia (22.0%), suspected GIB/miscellaneous reasons (12.3%), iron deficiency with or without anemia (7.6%), and non-bloody diarrhea (2.1%). In 1.8% of cases, no indication for FIT was found in the EMR. No FITs were ordered for the indication of CRC detection. Of these indication categories, overt GIB yielded the highest percentage of FIT positive results (44.0%), and non-bloody diarrhea yielded the lowest (0%).

A total of 110 patients (28.7%) underwent FIT and received invasive diagnostic follow-up. Of these 110 patients, 57 (51.8%) underwent EGD (2 of whom had further surgical intervention), 21 (19.1%) underwent colonoscopy (1 of whom had further surgical intervention), 25 (22.7%) underwent dual EGD and colonoscopy, 1 (0.9%) underwent flexible sigmoidoscopy, and 6 (5.5%) directly underwent abdominal surgical intervention. There was a significantly higher rate of diagnostic follow-up for FIT-positive vs FIT-negative patients (42.9% vs 21.3%; P < 0.001). However, of the 110 patients who underwent subsequent diagnostic follow-up, 48.2% were FIT negative. FIT-negative patients who received diagnostic follow-up were just as likely to have an abnormal finding as FIT-positive patients (90.6% vs 91.2%; P = 0.86).

Of the 382 patients in the study, 4 were diagnosed with CRC through diagnostic follow-up (1.0%). Of those 4 patients, 1 was FIT positive.

The results of the multivariable analyses to evaluate predictors of diagnostic colonoscopy are described in Table 3. Variables in the final model were FITresult, age, and FIT indication. After adjusting for other variables in the model, receipt of diagnostic follow-up was significantly associated with having a positive FIT (adjusted RR, 1.72; P < 0.001) and an overt GIB as an indication (adjusted RR, 2.00; P < 0.01).

Discussion

During the time frame of our study, 5.7% of all FITs ordered within our health system were ordered in the acute patient care setting at our hospitals. The most common indication was overt GIB, which was the indication for 54.2% of patients. Of note, none of the FITs ordered in the acute patient care setting were ordered for CRC screening. These findings support the evidence in the literature that stool-based screening tests, including FIT, are commonly used in US health care systems for diagnostic purposes and risk stratification in acute patient care to detect GIBs.^13-18

Our data suggest that FIT was not a clinically useful test in determining a patient’s need for diagnostic follow-up. While having a positive FIT was significantly associated with obtaining a diagnostic exam in multivariate analysis (RR, 1.72), having signs of overt GI bleeding was a stronger predictor of diagnostic follow-up (RR, 2.00). This salient finding is evidence that a thorough clinical history and physical exam may more strongly predict whether a patient will undergo endoscopy or other follow-up than a FIT result. These findings support other studies in the literature that have called into question the utility of FOBTs in these acute settings.^13-19 Under such circumstances, FOBTs have been shown to rarely influence patient management and thus represent an unnecessary expense.^13–17 Additionally, in some cases, FOBT use in these settings may negatively affect patient outcomes. Such adverse effects include delaying treatment until results are returned or obfuscating indicated management with the results (eg, a patient with indications for colonoscopy not being referred due to a negative FOBT).^13,14,17

We found that, for patients who subsequently went on to have diagnostic follow-up (most commonly endoscopy), there was no difference in the likelihood of FIT-positive and FIT-negative patients to have an abnormality discovered (91.2% vs 90.6%; P = 0.86). This analysis demonstrates no post-hoc support for FIT positivity as a predictor of presence of pathology in patients who were discriminately selected for diagnostic follow-up on clinical grounds by gastroenterologists and surgeons. It does, however, further support that clinical judgment about the need for diagnostic follow-up—irrespective of FIT result—has a very high yield for discovery of pathology in the acute setting.

There are multiple reasons why FOBTs, and specifically FIT, contribute little in management decisions for patients with suspected GI blood loss. Use of FIT raises concern for both false-negatives and false-positives when used outside of its indication. Regarding false- negatives, FIT is an unreliable test for detection of blood loss from the upper GI tract. As FITs utilize antibodies to detect the presence of globin, a byproduct of red blood cell breakdown, it is expected that FIT would fail to detect many cases of upper GI bleeding, as globin is broken down in the upper GI tract.²⁴ This fact is part of what has made FIT a more effective CRC screening test than its guaiac-based counterparts—it has greater specificity for lower GI tract blood loss compared to tests relying on detection of heme.⁸ While guaiac-based assays like Hemoccult have also been shown to be poor tests in acute patient care, they may more frequently, though still unreliably, detect blood of upper GI origin. We believe that part of the ongoing use of FIT in patients with a suspected upper GIB may be from lack of understanding among providers on the mechanistic difference between gFOBTs and FITs, even though gFOBTs also yield highly unreliable results.

FIT does not have the same risk of false-positive results that guaiac-based tests have, which can yield positive results with extra-intestinal blood ingestion, aspirin, or alcohol use; insignificant GI bleeding; and consumption of peroxidase-containing foods.^13,17,25 However, from a clinical standpoint, there are several scenarios of insignificant bleeding that would yield a positive FIT result, such as hemorrhoids, which are common in the US population.^26,27 Additionally, in the ED, where most FITs were performed in our study, it is possible that samples for FITs are being obtained via digital rectal exam (DRE) given patients’ acuity of medical conditions and time constraints. However, FIT has been validated when using a formed stool sample. Obtaining FIT via DRE may lead to microtrauma to the rectum, which could hypothetically yield a positive FIT.

Strengths of this study include its use of in-depth chart data on a large number of FIT-positive patients, which allowed us to discern indications, outcomes, and other clinical data that may have influenced clinical decision-making. Additionally, whereas other studies that address FOBT use in acute patient care have focused on guaiac-based assays, our findings regarding the lack of utility of FIT are novel and have particular relevance as FITs continue to grow in popularity. Nonetheless, there are certain limitations future research should seek to address. In this study, the diagnostic follow-up result was coded by presence or absence of pathologic findings but did not qualify findings by severity or attempt to determine whether the pathology noted on diagnostic follow-up was the definitive source of the suspected GI bleed. These variables could help determine whether there was a difference in severity of bleeding between FIT-positive and FIT-negative patients and could potentially be studied with a prospective research design. Our own study was not designed to address the question of whether FIT result informs patient management decisions. To answer this directly, interviews would have to be conducted with those making the follow-up decision (ie, endoscopists and surgeons). Additionally, this study was not adequately powered to make determinations on the efficacy of FIT in the acute care setting for detection of CRC. As mentioned, only 1 of the 4 patients (25%) who went on to be diagnosed with CRC on follow-up was initially FIT-positive. This would require further investigation.

Conclusion

FIT is being utilized for diagnostic purposes in the acute care of symptomatic patients, which is a misuse of an established screening test for CRC. While our study was not designed to answer whether and how often a FIT result informs subsequent patient management, our results indicate that FIT is an ineffective diagnostic and risk-stratification tool when used in the acute care setting. Our findings add to existing evidence that indicates FOBTs should not be used in acute patient care.

Taken as a whole, the results of our study add to a growing body of evidence demonstrating no role for FOBTs, and specifically FIT, in acute patient care. In light of this evidence, some health care systems have already demonstrated success with system-wide disinvestment from the test in acute patient care settings, with one group publishing about their disinvestment process.²⁸ After completion of our study, our preliminary data were presented to leadership from the internal medicine, emergency medicine, and laboratory divisions within our health care delivery system to galvanize complete disinvestment of FIT from acute care at our hospitals, a policy that was put into effect in July 2019.

Corresponding author: Nathaniel J. Spezia-Lindner, MD, Baylor College of Medicine, 7200 Cambridge St, BCM 903, Ste A10.197, Houston, TX 77030; speziali@bcm.edu.

Financial disclosures: None.

Funding: Cancer Prevention and Research Institute of Texas, CPRIT (PP170094, PDs: ML Jibaja-Weiss and JR Montealegre).

From Baylor College of Medicine, Houston, TX (Drs. Spezia-Lindner, Montealegre, Muldrew, and Suarez) and Harris Health System, Houston, TX (Shanna L. Harris, Maria Daheri, and Drs. Muldrew and Suarez).

Abstract

Objective: To characterize and analyze the prevalence, indications for, and outcomes of fecal immunochemical testing (FIT) in acute patient care within a safety net health care system’s emergency departments (EDs) and inpatient settings.

Design: Retrospective cohort study derived from administrative data.

Setting: A large, urban, safety net health care delivery system in Texas. The data gathered were from the health care system’s 2 primary hospitals and their associated EDs. This health care system utilizes FIT exclusively for fecal occult blood testing.

Participants: Adults ≥18 years who underwent FIT in the ED or inpatient setting between August 2016 and March 2017. Chart review abstractions were performed on a sample (n = 382) from the larger subset.

Measurements: Primary data points included total FITs performed in acute patient care during the study period, basic demographic data, FIT indications, FIT result, receipt of invasive diagnostic follow-up, and result of invasive diagnostic follow-up. Multivariable log-binomial regression was used to calculate risk ratios (RRs) to assess the association between FIT result and receipt of diagnostic follow-up. Chi-square analysis was used to compare the proportion of abnormal findings on diagnostic follow-up by FIT result.

Results: During the 8-month study period, 2718 FITs were performed in the ED and inpatient setting, comprising 5.7% of system-wide FITs. Of the 382 patients included in the chart review who underwent acute care FIT, a majority had their test performed in the ED (304, 79.6%), 133 of which were positive (34.8%). The most common indication for FIT was evidence of overt gastrointestinal (GI) bleed (207, 54.2%), followed by anemia (84, 22.0%). While a positive FIT result was significantly associated with obtaining a diagnostic exam in multivariate analysis (RR, 1.72; P < 0.001), having signs of overt GI bleeding was a stronger predictor of diagnostic follow-up (RR, 2.00; P = 0.003). Of patients who underwent FIT and received diagnostic follow-up (n = 110), 48.2% were FIT negative. These patients were just as likely to have an abnormal finding as FIT-positive patients (90.6% vs 91.2%; P = 0.86). Of the 382 patients in the study, 4 (1.0%) were subsequently diagnosed with colorectal cancer (CRC). Of those 4 patients, 1 (25%) was FIT positive.

Conclusion: FIT is being utilized in acute patient care outside of its established indication for CRC screening in asymptomatic, average-risk adults. Our study demonstrates that FIT is not useful in acute patient care.

Keywords: FOBT; FIT; fecal immunochemical testing; inpatient.

Despite the exclusive validation of FOBTs for use in CRC screening, studies have demonstrated that they are commonly used for a multitude of additional indications in emergency department (ED) and inpatient settings, most aimed at detecting or confirming GI blood loss. This may lead to inappropriate patient management, including the receipt of unnecessary follow-up procedures, which can incur significant costs to the patient and the health system.^13-19 These costs may be particularly burdensome in safety net health systems (ie, those that offer access to care regardless of the patient’s ability to pay), which serve a large proportion of socioeconomically disadvantaged individuals in the United States.^20,21 To our knowledge, no published study to date has specifically investigated the role of FIT in acute patient management.

This study characterizes the use of FIT in acute patient care within a large, urban, safety net health care system. Through a retrospective review of administrative data and patient charts, we evaluated FIT use prevalence, indications, and patient outcomes in the ED and inpatient settings.

Methods

Setting

This study was conducted in a large, urban, county-based integrated delivery system in Houston, Texas, that provides health care services to one of the largest uninsured and underinsured populations in the country.²² The health system includes 2 main hospitals and more than 20 ambulatory care clinics. Within its ambulatory care clinics, the health system implements a population-based screening strategy using stool-based testing. All adults aged 50 years or older who are due for FIT are identified through the health-maintenance module of the electronic medical record (EMR) and offered a take-home FIT. The health system utilizes FIT exclusively (OC-Light S FIT, Polymedco, Cortlandt Manor, NY); no guaiac-based assays are available.

Design and Data Collection

We began by using administrative records to determine the proportion of FITs conducted health system-wide that were ordered and completed in the acute care setting over the study period (August 2016-March 2017). Specifically, we used aggregate quality metric reports, which quantify the number of FITs conducted at each health system clinic and hospital each month, to calculate the proportion of FITs done in the ED and inpatient hospital setting.

We then conducted a retrospective cohort study of 382 adult patients who received FIT in the EDs and inpatient wards in both of the health system’s hospitals over the study period. All data were collected by retrospective chart review in Epic (Madison, WI) EMRs. Sampling was performed by selecting the medical record numbers corresponding to the first 50 completed FITs chronologically each month over the 8-month period, with a total of 400 charts reviewed.

Data collected included basic patient demographics, location of FIT ordering (ED vs inpatient), primary service ordering FIT, FIT indication, FIT result, and receipt and results of invasive diagnostic follow-up. Demographics collected included age, biological sex, race (self-selected), and insurance coverage.

FIT indication was determined based on resident or attending physician notes. The history of present illness, physical exam, and assessment and plan section of notes were reviewed by the lead author for a specific statement of indication for FIT or for evidence of clinical presentation for which FIT could reasonably be ordered. Indications were iteratively reviewed and collapsed into 6 different categories: anemia, iron deficiency with or without anemia, overt GIB, suspected GIB/miscellaneous, non-bloody diarrhea, and no indication identified. Overt GIB was defined as reported or witnessed hematemesis, coffee-ground emesis, hematochezia, bright red blood per rectum, or melena irrespective of time frame (current or remote) or chronicity (acute, subacute, or chronic). In cases where signs of overt bleed were not witnessed by medical professionals, determination of conditions such as melena or coffee-ground emesis were made based on health care providers’ assessment of patient history as documented in his or her notes. Suspected GIB/miscellaneous was defined with the following parameters: any new drop in hemoglobin, abdominal pain, anorectal pain, non-bloody vomiting, hemoptysis, isolated rising blood urea nitrogen, or patient noticing blood on self, clothing, or in the commode without an identified source. Patients who were anemic and found to have iron deficiency on recent lab studies (within 6 months) were reflexively categorized into iron deficiency with or without anemia as opposed to the “anemia” category, which was comprised of any anemia without recent iron studies or non-iron deficient anemia. FIT result was determined by test result entry in Epic, with results either reading positive or negative.

Diagnostic follow-up, for our purposes, was defined as receipt of an invasive procedure or surgery, including esophagogastroduodenoscopy (EGD), colonoscopy, flexible sigmoidoscopy, diagnostic and/or therapeutic abdominal surgical intervention, or any combination of these. Results of diagnostic follow-up were coded as normal or abnormal. A normal result was determined if all procedures performed were listed as normal or as “no pathological findings” on the operative or endoscopic report. Any reported pathologic findings on the operative/endoscopic report were coded as abnormal.

Statistical Analysis

Proportions were used to describe demographic characteristics of patients who received a FIT in acute hospital settings. Bivariable tables and Chi-square tests were used to compare indications and outcomes for FIT-positive and FIT-negative patients. The association between receipt of an invasive diagnostic follow-up (outcome) and the results of an inpatient FIT (predictor) was assessed using multivariable log-binomial regression to calculate risk ratios (RRs) and corresponding 95% confidence intervals. Log-binomial regression was used over logistic regression given that adjusted odds ratios generated by logistic regression often overestimate the association between the risk factor and the outcome when the outcome is common,²³ as in the case of diagnostic follow-up. The model was adjusted for variables selected a priori, specifically, age, gender, and FIT indication. Chi-square analysis was used to compare the proportion of abnormal findings on diagnostic follow-up by FIT result (negative vs positive).

Results

During the 8-month study period, there were 2718 FITs ordered and completed in the acute care setting, compared to 44,662 FITs ordered and completed in the outpatient setting (5.7% performed during acute care).

Among the 400 charts reviewed, 7 were excluded from the analysis because they were duplicates from the same patient, and 11 were excluded due to insufficient information in the patient’s medical record, resulting in 382 patients included in the analysis. Patient demographic characteristics are described in Table 1. Patients were predominantly Hispanic/Latino or Black/African American (51.0% and 32.5%, respectively), a majority had insurance through the county health system (50.5%), and most were male (58.1%). The average age of those receiving FIT was 52 years (standard deviation, 14.8 years), with 40.8% being under the age of 50. For a majority of patients, FIT was ordered in the ED by emergency medicine providers (79.8%). The remaining FITs were ordered by providers in 12 different inpatient departments. Of the FITs ordered, 35.1% were positive.

Indications for ordering FIT are listed in Table 2. The largest proportion of FITs were ordered for overt signs of GIB (54.2%), followed by anemia (22.0%), suspected GIB/miscellaneous reasons (12.3%), iron deficiency with or without anemia (7.6%), and non-bloody diarrhea (2.1%). In 1.8% of cases, no indication for FIT was found in the EMR. No FITs were ordered for the indication of CRC detection. Of these indication categories, overt GIB yielded the highest percentage of FIT positive results (44.0%), and non-bloody diarrhea yielded the lowest (0%).

A total of 110 patients (28.7%) underwent FIT and received invasive diagnostic follow-up. Of these 110 patients, 57 (51.8%) underwent EGD (2 of whom had further surgical intervention), 21 (19.1%) underwent colonoscopy (1 of whom had further surgical intervention), 25 (22.7%) underwent dual EGD and colonoscopy, 1 (0.9%) underwent flexible sigmoidoscopy, and 6 (5.5%) directly underwent abdominal surgical intervention. There was a significantly higher rate of diagnostic follow-up for FIT-positive vs FIT-negative patients (42.9% vs 21.3%; P < 0.001). However, of the 110 patients who underwent subsequent diagnostic follow-up, 48.2% were FIT negative. FIT-negative patients who received diagnostic follow-up were just as likely to have an abnormal finding as FIT-positive patients (90.6% vs 91.2%; P = 0.86).

Of the 382 patients in the study, 4 were diagnosed with CRC through diagnostic follow-up (1.0%). Of those 4 patients, 1 was FIT positive.

The results of the multivariable analyses to evaluate predictors of diagnostic colonoscopy are described in Table 3. Variables in the final model were FITresult, age, and FIT indication. After adjusting for other variables in the model, receipt of diagnostic follow-up was significantly associated with having a positive FIT (adjusted RR, 1.72; P < 0.001) and an overt GIB as an indication (adjusted RR, 2.00; P < 0.01).

Discussion

During the time frame of our study, 5.7% of all FITs ordered within our health system were ordered in the acute patient care setting at our hospitals. The most common indication was overt GIB, which was the indication for 54.2% of patients. Of note, none of the FITs ordered in the acute patient care setting were ordered for CRC screening. These findings support the evidence in the literature that stool-based screening tests, including FIT, are commonly used in US health care systems for diagnostic purposes and risk stratification in acute patient care to detect GIBs.^13-18

Our data suggest that FIT was not a clinically useful test in determining a patient’s need for diagnostic follow-up. While having a positive FIT was significantly associated with obtaining a diagnostic exam in multivariate analysis (RR, 1.72), having signs of overt GI bleeding was a stronger predictor of diagnostic follow-up (RR, 2.00). This salient finding is evidence that a thorough clinical history and physical exam may more strongly predict whether a patient will undergo endoscopy or other follow-up than a FIT result. These findings support other studies in the literature that have called into question the utility of FOBTs in these acute settings.^13-19 Under such circumstances, FOBTs have been shown to rarely influence patient management and thus represent an unnecessary expense.^13–17 Additionally, in some cases, FOBT use in these settings may negatively affect patient outcomes. Such adverse effects include delaying treatment until results are returned or obfuscating indicated management with the results (eg, a patient with indications for colonoscopy not being referred due to a negative FOBT).^13,14,17

We found that, for patients who subsequently went on to have diagnostic follow-up (most commonly endoscopy), there was no difference in the likelihood of FIT-positive and FIT-negative patients to have an abnormality discovered (91.2% vs 90.6%; P = 0.86). This analysis demonstrates no post-hoc support for FIT positivity as a predictor of presence of pathology in patients who were discriminately selected for diagnostic follow-up on clinical grounds by gastroenterologists and surgeons. It does, however, further support that clinical judgment about the need for diagnostic follow-up—irrespective of FIT result—has a very high yield for discovery of pathology in the acute setting.

There are multiple reasons why FOBTs, and specifically FIT, contribute little in management decisions for patients with suspected GI blood loss. Use of FIT raises concern for both false-negatives and false-positives when used outside of its indication. Regarding false- negatives, FIT is an unreliable test for detection of blood loss from the upper GI tract. As FITs utilize antibodies to detect the presence of globin, a byproduct of red blood cell breakdown, it is expected that FIT would fail to detect many cases of upper GI bleeding, as globin is broken down in the upper GI tract.²⁴ This fact is part of what has made FIT a more effective CRC screening test than its guaiac-based counterparts—it has greater specificity for lower GI tract blood loss compared to tests relying on detection of heme.⁸ While guaiac-based assays like Hemoccult have also been shown to be poor tests in acute patient care, they may more frequently, though still unreliably, detect blood of upper GI origin. We believe that part of the ongoing use of FIT in patients with a suspected upper GIB may be from lack of understanding among providers on the mechanistic difference between gFOBTs and FITs, even though gFOBTs also yield highly unreliable results.

FIT does not have the same risk of false-positive results that guaiac-based tests have, which can yield positive results with extra-intestinal blood ingestion, aspirin, or alcohol use; insignificant GI bleeding; and consumption of peroxidase-containing foods.^13,17,25 However, from a clinical standpoint, there are several scenarios of insignificant bleeding that would yield a positive FIT result, such as hemorrhoids, which are common in the US population.^26,27 Additionally, in the ED, where most FITs were performed in our study, it is possible that samples for FITs are being obtained via digital rectal exam (DRE) given patients’ acuity of medical conditions and time constraints. However, FIT has been validated when using a formed stool sample. Obtaining FIT via DRE may lead to microtrauma to the rectum, which could hypothetically yield a positive FIT.

Strengths of this study include its use of in-depth chart data on a large number of FIT-positive patients, which allowed us to discern indications, outcomes, and other clinical data that may have influenced clinical decision-making. Additionally, whereas other studies that address FOBT use in acute patient care have focused on guaiac-based assays, our findings regarding the lack of utility of FIT are novel and have particular relevance as FITs continue to grow in popularity. Nonetheless, there are certain limitations future research should seek to address. In this study, the diagnostic follow-up result was coded by presence or absence of pathologic findings but did not qualify findings by severity or attempt to determine whether the pathology noted on diagnostic follow-up was the definitive source of the suspected GI bleed. These variables could help determine whether there was a difference in severity of bleeding between FIT-positive and FIT-negative patients and could potentially be studied with a prospective research design. Our own study was not designed to address the question of whether FIT result informs patient management decisions. To answer this directly, interviews would have to be conducted with those making the follow-up decision (ie, endoscopists and surgeons). Additionally, this study was not adequately powered to make determinations on the efficacy of FIT in the acute care setting for detection of CRC. As mentioned, only 1 of the 4 patients (25%) who went on to be diagnosed with CRC on follow-up was initially FIT-positive. This would require further investigation.

Conclusion

FIT is being utilized for diagnostic purposes in the acute care of symptomatic patients, which is a misuse of an established screening test for CRC. While our study was not designed to answer whether and how often a FIT result informs subsequent patient management, our results indicate that FIT is an ineffective diagnostic and risk-stratification tool when used in the acute care setting. Our findings add to existing evidence that indicates FOBTs should not be used in acute patient care.

Taken as a whole, the results of our study add to a growing body of evidence demonstrating no role for FOBTs, and specifically FIT, in acute patient care. In light of this evidence, some health care systems have already demonstrated success with system-wide disinvestment from the test in acute patient care settings, with one group publishing about their disinvestment process.²⁸ After completion of our study, our preliminary data were presented to leadership from the internal medicine, emergency medicine, and laboratory divisions within our health care delivery system to galvanize complete disinvestment of FIT from acute care at our hospitals, a policy that was put into effect in July 2019.

Corresponding author: Nathaniel J. Spezia-Lindner, MD, Baylor College of Medicine, 7200 Cambridge St, BCM 903, Ste A10.197, Houston, TX 77030; speziali@bcm.edu.

Financial disclosures: None.

Funding: Cancer Prevention and Research Institute of Texas, CPRIT (PP170094, PDs: ML Jibaja-Weiss and JR Montealegre).

From The Johns Hopkins Hospital, Baltimore, MD (Dr. Muganlinskaya and Dr. Skojec, retired); The George Washington University, Washington, DC (Dr. Posey); and Johns Hopkins University, Baltimore, MD (Dr. Resar).

Abstract

Objective: Assessing the risk characteristics of patients with acute myocardial infarction (MI) can help providers make appropriate referral decisions. This quality improvement project sought to improve timely, appropriate referrals among patients with type I MI by adding a risk assessment, the AMI READMITS score, to the existing referral protocol.

Methods: Patients’ chart data were analyzed to assess changes in referrals and timely follow-up appointments from pre-intervention to intervention. A survey assessed providers’ satisfaction with the new referral protocol.

Results: Among 57 patients (n = 29 preintervention; n = 28 intervention), documented referrals increased significantly from 66% to 89% (χ2 = 4.571, df = 1, P = 0.033); and timely appointments increased by 10%, which was not significant (χ2 = 3.550, df = 2, P = 0.169). Most providers agreed that the new protocol was easy to use, useful in making referral decisions, and improved the referral process. All agreed the risk score should be incorporated into electronic clinical notes. Provider opinions related to implementing the risk score in clinical practice were mixed. Qualitative feedback suggests this was due to limited validation of the AMI READMITS score in reducing readmissions.

Conclusions: Our risk-based referral protocol helped to increase appropriate referrals among patients with type I MI. Provider adoption may be enhanced by incorporating the protocol into electronic clinical notes. Research to further validate the accuracy of the AMI READMITS score in predicting readmissions may support adoption of the protocol in clinical practice.

Keywords: quality improvement; type I myocardial infarction; referral process; readmission risk; risk assessment; chart review.

Early follow-up after discharge is an important strategy to reduce the risk of unplanned hospital readmissions among patients with various conditions.^1-3 While patient confounding factors, such as chronic health problems, environment, socioeconomic status, and literacy, make it difficult to avoid all unplanned readmissions, early follow-up may help providers identify and appropriately manage some health-related issues, and as such is a pivotal element of a readmission prevention strategy.⁴ There is evidence that patients with non-ST elevation myocardial infarction (NSTEMI) who have an outpatient appointment with a physician within 7 days after discharge have a lower risk of 30-day readmission.⁵

Our hospital’s postmyocardial infarction clinic was created to prevent unplanned readmissions within 30 days after discharge among patients with type I myocardial infarction (MI). Since inception, the number of referrals has been much lower than expected. In 2018, the total number of patients discharged from the hospital with type I MI and any troponin I level above 0.40 ng/mL was 313. Most of these patients were discharged from the hospital’s cardiac units; however, only 91 referrals were made. To increase referrals, the cardiology nurse practitioners (NPs) developed a post-MI referral protocol (Figure 1). However, this protocol was not consistently used and referrals to the clinic remained low.

Evidence-based risk assessment tools have the potential to increase effective patient management. For example, cardiology providers at the hospital utilize various scores, such as CHA₂DS₂-VASc⁶ and the Society of Thoracic Surgery risk score,⁷ to plan patient management. Among the scores used to predict unplanned readmissions for MI patients, the most promising is the AMI READMITS score.⁸ Unlike other nonspecific prediction models, the AMI READMITS score was developed based on variables extracted from the electronic health records (EHRs) of patients who were hospitalized for MI and readmitted within 30 days after discharge. Recognizing the potential to increase referrals by integrating an MI-specific risk assessment, this quality improvement study modified the existing referral protocol to include the patients’ AMI READMITS score and recommendations for follow-up.

Currently, there are no clear recommendations on how soon after discharge patients with MI should undergo follow-up. As research data vary, we selected 7 days follow-up for patients from high risk groups based on the “See you in 7” initiative for patients with heart failure (HF) and MI,^9,10 as well as evidence that patients with NSTEMI have a lower risk of 30-day readmission if they have follow-up within 7 days after discharge⁵; and we selected 14 days follow-up for patients from low-risk groups based on evidence that postdischarge follow-up within 14 days reduces risk of 30-day readmission in patients with acute myocardial infarction (AMI) and/or acutely decompensated HF.¹¹

Methods

This project was designed to answer the following question: For adult patients with type I MI, does implementation of a readmission risk assessment referral protocol increase the percentage of referrals and appointments scheduled within a recommended time? Anticipated outcomes included: (1) increased referrals to a cardiologist or the post-MI clinic; (2) increased scheduled follow-up appointments within 7 to 14 days; (3) provider satisfaction with the usability and usefulness of the new protocol; and (4) consistent provider adoption of the new risk assessment referral protocol.

To evaluate the degree to which these outcomes were achieved, we reviewed patient charts for 2 months prior and 2 months during implementation of the new referral protocol. As shown in Figure 2, the new protocol added the following process steps to the existing protocol: calculation of the AMI READMITS score, recommendations for follow-up based on patients’ risk score, and guidance to refer patients to the post-MI clinic if patients did not have an appointment with a cardiologist within 7 to 14 days after discharge. Patients’ risk assessment scores were obtained from forms completed by clinicians during the intervention. Clinician’s perceptions related to the usability and usefulness of the new protocol and feedback related to its long-term adoption were assessed using a descriptive survey.

Population

The project population included all adult patients (≥ 18 years old) with type I MI who were admitted or transferred to the hospital, had a percutaneous coronary intervention (PCI), or were managed without PCI and discharged from the hospital’s cardiac care unit (CCU) and progressive cardiac care unit (PCCU). The criteria for type I MI included the “detection of a rise and/or fall of cardiac troponin with at least 1 value above the 99th percentile and with at least 1 of the following: symptoms of acute myocardial ischemia; new ischemic electrocardiographic (ECG) changes; development of new pathological Q waves; imaging evidence of new loss of viable myocardium or new regional wall motion abnormality in a pattern consistent with an ischemic etiology; identification of a coronary thrombus by angiography including intracoronary imaging or by autopsy.”¹² The study excluded patients with type I MI who were referred for coronary bypass surgery.

Intervention

The revised risk assessment protocol was implemented within the CCU and PCCU. The lead investigator met with each provider to discuss the role of the post-MI clinic, current referral rates, the purpose of the project, and the new referral process to be completed during the project for each patient discharged with type I MI. Cardiology NPs, fellows, and residents were asked to use the risk-assessment form to calculate patients’ risk for readmission, and refer patients to the post-MI clinic if an appointment with a cardiologist was not available within 7 to 14 days after discharge. Every week during the intervention phase, the investigator sent reminder emails to ensure form completion. Providers were asked to calculate and write the score, the discharge and referral dates, where referrals were made (a cardiologist or the post-MI clinic), date of appointment, and reason for not scheduling an appointment or not referring on the risk assessment form, and to drop the completed forms in specific labeled boxes located at the CCU and PCCU work stations. The investigator collected the completed forms weekly. When the number of discharged patients did not match the number of completed forms, the investigator followed up with discharging providers to understand why.

Data and Data Collection

Data to determine whether the use of the new protocol increased discharge referrals among patients with type I MI within the recommended timeframes were collected by electronic chart review. Data included discharging unit, patients’ age, gender, admission and discharge date, diagnosis, referral to a cardiologist and the post-MI clinic, and appointment date. Clinical data needed to calculate the AMI READMITS score was also collected: PCI within 24 hours, serum creatinine, systolic blood pressure (SBP), brain natriuretic peptide (BNP), and diabetes status.

Data to assess provider satisfaction with the usability and usefulness of the new protocol were gathered through an online survey. The survey included 1 question related to the providers’ role, 1 question asking whether they used the risk assessment for each patient, and 5 Likert-items assessing the ease of usage. An additional open-ended question asked providers to share feedback related to integrating the AMI READMITS risk assessment score to the post-MI referral protocol long term.

To evaluate how consistently providers utilized the new referral protocol when discharging patients with type I MI, the number of completed forms was compared with the number of those patients who were discharged.

Statistical Analysis

Descriptive statistics were used to summarize patient demographics and to calculate the frequency of referrals before and during the intervention. Chi-square statistics were calculated to determine whether the change in percentage of referrals and timely referrals was significant. Descriptive statistics were used to determine the level of provider satisfaction related to each survey item. A content analysis method was used to synthesize themes from the open-ended question asking clinicians to share their feedback related to the new protocol.

Results

Fifty-seven patients met the study inclusion criteria: 29 patients during the preintervention phase and 28 patients during the intervention phase. There were 35 male (61.4%) and 22 female (38.6%) patients. Twenty-five patients (43.9%) were from age groups 41 through 60 years and 61 through 80 years, respectively, representing the majority of included patients. Seven patients (12.3%) were from the 81 years and older age group. There were no patients in the age group 18 through 40 years. Based on the AMI READMITS score calculation, 57.9% (n = 33) patients were from a low-risk group (includes extremely low and low risk for readmission) and 42.1% (n = 24) were from a high-risk group (includes moderate, high, and extremely high risk for readmission).

Provider adoption of the new protocol during the intervention was high. Referral forms were completed for 82% (n = 23) of the 28 patients during the intervention. Analysis findings showed a statistically significant increase in documented referrals after implementing the new referral protocol. During the preintervention phase, 66% (n = 19) of patients with type I MI were referred to see a cardiologist or an NP at a post-MI clinic and there was no documented referral for 34% (n = 10) of patients. During the intervention phase, 89% (n = 25) of patients were referred and there was no documented referral for 11% (n = 3) of patients. Chi-square results indicated that the increase in referrals was significant (χ2 = 4.571, df = 1, P = 0.033).

Data analysis examined whether patient referrals fell within the recommended timeframe of 7 days for the high-risk group (included moderate-to-extremely high risk) and 14 days for the low-risk group (included low-to-extremely low risk). During the preintervention phase, 31% (n = 9) of patient referrals were scheduled as recommended; 28% (n = 8) of patient referrals were scheduled but delayed; and there was no referral date documented for 41% (n = 12) of patients. During the intervention phase, referrals scheduled as recommended increased to 53% (n = 15); 25% (n = 7) of referrals were scheduled but delayed; and there was no referral date documented for 21.4% (n = 6) of patients. The change in appointments scheduled as recommended was not significant (χ2 = 3.550, df = 2, P = 0.169).

Surveys were emailed to 25 cardiology fellows and 3 cardiology NPs who participated in this study. Eighteen of the 28 clinicians (15 cardiology fellows and 3 cardiology NPs) responded for a response rate of 64%. One of several residents who rotated through the CCU and PCCU during the intervention also completed the survey, for a total of 19 participants. When asked if the protocol was easy to use, 79% agreed or strongly agreed. Eighteen of the 19 participants (95%) agreed or strongly agreed that the protocol was useful in making referral decisions. Sixty-eight percent agreed or strongly agreed that the AMI READMITS risk assessment score improves referral process. All participants agreed or strongly agreed that there should be an option to incorporate the AMI READMITS risk assessment score into electronic clinical notes. When asked whether the AMI READMITS risk score should be implemented in clinical practice, responses were mixed (Figure 3). A common theme among the 4 participants who responded with comments was the need for additional data to validate the usefulness of the AMI READMITS to reduce readmissions. In addition, 1 participant commented that “manual calculation [of the risk score] is not ideal.”

Discussion

This project demonstrated that implementing an evidence-based referral protocol integrating the AMI-READMITS score can increase timely postdischarge referrals among patients with type I MI. The percentage of appropriately scheduled appointments increased during the intervention phase; however, a relatively high number of appointments were scheduled outside of the recommended timeframe, similar to preintervention. Thus, while the new protocol increased referrals and provider documentation of these referrals, it appears that challenges in scheduling timely referral appointments remained. This project did not examine the reasons for delayed appointments.

The survey findings indicated that providers were generally satisfied with the usability and usefulness of the new risk assessment protocol. A large majority agreed or strongly agreed that it was easy to use and useful in making referral decisions, and most agreed or strongly agreed that it improves the referral process. Mixed opinions regarding implementing the AMI READMITS score in clinical practice, combined with qualitative findings, suggest that a lack of external validation of the AMI READMITS presents a barrier to its long-term adoption. All providers who participated in the survey agreed or strongly agreed that the risk assessment should be incorporated into electronic clinical notes. We have begun the process of working with the EHR vendor to automate the AMI risk-assessment within the referral work-flow, which will provide an opportunity for a follow-up quality improvement study.

This quality improvement project has several limitations. First, it implemented a small change in 2 inpatient units at 1 hospital using a simple pre- posttest design. Therefore, the findings are not generalizable to other settings. Prior to the intervention, some referrals may have been made without documentation. While the authors were able to trace undocumented referrals for patients who were referred to the post-MI clinic or to a cardiologist affiliated with the hospital, some patients may have been referred to cardiologists who were not affiliated with the hospital. Another limitation was that the self-created provider survey used was not tested in other clinical settings; thus, it cannot be determined whether the sensitivity and specificity of the survey questions are high. In addition, the clinical providers who participated in the study knew the study team, which may have influenced their behavior during the study period. Furthermore, the identified improvement in clinicians’ referral practices may not be sustainable due to the complexity and effort required to manually calculate the risk score. This limitation could be eliminated by integrating the risk score calculation into the EHR.

Conclusion

Early follow-up after discharge plays an important role in supporting patients’ self-management of some risk factors (ie, diet, weight, and smoking) and identifying gaps in postdischarge care which may lead to readmission. This project provides evidence that integrating the AMI READMITS risk assessment score into the referral process can help to guide discharge decision-making and increase timely, appropriate referrals for patients with MI. Integration of a specific risk assessment, such as the AMI READMITS, within the post-MI referral protocol may help clinicians make more efficient, educated referral decisions. Future studies should explore more specifically how and why the new protocol impacts clinicians’ decision-making and behavior related to post-MI referrals. In addition, future studies should investigate challenges associated with scheduling postdischarge appointments. It will be important to investigate how integration of the new protocol within the EHR may increase efficiency, consistency, and provider satisfaction with the new referral process. Additional research investigating the effects of the AMI READMITS score on readmissions reduction will be important to promote long-term adoption of the improved referral protocol in clinical practice.

Acknowledgments: The authors thank Shelly Conaway, ANP-BC, MSN, Angela Street, ANP-BC, MSN, Andrew Geis, ACNP-BC, MSN, Richard P. Jones II, MD, Eunice Young, MD, Joy Rothwell, MSN, RN-BC, Allison Olazo, MBA, MSN, RN-BC, Elizabeth Heck, RN-BC, and Matthew Trojanowski, MHA, MS, RRT, CSSBB for their support of this study.

Corresponding author: Nailya Muganlinskaya, DNP, MPH, ACNP-BC, MSN, The Johns Hopkins Hospital, 1800 Orleans St, Baltimore, MD 21287; nmuganl1@jhmi.edu.

Financial disclosures: None.

From The Johns Hopkins Hospital, Baltimore, MD (Dr. Muganlinskaya and Dr. Skojec, retired); The George Washington University, Washington, DC (Dr. Posey); and Johns Hopkins University, Baltimore, MD (Dr. Resar).

Abstract

Objective: Assessing the risk characteristics of patients with acute myocardial infarction (MI) can help providers make appropriate referral decisions. This quality improvement project sought to improve timely, appropriate referrals among patients with type I MI by adding a risk assessment, the AMI READMITS score, to the existing referral protocol.

Methods: Patients’ chart data were analyzed to assess changes in referrals and timely follow-up appointments from pre-intervention to intervention. A survey assessed providers’ satisfaction with the new referral protocol.

Results: Among 57 patients (n = 29 preintervention; n = 28 intervention), documented referrals increased significantly from 66% to 89% (χ2 = 4.571, df = 1, P = 0.033); and timely appointments increased by 10%, which was not significant (χ2 = 3.550, df = 2, P = 0.169). Most providers agreed that the new protocol was easy to use, useful in making referral decisions, and improved the referral process. All agreed the risk score should be incorporated into electronic clinical notes. Provider opinions related to implementing the risk score in clinical practice were mixed. Qualitative feedback suggests this was due to limited validation of the AMI READMITS score in reducing readmissions.

Conclusions: Our risk-based referral protocol helped to increase appropriate referrals among patients with type I MI. Provider adoption may be enhanced by incorporating the protocol into electronic clinical notes. Research to further validate the accuracy of the AMI READMITS score in predicting readmissions may support adoption of the protocol in clinical practice.

Keywords: quality improvement; type I myocardial infarction; referral process; readmission risk; risk assessment; chart review.

Early follow-up after discharge is an important strategy to reduce the risk of unplanned hospital readmissions among patients with various conditions.^1-3 While patient confounding factors, such as chronic health problems, environment, socioeconomic status, and literacy, make it difficult to avoid all unplanned readmissions, early follow-up may help providers identify and appropriately manage some health-related issues, and as such is a pivotal element of a readmission prevention strategy.⁴ There is evidence that patients with non-ST elevation myocardial infarction (NSTEMI) who have an outpatient appointment with a physician within 7 days after discharge have a lower risk of 30-day readmission.⁵

Our hospital’s postmyocardial infarction clinic was created to prevent unplanned readmissions within 30 days after discharge among patients with type I myocardial infarction (MI). Since inception, the number of referrals has been much lower than expected. In 2018, the total number of patients discharged from the hospital with type I MI and any troponin I level above 0.40 ng/mL was 313. Most of these patients were discharged from the hospital’s cardiac units; however, only 91 referrals were made. To increase referrals, the cardiology nurse practitioners (NPs) developed a post-MI referral protocol (Figure 1). However, this protocol was not consistently used and referrals to the clinic remained low.

Evidence-based risk assessment tools have the potential to increase effective patient management. For example, cardiology providers at the hospital utilize various scores, such as CHA₂DS₂-VASc⁶ and the Society of Thoracic Surgery risk score,⁷ to plan patient management. Among the scores used to predict unplanned readmissions for MI patients, the most promising is the AMI READMITS score.⁸ Unlike other nonspecific prediction models, the AMI READMITS score was developed based on variables extracted from the electronic health records (EHRs) of patients who were hospitalized for MI and readmitted within 30 days after discharge. Recognizing the potential to increase referrals by integrating an MI-specific risk assessment, this quality improvement study modified the existing referral protocol to include the patients’ AMI READMITS score and recommendations for follow-up.

Currently, there are no clear recommendations on how soon after discharge patients with MI should undergo follow-up. As research data vary, we selected 7 days follow-up for patients from high risk groups based on the “See you in 7” initiative for patients with heart failure (HF) and MI,^9,10 as well as evidence that patients with NSTEMI have a lower risk of 30-day readmission if they have follow-up within 7 days after discharge⁵; and we selected 14 days follow-up for patients from low-risk groups based on evidence that postdischarge follow-up within 14 days reduces risk of 30-day readmission in patients with acute myocardial infarction (AMI) and/or acutely decompensated HF.¹¹

Methods

This project was designed to answer the following question: For adult patients with type I MI, does implementation of a readmission risk assessment referral protocol increase the percentage of referrals and appointments scheduled within a recommended time? Anticipated outcomes included: (1) increased referrals to a cardiologist or the post-MI clinic; (2) increased scheduled follow-up appointments within 7 to 14 days; (3) provider satisfaction with the usability and usefulness of the new protocol; and (4) consistent provider adoption of the new risk assessment referral protocol.

To evaluate the degree to which these outcomes were achieved, we reviewed patient charts for 2 months prior and 2 months during implementation of the new referral protocol. As shown in Figure 2, the new protocol added the following process steps to the existing protocol: calculation of the AMI READMITS score, recommendations for follow-up based on patients’ risk score, and guidance to refer patients to the post-MI clinic if patients did not have an appointment with a cardiologist within 7 to 14 days after discharge. Patients’ risk assessment scores were obtained from forms completed by clinicians during the intervention. Clinician’s perceptions related to the usability and usefulness of the new protocol and feedback related to its long-term adoption were assessed using a descriptive survey.

Population

The project population included all adult patients (≥ 18 years old) with type I MI who were admitted or transferred to the hospital, had a percutaneous coronary intervention (PCI), or were managed without PCI and discharged from the hospital’s cardiac care unit (CCU) and progressive cardiac care unit (PCCU). The criteria for type I MI included the “detection of a rise and/or fall of cardiac troponin with at least 1 value above the 99th percentile and with at least 1 of the following: symptoms of acute myocardial ischemia; new ischemic electrocardiographic (ECG) changes; development of new pathological Q waves; imaging evidence of new loss of viable myocardium or new regional wall motion abnormality in a pattern consistent with an ischemic etiology; identification of a coronary thrombus by angiography including intracoronary imaging or by autopsy.”¹² The study excluded patients with type I MI who were referred for coronary bypass surgery.

Intervention

The revised risk assessment protocol was implemented within the CCU and PCCU. The lead investigator met with each provider to discuss the role of the post-MI clinic, current referral rates, the purpose of the project, and the new referral process to be completed during the project for each patient discharged with type I MI. Cardiology NPs, fellows, and residents were asked to use the risk-assessment form to calculate patients’ risk for readmission, and refer patients to the post-MI clinic if an appointment with a cardiologist was not available within 7 to 14 days after discharge. Every week during the intervention phase, the investigator sent reminder emails to ensure form completion. Providers were asked to calculate and write the score, the discharge and referral dates, where referrals were made (a cardiologist or the post-MI clinic), date of appointment, and reason for not scheduling an appointment or not referring on the risk assessment form, and to drop the completed forms in specific labeled boxes located at the CCU and PCCU work stations. The investigator collected the completed forms weekly. When the number of discharged patients did not match the number of completed forms, the investigator followed up with discharging providers to understand why.

Data and Data Collection

Data to determine whether the use of the new protocol increased discharge referrals among patients with type I MI within the recommended timeframes were collected by electronic chart review. Data included discharging unit, patients’ age, gender, admission and discharge date, diagnosis, referral to a cardiologist and the post-MI clinic, and appointment date. Clinical data needed to calculate the AMI READMITS score was also collected: PCI within 24 hours, serum creatinine, systolic blood pressure (SBP), brain natriuretic peptide (BNP), and diabetes status.

Data to assess provider satisfaction with the usability and usefulness of the new protocol were gathered through an online survey. The survey included 1 question related to the providers’ role, 1 question asking whether they used the risk assessment for each patient, and 5 Likert-items assessing the ease of usage. An additional open-ended question asked providers to share feedback related to integrating the AMI READMITS risk assessment score to the post-MI referral protocol long term.

To evaluate how consistently providers utilized the new referral protocol when discharging patients with type I MI, the number of completed forms was compared with the number of those patients who were discharged.

Statistical Analysis

Descriptive statistics were used to summarize patient demographics and to calculate the frequency of referrals before and during the intervention. Chi-square statistics were calculated to determine whether the change in percentage of referrals and timely referrals was significant. Descriptive statistics were used to determine the level of provider satisfaction related to each survey item. A content analysis method was used to synthesize themes from the open-ended question asking clinicians to share their feedback related to the new protocol.

Results

Fifty-seven patients met the study inclusion criteria: 29 patients during the preintervention phase and 28 patients during the intervention phase. There were 35 male (61.4%) and 22 female (38.6%) patients. Twenty-five patients (43.9%) were from age groups 41 through 60 years and 61 through 80 years, respectively, representing the majority of included patients. Seven patients (12.3%) were from the 81 years and older age group. There were no patients in the age group 18 through 40 years. Based on the AMI READMITS score calculation, 57.9% (n = 33) patients were from a low-risk group (includes extremely low and low risk for readmission) and 42.1% (n = 24) were from a high-risk group (includes moderate, high, and extremely high risk for readmission).

Provider adoption of the new protocol during the intervention was high. Referral forms were completed for 82% (n = 23) of the 28 patients during the intervention. Analysis findings showed a statistically significant increase in documented referrals after implementing the new referral protocol. During the preintervention phase, 66% (n = 19) of patients with type I MI were referred to see a cardiologist or an NP at a post-MI clinic and there was no documented referral for 34% (n = 10) of patients. During the intervention phase, 89% (n = 25) of patients were referred and there was no documented referral for 11% (n = 3) of patients. Chi-square results indicated that the increase in referrals was significant (χ2 = 4.571, df = 1, P = 0.033).

Data analysis examined whether patient referrals fell within the recommended timeframe of 7 days for the high-risk group (included moderate-to-extremely high risk) and 14 days for the low-risk group (included low-to-extremely low risk). During the preintervention phase, 31% (n = 9) of patient referrals were scheduled as recommended; 28% (n = 8) of patient referrals were scheduled but delayed; and there was no referral date documented for 41% (n = 12) of patients. During the intervention phase, referrals scheduled as recommended increased to 53% (n = 15); 25% (n = 7) of referrals were scheduled but delayed; and there was no referral date documented for 21.4% (n = 6) of patients. The change in appointments scheduled as recommended was not significant (χ2 = 3.550, df = 2, P = 0.169).

Surveys were emailed to 25 cardiology fellows and 3 cardiology NPs who participated in this study. Eighteen of the 28 clinicians (15 cardiology fellows and 3 cardiology NPs) responded for a response rate of 64%. One of several residents who rotated through the CCU and PCCU during the intervention also completed the survey, for a total of 19 participants. When asked if the protocol was easy to use, 79% agreed or strongly agreed. Eighteen of the 19 participants (95%) agreed or strongly agreed that the protocol was useful in making referral decisions. Sixty-eight percent agreed or strongly agreed that the AMI READMITS risk assessment score improves referral process. All participants agreed or strongly agreed that there should be an option to incorporate the AMI READMITS risk assessment score into electronic clinical notes. When asked whether the AMI READMITS risk score should be implemented in clinical practice, responses were mixed (Figure 3). A common theme among the 4 participants who responded with comments was the need for additional data to validate the usefulness of the AMI READMITS to reduce readmissions. In addition, 1 participant commented that “manual calculation [of the risk score] is not ideal.”

Discussion

This project demonstrated that implementing an evidence-based referral protocol integrating the AMI-READMITS score can increase timely postdischarge referrals among patients with type I MI. The percentage of appropriately scheduled appointments increased during the intervention phase; however, a relatively high number of appointments were scheduled outside of the recommended timeframe, similar to preintervention. Thus, while the new protocol increased referrals and provider documentation of these referrals, it appears that challenges in scheduling timely referral appointments remained. This project did not examine the reasons for delayed appointments.

The survey findings indicated that providers were generally satisfied with the usability and usefulness of the new risk assessment protocol. A large majority agreed or strongly agreed that it was easy to use and useful in making referral decisions, and most agreed or strongly agreed that it improves the referral process. Mixed opinions regarding implementing the AMI READMITS score in clinical practice, combined with qualitative findings, suggest that a lack of external validation of the AMI READMITS presents a barrier to its long-term adoption. All providers who participated in the survey agreed or strongly agreed that the risk assessment should be incorporated into electronic clinical notes. We have begun the process of working with the EHR vendor to automate the AMI risk-assessment within the referral work-flow, which will provide an opportunity for a follow-up quality improvement study.

This quality improvement project has several limitations. First, it implemented a small change in 2 inpatient units at 1 hospital using a simple pre- posttest design. Therefore, the findings are not generalizable to other settings. Prior to the intervention, some referrals may have been made without documentation. While the authors were able to trace undocumented referrals for patients who were referred to the post-MI clinic or to a cardiologist affiliated with the hospital, some patients may have been referred to cardiologists who were not affiliated with the hospital. Another limitation was that the self-created provider survey used was not tested in other clinical settings; thus, it cannot be determined whether the sensitivity and specificity of the survey questions are high. In addition, the clinical providers who participated in the study knew the study team, which may have influenced their behavior during the study period. Furthermore, the identified improvement in clinicians’ referral practices may not be sustainable due to the complexity and effort required to manually calculate the risk score. This limitation could be eliminated by integrating the risk score calculation into the EHR.

Conclusion

Early follow-up after discharge plays an important role in supporting patients’ self-management of some risk factors (ie, diet, weight, and smoking) and identifying gaps in postdischarge care which may lead to readmission. This project provides evidence that integrating the AMI READMITS risk assessment score into the referral process can help to guide discharge decision-making and increase timely, appropriate referrals for patients with MI. Integration of a specific risk assessment, such as the AMI READMITS, within the post-MI referral protocol may help clinicians make more efficient, educated referral decisions. Future studies should explore more specifically how and why the new protocol impacts clinicians’ decision-making and behavior related to post-MI referrals. In addition, future studies should investigate challenges associated with scheduling postdischarge appointments. It will be important to investigate how integration of the new protocol within the EHR may increase efficiency, consistency, and provider satisfaction with the new referral process. Additional research investigating the effects of the AMI READMITS score on readmissions reduction will be important to promote long-term adoption of the improved referral protocol in clinical practice.

Acknowledgments: The authors thank Shelly Conaway, ANP-BC, MSN, Angela Street, ANP-BC, MSN, Andrew Geis, ACNP-BC, MSN, Richard P. Jones II, MD, Eunice Young, MD, Joy Rothwell, MSN, RN-BC, Allison Olazo, MBA, MSN, RN-BC, Elizabeth Heck, RN-BC, and Matthew Trojanowski, MHA, MS, RRT, CSSBB for their support of this study.

Corresponding author: Nailya Muganlinskaya, DNP, MPH, ACNP-BC, MSN, The Johns Hopkins Hospital, 1800 Orleans St, Baltimore, MD 21287; nmuganl1@jhmi.edu.

Financial disclosures: None.

From The Johns Hopkins Hospital, Baltimore, MD (Dr. Muganlinskaya and Dr. Skojec, retired); The George Washington University, Washington, DC (Dr. Posey); and Johns Hopkins University, Baltimore, MD (Dr. Resar).

Abstract

Objective: Assessing the risk characteristics of patients with acute myocardial infarction (MI) can help providers make appropriate referral decisions. This quality improvement project sought to improve timely, appropriate referrals among patients with type I MI by adding a risk assessment, the AMI READMITS score, to the existing referral protocol.

Methods: Patients’ chart data were analyzed to assess changes in referrals and timely follow-up appointments from pre-intervention to intervention. A survey assessed providers’ satisfaction with the new referral protocol.

Results: Among 57 patients (n = 29 preintervention; n = 28 intervention), documented referrals increased significantly from 66% to 89% (χ2 = 4.571, df = 1, P = 0.033); and timely appointments increased by 10%, which was not significant (χ2 = 3.550, df = 2, P = 0.169). Most providers agreed that the new protocol was easy to use, useful in making referral decisions, and improved the referral process. All agreed the risk score should be incorporated into electronic clinical notes. Provider opinions related to implementing the risk score in clinical practice were mixed. Qualitative feedback suggests this was due to limited validation of the AMI READMITS score in reducing readmissions.

Conclusions: Our risk-based referral protocol helped to increase appropriate referrals among patients with type I MI. Provider adoption may be enhanced by incorporating the protocol into electronic clinical notes. Research to further validate the accuracy of the AMI READMITS score in predicting readmissions may support adoption of the protocol in clinical practice.

Keywords: quality improvement; type I myocardial infarction; referral process; readmission risk; risk assessment; chart review.

Early follow-up after discharge is an important strategy to reduce the risk of unplanned hospital readmissions among patients with various conditions.^1-3 While patient confounding factors, such as chronic health problems, environment, socioeconomic status, and literacy, make it difficult to avoid all unplanned readmissions, early follow-up may help providers identify and appropriately manage some health-related issues, and as such is a pivotal element of a readmission prevention strategy.⁴ There is evidence that patients with non-ST elevation myocardial infarction (NSTEMI) who have an outpatient appointment with a physician within 7 days after discharge have a lower risk of 30-day readmission.⁵

Our hospital’s postmyocardial infarction clinic was created to prevent unplanned readmissions within 30 days after discharge among patients with type I myocardial infarction (MI). Since inception, the number of referrals has been much lower than expected. In 2018, the total number of patients discharged from the hospital with type I MI and any troponin I level above 0.40 ng/mL was 313. Most of these patients were discharged from the hospital’s cardiac units; however, only 91 referrals were made. To increase referrals, the cardiology nurse practitioners (NPs) developed a post-MI referral protocol (Figure 1). However, this protocol was not consistently used and referrals to the clinic remained low.

Evidence-based risk assessment tools have the potential to increase effective patient management. For example, cardiology providers at the hospital utilize various scores, such as CHA₂DS₂-VASc⁶ and the Society of Thoracic Surgery risk score,⁷ to plan patient management. Among the scores used to predict unplanned readmissions for MI patients, the most promising is the AMI READMITS score.⁸ Unlike other nonspecific prediction models, the AMI READMITS score was developed based on variables extracted from the electronic health records (EHRs) of patients who were hospitalized for MI and readmitted within 30 days after discharge. Recognizing the potential to increase referrals by integrating an MI-specific risk assessment, this quality improvement study modified the existing referral protocol to include the patients’ AMI READMITS score and recommendations for follow-up.

Currently, there are no clear recommendations on how soon after discharge patients with MI should undergo follow-up. As research data vary, we selected 7 days follow-up for patients from high risk groups based on the “See you in 7” initiative for patients with heart failure (HF) and MI,^9,10 as well as evidence that patients with NSTEMI have a lower risk of 30-day readmission if they have follow-up within 7 days after discharge⁵; and we selected 14 days follow-up for patients from low-risk groups based on evidence that postdischarge follow-up within 14 days reduces risk of 30-day readmission in patients with acute myocardial infarction (AMI) and/or acutely decompensated HF.¹¹

Methods

This project was designed to answer the following question: For adult patients with type I MI, does implementation of a readmission risk assessment referral protocol increase the percentage of referrals and appointments scheduled within a recommended time? Anticipated outcomes included: (1) increased referrals to a cardiologist or the post-MI clinic; (2) increased scheduled follow-up appointments within 7 to 14 days; (3) provider satisfaction with the usability and usefulness of the new protocol; and (4) consistent provider adoption of the new risk assessment referral protocol.

To evaluate the degree to which these outcomes were achieved, we reviewed patient charts for 2 months prior and 2 months during implementation of the new referral protocol. As shown in Figure 2, the new protocol added the following process steps to the existing protocol: calculation of the AMI READMITS score, recommendations for follow-up based on patients’ risk score, and guidance to refer patients to the post-MI clinic if patients did not have an appointment with a cardiologist within 7 to 14 days after discharge. Patients’ risk assessment scores were obtained from forms completed by clinicians during the intervention. Clinician’s perceptions related to the usability and usefulness of the new protocol and feedback related to its long-term adoption were assessed using a descriptive survey.

Population

The project population included all adult patients (≥ 18 years old) with type I MI who were admitted or transferred to the hospital, had a percutaneous coronary intervention (PCI), or were managed without PCI and discharged from the hospital’s cardiac care unit (CCU) and progressive cardiac care unit (PCCU). The criteria for type I MI included the “detection of a rise and/or fall of cardiac troponin with at least 1 value above the 99th percentile and with at least 1 of the following: symptoms of acute myocardial ischemia; new ischemic electrocardiographic (ECG) changes; development of new pathological Q waves; imaging evidence of new loss of viable myocardium or new regional wall motion abnormality in a pattern consistent with an ischemic etiology; identification of a coronary thrombus by angiography including intracoronary imaging or by autopsy.”¹² The study excluded patients with type I MI who were referred for coronary bypass surgery.

Intervention

The revised risk assessment protocol was implemented within the CCU and PCCU. The lead investigator met with each provider to discuss the role of the post-MI clinic, current referral rates, the purpose of the project, and the new referral process to be completed during the project for each patient discharged with type I MI. Cardiology NPs, fellows, and residents were asked to use the risk-assessment form to calculate patients’ risk for readmission, and refer patients to the post-MI clinic if an appointment with a cardiologist was not available within 7 to 14 days after discharge. Every week during the intervention phase, the investigator sent reminder emails to ensure form completion. Providers were asked to calculate and write the score, the discharge and referral dates, where referrals were made (a cardiologist or the post-MI clinic), date of appointment, and reason for not scheduling an appointment or not referring on the risk assessment form, and to drop the completed forms in specific labeled boxes located at the CCU and PCCU work stations. The investigator collected the completed forms weekly. When the number of discharged patients did not match the number of completed forms, the investigator followed up with discharging providers to understand why.

Data and Data Collection

Data to determine whether the use of the new protocol increased discharge referrals among patients with type I MI within the recommended timeframes were collected by electronic chart review. Data included discharging unit, patients’ age, gender, admission and discharge date, diagnosis, referral to a cardiologist and the post-MI clinic, and appointment date. Clinical data needed to calculate the AMI READMITS score was also collected: PCI within 24 hours, serum creatinine, systolic blood pressure (SBP), brain natriuretic peptide (BNP), and diabetes status.

Data to assess provider satisfaction with the usability and usefulness of the new protocol were gathered through an online survey. The survey included 1 question related to the providers’ role, 1 question asking whether they used the risk assessment for each patient, and 5 Likert-items assessing the ease of usage. An additional open-ended question asked providers to share feedback related to integrating the AMI READMITS risk assessment score to the post-MI referral protocol long term.

To evaluate how consistently providers utilized the new referral protocol when discharging patients with type I MI, the number of completed forms was compared with the number of those patients who were discharged.

Statistical Analysis

Descriptive statistics were used to summarize patient demographics and to calculate the frequency of referrals before and during the intervention. Chi-square statistics were calculated to determine whether the change in percentage of referrals and timely referrals was significant. Descriptive statistics were used to determine the level of provider satisfaction related to each survey item. A content analysis method was used to synthesize themes from the open-ended question asking clinicians to share their feedback related to the new protocol.

Results

Fifty-seven patients met the study inclusion criteria: 29 patients during the preintervention phase and 28 patients during the intervention phase. There were 35 male (61.4%) and 22 female (38.6%) patients. Twenty-five patients (43.9%) were from age groups 41 through 60 years and 61 through 80 years, respectively, representing the majority of included patients. Seven patients (12.3%) were from the 81 years and older age group. There were no patients in the age group 18 through 40 years. Based on the AMI READMITS score calculation, 57.9% (n = 33) patients were from a low-risk group (includes extremely low and low risk for readmission) and 42.1% (n = 24) were from a high-risk group (includes moderate, high, and extremely high risk for readmission).

Provider adoption of the new protocol during the intervention was high. Referral forms were completed for 82% (n = 23) of the 28 patients during the intervention. Analysis findings showed a statistically significant increase in documented referrals after implementing the new referral protocol. During the preintervention phase, 66% (n = 19) of patients with type I MI were referred to see a cardiologist or an NP at a post-MI clinic and there was no documented referral for 34% (n = 10) of patients. During the intervention phase, 89% (n = 25) of patients were referred and there was no documented referral for 11% (n = 3) of patients. Chi-square results indicated that the increase in referrals was significant (χ2 = 4.571, df = 1, P = 0.033).

Data analysis examined whether patient referrals fell within the recommended timeframe of 7 days for the high-risk group (included moderate-to-extremely high risk) and 14 days for the low-risk group (included low-to-extremely low risk). During the preintervention phase, 31% (n = 9) of patient referrals were scheduled as recommended; 28% (n = 8) of patient referrals were scheduled but delayed; and there was no referral date documented for 41% (n = 12) of patients. During the intervention phase, referrals scheduled as recommended increased to 53% (n = 15); 25% (n = 7) of referrals were scheduled but delayed; and there was no referral date documented for 21.4% (n = 6) of patients. The change in appointments scheduled as recommended was not significant (χ2 = 3.550, df = 2, P = 0.169).

Surveys were emailed to 25 cardiology fellows and 3 cardiology NPs who participated in this study. Eighteen of the 28 clinicians (15 cardiology fellows and 3 cardiology NPs) responded for a response rate of 64%. One of several residents who rotated through the CCU and PCCU during the intervention also completed the survey, for a total of 19 participants. When asked if the protocol was easy to use, 79% agreed or strongly agreed. Eighteen of the 19 participants (95%) agreed or strongly agreed that the protocol was useful in making referral decisions. Sixty-eight percent agreed or strongly agreed that the AMI READMITS risk assessment score improves referral process. All participants agreed or strongly agreed that there should be an option to incorporate the AMI READMITS risk assessment score into electronic clinical notes. When asked whether the AMI READMITS risk score should be implemented in clinical practice, responses were mixed (Figure 3). A common theme among the 4 participants who responded with comments was the need for additional data to validate the usefulness of the AMI READMITS to reduce readmissions. In addition, 1 participant commented that “manual calculation [of the risk score] is not ideal.”

Discussion

This project demonstrated that implementing an evidence-based referral protocol integrating the AMI-READMITS score can increase timely postdischarge referrals among patients with type I MI. The percentage of appropriately scheduled appointments increased during the intervention phase; however, a relatively high number of appointments were scheduled outside of the recommended timeframe, similar to preintervention. Thus, while the new protocol increased referrals and provider documentation of these referrals, it appears that challenges in scheduling timely referral appointments remained. This project did not examine the reasons for delayed appointments.

The survey findings indicated that providers were generally satisfied with the usability and usefulness of the new risk assessment protocol. A large majority agreed or strongly agreed that it was easy to use and useful in making referral decisions, and most agreed or strongly agreed that it improves the referral process. Mixed opinions regarding implementing the AMI READMITS score in clinical practice, combined with qualitative findings, suggest that a lack of external validation of the AMI READMITS presents a barrier to its long-term adoption. All providers who participated in the survey agreed or strongly agreed that the risk assessment should be incorporated into electronic clinical notes. We have begun the process of working with the EHR vendor to automate the AMI risk-assessment within the referral work-flow, which will provide an opportunity for a follow-up quality improvement study.

This quality improvement project has several limitations. First, it implemented a small change in 2 inpatient units at 1 hospital using a simple pre- posttest design. Therefore, the findings are not generalizable to other settings. Prior to the intervention, some referrals may have been made without documentation. While the authors were able to trace undocumented referrals for patients who were referred to the post-MI clinic or to a cardiologist affiliated with the hospital, some patients may have been referred to cardiologists who were not affiliated with the hospital. Another limitation was that the self-created provider survey used was not tested in other clinical settings; thus, it cannot be determined whether the sensitivity and specificity of the survey questions are high. In addition, the clinical providers who participated in the study knew the study team, which may have influenced their behavior during the study period. Furthermore, the identified improvement in clinicians’ referral practices may not be sustainable due to the complexity and effort required to manually calculate the risk score. This limitation could be eliminated by integrating the risk score calculation into the EHR.

Conclusion

Early follow-up after discharge plays an important role in supporting patients’ self-management of some risk factors (ie, diet, weight, and smoking) and identifying gaps in postdischarge care which may lead to readmission. This project provides evidence that integrating the AMI READMITS risk assessment score into the referral process can help to guide discharge decision-making and increase timely, appropriate referrals for patients with MI. Integration of a specific risk assessment, such as the AMI READMITS, within the post-MI referral protocol may help clinicians make more efficient, educated referral decisions. Future studies should explore more specifically how and why the new protocol impacts clinicians’ decision-making and behavior related to post-MI referrals. In addition, future studies should investigate challenges associated with scheduling postdischarge appointments. It will be important to investigate how integration of the new protocol within the EHR may increase efficiency, consistency, and provider satisfaction with the new referral process. Additional research investigating the effects of the AMI READMITS score on readmissions reduction will be important to promote long-term adoption of the improved referral protocol in clinical practice.

Acknowledgments: The authors thank Shelly Conaway, ANP-BC, MSN, Angela Street, ANP-BC, MSN, Andrew Geis, ACNP-BC, MSN, Richard P. Jones II, MD, Eunice Young, MD, Joy Rothwell, MSN, RN-BC, Allison Olazo, MBA, MSN, RN-BC, Elizabeth Heck, RN-BC, and Matthew Trojanowski, MHA, MS, RRT, CSSBB for their support of this study.

Corresponding author: Nailya Muganlinskaya, DNP, MPH, ACNP-BC, MSN, The Johns Hopkins Hospital, 1800 Orleans St, Baltimore, MD 21287; nmuganl1@jhmi.edu.

Financial disclosures: None.

From the Milwaukee County Behavioral Health Division, Milwaukee, WI.

Abstract

Objective: Implementation of the Quadruple Aim of health care must begin with a clearly articulated set of concepts, or core domains (CDs), that comprise each aim. These CDs can then be operationalized with existing or new measures. If aligned to the organization’s mission and strategic goals, these CDs have the potential to focus quality improvement activities and reduce measurement burden. This article represents the efforts of a publicly funded behavioral health system to operationalize the Quadruple Aim through the development of CDs.

Methods: Various stakeholders across the organization were consulted on their perceptions of the Quadruple Aim and the CDs they believed should support it. Then, a review of existing literature on core metrics for health care and population health was completed, summarized, and integrated with the stakeholder feedback.

Results: These efforts led to the development and adoption of 15 CDs, with an accompanying literature review and set of recommendations of new and existing measures for each domain.

Conclusions: It is possible to create a comprehensive yet economical set of CDs and attendant measures that can be implemented in a staged, scalable, enterprise manner. It is hoped that the process articulated here, and the accompanying literature review, may be of some benefit to other public or government-run health systems in their own quality improvement journey to operationalize the Quadruple Aim by developing a set of CDs.

Keywords: quality measures; quality improvement; adult behavioral health.

First articulated in 2008, the Triple Aim proposes that health care systems should simultaneously seek to improve the patient’s experience of care, improve the health of populations, and reduce the per capita costs of care for populations.¹ More recently, some have argued that health care provider burnout can deleteriously impact the attainment of the Triple Aim and have therefore advocated for an expanded focus to include a fourth Aim, the work life quality of the staff.² Milwaukee County Behavioral Health Division (BHD), a publicly funded, county-based behavioral health care system in Milwaukee, Wisconsin, recently adopted the Quadruple Aim as the framework by which it will organize its quality activities.

Although originally developed for medical organizations, BHD believes that the Quadruple Aim has strong applicability to county-level behavioral health services. Many county-based behavioral health divisions provide a variety of programs to large segments of the county based on financial eligibility and/or clinical need, and thus often have responsibilities to populations or subpopulations, rather than programs. County health divisions, such as Milwaukee County’s Department of Health and Human Services, are often asked to improve outcomes and client experience of care with neutral growth budgets and less reliance on taxes to fund programs, while simultaneously attracting and retaining competent staff.

Crucial to the effective implementation of the Quadruple Aim, however, is a clear set of population- level measures that help organizations assess their progress.³ Unfortunately, as some authors have noted, evaluation of the Quadruple Aim remains a challenge because the “concepts of (population) health, quality of care and costs are not unanimously defined and measures for these concepts are under construction.”⁴ Several authors have provided some guidance to assist in the development of a set of measures that effectively capture the elements of the Quadruple Aim.^5,6 However, the recent rapid proliferation of quality measures in health care^7,8 has been both burdensome and costly for providers.^9,10 Any measures adopted should not only be as meaningful as possible with regards to assessing progress towards the basic aims of health care, but should also be parsimonious, to limit measurement burden for providers (and patients) and focus attention on important issues.^11,12

To select the most effective, parsimonious set of measures possible, one must first select a set of key foci from among the many possible areas of focus that the core measure is intended to represent. The core domains (CDs), if appropriately consistent with the strategic goals of the organization, provide a mechanism to orient the efforts of the organization at every level and help every staff member of the organization understand how his or her work impacts the progress towards these goals.¹¹ The CDs, therefore, represent the opportunity to affect a greater integration of efforts across the organization toward these shared aims, creating uniformity of purpose at every level. Further, increasing organizational attention on the CDs can also help to reduce measurement burden by streamlining and focusing the data capture processes on the most valuable elements of quality and health, and discarding other extraneous measures (albeit not at the expense of other reporting requirements).¹¹ The remainder of this article describes the CDs selected by BHD to assess its progress toward implementation of the Quadruple Aim and are organized by the Aim which they best represent.

Methods

To effectively implement the Quadruple Aim at BHD, it was necessary to clearly define the subpopulation of focus for our efforts.⁶ In this case, the subpopulation of interest was defined as all adult clients (18 years and older) who received at least 1 service encounter within a specified time frame from a program that BHD either operated or contracted with to provide care. Services provided by the BHD network include everything from psychiatric inpatient services to mental health and addiction treatment and care management. A limited array of social services, including housing and employment services, is also available to eligible consumers. BHD is the county-run behavioral health provider for individuals who are uninsured or underinsured in Milwaukee County, a demographically diverse, primarily urban county of approximately 950,000 people located in Wisconsin. Approximately 15,000 adults receive services at BHD each year.

This work began by obtaining executive sponsorship for the project, in this case from the Chief Operations Officer and Executive Medical Director of BHD. With their backing, an initial review of the literature produced a preliminary set of possible domains, for which we created working definitions. We then made a list of key stakeholders throughout BHD to whom we needed to present the idea of the Quadruple Aim, and the CDs under each Aim, to secure their support. These stakeholders, which included individuals involved in quality activities, program managers, and executive leadership, were strategically selected based on their relative influence within the organization. A set of brief presentations and handouts explaining the project were then developed and shared at different focus groups with these stakeholders over the course of 6 months. These focus groups served to not only educate the organization about the Quadruple Aim and the CDs but afforded participants an opportunity to provide feedback as well.

During the focus groups, we asked participants which domains they believed were most important (were “core”) when operationalizing the Quadruple Aim. The focus groups provided feedback on the domain definitions, feedback that was used to develop uniform, mutually agreed upon definitions for the CDs that were generalizable to all departments at BHD, regardless of the focus of their services within the continuum of care or the continuum of age. This was a crucial step, as it will eventually enable BHD to aggregate data across departments, even if there are minor discrepancies in the specific items they use to assess the CDs. Comments from the focus groups ultimately resulted in a truncated list of domains and definitions, which, coupled with the literature review, resulted in our final set of CDs.

During our review of the literature, we also looked for items that we felt could best represent each CD in the briefest, most meaningful way. (These items were not meant to supersede existing data, but to provide examples that could be implemented with existing data or recommendations that could be utilized in the absence of existing data.) During this process, we made every effort to make use of existing data-reporting requirements. For example, if we had a state mandate to collect data on housing status, we attempted to leverage this required data point to represent the CD related to housing. In other cases, we attempted to utilize claims or other administrative data to operationalize the CD, such as in the cost-of-care metric articulated in the section the Third Aim. For CDs for which no data existed or were insufficient, we emphasized the use of single- versus multi-item scales. For example, if we found a single-item global assessment of quality of life that had good psychometric properties relative to its longer parent scale, we selected the single item. This approach to item selection allowed us to create the most efficient, parsimonious set of measures possible, which we believed would enable us to comprehensively assess all the CDs with the least amount of burden to staff and clients. These items were presented at stakeholder focus groups, during which we asked for comments on the existing measures in their program or department and gave them the opportunity to comment on the new recommended measures.

A working definition is provided for each CD, followed by a brief review of the research base supporting its inclusion in the final list. The item(s) selected by BHD to represent each CD and the source of the item(s) are then supplied. These items were based either on measures currently collected because of existing reporting mandates or, in the case where extant measures were not available, on new items that demonstrated acceptable psychometric properties in the research literature. The CDs and items are organized by the Aim they best represent. A full list of the CDs by Quadruple Aim and items by CD is provided in the Appendix of the online version of this article. This article concludes with a brief summary of this effort and a discussion of how staff will utilize these items at different levels throughout the BHD system.

The First Aim: Population Health

Health Outcomes

Deaths. This can be defined as the cause of death, as determined by the medical examiner’s office (where appropriate) or as the age at time of death. This CD can also be reported as proportion of deaths considered premature (eg, before age 75) or calculated as total years of potential life lost.

Brief review and suggested item(s). Rates and causes of premature mortality are critical foci for the County Health Rankings & Roadmaps,¹³ the Institute for Healthcare Improvement’s “Guide to Measuring the Triple Aim,”⁶ the Centers for Disease Control and Prevention’s “Community Health Assessment for Population Health Improvement,”¹⁴ and the Institute of Medicine’s (IOM) “Vital Signs: Core Metrics for Health and Health Care Progress.”¹¹ There is ample evidence that individuals with serious mental illness are at increased risk of early mortality relative to the general population,^15-18 and this risk applies to those with substance use disorders as well.^15,19-20 BHD tracks all deaths that occur while patients are receiving BHD-funded, community-based services.

Self-Reported Health and Well-Being. This CD asks patients to rate their current physical and mental health status, as well as their overall quality of life.

Brief review and suggested item(s): Self-rated physical health. Premature mortality among individuals with behavioral health issues appears to be due, in large part, to their increased vulnerability to the development of medical comorbidities.^16,21 A single self-rating question has demonstrated considerable sensitivity to premature mortality,^22,23 with predictive properties up to a decade prior to death.^24,25 Further, self-rated health has been associated with subsequent functional decline,^26,27 acute service utilization,^28,29 and overall health care costs.²⁸

Brief review and suggested item(s): Self-rated mental health. Mental health disorders are associated with significant disability worldwide,³⁰ and comorbid mental health issues can exacerbate the course of other medical problems. For example, depression is associated with increased rates of mortality among individuals with diabetes and³¹ cardiovascular disease,³² as well as with rates of overall mortality,³³ and psychiatric comorbidity is associated with longer lengths of stay and higher costs among patients hospitalized for medical problems.³⁴ Research has found that a single-item measure of self-rated mental health is associated with the presence of psychiatric diagnoses, psychiatric symptoms, and subsequent depression and serious mental illness up to 1 year post-assessment.^35,36 There is even evidence that self-rated mental health may be more strongly associated with self-ratings of overall health than self-ratings of physical health.³⁷

Brief review and suggested item(s): Self-rated quality of life. Quality of life is a critical component of the recovery journey and overall health.³⁸ For example, the County Health Rankings & Roadmaps lists “quality of life” as 1 of its key “health outcomes” in its County Health Rankings.¹³ As some authors have noted, quality of life is often inferred from other “objective” recovery domains, such as employment, health status, or housing status. However, there is evidence that these objective domains are functionally distinct from the inherently subjective construct of quality of life.³⁹ This has led other authors to conclude that these domains should be assessed separately when evaluating outcomes.⁴⁰ Single-item quality of life assessments have been used in research with individuals with cancer,⁴¹ adults with disabilities,⁴² patients with cystic fibrosis,⁴³ and children with epilepsy.⁴⁴ For this effort, BHD selected the first global quality of life item from the World Health Organization’s WHOQOL-BREF quality of life assessment,⁴⁵ an item used in other quality of life research.⁴⁶

Health Factors

Substance Use. This CD is a composite of 4 different types of substance use, any recent heavy alcohol use (defined as 5 or more drinks in one sitting), any recent drug use, any recent prescription drug abuse, and any recent tobacco use.

Brief review and suggested item(s). As noted, substance use disorders confer an increased risk for early mortality^15,19 and are significantly implicated in disease disability burden worldwide.³⁰ Substance use has also been associated with both the onset^47,48 and exacerbation of mental health diagnoses.^49-51 Further, substance use appears to heighten the risk of violence in the general population⁵² and especially among those with a co-occurring mental illness.^53,54 The County Health Rankings & Roadmaps list alcohol and drug use as key behaviors to address to improve the overall health of a given county,¹³ and the Centers for Medicare & Medicaid Services (CMS) has endorsed initiation and engagement in addiction treatment as one of the measures in its Adult Core Set.⁵⁵

Tobacco use continues to be one of the most significant risk factors for early mortality worldwide, and evidence indicates that it is associated with a lower life expectancy of nearly 10 years.⁵⁶ Unfortunately, rates of tobacco use are even higher among those with severe mental illness relative to the general population, and their rates of smoking cessation are lower.^57,58 Tobacco use is a significant risk factor for the high rates of early mortality in individuals with severe mental illness.¹⁸ Further, a recent meta-analysis noted that, relative to those who continued to smoke, those who ceased smoking had reduced rates of psychological distress and increased quality of life rankings.⁵⁹ Reducing tobacco use is one of the key components of the County Health Rankings & Roadmaps, and medication assistance with smoking and tobacco use cessation is also listed in the CMS Adult Core Set.^13,55

An accumulating body of evidence suggests that single-item measures can adequately detect alcohol^60-62 and drug use disorders.^60-64 McNeely and colleagues recently developed and tested a brief 4-item screen, the Tobacco, Alcohol, Prescription medication, and other Substance use (TAPS) tool.^65,66 Preliminary evidence suggests that the TAPS tool can effectively identify the presence of problematic and disordered use of tobacco, alcohol, prescription medications, and other drugs.^65-67 BHD will use the 4 items from the TAPS tool to represent its substance use CD.

Education/Employment Status. This CD assesses the proportion of BHD members who have completed high school, who are in some type of educational or training program, or who are engaged in some type of employment activity (defined as full-time, part-time, supported, sheltered workshop, or as a full-time homemaker).

Brief review and suggested item(s). Research indicates that unemployment is a risk factor for mortality, even after controlling for other risk factors (eg, age, sex, socioeconomic status [SES], health).⁶⁸ Unemployment is associated with poorer physical and mental health in the general population and among those with disabilities.^69-71 Promisingly, evidence suggests that gaining employment or re-employment is associated with better health,⁷² even for individuals with substance use disorders⁷³ or moderate⁷⁴ to severe mental health disorders.^75-78 Some authors have even proposed that, above and beyond the associated health benefits, employment may also help to realize a modest cost savings due to reduced service utilization and disability.^79,80 Employment is a core tenet in the Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) model of recovery,⁸¹ and is also listed as an important recovery goal for individuals with behavioral health issues.⁸² BHD collects data on employment status on all the patients it serves as part of its state-mandated reporting requirements and will use this item in the CD data set.⁸³

Living Situation. This is measured as the proportion of people who live in permanent, supportive, stable housing; it may also be measured as the percentage of the population living with severe housing problems or who are homeless.

Brief review and suggested item(s). Housing problems can be conceptualized as 3 inter-related components: conditions within the home, neighborhood conditions, and housing affordability, each of which can contribute uniquely to poorer physical and mental health of individuals and families⁸⁴ and to educational outcomes for children.^85,86 Further, individuals who are homeless have a standardized mortality ratio 2 to 5 times that of the general population,^87-89 even after controlling for low income status,⁹⁰ and some evidence suggests these rates are even higher among unsheltered versus sheltered homeless individuals.⁹¹ Interventions to improve the condition of housing have demonstrated positive impacts on both physical and mental health,⁹² and a recent study found that individuals receiving housing assistance in the form of public housing or multifamily housing from the Department of Housing and Urban Development had better self-rated physical and mental health relative to individuals on the wait list for housing assistance.⁹³ Moreover, the provision of housing has been shown to promote reductions in substance use and health service utilization among homeless individuals with substance use disorders.⁹⁴ Rog and colleagues reviewed the literature on permanent supportive housing for individuals with substance use or mental health disorders who were homeless or disabled, and found that provision of housing led to reduced rates of homelessness, emergency department (ED) and inpatient utilization and increased consumer satisfaction.⁹⁵

Importantly, evidence suggests that housing is viewed as facilitative of recovery. For example, in a recent qualitative study of homeless individuals with mental illness, housing was seen as a critical first step in recovery, providing a sense of security, increasing feelings of personal independence and autonomy, improving perceptions of health and well-being, and affording a stable environment to rebuild relationships with important others.⁹⁶ BHD collects data on housing status on all the patients it serves as part of its state-mandated reporting requirements and will utilize this item in the CD data set.⁸³

Social Relationships. This is defined as recent interactions with family, supportive networks (formal and informal), and other recovery services.

Brief review and suggested item(s). Research has long established that social relationships have a significant impact on health, including rates of mortality as well as physical and mental health morbidity.^97-99 Social connectedness is another of the pillars supporting an individual’s recovery in SAMHSA’s formulation. Several reviews of the recovery literature^38,82 support its importance to the recovery process and inclusion in any assessment of holistic recovery. Social support has been shown to promote recovery among individuals with severe mental illness^100-102 and substance use disorders,¹⁰³ and may mitigate the progression of chronic, life-threatening physical illnesses.⁹⁷ For the purposes of BHD’s CD data set, the social support question from the “100 Million Healthier Lives Common Questionnaire for Adults” will be used to assess individuals’ perceived adequacy of social support.¹⁰⁴

Legal Involvement. Defined as involvement with the civil or criminal justice system, including arrests, imprisonment, or detainment.

Brief review and suggested item(s). Involvement in the criminal justice system is both disruptive for the individual in recovery and expensive to the larger health care system.¹⁰⁵ Individuals with substance use¹⁰⁶ and severe mental health disorders¹⁰⁷ are over-represented in the prison system, and evidence suggests that general physical and mental health declines while individuals are in prison.^108,109 Perhaps even more concerning, numerous studies have demonstrated an increase in mortality rates for individuals recently released from prison relative to the general population, particularly during the period immediately following release.^108-110 This relationship may even persist long term.¹¹¹ Further, research indicates that individuals recently released from prison have increased emergency care and hospital utilization.^112,113

Incarceration can have significant impacts on the health of the broader community as well. For example, research has found an association between parental incarceration to rates of infant mortality,¹¹⁴ increased behavioral and developmental problems of children of incarcerated parents,^115,116 lower rates of child support payments,¹¹⁷ and poorer cardiovascular health of female partners of incarcerated individuals.¹¹⁸ Formerly incarcerated individuals experience slower wage growth as well.¹¹⁹ However, evidence also indicates that engagement in mental health¹²⁰ and substance abuse¹²¹ treatment can reduce the likelihood of subsequent recidivism. As part of its state-mandated reporting, BHD is required to provide information on the criminal justice system involvement of its clients in the previous 6 months, including whether they have been jailed or imprisoned,⁸³ and this will function as its measure of legal involvement in its CD data set.

Socioeconomic Status. Socioeconomic status is the social standing or class of an individual or group. It is often measured as a combination of education, income, and occupation. It can also be defined subjectively, such as one’s evaluation of status relative to similar others or based on an individual’s interpretation of her or his financial needs.

Brief review and suggested item(s). A large body of evidence supports the existence of a robust relationship between lower SES and poor health, including mortality and chronic medical diseases,^122-124 as well as mental illness.^125-127 Although previous research has examined this relationship using objective indicators of SES (eg, income, education level, occupation), there has recently been an increased interest in exploring the relationship of subjective SES with health indices. Subjective SES is generally assessed by asking individuals to rate themselves relative to others in the society in which they live, in terms of wealth, occupation, educational level, or other indicators of social status. Evidence suggests that subjective SES is associated with objective measures of SES,^128-130 and relates to measures of physical and mental health as well, even after controlling for objective SES.^130-135 BHD will be using a modified version of the Subject SES Scale,^131,135 which is deployed in the “100 Million Healthier Lives Common Questionnaire for Adults.”¹⁰⁴

Acute Service Use. This is defined as an admission to a medical or psychiatric emergency room or to a medical or psychiatric hospital or to a detoxification facility.

Brief review and suggested item(s). The CMS Adult Core Set includes “plan all cause readmissions” as a key quality metric.⁵⁵ Hospital readmissions are also endorsed by the National Committee on Quality Assurance as one of its Health Effectiveness Data and Information Set (HEDIS) measures and by the National Quality Forum. Readmissions, despite their widespread endorsement, are a somewhat controversial measure. Although readmissions are costly to the health care system,¹³⁶ the relationship between readmissions and quality is inconsistent. For example, Krumholz and colleagues¹³⁷ found differential rates of readmission for the same patient discharged from 2 different hospitals, which were categorized based on previous readmission rates, suggesting that hospitals do have different levels of performance even when treating the same patient. However, other data indicate that 30-day, all-cause, risk-standardized readmission rates are not associated with hospital 30-day, all-cause, risk-standardized mortality rates.¹³⁸

Chin and colleague found that readmissions to the hospital that occurred more than 7 days post-discharge were likely due to community- and household-related factors, rather than hospital-related quality factors.¹³⁹ Transitional care interventions that have successfully reduced 30-day readmission rates are most often multicomponent and focus not just on hospital-based interventions (eg, discharge planning, education) but on follow-up care in the community by formal supports (eg, in-home visits, telephone calls, outpatient clinic appointments, case management) and informal supports (eg, family and friends).^140-143 Further, qualitative evidence suggests that some individuals perceive psychiatric hospitalizations to be the result of insufficient resources or unsuccessful attempts to maintain their stability in the community.¹⁴⁴ Thus, unplanned or avoidable hospital readmissions may represent a failure of the continuum of care not only from the perspective of the health care system, but from the patient perspective as well.

Frequent or nonurgent use of EDs is conceptually similar to excessive or avoidable inpatient utilization in several ways. For example, overuse of EDs is costly, with some estimates suggesting that it is responsible for up to $38 billion in wasteful spending each year.¹⁴⁵ Individuals with frequent ED visits have a greater disease burden¹⁴⁶ and an increased risk of mortality compared to nonfrequent users.¹⁴⁷ Research suggests that individuals who visit the ED for non-urgent issues do so because of perceived difficulties associated with accessing primary care, and the convenience of EDs relative to primary care.^148-150 Moreover, similar to the hospital readmission literature discussed earlier, successful strategies to reduce high rates of ED utilization generally focus on continuum of care interventions, such as provision of case management services.^151-155

This evidence implies that frequent ED utilization and hospital readmissions may not be a fundamental issue of quality (or lack thereof) in hospitals or EDs but rather a lack of, or ineffectual, transitional and continuum of care strategies and services. To underscore this point, some authors have argued that a system that is excessively crisis-oriented hinders recovery because it is reactive rather than proactive, predicated on the notion that one’s condition must deteriorate to receive care.¹⁵⁶

Although some organizations may have access to claims data or may function as self-contained health systems (eg, the Veterans Health Administration [VHA] ), others may not have access to such data. In the absence of claims data, patient self-report of service utilization has been used as a proxy for actual agency records.¹⁵⁷ Although concordance between medical and/or agency records and patient self-report has been variable,¹⁵⁷ evidence generally suggests that rates of agreement are higher the shorter the recall time interval.^158,159 BHD does not have access to comprehensive claims data and has therefore chosen to use 5 dichotomously scored (yes/no) questions—related to medical inpatient, medical ED, psychiatric inpatient, psychiatric ED, and detoxification use in the last 30 days—to represent the CD of acute service utilization.

The Second Aim: Quality of Care

Safety

Safety is defined as avoiding injuries to patients from the care that is intended to help them.

Brief review and suggested item(s). As noted in “Crossing the Quality Chasm,” the IOM’s seminal document, “the health care environment should be safe for all patients, in all of its processes, all the time.”¹⁶⁰ The landmark Harvard Medical Practice Study in 1991 found that adverse events occurred in nearly 4% of all hospital admissions and, among these, over a quarter were due to negligence.¹⁶¹ Other estimates of adverse events range as high as 17%.¹⁶² Indeed, a recent article by Makary and Daniel estimated that medical errors may be the third leading cause of death in the United States.¹⁶³ Unfortunately, research on safety in the mental health field has lagged behind that of physical health,¹⁶⁴ with evidence indicating that research in nonhospital settings in mental health care may be particularly scarce.¹⁶⁵ In a study of adverse events that occurred in psychiatric inpatient units in the VHA system between 2015 and 2016, Mills and colleagues found that of the 87 root cause analysis reports, suicide attempts were the most frequent, and, among safety events, falls were the most frequently reported, followed by medication events.¹⁶⁶ Another report on data collected from psychiatric inpatient units in the VHA revealed that nearly one-fifth of patients experienced a safety event, over half of which were deemed preventable.¹⁶⁷ These numbers likely represent an underestimation of the true volume of safety events, as another study by the same research group found that less than 40% of safety events described in patient medical records were documented in the incident reporting system.¹⁶⁸ BHD will utilize the total number of complaints and incident reports submitted within a given time frame as its “safety” metric in the CD data set.

Wait Time for Service

The CD is defined as the length of time between the date a patient first contacted BHD for services and the date of their first clinical service.

Brief review and suggested item(s). “Timeliness” was listed among the 6 aims for improvement in “Crossing the Quality Chasm” in 2001, and it remains no less relevant today.¹⁶⁰ For example, evidence indicates that access to primary care is inversely related to avoidable hospitalizations.¹⁶⁹ One study found that, of patients hospitalized for cardiovascular problems, those who had difficulty accessing routine care post discharge had higher 30-day readmission rates.¹⁷⁰ Among VHA patients, longer wait times are associated with more avoidable hospitalizations and higher rates of mortality.¹⁷¹ Longer wait times appear to decrease the likelihood of attending a first appointment for individuals with substance use^172,173 and mental health disorders.¹⁷⁴ Importantly, longer wait times are associated with lower ratings of the patient experience of care, including perceptions of the quality of and satisfaction with care,¹⁷⁵ and may be associated with worse outcomes for individuals in early intervention for psychosis treatment.¹⁷⁶ For the purposes of the CD data set, BHD will monitor the length of time between the date a patient first contacted BHD for services and the date of their first clinical service.

Patient satisfaction is defined as the degree of patients’ satisfaction with the care they have received.

Brief review and suggested item(s). Research has consistently demonstrated the relationship of the patient’s experience of care to a variety of safety and clinical effectiveness measures in medical health care,¹⁷⁷ and the therapeutic alliance is one of the most consistent predictors of outcomes in behavioral health, regardless of therapeutic modality.¹⁷⁸ Patient satisfaction is a commonly assessed aspect of the patient experience of care. Patient satisfaction scores have been correlated with patient adherence to recommended treatment regimens, care quality, and health outcomes.¹⁷⁹ For example, Aiken et al found that patient satisfaction with hospital care was associated with higher ratings of the quality and safety of nursing care in these hospitals.¹⁸⁰ Increased satisfaction with inpatient care has been associated with lower 30-day readmission rates for patients with acute myocardial infarction, heart failure, and pneumonia,¹⁸¹ and patients with schizophrenia who reported higher treatment satisfaction also reported better quality of life.^182,183 Many satisfaction survey options exist to evaluate this CD, including the Consumer Assessment of Healthcare Providers and Systems and the Client Satisfaction Questionnaire; BHD will utilize an outpatient behavioral health survey from a third-party vendor.

The Third Aim: Cost of Care

Cost of Care

This can be defined as the average cost to provide care per patient per month.

Brief review and suggested item(s). Per capita cost, or rather, the total cost of providing care to a circumscribed population divided by the total population, has been espoused as an important metric for the Triple Aim and the County Health Rankings.^6,13 Indeed, between 1960 and 2016, per capita expenditures for health care have grown 70-fold, and the percent of the national gross domestic product accounted for by health expenditures has more than tripled (5.0% to 17.9%).¹⁸⁴ One of the more common metrics deployed for assessing health care cost is the per capita per month cost, or rather, the per member per month cost of the predefined population for a given health care system.^6,185,186 In fact, some authors have proposed that cost of care can be used not only to track efficient resource allocation, but can also be a proxy for a healthier population as well (ie, as health improves, individuals use fewer and less-expensive services, thus costing the system less).¹⁸⁷ To assess this metric, BHD will calculate the total amount billed for patient care provided within BHD’s health network each month (irrespective of funding source) and then divide this sum by the number of members served each month. Although this measure does not account for care received at other health care facilities outside BHD’s provider network, nor does it include all the overhead costs associated with the care provided by BHD itself, it is consistent with the claims-based approach used or recommended by other authors.^6,188

The Fourth Aim: Staff Well-being

Staff Quality of Work Life

This can be defined as the quality of the work life of health care clinicians and staff.

Brief review and suggested item(s). Some authors have suggested that the Triple Aim framework is incomplete and have proffered compelling arguments that provider well-being and the quality of work life constitutes a fourth aim.² Provider burnout is prevalent in both medical^2,189 and behavioral health care.^190,191 Burnout among health care professionals has been associated with higher rates of perceived medical errors,¹⁹² lower patient satisfaction scores,^189,193 lower rates of provider empathy,¹⁹⁴ more negative attitudes towards patients,¹⁹⁵ and poorer staff mental and physical health.¹⁹¹

Burnout is also associated with higher rates of absenteeism, turnover intentions, and turnover.^{190,191,196,197} However, burnout is not the only predictor of staff turnover; for example, turnover rates are a useful proxy for staff quality of work life for several reasons.¹⁹⁸ First, turnover is associated with substantial direct and indirect costs, including lost productivity, increased errors, and lost revenue and recruitment costs, with some turnover cost estimates as high as $17 billion for physicians and $14 billion for nurses annually.^199-201 Second, research indicates that staff turnover can have a deleterious impact on implementation of evidence-based interventions.^202-205 Finally, consistent with the philosophy of utilizing existing data sources for the CD measures, turnover can be relatively easily extracted from administrative data for operated or contracted programs, and its collection does not place any additional burden on staff. As a large behavioral health system that is both a provider and payer of care, BHD will therefore examine the turnover rates of its internal administrative and clinical staff as well as the turnover of staff in its contracted provider network as its measures for the Staff Quality of Work Life CD.

Clinical Implications

These metrics can be deployed at any level of the organization. Clinicians may use 1 or more of the measures to track the recovery of individual clients, or in aggregate for their entire caseload. Similarly, managers can use these measures to assess the overall effectiveness of the programs for which they are responsible. Executive leaders can evaluate the impact of several programs or the system of care on the health of a subpopulation of clients with a specific condition, or for all their enrolled members. Further, not all measures need be utilized for every dashboard or evaluative effort. The benefit of a comprehensive set of measures lies in their flexibility—1 or more of the measures may be selected depending on the project being implemented or the interests of the stakeholder.

It is important to note that many of the CDs (and their accompanying measures) are aligned to/consistent with social determinants of health.^206,207 Evidence suggests that social determinants make substantial contributions to the overall health of individuals and populations and may even account for a greater proportion of variance in health outcomes than health care itself.²⁰⁸ The measures articulated here, therefore, can be used to assess whether and how effectively care provision has addressed these social determinants, as well as the relative impact their resolution may have on other health outcomes (eg, mortality, self-rated health).

These measures can also be used to stratify clients by clinical severity or degree of socioeconomic deprivation. The ability to adjust for risk has many applications in health care, particularly when organizations are attempting to implement value-based purchasing models, such as pay-for-performance contracts or other alternative payment models (population health-based payment models).²⁰⁹ Indeed, once fully implemented, the CDs and measures will enable BHD to more effectively build and execute different conceptual models of “value” (see references 210 and 211 for examples). We will be able to assess the progress our clients have made in care, the cost associated with that degree of improvement, the experience of those clients receiving that care, and the clinical and social variables that may influence the relative degree of improvement (or lack thereof). Thus, the CDs provide a conceptual and data-driven foundation for the Quadruple Aim and any quality initiatives that either catalyze or augment its implementation.

This article provides an overview of the CDs selected by BHD to help organize, focus, advance, and track its quality efforts within the framework of the Quadruple Aim. Although items aligned to each of these CDs are offered, the CDs themselves have been broadly conceptualized such that they can flexibly admit a variety of possible items and/or assessments to operationalize each CD and thus have potential applicability to other behavioral health systems, particularly public systems that have state-mandated and other data reporting requirements.

Bearing in mind the burden that growing data collection requirements can have on the provision of quality care and staff work satisfaction and burnout,^10,212 the CDs (and the items selected to represent each) are designed with “strategic parsimony” in mind. Although the CDs are inclusive in that they cover care quality, cost of care, staff quality of life, and general population health, only CDs and items undergirded by a solid evidence base and high value with regards to BHD’s mission and values, as determined by key stakeholders, were selected. Moreover, BHD attempted to make use of existing data collection and reporting mandates when selecting the final pool of items to reduce the measurement burden on staff and clients. Thus, the final set of CDs and items are designed to be comprehensive yet economical.

The CDs are deeply interrelated. Although each CD may be individually viewed as a valuable metric, improvements in any 1 CD will impact the others (eg, increasing care quality should impact population health, increasing staff quality of life should impact the quality of care). Moreover, this idea of interrelatedness acknowledges the need to view health systems and the populations they serve holistically, in that improvement is not simply the degree of change in any given metric (whether individually or collectively), but rather something more entirely. The concepts of value, quality, and health are complex, multidimensional, and dynamic, and the CDs that comprise these concepts should not be considered independently from one another. The CDs (and items) offered in this article are scalable in that they can be used at different levels of an organization depending on the question or stakeholder, and can be used individually or in combination with one another. Moreover, they are adaptable to a variety of risk-adjusted program, population health, and value-based evaluation models. It is hoped that the process articulated here, and the accompanying literature review, may benefit other public or government-run health systems in their own quality journey to operationalize the Quadruple Aim by developing a set of CDs.

Corresponding author: Walter Matthew Drymalski, PhD; walter.drymalski@milwaukeecountywi.gov.

Financial disclosures: None.

From the Milwaukee County Behavioral Health Division, Milwaukee, WI.

Abstract

Objective: Implementation of the Quadruple Aim of health care must begin with a clearly articulated set of concepts, or core domains (CDs), that comprise each aim. These CDs can then be operationalized with existing or new measures. If aligned to the organization’s mission and strategic goals, these CDs have the potential to focus quality improvement activities and reduce measurement burden. This article represents the efforts of a publicly funded behavioral health system to operationalize the Quadruple Aim through the development of CDs.

Methods: Various stakeholders across the organization were consulted on their perceptions of the Quadruple Aim and the CDs they believed should support it. Then, a review of existing literature on core metrics for health care and population health was completed, summarized, and integrated with the stakeholder feedback.

Results: These efforts led to the development and adoption of 15 CDs, with an accompanying literature review and set of recommendations of new and existing measures for each domain.

Conclusions: It is possible to create a comprehensive yet economical set of CDs and attendant measures that can be implemented in a staged, scalable, enterprise manner. It is hoped that the process articulated here, and the accompanying literature review, may be of some benefit to other public or government-run health systems in their own quality improvement journey to operationalize the Quadruple Aim by developing a set of CDs.

Keywords: quality measures; quality improvement; adult behavioral health.

First articulated in 2008, the Triple Aim proposes that health care systems should simultaneously seek to improve the patient’s experience of care, improve the health of populations, and reduce the per capita costs of care for populations.¹ More recently, some have argued that health care provider burnout can deleteriously impact the attainment of the Triple Aim and have therefore advocated for an expanded focus to include a fourth Aim, the work life quality of the staff.² Milwaukee County Behavioral Health Division (BHD), a publicly funded, county-based behavioral health care system in Milwaukee, Wisconsin, recently adopted the Quadruple Aim as the framework by which it will organize its quality activities.

Although originally developed for medical organizations, BHD believes that the Quadruple Aim has strong applicability to county-level behavioral health services. Many county-based behavioral health divisions provide a variety of programs to large segments of the county based on financial eligibility and/or clinical need, and thus often have responsibilities to populations or subpopulations, rather than programs. County health divisions, such as Milwaukee County’s Department of Health and Human Services, are often asked to improve outcomes and client experience of care with neutral growth budgets and less reliance on taxes to fund programs, while simultaneously attracting and retaining competent staff.

Crucial to the effective implementation of the Quadruple Aim, however, is a clear set of population- level measures that help organizations assess their progress.³ Unfortunately, as some authors have noted, evaluation of the Quadruple Aim remains a challenge because the “concepts of (population) health, quality of care and costs are not unanimously defined and measures for these concepts are under construction.”⁴ Several authors have provided some guidance to assist in the development of a set of measures that effectively capture the elements of the Quadruple Aim.^5,6 However, the recent rapid proliferation of quality measures in health care^7,8 has been both burdensome and costly for providers.^9,10 Any measures adopted should not only be as meaningful as possible with regards to assessing progress towards the basic aims of health care, but should also be parsimonious, to limit measurement burden for providers (and patients) and focus attention on important issues.^11,12

To select the most effective, parsimonious set of measures possible, one must first select a set of key foci from among the many possible areas of focus that the core measure is intended to represent. The core domains (CDs), if appropriately consistent with the strategic goals of the organization, provide a mechanism to orient the efforts of the organization at every level and help every staff member of the organization understand how his or her work impacts the progress towards these goals.¹¹ The CDs, therefore, represent the opportunity to affect a greater integration of efforts across the organization toward these shared aims, creating uniformity of purpose at every level. Further, increasing organizational attention on the CDs can also help to reduce measurement burden by streamlining and focusing the data capture processes on the most valuable elements of quality and health, and discarding other extraneous measures (albeit not at the expense of other reporting requirements).¹¹ The remainder of this article describes the CDs selected by BHD to assess its progress toward implementation of the Quadruple Aim and are organized by the Aim which they best represent.

Methods

To effectively implement the Quadruple Aim at BHD, it was necessary to clearly define the subpopulation of focus for our efforts.⁶ In this case, the subpopulation of interest was defined as all adult clients (18 years and older) who received at least 1 service encounter within a specified time frame from a program that BHD either operated or contracted with to provide care. Services provided by the BHD network include everything from psychiatric inpatient services to mental health and addiction treatment and care management. A limited array of social services, including housing and employment services, is also available to eligible consumers. BHD is the county-run behavioral health provider for individuals who are uninsured or underinsured in Milwaukee County, a demographically diverse, primarily urban county of approximately 950,000 people located in Wisconsin. Approximately 15,000 adults receive services at BHD each year.

This work began by obtaining executive sponsorship for the project, in this case from the Chief Operations Officer and Executive Medical Director of BHD. With their backing, an initial review of the literature produced a preliminary set of possible domains, for which we created working definitions. We then made a list of key stakeholders throughout BHD to whom we needed to present the idea of the Quadruple Aim, and the CDs under each Aim, to secure their support. These stakeholders, which included individuals involved in quality activities, program managers, and executive leadership, were strategically selected based on their relative influence within the organization. A set of brief presentations and handouts explaining the project were then developed and shared at different focus groups with these stakeholders over the course of 6 months. These focus groups served to not only educate the organization about the Quadruple Aim and the CDs but afforded participants an opportunity to provide feedback as well.

During the focus groups, we asked participants which domains they believed were most important (were “core”) when operationalizing the Quadruple Aim. The focus groups provided feedback on the domain definitions, feedback that was used to develop uniform, mutually agreed upon definitions for the CDs that were generalizable to all departments at BHD, regardless of the focus of their services within the continuum of care or the continuum of age. This was a crucial step, as it will eventually enable BHD to aggregate data across departments, even if there are minor discrepancies in the specific items they use to assess the CDs. Comments from the focus groups ultimately resulted in a truncated list of domains and definitions, which, coupled with the literature review, resulted in our final set of CDs.

During our review of the literature, we also looked for items that we felt could best represent each CD in the briefest, most meaningful way. (These items were not meant to supersede existing data, but to provide examples that could be implemented with existing data or recommendations that could be utilized in the absence of existing data.) During this process, we made every effort to make use of existing data-reporting requirements. For example, if we had a state mandate to collect data on housing status, we attempted to leverage this required data point to represent the CD related to housing. In other cases, we attempted to utilize claims or other administrative data to operationalize the CD, such as in the cost-of-care metric articulated in the section the Third Aim. For CDs for which no data existed or were insufficient, we emphasized the use of single- versus multi-item scales. For example, if we found a single-item global assessment of quality of life that had good psychometric properties relative to its longer parent scale, we selected the single item. This approach to item selection allowed us to create the most efficient, parsimonious set of measures possible, which we believed would enable us to comprehensively assess all the CDs with the least amount of burden to staff and clients. These items were presented at stakeholder focus groups, during which we asked for comments on the existing measures in their program or department and gave them the opportunity to comment on the new recommended measures.

A working definition is provided for each CD, followed by a brief review of the research base supporting its inclusion in the final list. The item(s) selected by BHD to represent each CD and the source of the item(s) are then supplied. These items were based either on measures currently collected because of existing reporting mandates or, in the case where extant measures were not available, on new items that demonstrated acceptable psychometric properties in the research literature. The CDs and items are organized by the Aim they best represent. A full list of the CDs by Quadruple Aim and items by CD is provided in the Appendix of the online version of this article. This article concludes with a brief summary of this effort and a discussion of how staff will utilize these items at different levels throughout the BHD system.

The First Aim: Population Health

Health Outcomes

Deaths. This can be defined as the cause of death, as determined by the medical examiner’s office (where appropriate) or as the age at time of death. This CD can also be reported as proportion of deaths considered premature (eg, before age 75) or calculated as total years of potential life lost.

Brief review and suggested item(s). Rates and causes of premature mortality are critical foci for the County Health Rankings & Roadmaps,¹³ the Institute for Healthcare Improvement’s “Guide to Measuring the Triple Aim,”⁶ the Centers for Disease Control and Prevention’s “Community Health Assessment for Population Health Improvement,”¹⁴ and the Institute of Medicine’s (IOM) “Vital Signs: Core Metrics for Health and Health Care Progress.”¹¹ There is ample evidence that individuals with serious mental illness are at increased risk of early mortality relative to the general population,^15-18 and this risk applies to those with substance use disorders as well.^15,19-20 BHD tracks all deaths that occur while patients are receiving BHD-funded, community-based services.

Self-Reported Health and Well-Being. This CD asks patients to rate their current physical and mental health status, as well as their overall quality of life.

Brief review and suggested item(s): Self-rated physical health. Premature mortality among individuals with behavioral health issues appears to be due, in large part, to their increased vulnerability to the development of medical comorbidities.^16,21 A single self-rating question has demonstrated considerable sensitivity to premature mortality,^22,23 with predictive properties up to a decade prior to death.^24,25 Further, self-rated health has been associated with subsequent functional decline,^26,27 acute service utilization,^28,29 and overall health care costs.²⁸

Brief review and suggested item(s): Self-rated mental health. Mental health disorders are associated with significant disability worldwide,³⁰ and comorbid mental health issues can exacerbate the course of other medical problems. For example, depression is associated with increased rates of mortality among individuals with diabetes and³¹ cardiovascular disease,³² as well as with rates of overall mortality,³³ and psychiatric comorbidity is associated with longer lengths of stay and higher costs among patients hospitalized for medical problems.³⁴ Research has found that a single-item measure of self-rated mental health is associated with the presence of psychiatric diagnoses, psychiatric symptoms, and subsequent depression and serious mental illness up to 1 year post-assessment.^35,36 There is even evidence that self-rated mental health may be more strongly associated with self-ratings of overall health than self-ratings of physical health.³⁷

Brief review and suggested item(s): Self-rated quality of life. Quality of life is a critical component of the recovery journey and overall health.³⁸ For example, the County Health Rankings & Roadmaps lists “quality of life” as 1 of its key “health outcomes” in its County Health Rankings.¹³ As some authors have noted, quality of life is often inferred from other “objective” recovery domains, such as employment, health status, or housing status. However, there is evidence that these objective domains are functionally distinct from the inherently subjective construct of quality of life.³⁹ This has led other authors to conclude that these domains should be assessed separately when evaluating outcomes.⁴⁰ Single-item quality of life assessments have been used in research with individuals with cancer,⁴¹ adults with disabilities,⁴² patients with cystic fibrosis,⁴³ and children with epilepsy.⁴⁴ For this effort, BHD selected the first global quality of life item from the World Health Organization’s WHOQOL-BREF quality of life assessment,⁴⁵ an item used in other quality of life research.⁴⁶

Health Factors

Substance Use. This CD is a composite of 4 different types of substance use, any recent heavy alcohol use (defined as 5 or more drinks in one sitting), any recent drug use, any recent prescription drug abuse, and any recent tobacco use.

Brief review and suggested item(s). As noted, substance use disorders confer an increased risk for early mortality^15,19 and are significantly implicated in disease disability burden worldwide.³⁰ Substance use has also been associated with both the onset^47,48 and exacerbation of mental health diagnoses.^49-51 Further, substance use appears to heighten the risk of violence in the general population⁵² and especially among those with a co-occurring mental illness.^53,54 The County Health Rankings & Roadmaps list alcohol and drug use as key behaviors to address to improve the overall health of a given county,¹³ and the Centers for Medicare & Medicaid Services (CMS) has endorsed initiation and engagement in addiction treatment as one of the measures in its Adult Core Set.⁵⁵

Tobacco use continues to be one of the most significant risk factors for early mortality worldwide, and evidence indicates that it is associated with a lower life expectancy of nearly 10 years.⁵⁶ Unfortunately, rates of tobacco use are even higher among those with severe mental illness relative to the general population, and their rates of smoking cessation are lower.^57,58 Tobacco use is a significant risk factor for the high rates of early mortality in individuals with severe mental illness.¹⁸ Further, a recent meta-analysis noted that, relative to those who continued to smoke, those who ceased smoking had reduced rates of psychological distress and increased quality of life rankings.⁵⁹ Reducing tobacco use is one of the key components of the County Health Rankings & Roadmaps, and medication assistance with smoking and tobacco use cessation is also listed in the CMS Adult Core Set.^13,55

An accumulating body of evidence suggests that single-item measures can adequately detect alcohol^60-62 and drug use disorders.^60-64 McNeely and colleagues recently developed and tested a brief 4-item screen, the Tobacco, Alcohol, Prescription medication, and other Substance use (TAPS) tool.^65,66 Preliminary evidence suggests that the TAPS tool can effectively identify the presence of problematic and disordered use of tobacco, alcohol, prescription medications, and other drugs.^65-67 BHD will use the 4 items from the TAPS tool to represent its substance use CD.

Education/Employment Status. This CD assesses the proportion of BHD members who have completed high school, who are in some type of educational or training program, or who are engaged in some type of employment activity (defined as full-time, part-time, supported, sheltered workshop, or as a full-time homemaker).

Brief review and suggested item(s). Research indicates that unemployment is a risk factor for mortality, even after controlling for other risk factors (eg, age, sex, socioeconomic status [SES], health).⁶⁸ Unemployment is associated with poorer physical and mental health in the general population and among those with disabilities.^69-71 Promisingly, evidence suggests that gaining employment or re-employment is associated with better health,⁷² even for individuals with substance use disorders⁷³ or moderate⁷⁴ to severe mental health disorders.^75-78 Some authors have even proposed that, above and beyond the associated health benefits, employment may also help to realize a modest cost savings due to reduced service utilization and disability.^79,80 Employment is a core tenet in the Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) model of recovery,⁸¹ and is also listed as an important recovery goal for individuals with behavioral health issues.⁸² BHD collects data on employment status on all the patients it serves as part of its state-mandated reporting requirements and will use this item in the CD data set.⁸³

Living Situation. This is measured as the proportion of people who live in permanent, supportive, stable housing; it may also be measured as the percentage of the population living with severe housing problems or who are homeless.

Brief review and suggested item(s). Housing problems can be conceptualized as 3 inter-related components: conditions within the home, neighborhood conditions, and housing affordability, each of which can contribute uniquely to poorer physical and mental health of individuals and families⁸⁴ and to educational outcomes for children.^85,86 Further, individuals who are homeless have a standardized mortality ratio 2 to 5 times that of the general population,^87-89 even after controlling for low income status,⁹⁰ and some evidence suggests these rates are even higher among unsheltered versus sheltered homeless individuals.⁹¹ Interventions to improve the condition of housing have demonstrated positive impacts on both physical and mental health,⁹² and a recent study found that individuals receiving housing assistance in the form of public housing or multifamily housing from the Department of Housing and Urban Development had better self-rated physical and mental health relative to individuals on the wait list for housing assistance.⁹³ Moreover, the provision of housing has been shown to promote reductions in substance use and health service utilization among homeless individuals with substance use disorders.⁹⁴ Rog and colleagues reviewed the literature on permanent supportive housing for individuals with substance use or mental health disorders who were homeless or disabled, and found that provision of housing led to reduced rates of homelessness, emergency department (ED) and inpatient utilization and increased consumer satisfaction.⁹⁵

Importantly, evidence suggests that housing is viewed as facilitative of recovery. For example, in a recent qualitative study of homeless individuals with mental illness, housing was seen as a critical first step in recovery, providing a sense of security, increasing feelings of personal independence and autonomy, improving perceptions of health and well-being, and affording a stable environment to rebuild relationships with important others.⁹⁶ BHD collects data on housing status on all the patients it serves as part of its state-mandated reporting requirements and will utilize this item in the CD data set.⁸³

Social Relationships. This is defined as recent interactions with family, supportive networks (formal and informal), and other recovery services.

Brief review and suggested item(s). Research has long established that social relationships have a significant impact on health, including rates of mortality as well as physical and mental health morbidity.^97-99 Social connectedness is another of the pillars supporting an individual’s recovery in SAMHSA’s formulation. Several reviews of the recovery literature^38,82 support its importance to the recovery process and inclusion in any assessment of holistic recovery. Social support has been shown to promote recovery among individuals with severe mental illness^100-102 and substance use disorders,¹⁰³ and may mitigate the progression of chronic, life-threatening physical illnesses.⁹⁷ For the purposes of BHD’s CD data set, the social support question from the “100 Million Healthier Lives Common Questionnaire for Adults” will be used to assess individuals’ perceived adequacy of social support.¹⁰⁴

Legal Involvement. Defined as involvement with the civil or criminal justice system, including arrests, imprisonment, or detainment.

Brief review and suggested item(s). Involvement in the criminal justice system is both disruptive for the individual in recovery and expensive to the larger health care system.¹⁰⁵ Individuals with substance use¹⁰⁶ and severe mental health disorders¹⁰⁷ are over-represented in the prison system, and evidence suggests that general physical and mental health declines while individuals are in prison.^108,109 Perhaps even more concerning, numerous studies have demonstrated an increase in mortality rates for individuals recently released from prison relative to the general population, particularly during the period immediately following release.^108-110 This relationship may even persist long term.¹¹¹ Further, research indicates that individuals recently released from prison have increased emergency care and hospital utilization.^112,113

Incarceration can have significant impacts on the health of the broader community as well. For example, research has found an association between parental incarceration to rates of infant mortality,¹¹⁴ increased behavioral and developmental problems of children of incarcerated parents,^115,116 lower rates of child support payments,¹¹⁷ and poorer cardiovascular health of female partners of incarcerated individuals.¹¹⁸ Formerly incarcerated individuals experience slower wage growth as well.¹¹⁹ However, evidence also indicates that engagement in mental health¹²⁰ and substance abuse¹²¹ treatment can reduce the likelihood of subsequent recidivism. As part of its state-mandated reporting, BHD is required to provide information on the criminal justice system involvement of its clients in the previous 6 months, including whether they have been jailed or imprisoned,⁸³ and this will function as its measure of legal involvement in its CD data set.

Socioeconomic Status. Socioeconomic status is the social standing or class of an individual or group. It is often measured as a combination of education, income, and occupation. It can also be defined subjectively, such as one’s evaluation of status relative to similar others or based on an individual’s interpretation of her or his financial needs.

Brief review and suggested item(s). A large body of evidence supports the existence of a robust relationship between lower SES and poor health, including mortality and chronic medical diseases,^122-124 as well as mental illness.^125-127 Although previous research has examined this relationship using objective indicators of SES (eg, income, education level, occupation), there has recently been an increased interest in exploring the relationship of subjective SES with health indices. Subjective SES is generally assessed by asking individuals to rate themselves relative to others in the society in which they live, in terms of wealth, occupation, educational level, or other indicators of social status. Evidence suggests that subjective SES is associated with objective measures of SES,^128-130 and relates to measures of physical and mental health as well, even after controlling for objective SES.^130-135 BHD will be using a modified version of the Subject SES Scale,^131,135 which is deployed in the “100 Million Healthier Lives Common Questionnaire for Adults.”¹⁰⁴

Acute Service Use. This is defined as an admission to a medical or psychiatric emergency room or to a medical or psychiatric hospital or to a detoxification facility.

Brief review and suggested item(s). The CMS Adult Core Set includes “plan all cause readmissions” as a key quality metric.⁵⁵ Hospital readmissions are also endorsed by the National Committee on Quality Assurance as one of its Health Effectiveness Data and Information Set (HEDIS) measures and by the National Quality Forum. Readmissions, despite their widespread endorsement, are a somewhat controversial measure. Although readmissions are costly to the health care system,¹³⁶ the relationship between readmissions and quality is inconsistent. For example, Krumholz and colleagues¹³⁷ found differential rates of readmission for the same patient discharged from 2 different hospitals, which were categorized based on previous readmission rates, suggesting that hospitals do have different levels of performance even when treating the same patient. However, other data indicate that 30-day, all-cause, risk-standardized readmission rates are not associated with hospital 30-day, all-cause, risk-standardized mortality rates.¹³⁸

Chin and colleague found that readmissions to the hospital that occurred more than 7 days post-discharge were likely due to community- and household-related factors, rather than hospital-related quality factors.¹³⁹ Transitional care interventions that have successfully reduced 30-day readmission rates are most often multicomponent and focus not just on hospital-based interventions (eg, discharge planning, education) but on follow-up care in the community by formal supports (eg, in-home visits, telephone calls, outpatient clinic appointments, case management) and informal supports (eg, family and friends).^140-143 Further, qualitative evidence suggests that some individuals perceive psychiatric hospitalizations to be the result of insufficient resources or unsuccessful attempts to maintain their stability in the community.¹⁴⁴ Thus, unplanned or avoidable hospital readmissions may represent a failure of the continuum of care not only from the perspective of the health care system, but from the patient perspective as well.

Frequent or nonurgent use of EDs is conceptually similar to excessive or avoidable inpatient utilization in several ways. For example, overuse of EDs is costly, with some estimates suggesting that it is responsible for up to $38 billion in wasteful spending each year.¹⁴⁵ Individuals with frequent ED visits have a greater disease burden¹⁴⁶ and an increased risk of mortality compared to nonfrequent users.¹⁴⁷ Research suggests that individuals who visit the ED for non-urgent issues do so because of perceived difficulties associated with accessing primary care, and the convenience of EDs relative to primary care.^148-150 Moreover, similar to the hospital readmission literature discussed earlier, successful strategies to reduce high rates of ED utilization generally focus on continuum of care interventions, such as provision of case management services.^151-155

This evidence implies that frequent ED utilization and hospital readmissions may not be a fundamental issue of quality (or lack thereof) in hospitals or EDs but rather a lack of, or ineffectual, transitional and continuum of care strategies and services. To underscore this point, some authors have argued that a system that is excessively crisis-oriented hinders recovery because it is reactive rather than proactive, predicated on the notion that one’s condition must deteriorate to receive care.¹⁵⁶

Although some organizations may have access to claims data or may function as self-contained health systems (eg, the Veterans Health Administration [VHA] ), others may not have access to such data. In the absence of claims data, patient self-report of service utilization has been used as a proxy for actual agency records.¹⁵⁷ Although concordance between medical and/or agency records and patient self-report has been variable,¹⁵⁷ evidence generally suggests that rates of agreement are higher the shorter the recall time interval.^158,159 BHD does not have access to comprehensive claims data and has therefore chosen to use 5 dichotomously scored (yes/no) questions—related to medical inpatient, medical ED, psychiatric inpatient, psychiatric ED, and detoxification use in the last 30 days—to represent the CD of acute service utilization.

The Second Aim: Quality of Care

Safety

Safety is defined as avoiding injuries to patients from the care that is intended to help them.

Brief review and suggested item(s). As noted in “Crossing the Quality Chasm,” the IOM’s seminal document, “the health care environment should be safe for all patients, in all of its processes, all the time.”¹⁶⁰ The landmark Harvard Medical Practice Study in 1991 found that adverse events occurred in nearly 4% of all hospital admissions and, among these, over a quarter were due to negligence.¹⁶¹ Other estimates of adverse events range as high as 17%.¹⁶² Indeed, a recent article by Makary and Daniel estimated that medical errors may be the third leading cause of death in the United States.¹⁶³ Unfortunately, research on safety in the mental health field has lagged behind that of physical health,¹⁶⁴ with evidence indicating that research in nonhospital settings in mental health care may be particularly scarce.¹⁶⁵ In a study of adverse events that occurred in psychiatric inpatient units in the VHA system between 2015 and 2016, Mills and colleagues found that of the 87 root cause analysis reports, suicide attempts were the most frequent, and, among safety events, falls were the most frequently reported, followed by medication events.¹⁶⁶ Another report on data collected from psychiatric inpatient units in the VHA revealed that nearly one-fifth of patients experienced a safety event, over half of which were deemed preventable.¹⁶⁷ These numbers likely represent an underestimation of the true volume of safety events, as another study by the same research group found that less than 40% of safety events described in patient medical records were documented in the incident reporting system.¹⁶⁸ BHD will utilize the total number of complaints and incident reports submitted within a given time frame as its “safety” metric in the CD data set.

Wait Time for Service

The CD is defined as the length of time between the date a patient first contacted BHD for services and the date of their first clinical service.

Brief review and suggested item(s). “Timeliness” was listed among the 6 aims for improvement in “Crossing the Quality Chasm” in 2001, and it remains no less relevant today.¹⁶⁰ For example, evidence indicates that access to primary care is inversely related to avoidable hospitalizations.¹⁶⁹ One study found that, of patients hospitalized for cardiovascular problems, those who had difficulty accessing routine care post discharge had higher 30-day readmission rates.¹⁷⁰ Among VHA patients, longer wait times are associated with more avoidable hospitalizations and higher rates of mortality.¹⁷¹ Longer wait times appear to decrease the likelihood of attending a first appointment for individuals with substance use^172,173 and mental health disorders.¹⁷⁴ Importantly, longer wait times are associated with lower ratings of the patient experience of care, including perceptions of the quality of and satisfaction with care,¹⁷⁵ and may be associated with worse outcomes for individuals in early intervention for psychosis treatment.¹⁷⁶ For the purposes of the CD data set, BHD will monitor the length of time between the date a patient first contacted BHD for services and the date of their first clinical service.

Patient satisfaction is defined as the degree of patients’ satisfaction with the care they have received.

Brief review and suggested item(s). Research has consistently demonstrated the relationship of the patient’s experience of care to a variety of safety and clinical effectiveness measures in medical health care,¹⁷⁷ and the therapeutic alliance is one of the most consistent predictors of outcomes in behavioral health, regardless of therapeutic modality.¹⁷⁸ Patient satisfaction is a commonly assessed aspect of the patient experience of care. Patient satisfaction scores have been correlated with patient adherence to recommended treatment regimens, care quality, and health outcomes.¹⁷⁹ For example, Aiken et al found that patient satisfaction with hospital care was associated with higher ratings of the quality and safety of nursing care in these hospitals.¹⁸⁰ Increased satisfaction with inpatient care has been associated with lower 30-day readmission rates for patients with acute myocardial infarction, heart failure, and pneumonia,¹⁸¹ and patients with schizophrenia who reported higher treatment satisfaction also reported better quality of life.^182,183 Many satisfaction survey options exist to evaluate this CD, including the Consumer Assessment of Healthcare Providers and Systems and the Client Satisfaction Questionnaire; BHD will utilize an outpatient behavioral health survey from a third-party vendor.

The Third Aim: Cost of Care

Cost of Care

This can be defined as the average cost to provide care per patient per month.

Brief review and suggested item(s). Per capita cost, or rather, the total cost of providing care to a circumscribed population divided by the total population, has been espoused as an important metric for the Triple Aim and the County Health Rankings.^6,13 Indeed, between 1960 and 2016, per capita expenditures for health care have grown 70-fold, and the percent of the national gross domestic product accounted for by health expenditures has more than tripled (5.0% to 17.9%).¹⁸⁴ One of the more common metrics deployed for assessing health care cost is the per capita per month cost, or rather, the per member per month cost of the predefined population for a given health care system.^6,185,186 In fact, some authors have proposed that cost of care can be used not only to track efficient resource allocation, but can also be a proxy for a healthier population as well (ie, as health improves, individuals use fewer and less-expensive services, thus costing the system less).¹⁸⁷ To assess this metric, BHD will calculate the total amount billed for patient care provided within BHD’s health network each month (irrespective of funding source) and then divide this sum by the number of members served each month. Although this measure does not account for care received at other health care facilities outside BHD’s provider network, nor does it include all the overhead costs associated with the care provided by BHD itself, it is consistent with the claims-based approach used or recommended by other authors.^6,188

The Fourth Aim: Staff Well-being

Staff Quality of Work Life

This can be defined as the quality of the work life of health care clinicians and staff.

Brief review and suggested item(s). Some authors have suggested that the Triple Aim framework is incomplete and have proffered compelling arguments that provider well-being and the quality of work life constitutes a fourth aim.² Provider burnout is prevalent in both medical^2,189 and behavioral health care.^190,191 Burnout among health care professionals has been associated with higher rates of perceived medical errors,¹⁹² lower patient satisfaction scores,^189,193 lower rates of provider empathy,¹⁹⁴ more negative attitudes towards patients,¹⁹⁵ and poorer staff mental and physical health.¹⁹¹

Burnout is also associated with higher rates of absenteeism, turnover intentions, and turnover.^{190,191,196,197} However, burnout is not the only predictor of staff turnover; for example, turnover rates are a useful proxy for staff quality of work life for several reasons.¹⁹⁸ First, turnover is associated with substantial direct and indirect costs, including lost productivity, increased errors, and lost revenue and recruitment costs, with some turnover cost estimates as high as $17 billion for physicians and $14 billion for nurses annually.^199-201 Second, research indicates that staff turnover can have a deleterious impact on implementation of evidence-based interventions.^202-205 Finally, consistent with the philosophy of utilizing existing data sources for the CD measures, turnover can be relatively easily extracted from administrative data for operated or contracted programs, and its collection does not place any additional burden on staff. As a large behavioral health system that is both a provider and payer of care, BHD will therefore examine the turnover rates of its internal administrative and clinical staff as well as the turnover of staff in its contracted provider network as its measures for the Staff Quality of Work Life CD.

Clinical Implications

These metrics can be deployed at any level of the organization. Clinicians may use 1 or more of the measures to track the recovery of individual clients, or in aggregate for their entire caseload. Similarly, managers can use these measures to assess the overall effectiveness of the programs for which they are responsible. Executive leaders can evaluate the impact of several programs or the system of care on the health of a subpopulation of clients with a specific condition, or for all their enrolled members. Further, not all measures need be utilized for every dashboard or evaluative effort. The benefit of a comprehensive set of measures lies in their flexibility—1 or more of the measures may be selected depending on the project being implemented or the interests of the stakeholder.

It is important to note that many of the CDs (and their accompanying measures) are aligned to/consistent with social determinants of health.^206,207 Evidence suggests that social determinants make substantial contributions to the overall health of individuals and populations and may even account for a greater proportion of variance in health outcomes than health care itself.²⁰⁸ The measures articulated here, therefore, can be used to assess whether and how effectively care provision has addressed these social determinants, as well as the relative impact their resolution may have on other health outcomes (eg, mortality, self-rated health).

These measures can also be used to stratify clients by clinical severity or degree of socioeconomic deprivation. The ability to adjust for risk has many applications in health care, particularly when organizations are attempting to implement value-based purchasing models, such as pay-for-performance contracts or other alternative payment models (population health-based payment models).²⁰⁹ Indeed, once fully implemented, the CDs and measures will enable BHD to more effectively build and execute different conceptual models of “value” (see references 210 and 211 for examples). We will be able to assess the progress our clients have made in care, the cost associated with that degree of improvement, the experience of those clients receiving that care, and the clinical and social variables that may influence the relative degree of improvement (or lack thereof). Thus, the CDs provide a conceptual and data-driven foundation for the Quadruple Aim and any quality initiatives that either catalyze or augment its implementation.

This article provides an overview of the CDs selected by BHD to help organize, focus, advance, and track its quality efforts within the framework of the Quadruple Aim. Although items aligned to each of these CDs are offered, the CDs themselves have been broadly conceptualized such that they can flexibly admit a variety of possible items and/or assessments to operationalize each CD and thus have potential applicability to other behavioral health systems, particularly public systems that have state-mandated and other data reporting requirements.

Bearing in mind the burden that growing data collection requirements can have on the provision of quality care and staff work satisfaction and burnout,^10,212 the CDs (and the items selected to represent each) are designed with “strategic parsimony” in mind. Although the CDs are inclusive in that they cover care quality, cost of care, staff quality of life, and general population health, only CDs and items undergirded by a solid evidence base and high value with regards to BHD’s mission and values, as determined by key stakeholders, were selected. Moreover, BHD attempted to make use of existing data collection and reporting mandates when selecting the final pool of items to reduce the measurement burden on staff and clients. Thus, the final set of CDs and items are designed to be comprehensive yet economical.

The CDs are deeply interrelated. Although each CD may be individually viewed as a valuable metric, improvements in any 1 CD will impact the others (eg, increasing care quality should impact population health, increasing staff quality of life should impact the quality of care). Moreover, this idea of interrelatedness acknowledges the need to view health systems and the populations they serve holistically, in that improvement is not simply the degree of change in any given metric (whether individually or collectively), but rather something more entirely. The concepts of value, quality, and health are complex, multidimensional, and dynamic, and the CDs that comprise these concepts should not be considered independently from one another. The CDs (and items) offered in this article are scalable in that they can be used at different levels of an organization depending on the question or stakeholder, and can be used individually or in combination with one another. Moreover, they are adaptable to a variety of risk-adjusted program, population health, and value-based evaluation models. It is hoped that the process articulated here, and the accompanying literature review, may benefit other public or government-run health systems in their own quality journey to operationalize the Quadruple Aim by developing a set of CDs.

Corresponding author: Walter Matthew Drymalski, PhD; walter.drymalski@milwaukeecountywi.gov.

Financial disclosures: None.

From the Milwaukee County Behavioral Health Division, Milwaukee, WI.

Abstract

Objective: Implementation of the Quadruple Aim of health care must begin with a clearly articulated set of concepts, or core domains (CDs), that comprise each aim. These CDs can then be operationalized with existing or new measures. If aligned to the organization’s mission and strategic goals, these CDs have the potential to focus quality improvement activities and reduce measurement burden. This article represents the efforts of a publicly funded behavioral health system to operationalize the Quadruple Aim through the development of CDs.

Methods: Various stakeholders across the organization were consulted on their perceptions of the Quadruple Aim and the CDs they believed should support it. Then, a review of existing literature on core metrics for health care and population health was completed, summarized, and integrated with the stakeholder feedback.

Results: These efforts led to the development and adoption of 15 CDs, with an accompanying literature review and set of recommendations of new and existing measures for each domain.

Conclusions: It is possible to create a comprehensive yet economical set of CDs and attendant measures that can be implemented in a staged, scalable, enterprise manner. It is hoped that the process articulated here, and the accompanying literature review, may be of some benefit to other public or government-run health systems in their own quality improvement journey to operationalize the Quadruple Aim by developing a set of CDs.

Keywords: quality measures; quality improvement; adult behavioral health.

First articulated in 2008, the Triple Aim proposes that health care systems should simultaneously seek to improve the patient’s experience of care, improve the health of populations, and reduce the per capita costs of care for populations.¹ More recently, some have argued that health care provider burnout can deleteriously impact the attainment of the Triple Aim and have therefore advocated for an expanded focus to include a fourth Aim, the work life quality of the staff.² Milwaukee County Behavioral Health Division (BHD), a publicly funded, county-based behavioral health care system in Milwaukee, Wisconsin, recently adopted the Quadruple Aim as the framework by which it will organize its quality activities.

Although originally developed for medical organizations, BHD believes that the Quadruple Aim has strong applicability to county-level behavioral health services. Many county-based behavioral health divisions provide a variety of programs to large segments of the county based on financial eligibility and/or clinical need, and thus often have responsibilities to populations or subpopulations, rather than programs. County health divisions, such as Milwaukee County’s Department of Health and Human Services, are often asked to improve outcomes and client experience of care with neutral growth budgets and less reliance on taxes to fund programs, while simultaneously attracting and retaining competent staff.

Crucial to the effective implementation of the Quadruple Aim, however, is a clear set of population- level measures that help organizations assess their progress.³ Unfortunately, as some authors have noted, evaluation of the Quadruple Aim remains a challenge because the “concepts of (population) health, quality of care and costs are not unanimously defined and measures for these concepts are under construction.”⁴ Several authors have provided some guidance to assist in the development of a set of measures that effectively capture the elements of the Quadruple Aim.^5,6 However, the recent rapid proliferation of quality measures in health care^7,8 has been both burdensome and costly for providers.^9,10 Any measures adopted should not only be as meaningful as possible with regards to assessing progress towards the basic aims of health care, but should also be parsimonious, to limit measurement burden for providers (and patients) and focus attention on important issues.^11,12

To select the most effective, parsimonious set of measures possible, one must first select a set of key foci from among the many possible areas of focus that the core measure is intended to represent. The core domains (CDs), if appropriately consistent with the strategic goals of the organization, provide a mechanism to orient the efforts of the organization at every level and help every staff member of the organization understand how his or her work impacts the progress towards these goals.¹¹ The CDs, therefore, represent the opportunity to affect a greater integration of efforts across the organization toward these shared aims, creating uniformity of purpose at every level. Further, increasing organizational attention on the CDs can also help to reduce measurement burden by streamlining and focusing the data capture processes on the most valuable elements of quality and health, and discarding other extraneous measures (albeit not at the expense of other reporting requirements).¹¹ The remainder of this article describes the CDs selected by BHD to assess its progress toward implementation of the Quadruple Aim and are organized by the Aim which they best represent.

Methods

To effectively implement the Quadruple Aim at BHD, it was necessary to clearly define the subpopulation of focus for our efforts.⁶ In this case, the subpopulation of interest was defined as all adult clients (18 years and older) who received at least 1 service encounter within a specified time frame from a program that BHD either operated or contracted with to provide care. Services provided by the BHD network include everything from psychiatric inpatient services to mental health and addiction treatment and care management. A limited array of social services, including housing and employment services, is also available to eligible consumers. BHD is the county-run behavioral health provider for individuals who are uninsured or underinsured in Milwaukee County, a demographically diverse, primarily urban county of approximately 950,000 people located in Wisconsin. Approximately 15,000 adults receive services at BHD each year.

This work began by obtaining executive sponsorship for the project, in this case from the Chief Operations Officer and Executive Medical Director of BHD. With their backing, an initial review of the literature produced a preliminary set of possible domains, for which we created working definitions. We then made a list of key stakeholders throughout BHD to whom we needed to present the idea of the Quadruple Aim, and the CDs under each Aim, to secure their support. These stakeholders, which included individuals involved in quality activities, program managers, and executive leadership, were strategically selected based on their relative influence within the organization. A set of brief presentations and handouts explaining the project were then developed and shared at different focus groups with these stakeholders over the course of 6 months. These focus groups served to not only educate the organization about the Quadruple Aim and the CDs but afforded participants an opportunity to provide feedback as well.

During the focus groups, we asked participants which domains they believed were most important (were “core”) when operationalizing the Quadruple Aim. The focus groups provided feedback on the domain definitions, feedback that was used to develop uniform, mutually agreed upon definitions for the CDs that were generalizable to all departments at BHD, regardless of the focus of their services within the continuum of care or the continuum of age. This was a crucial step, as it will eventually enable BHD to aggregate data across departments, even if there are minor discrepancies in the specific items they use to assess the CDs. Comments from the focus groups ultimately resulted in a truncated list of domains and definitions, which, coupled with the literature review, resulted in our final set of CDs.

During our review of the literature, we also looked for items that we felt could best represent each CD in the briefest, most meaningful way. (These items were not meant to supersede existing data, but to provide examples that could be implemented with existing data or recommendations that could be utilized in the absence of existing data.) During this process, we made every effort to make use of existing data-reporting requirements. For example, if we had a state mandate to collect data on housing status, we attempted to leverage this required data point to represent the CD related to housing. In other cases, we attempted to utilize claims or other administrative data to operationalize the CD, such as in the cost-of-care metric articulated in the section the Third Aim. For CDs for which no data existed or were insufficient, we emphasized the use of single- versus multi-item scales. For example, if we found a single-item global assessment of quality of life that had good psychometric properties relative to its longer parent scale, we selected the single item. This approach to item selection allowed us to create the most efficient, parsimonious set of measures possible, which we believed would enable us to comprehensively assess all the CDs with the least amount of burden to staff and clients. These items were presented at stakeholder focus groups, during which we asked for comments on the existing measures in their program or department and gave them the opportunity to comment on the new recommended measures.

A working definition is provided for each CD, followed by a brief review of the research base supporting its inclusion in the final list. The item(s) selected by BHD to represent each CD and the source of the item(s) are then supplied. These items were based either on measures currently collected because of existing reporting mandates or, in the case where extant measures were not available, on new items that demonstrated acceptable psychometric properties in the research literature. The CDs and items are organized by the Aim they best represent. A full list of the CDs by Quadruple Aim and items by CD is provided in the Appendix of the online version of this article. This article concludes with a brief summary of this effort and a discussion of how staff will utilize these items at different levels throughout the BHD system.

The First Aim: Population Health

Health Outcomes

Deaths. This can be defined as the cause of death, as determined by the medical examiner’s office (where appropriate) or as the age at time of death. This CD can also be reported as proportion of deaths considered premature (eg, before age 75) or calculated as total years of potential life lost.

Brief review and suggested item(s). Rates and causes of premature mortality are critical foci for the County Health Rankings & Roadmaps,¹³ the Institute for Healthcare Improvement’s “Guide to Measuring the Triple Aim,”⁶ the Centers for Disease Control and Prevention’s “Community Health Assessment for Population Health Improvement,”¹⁴ and the Institute of Medicine’s (IOM) “Vital Signs: Core Metrics for Health and Health Care Progress.”¹¹ There is ample evidence that individuals with serious mental illness are at increased risk of early mortality relative to the general population,^15-18 and this risk applies to those with substance use disorders as well.^15,19-20 BHD tracks all deaths that occur while patients are receiving BHD-funded, community-based services.

Self-Reported Health and Well-Being. This CD asks patients to rate their current physical and mental health status, as well as their overall quality of life.

Brief review and suggested item(s): Self-rated physical health. Premature mortality among individuals with behavioral health issues appears to be due, in large part, to their increased vulnerability to the development of medical comorbidities.^16,21 A single self-rating question has demonstrated considerable sensitivity to premature mortality,^22,23 with predictive properties up to a decade prior to death.^24,25 Further, self-rated health has been associated with subsequent functional decline,^26,27 acute service utilization,^28,29 and overall health care costs.²⁸

Brief review and suggested item(s): Self-rated mental health. Mental health disorders are associated with significant disability worldwide,³⁰ and comorbid mental health issues can exacerbate the course of other medical problems. For example, depression is associated with increased rates of mortality among individuals with diabetes and³¹ cardiovascular disease,³² as well as with rates of overall mortality,³³ and psychiatric comorbidity is associated with longer lengths of stay and higher costs among patients hospitalized for medical problems.³⁴ Research has found that a single-item measure of self-rated mental health is associated with the presence of psychiatric diagnoses, psychiatric symptoms, and subsequent depression and serious mental illness up to 1 year post-assessment.^35,36 There is even evidence that self-rated mental health may be more strongly associated with self-ratings of overall health than self-ratings of physical health.³⁷

Brief review and suggested item(s): Self-rated quality of life. Quality of life is a critical component of the recovery journey and overall health.³⁸ For example, the County Health Rankings & Roadmaps lists “quality of life” as 1 of its key “health outcomes” in its County Health Rankings.¹³ As some authors have noted, quality of life is often inferred from other “objective” recovery domains, such as employment, health status, or housing status. However, there is evidence that these objective domains are functionally distinct from the inherently subjective construct of quality of life.³⁹ This has led other authors to conclude that these domains should be assessed separately when evaluating outcomes.⁴⁰ Single-item quality of life assessments have been used in research with individuals with cancer,⁴¹ adults with disabilities,⁴² patients with cystic fibrosis,⁴³ and children with epilepsy.⁴⁴ For this effort, BHD selected the first global quality of life item from the World Health Organization’s WHOQOL-BREF quality of life assessment,⁴⁵ an item used in other quality of life research.⁴⁶

Health Factors

Substance Use. This CD is a composite of 4 different types of substance use, any recent heavy alcohol use (defined as 5 or more drinks in one sitting), any recent drug use, any recent prescription drug abuse, and any recent tobacco use.

Brief review and suggested item(s). As noted, substance use disorders confer an increased risk for early mortality^15,19 and are significantly implicated in disease disability burden worldwide.³⁰ Substance use has also been associated with both the onset^47,48 and exacerbation of mental health diagnoses.^49-51 Further, substance use appears to heighten the risk of violence in the general population⁵² and especially among those with a co-occurring mental illness.^53,54 The County Health Rankings & Roadmaps list alcohol and drug use as key behaviors to address to improve the overall health of a given county,¹³ and the Centers for Medicare & Medicaid Services (CMS) has endorsed initiation and engagement in addiction treatment as one of the measures in its Adult Core Set.⁵⁵

Tobacco use continues to be one of the most significant risk factors for early mortality worldwide, and evidence indicates that it is associated with a lower life expectancy of nearly 10 years.⁵⁶ Unfortunately, rates of tobacco use are even higher among those with severe mental illness relative to the general population, and their rates of smoking cessation are lower.^57,58 Tobacco use is a significant risk factor for the high rates of early mortality in individuals with severe mental illness.¹⁸ Further, a recent meta-analysis noted that, relative to those who continued to smoke, those who ceased smoking had reduced rates of psychological distress and increased quality of life rankings.⁵⁹ Reducing tobacco use is one of the key components of the County Health Rankings & Roadmaps, and medication assistance with smoking and tobacco use cessation is also listed in the CMS Adult Core Set.^13,55

An accumulating body of evidence suggests that single-item measures can adequately detect alcohol^60-62 and drug use disorders.^60-64 McNeely and colleagues recently developed and tested a brief 4-item screen, the Tobacco, Alcohol, Prescription medication, and other Substance use (TAPS) tool.^65,66 Preliminary evidence suggests that the TAPS tool can effectively identify the presence of problematic and disordered use of tobacco, alcohol, prescription medications, and other drugs.^65-67 BHD will use the 4 items from the TAPS tool to represent its substance use CD.

Education/Employment Status. This CD assesses the proportion of BHD members who have completed high school, who are in some type of educational or training program, or who are engaged in some type of employment activity (defined as full-time, part-time, supported, sheltered workshop, or as a full-time homemaker).

Brief review and suggested item(s). Research indicates that unemployment is a risk factor for mortality, even after controlling for other risk factors (eg, age, sex, socioeconomic status [SES], health).⁶⁸ Unemployment is associated with poorer physical and mental health in the general population and among those with disabilities.^69-71 Promisingly, evidence suggests that gaining employment or re-employment is associated with better health,⁷² even for individuals with substance use disorders⁷³ or moderate⁷⁴ to severe mental health disorders.^75-78 Some authors have even proposed that, above and beyond the associated health benefits, employment may also help to realize a modest cost savings due to reduced service utilization and disability.^79,80 Employment is a core tenet in the Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) model of recovery,⁸¹ and is also listed as an important recovery goal for individuals with behavioral health issues.⁸² BHD collects data on employment status on all the patients it serves as part of its state-mandated reporting requirements and will use this item in the CD data set.⁸³

Living Situation. This is measured as the proportion of people who live in permanent, supportive, stable housing; it may also be measured as the percentage of the population living with severe housing problems or who are homeless.

Brief review and suggested item(s). Housing problems can be conceptualized as 3 inter-related components: conditions within the home, neighborhood conditions, and housing affordability, each of which can contribute uniquely to poorer physical and mental health of individuals and families⁸⁴ and to educational outcomes for children.^85,86 Further, individuals who are homeless have a standardized mortality ratio 2 to 5 times that of the general population,^87-89 even after controlling for low income status,⁹⁰ and some evidence suggests these rates are even higher among unsheltered versus sheltered homeless individuals.⁹¹ Interventions to improve the condition of housing have demonstrated positive impacts on both physical and mental health,⁹² and a recent study found that individuals receiving housing assistance in the form of public housing or multifamily housing from the Department of Housing and Urban Development had better self-rated physical and mental health relative to individuals on the wait list for housing assistance.⁹³ Moreover, the provision of housing has been shown to promote reductions in substance use and health service utilization among homeless individuals with substance use disorders.⁹⁴ Rog and colleagues reviewed the literature on permanent supportive housing for individuals with substance use or mental health disorders who were homeless or disabled, and found that provision of housing led to reduced rates of homelessness, emergency department (ED) and inpatient utilization and increased consumer satisfaction.⁹⁵

Importantly, evidence suggests that housing is viewed as facilitative of recovery. For example, in a recent qualitative study of homeless individuals with mental illness, housing was seen as a critical first step in recovery, providing a sense of security, increasing feelings of personal independence and autonomy, improving perceptions of health and well-being, and affording a stable environment to rebuild relationships with important others.⁹⁶ BHD collects data on housing status on all the patients it serves as part of its state-mandated reporting requirements and will utilize this item in the CD data set.⁸³

Social Relationships. This is defined as recent interactions with family, supportive networks (formal and informal), and other recovery services.

Brief review and suggested item(s). Research has long established that social relationships have a significant impact on health, including rates of mortality as well as physical and mental health morbidity.^97-99 Social connectedness is another of the pillars supporting an individual’s recovery in SAMHSA’s formulation. Several reviews of the recovery literature^38,82 support its importance to the recovery process and inclusion in any assessment of holistic recovery. Social support has been shown to promote recovery among individuals with severe mental illness^100-102 and substance use disorders,¹⁰³ and may mitigate the progression of chronic, life-threatening physical illnesses.⁹⁷ For the purposes of BHD’s CD data set, the social support question from the “100 Million Healthier Lives Common Questionnaire for Adults” will be used to assess individuals’ perceived adequacy of social support.¹⁰⁴

Legal Involvement. Defined as involvement with the civil or criminal justice system, including arrests, imprisonment, or detainment.

Brief review and suggested item(s). Involvement in the criminal justice system is both disruptive for the individual in recovery and expensive to the larger health care system.¹⁰⁵ Individuals with substance use¹⁰⁶ and severe mental health disorders¹⁰⁷ are over-represented in the prison system, and evidence suggests that general physical and mental health declines while individuals are in prison.^108,109 Perhaps even more concerning, numerous studies have demonstrated an increase in mortality rates for individuals recently released from prison relative to the general population, particularly during the period immediately following release.^108-110 This relationship may even persist long term.¹¹¹ Further, research indicates that individuals recently released from prison have increased emergency care and hospital utilization.^112,113

Incarceration can have significant impacts on the health of the broader community as well. For example, research has found an association between parental incarceration to rates of infant mortality,¹¹⁴ increased behavioral and developmental problems of children of incarcerated parents,^115,116 lower rates of child support payments,¹¹⁷ and poorer cardiovascular health of female partners of incarcerated individuals.¹¹⁸ Formerly incarcerated individuals experience slower wage growth as well.¹¹⁹ However, evidence also indicates that engagement in mental health¹²⁰ and substance abuse¹²¹ treatment can reduce the likelihood of subsequent recidivism. As part of its state-mandated reporting, BHD is required to provide information on the criminal justice system involvement of its clients in the previous 6 months, including whether they have been jailed or imprisoned,⁸³ and this will function as its measure of legal involvement in its CD data set.

Socioeconomic Status. Socioeconomic status is the social standing or class of an individual or group. It is often measured as a combination of education, income, and occupation. It can also be defined subjectively, such as one’s evaluation of status relative to similar others or based on an individual’s interpretation of her or his financial needs.

Brief review and suggested item(s). A large body of evidence supports the existence of a robust relationship between lower SES and poor health, including mortality and chronic medical diseases,^122-124 as well as mental illness.^125-127 Although previous research has examined this relationship using objective indicators of SES (eg, income, education level, occupation), there has recently been an increased interest in exploring the relationship of subjective SES with health indices. Subjective SES is generally assessed by asking individuals to rate themselves relative to others in the society in which they live, in terms of wealth, occupation, educational level, or other indicators of social status. Evidence suggests that subjective SES is associated with objective measures of SES,^128-130 and relates to measures of physical and mental health as well, even after controlling for objective SES.^130-135 BHD will be using a modified version of the Subject SES Scale,^131,135 which is deployed in the “100 Million Healthier Lives Common Questionnaire for Adults.”¹⁰⁴

Acute Service Use. This is defined as an admission to a medical or psychiatric emergency room or to a medical or psychiatric hospital or to a detoxification facility.

Brief review and suggested item(s). The CMS Adult Core Set includes “plan all cause readmissions” as a key quality metric.⁵⁵ Hospital readmissions are also endorsed by the National Committee on Quality Assurance as one of its Health Effectiveness Data and Information Set (HEDIS) measures and by the National Quality Forum. Readmissions, despite their widespread endorsement, are a somewhat controversial measure. Although readmissions are costly to the health care system,¹³⁶ the relationship between readmissions and quality is inconsistent. For example, Krumholz and colleagues¹³⁷ found differential rates of readmission for the same patient discharged from 2 different hospitals, which were categorized based on previous readmission rates, suggesting that hospitals do have different levels of performance even when treating the same patient. However, other data indicate that 30-day, all-cause, risk-standardized readmission rates are not associated with hospital 30-day, all-cause, risk-standardized mortality rates.¹³⁸

Chin and colleague found that readmissions to the hospital that occurred more than 7 days post-discharge were likely due to community- and household-related factors, rather than hospital-related quality factors.¹³⁹ Transitional care interventions that have successfully reduced 30-day readmission rates are most often multicomponent and focus not just on hospital-based interventions (eg, discharge planning, education) but on follow-up care in the community by formal supports (eg, in-home visits, telephone calls, outpatient clinic appointments, case management) and informal supports (eg, family and friends).^140-143 Further, qualitative evidence suggests that some individuals perceive psychiatric hospitalizations to be the result of insufficient resources or unsuccessful attempts to maintain their stability in the community.¹⁴⁴ Thus, unplanned or avoidable hospital readmissions may represent a failure of the continuum of care not only from the perspective of the health care system, but from the patient perspective as well.

Frequent or nonurgent use of EDs is conceptually similar to excessive or avoidable inpatient utilization in several ways. For example, overuse of EDs is costly, with some estimates suggesting that it is responsible for up to $38 billion in wasteful spending each year.¹⁴⁵ Individuals with frequent ED visits have a greater disease burden¹⁴⁶ and an increased risk of mortality compared to nonfrequent users.¹⁴⁷ Research suggests that individuals who visit the ED for non-urgent issues do so because of perceived difficulties associated with accessing primary care, and the convenience of EDs relative to primary care.^148-150 Moreover, similar to the hospital readmission literature discussed earlier, successful strategies to reduce high rates of ED utilization generally focus on continuum of care interventions, such as provision of case management services.^151-155

This evidence implies that frequent ED utilization and hospital readmissions may not be a fundamental issue of quality (or lack thereof) in hospitals or EDs but rather a lack of, or ineffectual, transitional and continuum of care strategies and services. To underscore this point, some authors have argued that a system that is excessively crisis-oriented hinders recovery because it is reactive rather than proactive, predicated on the notion that one’s condition must deteriorate to receive care.¹⁵⁶

Although some organizations may have access to claims data or may function as self-contained health systems (eg, the Veterans Health Administration [VHA] ), others may not have access to such data. In the absence of claims data, patient self-report of service utilization has been used as a proxy for actual agency records.¹⁵⁷ Although concordance between medical and/or agency records and patient self-report has been variable,¹⁵⁷ evidence generally suggests that rates of agreement are higher the shorter the recall time interval.^158,159 BHD does not have access to comprehensive claims data and has therefore chosen to use 5 dichotomously scored (yes/no) questions—related to medical inpatient, medical ED, psychiatric inpatient, psychiatric ED, and detoxification use in the last 30 days—to represent the CD of acute service utilization.

The Second Aim: Quality of Care

Safety

Safety is defined as avoiding injuries to patients from the care that is intended to help them.

Brief review and suggested item(s). As noted in “Crossing the Quality Chasm,” the IOM’s seminal document, “the health care environment should be safe for all patients, in all of its processes, all the time.”¹⁶⁰ The landmark Harvard Medical Practice Study in 1991 found that adverse events occurred in nearly 4% of all hospital admissions and, among these, over a quarter were due to negligence.¹⁶¹ Other estimates of adverse events range as high as 17%.¹⁶² Indeed, a recent article by Makary and Daniel estimated that medical errors may be the third leading cause of death in the United States.¹⁶³ Unfortunately, research on safety in the mental health field has lagged behind that of physical health,¹⁶⁴ with evidence indicating that research in nonhospital settings in mental health care may be particularly scarce.¹⁶⁵ In a study of adverse events that occurred in psychiatric inpatient units in the VHA system between 2015 and 2016, Mills and colleagues found that of the 87 root cause analysis reports, suicide attempts were the most frequent, and, among safety events, falls were the most frequently reported, followed by medication events.¹⁶⁶ Another report on data collected from psychiatric inpatient units in the VHA revealed that nearly one-fifth of patients experienced a safety event, over half of which were deemed preventable.¹⁶⁷ These numbers likely represent an underestimation of the true volume of safety events, as another study by the same research group found that less than 40% of safety events described in patient medical records were documented in the incident reporting system.¹⁶⁸ BHD will utilize the total number of complaints and incident reports submitted within a given time frame as its “safety” metric in the CD data set.

Wait Time for Service

The CD is defined as the length of time between the date a patient first contacted BHD for services and the date of their first clinical service.

Brief review and suggested item(s). “Timeliness” was listed among the 6 aims for improvement in “Crossing the Quality Chasm” in 2001, and it remains no less relevant today.¹⁶⁰ For example, evidence indicates that access to primary care is inversely related to avoidable hospitalizations.¹⁶⁹ One study found that, of patients hospitalized for cardiovascular problems, those who had difficulty accessing routine care post discharge had higher 30-day readmission rates.¹⁷⁰ Among VHA patients, longer wait times are associated with more avoidable hospitalizations and higher rates of mortality.¹⁷¹ Longer wait times appear to decrease the likelihood of attending a first appointment for individuals with substance use^172,173 and mental health disorders.¹⁷⁴ Importantly, longer wait times are associated with lower ratings of the patient experience of care, including perceptions of the quality of and satisfaction with care,¹⁷⁵ and may be associated with worse outcomes for individuals in early intervention for psychosis treatment.¹⁷⁶ For the purposes of the CD data set, BHD will monitor the length of time between the date a patient first contacted BHD for services and the date of their first clinical service.

Patient satisfaction is defined as the degree of patients’ satisfaction with the care they have received.

Brief review and suggested item(s). Research has consistently demonstrated the relationship of the patient’s experience of care to a variety of safety and clinical effectiveness measures in medical health care,¹⁷⁷ and the therapeutic alliance is one of the most consistent predictors of outcomes in behavioral health, regardless of therapeutic modality.¹⁷⁸ Patient satisfaction is a commonly assessed aspect of the patient experience of care. Patient satisfaction scores have been correlated with patient adherence to recommended treatment regimens, care quality, and health outcomes.¹⁷⁹ For example, Aiken et al found that patient satisfaction with hospital care was associated with higher ratings of the quality and safety of nursing care in these hospitals.¹⁸⁰ Increased satisfaction with inpatient care has been associated with lower 30-day readmission rates for patients with acute myocardial infarction, heart failure, and pneumonia,¹⁸¹ and patients with schizophrenia who reported higher treatment satisfaction also reported better quality of life.^182,183 Many satisfaction survey options exist to evaluate this CD, including the Consumer Assessment of Healthcare Providers and Systems and the Client Satisfaction Questionnaire; BHD will utilize an outpatient behavioral health survey from a third-party vendor.

The Third Aim: Cost of Care

Cost of Care

This can be defined as the average cost to provide care per patient per month.

Brief review and suggested item(s). Per capita cost, or rather, the total cost of providing care to a circumscribed population divided by the total population, has been espoused as an important metric for the Triple Aim and the County Health Rankings.^6,13 Indeed, between 1960 and 2016, per capita expenditures for health care have grown 70-fold, and the percent of the national gross domestic product accounted for by health expenditures has more than tripled (5.0% to 17.9%).¹⁸⁴ One of the more common metrics deployed for assessing health care cost is the per capita per month cost, or rather, the per member per month cost of the predefined population for a given health care system.^6,185,186 In fact, some authors have proposed that cost of care can be used not only to track efficient resource allocation, but can also be a proxy for a healthier population as well (ie, as health improves, individuals use fewer and less-expensive services, thus costing the system less).¹⁸⁷ To assess this metric, BHD will calculate the total amount billed for patient care provided within BHD’s health network each month (irrespective of funding source) and then divide this sum by the number of members served each month. Although this measure does not account for care received at other health care facilities outside BHD’s provider network, nor does it include all the overhead costs associated with the care provided by BHD itself, it is consistent with the claims-based approach used or recommended by other authors.^6,188

The Fourth Aim: Staff Well-being

Staff Quality of Work Life

This can be defined as the quality of the work life of health care clinicians and staff.

Brief review and suggested item(s). Some authors have suggested that the Triple Aim framework is incomplete and have proffered compelling arguments that provider well-being and the quality of work life constitutes a fourth aim.² Provider burnout is prevalent in both medical^2,189 and behavioral health care.^190,191 Burnout among health care professionals has been associated with higher rates of perceived medical errors,¹⁹² lower patient satisfaction scores,^189,193 lower rates of provider empathy,¹⁹⁴ more negative attitudes towards patients,¹⁹⁵ and poorer staff mental and physical health.¹⁹¹

Burnout is also associated with higher rates of absenteeism, turnover intentions, and turnover.^{190,191,196,197} However, burnout is not the only predictor of staff turnover; for example, turnover rates are a useful proxy for staff quality of work life for several reasons.¹⁹⁸ First, turnover is associated with substantial direct and indirect costs, including lost productivity, increased errors, and lost revenue and recruitment costs, with some turnover cost estimates as high as $17 billion for physicians and $14 billion for nurses annually.^199-201 Second, research indicates that staff turnover can have a deleterious impact on implementation of evidence-based interventions.^202-205 Finally, consistent with the philosophy of utilizing existing data sources for the CD measures, turnover can be relatively easily extracted from administrative data for operated or contracted programs, and its collection does not place any additional burden on staff. As a large behavioral health system that is both a provider and payer of care, BHD will therefore examine the turnover rates of its internal administrative and clinical staff as well as the turnover of staff in its contracted provider network as its measures for the Staff Quality of Work Life CD.

Clinical Implications

These metrics can be deployed at any level of the organization. Clinicians may use 1 or more of the measures to track the recovery of individual clients, or in aggregate for their entire caseload. Similarly, managers can use these measures to assess the overall effectiveness of the programs for which they are responsible. Executive leaders can evaluate the impact of several programs or the system of care on the health of a subpopulation of clients with a specific condition, or for all their enrolled members. Further, not all measures need be utilized for every dashboard or evaluative effort. The benefit of a comprehensive set of measures lies in their flexibility—1 or more of the measures may be selected depending on the project being implemented or the interests of the stakeholder.

It is important to note that many of the CDs (and their accompanying measures) are aligned to/consistent with social determinants of health.^206,207 Evidence suggests that social determinants make substantial contributions to the overall health of individuals and populations and may even account for a greater proportion of variance in health outcomes than health care itself.²⁰⁸ The measures articulated here, therefore, can be used to assess whether and how effectively care provision has addressed these social determinants, as well as the relative impact their resolution may have on other health outcomes (eg, mortality, self-rated health).

These measures can also be used to stratify clients by clinical severity or degree of socioeconomic deprivation. The ability to adjust for risk has many applications in health care, particularly when organizations are attempting to implement value-based purchasing models, such as pay-for-performance contracts or other alternative payment models (population health-based payment models).²⁰⁹ Indeed, once fully implemented, the CDs and measures will enable BHD to more effectively build and execute different conceptual models of “value” (see references 210 and 211 for examples). We will be able to assess the progress our clients have made in care, the cost associated with that degree of improvement, the experience of those clients receiving that care, and the clinical and social variables that may influence the relative degree of improvement (or lack thereof). Thus, the CDs provide a conceptual and data-driven foundation for the Quadruple Aim and any quality initiatives that either catalyze or augment its implementation.

This article provides an overview of the CDs selected by BHD to help organize, focus, advance, and track its quality efforts within the framework of the Quadruple Aim. Although items aligned to each of these CDs are offered, the CDs themselves have been broadly conceptualized such that they can flexibly admit a variety of possible items and/or assessments to operationalize each CD and thus have potential applicability to other behavioral health systems, particularly public systems that have state-mandated and other data reporting requirements.

Bearing in mind the burden that growing data collection requirements can have on the provision of quality care and staff work satisfaction and burnout,^10,212 the CDs (and the items selected to represent each) are designed with “strategic parsimony” in mind. Although the CDs are inclusive in that they cover care quality, cost of care, staff quality of life, and general population health, only CDs and items undergirded by a solid evidence base and high value with regards to BHD’s mission and values, as determined by key stakeholders, were selected. Moreover, BHD attempted to make use of existing data collection and reporting mandates when selecting the final pool of items to reduce the measurement burden on staff and clients. Thus, the final set of CDs and items are designed to be comprehensive yet economical.

The CDs are deeply interrelated. Although each CD may be individually viewed as a valuable metric, improvements in any 1 CD will impact the others (eg, increasing care quality should impact population health, increasing staff quality of life should impact the quality of care). Moreover, this idea of interrelatedness acknowledges the need to view health systems and the populations they serve holistically, in that improvement is not simply the degree of change in any given metric (whether individually or collectively), but rather something more entirely. The concepts of value, quality, and health are complex, multidimensional, and dynamic, and the CDs that comprise these concepts should not be considered independently from one another. The CDs (and items) offered in this article are scalable in that they can be used at different levels of an organization depending on the question or stakeholder, and can be used individually or in combination with one another. Moreover, they are adaptable to a variety of risk-adjusted program, population health, and value-based evaluation models. It is hoped that the process articulated here, and the accompanying literature review, may benefit other public or government-run health systems in their own quality journey to operationalize the Quadruple Aim by developing a set of CDs.

Corresponding author: Walter Matthew Drymalski, PhD; walter.drymalski@milwaukeecountywi.gov.

Financial disclosures: None.

From Dignity Health Methodist Hospital of Sacramento Family Medicine Residency Program, Sacramento, CA (Dr. Oldach); Nationwide Children’s Hospital, Columbus, OH (Dr. Irwin); OhioHealth Research Institute, Columbus, OH (Dr. Pershing); Department of Clinical Transformation, OhioHealth, Columbus, OH (Dr. Zigmont and Dr. Gascon); and Department of Geriatrics, OhioHealth, Columbus, OH (Dr. Skully).

Abstract

Objective: An interdisciplinary committee was formed to identify factors contributing to surgical delays in urgent hip fracture repair at an urban, level 1 trauma center, with the goal of reducing preoperative time to less than 24 hours. Surgical optimization was identified as a primary, modifiable factor, as surgeons were reluctant to clear patients for surgery without cardiac consultation. Preoperative transthoracic echocardiogram (TTE) was recommended as a safe alternative to cardiac consultation in most patients.

Methods: A retrospective review was conducted for patients who underwent urgent hip fracture repair between January 2010 and April 2014 (n = 316). Time to medical optimization, time to surgery, hospital length of stay, and anesthesia induction were compared for 3 patient groups of interest: those who received (1) neither TTE nor cardiology consultation (ie, direct to surgery); (2) a preoperative TTE; or (3) preoperative cardiac consultation.

Results: There were significant between-group differences in medical optimization time (P = 0.001) and mean time to surgery (P < 0.001) when comparing the 3 groups of interest. Patients in the preoperative cardiac consult group had the longest times, followed by the TTE and direct-to-surgery groups. There were no differences in the type of induction agent used across treatment groups when stratifying by ejection fraction.

Conclusion: Preoperative TTE allows for decreased preoperative time compared to a cardiology consultation. It provides an easily implemented inter-departmental, intra-institutional intervention to decrease preoperative time in patients presenting with hip fractures.

Keywords: surgical delay; preoperative risk stratification; process improvement.

Hip fractures are common, expensive, and associated with poor outcomes.^1,2 Ample literature suggests that morbidity, mortality, and cost of care may be reduced by minimizing surgical delays.^3-5 While individual reports indicate mixed evidence, in a 2010 meta-analysis, surgery within 72 hours was associated with significant reductions in pneumonia and pressure sores, as well as a 19% reduction in all-cause mortality through 1 year.⁶ Additional reviews suggest evidence of improved patient outcomes (pain, length of stay, non-union, and/or mortality) when surgery occurs early, within 12 to 72 hours after injury.^4,6,7 Regardless of the definition of “early surgery” used, surgical delay remains a challenge, often due to organizational factors, including admission day of the week and hospital staffing, and patient characteristics, such as comorbidities, echocardiographic findings, age, and insurance status.^7-9

Among factors that contribute to surgical delays, the need for preoperative cardiovascular risk stratification is significantly modifiable.¹⁰ The American College of Cardiology (ACC)/American Heart Association (AHA) Task Force risk stratification framework for preoperative cardiac testing assists clinicians in determining surgical urgency, active cardiac conditions, cardiovascular risk factors, and functional capacity of each patient, and is well established for low- or intermediate-risk patients.¹¹ Specifically, metabolic equivalents (METs) measurements are used to identify medically stable patients with good or excellent functional capacity versus poor or unknown functional status. Patients with ≥ 4 METs may proceed to surgery without further testing; patients with < 4 METs may either proceed with planned surgery or undergo additional testing. Patients with a perceived increased risk profile who require urgent or semi-urgent hip fracture repair may be confounded by disagreement about required preoperative cardiac testing.

At OhioHealth Grant Medical Center (GMC), an urban, level 1 trauma center, the consideration of further preoperative noninvasive testing frequently contributed to surgical delays. In 2009, hip fracture patients arriving to the emergency department (ED) waited an average of 51 hours before being transferred to the operating room (OR) for surgery. Presuming prompt surgery is both desirable and feasible, the Grant Hip Fracture Management Committee (GHFMC) was developed in order to expedite surgeries in hip fracture patients. The GHFMC recommended a preoperative hip fracture protocol, and the outcomes from protocol implementation are described in this article.

Methods

This study was approved by the OhioHealth Institutional Review Board, with a waiver of the informed consent requirement. Medical records from patients treated at GMC during the time period between January 2010 and April 2014 (ie, following implementation of GHFMC recommendations) were retrospectively reviewed to identify the extent to which the use of preoperative transthoracic echocardiography (TTE) reduced average time to surgery and total length of stay, compared to cardiac consultation. This chart review included 316 participants and was used to identify primary induction agent utilized, time to medical optimization, time to surgery, and total length of hospital stay.

Intervention

The GHFMC conducted a 9-month quality improvement project to decrease ED-to-OR time to less than 24 hours for hip fracture patients. The multidisciplinary committee consisted of physicians from orthopedic surgery, anesthesia, hospital medicine, and geriatrics, along with key administrators and nurse outcomes managers. While there is lack of complete clarity surrounding optimal surgical timing, the committee decided that surgery within 24 hours would be beneficial for the majority of patients and therefore was considered a prudent goal.

Based on identified barriers that contributed to surgical delays, several process improvement strategies were implemented, including admitting patients to the hospitalist service, engaging the orthopedic trauma team, and implementing pre- and postoperative protocols and order sets (eg, ED and pain management order sets). Specific emphasis was placed on establishing guidelines for determining medical optimization. In the absence of established guidelines, medical optimization was determined at the discretion of the attending physician. The necessity of preoperative cardiac assessment was based, in part, on physician concerns about determining safe anesthesia protocols and hemodynamically managing patients who may have occult heart disease, specifically those patients with low functional capacity (< 4 METs) and/or inability to accurately communicate their medical history.

Many hip fractures result from a fall, and it may be unclear whether the fall causing a fracture was purely mechanical or indicative of a distinct acute or chronic illness. As a result, many patients received cardiac consultations, with or without pharmacologic stress testing, adding another 24 to 36 hours to preoperative time. As invasive preoperative cardiac procedures generally result in surgical delays without improving outcomes,¹¹ the committee recommended that clinicians reserve preoperative cardiac consultation for patients with active cardiac conditions.

In lieu of cardiac consultation, the committee suggested preoperative TTE. While use of TTE has not been shown to improve preoperative risk stratification in routine noncardiac surgeries, it has been shown to provide clinically useful information in patients at high risk for cardiac complications.¹¹ There was consensus for incorporating preoperative TTE for several reasons: (1) the patients with hip fractures were not “routine,” and often did not have a reliable medical history; (2) a large percentage of patients had cardiac risk factors; (3) patients with undiagnosed aortic stenosis, severe left ventricular dysfunction, or severe pulmonary hypertension would likely have altered intraoperative fluid management; and (4) in supplanting cardiac consultations, TTE would likely expedite patients’ ED-to-OR times. Therefore, the GHFMC created a recommendation of ordering urgent TTE for patients who were unable to exercise at ≥ 4 METs but needed urgent hip fracture surgery.

In order to evaluate the success of the new protocol, the ED-to-OR times were calculated for a cohort of patients who underwent surgery for hip fracture following algorithm implementation.

Participants

A chart review was conducted for patients admitted to GMC between January 2010 and April 2014 for operative treatment of a hip fracture. Exclusion criteria included lack of radiologist-diagnosed hip fracture, periprosthetic hip fracture, or multiple traumas. Electronic patient charts were reviewed by investigators (KI and BO) using a standardized, electronic abstraction form for 3 groups of patients who (1) proceeded directly to planned surgery without TTE or cardiac consultation (direct-to-surgery group); (2) received preoperative TTE but not a cardiac consultation (TTE-only group); or (3) received preoperative cardiac consultation (cardiac consult group).

Measures

Demographics, comorbid conditions, MET score, anesthesia protocol, and in-hospital morbidity and mortality were extracted from medical charts. Medical optimization time was determined by the latest time stamp of 1 of the following: time that the final consulting specialist stated that the patient was stable for surgery; time that the hospitalist described the patient as being ready for surgery; time that the TTE report was certified by the reading cardiologist; or time that the hospitalist described the outcome of completed preoperative risk stratification. Time elapsed prior to medical optimization, surgery, and discharge were calculated using differences between the patient’s arrival date and time at the ED, first recorded time of medical optimization, surgical start time (from the surgical report), and discharge time, respectively.

To assess whether the TTE protocol may have affected anesthesia selection, the induction agent (etomidate or propofol) was abstracted from anesthesia reports and stratified by the ejection fraction of each patient: very low (≤ 35%), low (36%–50%), or normal (> 50%). Patients without an echocardiogram report were assumed to have a normal ejection fraction for this analysis.

Analysis

Descriptive statistics were produced using mean and standard deviation (SD) for continuous variables and frequency and percentage for categorical variables. To determine whether statistically significant differences existed between the 3 groups, the Kruskal-Wallis test was used to compare skewed continuous variables, and Pearson’s chi-square test was used to compare categorical variables. Due to differences in baseline patient characteristics across the 3 treatment groups, inverse probability weights were used to adjust for group differences (using a multinomial logit treatment model) while comparing differences in outcome variables. This modeling strategy does not rely on any assumptions for the distribution of the outcome variable. Covariates were considered for inclusion in the treatment or outcome model if they were significantly associated (P < 0.05) with the group variable. Additionally, anesthetic agent (etomidate or propofol) was compared across the treatment groups after stratifying by ejection fraction to identify whether any differences existed in anesthesia regimen. Patients who were prescribed more than 1 anesthetic agent (n = 2) or an agent that was not of interest were removed from the analysis (n = 13). Stata (version 14) was used for analysis. All other missing data with respect to the tested variables were omitted in the analysis for that variable. Any disagreements about abstraction were resolved through consensus between the investigators.

Results

A total of 316 cases met inclusion criteria, including 108 direct-to-surgery patients, 143 preoperative TTE patients, and 65 cardiac consult patients. Patient demographics and preoperative characteristics are shown in Table 1. The average age for all patients was 76.5 years of age (SD, 12.89; IQR, 34-97); however, direct-to-surgery patients were significantly (P < 0.001) younger (71.2 years; SD, 14.2; interquartile range [IQR], 34-95 years) than TTE-only patients (79.0 years; SD, 11.5; IQR, 35-97 years) and cardiac consult patients (79.57 years; SD, 10.63; IQR, 49-97 years). The majority of patients were female (69.9%) and experienced a fall prior to admission (94%). Almost three-fourths of patients had 1 or more cardiac risk factors (73.7%), including history of congestive heart failure (CHF; 19%), coronary artery disease (CAD; 26.3%), chronic obstructive pulmonary disease (COPD; 19.3%), or aortic stenosis (AS; 3.5%). Due to between-group differences in these comorbid conditions, confounding factors were adjusted for in subsequent analyses.

As shown in Table 2, before adjustment for confounding factors, there were significant between-group differences in medical optimization time for patients in all 3 groups. After adjustment for treatment differences using age and number of comorbid diseases, and medical optimization time differences using age and COPD, fewer between-group differences were statistically significant. Patients who received a cardiac consult had an 18.44-hour longer medical optimization time compared to patients who went directly to surgery (29.136 vs 10.696 hours; P = 0.001). Optimization remained approximately 5 hours longer for the TTE-only group than for the direct-to-surgery group; however, this difference was not significant (P = 0.075).

When comparing differences in ED-to-OR time for the 3 groups after adjusting the probability of treatment for age and the number of comorbid conditions, and adjusting the probability of ED-to-OR time for age, COPD, and CHF, significant differences remained in ED-to-OR times across all groups. Specifically, patients in the direct-to-surgery group experienced the shortest time (mean, 20.64 hours), compared to patients in the TTE-only group (mean, 26.32; P = 0.04) or patients in the cardiac consult group (mean, 36.08; P < 0.001). TTE-only patients had a longer time of 5.68 hours, compared to the direct-to-surgery group, and patients in the preoperative cardiac consult group were on average 15.44 hours longer than the direct-to-surgery group.

When comparing differences in the length of stay for the 3 groups before statistical adjustments, differences were observed; however, after removing the confounding factors related to treatment (age and CAD) and the outcome (age and the number of comorbid conditions), there were no statistically significant differences in the length of stay for the 3 groups. Average length of stay was 131 hours for direct-to-surgery patients, 142 hours for TTE-only patients, and 141 hours for cardiac consult patients.

The use of different anesthetic agents was compared for patients in the 3 groups. The majority of patients in the study (87.7%) were given propofol, and there were no differences after stratifying by ejection fraction (Table 3).

Discussion

The GHFMC was created to reduce surgical delays for hip fracture. Medical optimization was considered a primary, modifiable factor given that surgeons were reluctant to proceed without a cardiac consult. To address this gap, the committee recommended a preoperative TTE for patients with low or unknown functional status. This threshold provides a quick and easy method for stratifying patients who previously required risk stratification by a cardiologist, which often resulted in surgery delays.

In their recommendations for implementation of hip fracture quality improvement projects, the Geriatric Fracture Center emphasizes the importance of multidisciplinary physician leadership along with standardization of approach across patients.¹² This recommendation is supported by increasing evidence that orthogeriatric collaborations are associated with decreased mortality and length of stay.¹³ The GHFMC and subsequent interventions reflect this approach, allowing for collaboration to identify cross-disciplinary procedural barriers to care. In our institution, addressing identified procedural barriers to care was associated with a reduction in the average time to surgery from 51 hours to 25.3 hours.

Multiple approaches have been attempted to decrease presurgical time in hip fracture patients in various settings. Prehospital interventions, such as providing ambulances with checklists and ability to bypass the ED, have not been shown to decrease time to surgery for hip fracture patients, though similar strategies have been successful in other conditions, such as stroke.^14,15 In-hospital procedures, such as implementation of a hip fracture protocol and reduction of preoperative interventions, have more consistently been found to decrease time to surgery and in-hospital mortality.^16,17 However, reduced delays have not been found universally. Luttrell and Nana found that preoperative TTE resulted in approximately 30.8-hour delays from the ED to OR, compared to patients who did not receive a preoperative TTE.¹⁸ However, in that study hospitalists used TTE at their own discretion, and there may have been confounding factors contributing to delays. When used as part of a protocol targeting patients with poor or unknown functional capacity, we believe that preoperative TTE results in modest surgical delays yet provides clinically useful information about each patient.

ACC/AHA preoperative guidelines were updated after we implemented our intervention and now recommend that patients with poor or unknown functional capacity in whom stress testing will not influence care proceed to surgery “according to guideline-directed medical care.”¹¹ While routine use of preoperative evaluation of left ventricular function is not recommended, assessing left ventricular function may be reasonable for patients with heart failure with a change in clinical status. Guidelines also recommend that patients with clinically suspected valvular stenosis undergo preoperative echocardiography.¹¹

Limitations

This study has several limitations. First, due to resource limitations, a substantial period of time elapsed between implementation of the new protocol and the analysis of the data set. That is, the hip fracture protocol assessed in this paper occurred from January 2010 through April 2014, and final analysis of the data set occurred in April 2020. This limitation precludes our ability to formally assess any pre- or post-protocol changes in patient outcomes. Second, randomization was not used to create groups that were balanced in differing health characteristics (ie, patients with noncardiac-related surgeries, patients in different age groups); however, the use of inverse probability treatment regression analysis was a way to statistically address these between-group differences. Moreover, this study is limited by the factors that were measured; unmeasured factors cannot be accounted for. Third, health care providers working at the hospital during this time were aware of the goal to decrease presurgical time, possibly creating exaggerated effects compared to a blinded trial. Finally, although this intervention is likely translatable to other centers, these results represent the experiences of a single level 1 trauma center and may not be replicable elsewhere.

Conclusion

Preoperative TTE in lieu of cardiac consultation has several advantages. First, it requires interdepartmental collaboration for implementation, but can be implemented through a single hospital or hospital system. Unlike prehospital interventions, preoperative urgent TTE for patients with low functional capacity does not require the support of emergency medical technicians, ambulance services, or other hospitals in the region. Second, while costs are associated with TTE, they are offset by a reduction in expensive consultations with specialists, surgical delays, and longer lengths of stay. Third, despite likely increased ED-to-OR times compared to no intervention, urgent TTE decreases time to surgery compared with cardiology consultation. Prior to the GHFMC, the ED-to-OR time at our institution was 51 hours. In contrast, the mean time following the GHFMC-led protocol was less than half that, at 25.3 hours (SD, 19.1 hours). In fact, nearly two-thirds (65.2%) of the patients evaluated in this study underwent surgery within 24 hours of admission. This improvement in presurgical time was attributed, in part, to the implementation of preoperative TTE over cardiology consultations.

Acknowledgments: The authors thank Jenny Williams, RN, who was instrumental in obtaining the data set for analysis, and Shauna Ayres, MPH, from the OhioHealth Research Institute, who provided writing and technical assistance.

Corresponding author: Robert Skully, MD, OhioHealth Family Medicine Grant, 290 East Town St., Columbus, OH 43215; robert.skully@ohiohealth.com.

Funding: This work was supported by the OhioHealth Summer Research Externship Program.

Financial disclosures: None.

From Dignity Health Methodist Hospital of Sacramento Family Medicine Residency Program, Sacramento, CA (Dr. Oldach); Nationwide Children’s Hospital, Columbus, OH (Dr. Irwin); OhioHealth Research Institute, Columbus, OH (Dr. Pershing); Department of Clinical Transformation, OhioHealth, Columbus, OH (Dr. Zigmont and Dr. Gascon); and Department of Geriatrics, OhioHealth, Columbus, OH (Dr. Skully).

Abstract

Objective: An interdisciplinary committee was formed to identify factors contributing to surgical delays in urgent hip fracture repair at an urban, level 1 trauma center, with the goal of reducing preoperative time to less than 24 hours. Surgical optimization was identified as a primary, modifiable factor, as surgeons were reluctant to clear patients for surgery without cardiac consultation. Preoperative transthoracic echocardiogram (TTE) was recommended as a safe alternative to cardiac consultation in most patients.

Methods: A retrospective review was conducted for patients who underwent urgent hip fracture repair between January 2010 and April 2014 (n = 316). Time to medical optimization, time to surgery, hospital length of stay, and anesthesia induction were compared for 3 patient groups of interest: those who received (1) neither TTE nor cardiology consultation (ie, direct to surgery); (2) a preoperative TTE; or (3) preoperative cardiac consultation.

Results: There were significant between-group differences in medical optimization time (P = 0.001) and mean time to surgery (P < 0.001) when comparing the 3 groups of interest. Patients in the preoperative cardiac consult group had the longest times, followed by the TTE and direct-to-surgery groups. There were no differences in the type of induction agent used across treatment groups when stratifying by ejection fraction.

Conclusion: Preoperative TTE allows for decreased preoperative time compared to a cardiology consultation. It provides an easily implemented inter-departmental, intra-institutional intervention to decrease preoperative time in patients presenting with hip fractures.

Keywords: surgical delay; preoperative risk stratification; process improvement.

Hip fractures are common, expensive, and associated with poor outcomes.^1,2 Ample literature suggests that morbidity, mortality, and cost of care may be reduced by minimizing surgical delays.^3-5 While individual reports indicate mixed evidence, in a 2010 meta-analysis, surgery within 72 hours was associated with significant reductions in pneumonia and pressure sores, as well as a 19% reduction in all-cause mortality through 1 year.⁶ Additional reviews suggest evidence of improved patient outcomes (pain, length of stay, non-union, and/or mortality) when surgery occurs early, within 12 to 72 hours after injury.^4,6,7 Regardless of the definition of “early surgery” used, surgical delay remains a challenge, often due to organizational factors, including admission day of the week and hospital staffing, and patient characteristics, such as comorbidities, echocardiographic findings, age, and insurance status.^7-9

Among factors that contribute to surgical delays, the need for preoperative cardiovascular risk stratification is significantly modifiable.¹⁰ The American College of Cardiology (ACC)/American Heart Association (AHA) Task Force risk stratification framework for preoperative cardiac testing assists clinicians in determining surgical urgency, active cardiac conditions, cardiovascular risk factors, and functional capacity of each patient, and is well established for low- or intermediate-risk patients.¹¹ Specifically, metabolic equivalents (METs) measurements are used to identify medically stable patients with good or excellent functional capacity versus poor or unknown functional status. Patients with ≥ 4 METs may proceed to surgery without further testing; patients with < 4 METs may either proceed with planned surgery or undergo additional testing. Patients with a perceived increased risk profile who require urgent or semi-urgent hip fracture repair may be confounded by disagreement about required preoperative cardiac testing.

At OhioHealth Grant Medical Center (GMC), an urban, level 1 trauma center, the consideration of further preoperative noninvasive testing frequently contributed to surgical delays. In 2009, hip fracture patients arriving to the emergency department (ED) waited an average of 51 hours before being transferred to the operating room (OR) for surgery. Presuming prompt surgery is both desirable and feasible, the Grant Hip Fracture Management Committee (GHFMC) was developed in order to expedite surgeries in hip fracture patients. The GHFMC recommended a preoperative hip fracture protocol, and the outcomes from protocol implementation are described in this article.

Methods

This study was approved by the OhioHealth Institutional Review Board, with a waiver of the informed consent requirement. Medical records from patients treated at GMC during the time period between January 2010 and April 2014 (ie, following implementation of GHFMC recommendations) were retrospectively reviewed to identify the extent to which the use of preoperative transthoracic echocardiography (TTE) reduced average time to surgery and total length of stay, compared to cardiac consultation. This chart review included 316 participants and was used to identify primary induction agent utilized, time to medical optimization, time to surgery, and total length of hospital stay.

Intervention

The GHFMC conducted a 9-month quality improvement project to decrease ED-to-OR time to less than 24 hours for hip fracture patients. The multidisciplinary committee consisted of physicians from orthopedic surgery, anesthesia, hospital medicine, and geriatrics, along with key administrators and nurse outcomes managers. While there is lack of complete clarity surrounding optimal surgical timing, the committee decided that surgery within 24 hours would be beneficial for the majority of patients and therefore was considered a prudent goal.

Based on identified barriers that contributed to surgical delays, several process improvement strategies were implemented, including admitting patients to the hospitalist service, engaging the orthopedic trauma team, and implementing pre- and postoperative protocols and order sets (eg, ED and pain management order sets). Specific emphasis was placed on establishing guidelines for determining medical optimization. In the absence of established guidelines, medical optimization was determined at the discretion of the attending physician. The necessity of preoperative cardiac assessment was based, in part, on physician concerns about determining safe anesthesia protocols and hemodynamically managing patients who may have occult heart disease, specifically those patients with low functional capacity (< 4 METs) and/or inability to accurately communicate their medical history.

Many hip fractures result from a fall, and it may be unclear whether the fall causing a fracture was purely mechanical or indicative of a distinct acute or chronic illness. As a result, many patients received cardiac consultations, with or without pharmacologic stress testing, adding another 24 to 36 hours to preoperative time. As invasive preoperative cardiac procedures generally result in surgical delays without improving outcomes,¹¹ the committee recommended that clinicians reserve preoperative cardiac consultation for patients with active cardiac conditions.

In lieu of cardiac consultation, the committee suggested preoperative TTE. While use of TTE has not been shown to improve preoperative risk stratification in routine noncardiac surgeries, it has been shown to provide clinically useful information in patients at high risk for cardiac complications.¹¹ There was consensus for incorporating preoperative TTE for several reasons: (1) the patients with hip fractures were not “routine,” and often did not have a reliable medical history; (2) a large percentage of patients had cardiac risk factors; (3) patients with undiagnosed aortic stenosis, severe left ventricular dysfunction, or severe pulmonary hypertension would likely have altered intraoperative fluid management; and (4) in supplanting cardiac consultations, TTE would likely expedite patients’ ED-to-OR times. Therefore, the GHFMC created a recommendation of ordering urgent TTE for patients who were unable to exercise at ≥ 4 METs but needed urgent hip fracture surgery.

In order to evaluate the success of the new protocol, the ED-to-OR times were calculated for a cohort of patients who underwent surgery for hip fracture following algorithm implementation.

Participants

A chart review was conducted for patients admitted to GMC between January 2010 and April 2014 for operative treatment of a hip fracture. Exclusion criteria included lack of radiologist-diagnosed hip fracture, periprosthetic hip fracture, or multiple traumas. Electronic patient charts were reviewed by investigators (KI and BO) using a standardized, electronic abstraction form for 3 groups of patients who (1) proceeded directly to planned surgery without TTE or cardiac consultation (direct-to-surgery group); (2) received preoperative TTE but not a cardiac consultation (TTE-only group); or (3) received preoperative cardiac consultation (cardiac consult group).

Measures

Demographics, comorbid conditions, MET score, anesthesia protocol, and in-hospital morbidity and mortality were extracted from medical charts. Medical optimization time was determined by the latest time stamp of 1 of the following: time that the final consulting specialist stated that the patient was stable for surgery; time that the hospitalist described the patient as being ready for surgery; time that the TTE report was certified by the reading cardiologist; or time that the hospitalist described the outcome of completed preoperative risk stratification. Time elapsed prior to medical optimization, surgery, and discharge were calculated using differences between the patient’s arrival date and time at the ED, first recorded time of medical optimization, surgical start time (from the surgical report), and discharge time, respectively.

To assess whether the TTE protocol may have affected anesthesia selection, the induction agent (etomidate or propofol) was abstracted from anesthesia reports and stratified by the ejection fraction of each patient: very low (≤ 35%), low (36%–50%), or normal (> 50%). Patients without an echocardiogram report were assumed to have a normal ejection fraction for this analysis.

Analysis

Descriptive statistics were produced using mean and standard deviation (SD) for continuous variables and frequency and percentage for categorical variables. To determine whether statistically significant differences existed between the 3 groups, the Kruskal-Wallis test was used to compare skewed continuous variables, and Pearson’s chi-square test was used to compare categorical variables. Due to differences in baseline patient characteristics across the 3 treatment groups, inverse probability weights were used to adjust for group differences (using a multinomial logit treatment model) while comparing differences in outcome variables. This modeling strategy does not rely on any assumptions for the distribution of the outcome variable. Covariates were considered for inclusion in the treatment or outcome model if they were significantly associated (P < 0.05) with the group variable. Additionally, anesthetic agent (etomidate or propofol) was compared across the treatment groups after stratifying by ejection fraction to identify whether any differences existed in anesthesia regimen. Patients who were prescribed more than 1 anesthetic agent (n = 2) or an agent that was not of interest were removed from the analysis (n = 13). Stata (version 14) was used for analysis. All other missing data with respect to the tested variables were omitted in the analysis for that variable. Any disagreements about abstraction were resolved through consensus between the investigators.

Results

A total of 316 cases met inclusion criteria, including 108 direct-to-surgery patients, 143 preoperative TTE patients, and 65 cardiac consult patients. Patient demographics and preoperative characteristics are shown in Table 1. The average age for all patients was 76.5 years of age (SD, 12.89; IQR, 34-97); however, direct-to-surgery patients were significantly (P < 0.001) younger (71.2 years; SD, 14.2; interquartile range [IQR], 34-95 years) than TTE-only patients (79.0 years; SD, 11.5; IQR, 35-97 years) and cardiac consult patients (79.57 years; SD, 10.63; IQR, 49-97 years). The majority of patients were female (69.9%) and experienced a fall prior to admission (94%). Almost three-fourths of patients had 1 or more cardiac risk factors (73.7%), including history of congestive heart failure (CHF; 19%), coronary artery disease (CAD; 26.3%), chronic obstructive pulmonary disease (COPD; 19.3%), or aortic stenosis (AS; 3.5%). Due to between-group differences in these comorbid conditions, confounding factors were adjusted for in subsequent analyses.

As shown in Table 2, before adjustment for confounding factors, there were significant between-group differences in medical optimization time for patients in all 3 groups. After adjustment for treatment differences using age and number of comorbid diseases, and medical optimization time differences using age and COPD, fewer between-group differences were statistically significant. Patients who received a cardiac consult had an 18.44-hour longer medical optimization time compared to patients who went directly to surgery (29.136 vs 10.696 hours; P = 0.001). Optimization remained approximately 5 hours longer for the TTE-only group than for the direct-to-surgery group; however, this difference was not significant (P = 0.075).

When comparing differences in ED-to-OR time for the 3 groups after adjusting the probability of treatment for age and the number of comorbid conditions, and adjusting the probability of ED-to-OR time for age, COPD, and CHF, significant differences remained in ED-to-OR times across all groups. Specifically, patients in the direct-to-surgery group experienced the shortest time (mean, 20.64 hours), compared to patients in the TTE-only group (mean, 26.32; P = 0.04) or patients in the cardiac consult group (mean, 36.08; P < 0.001). TTE-only patients had a longer time of 5.68 hours, compared to the direct-to-surgery group, and patients in the preoperative cardiac consult group were on average 15.44 hours longer than the direct-to-surgery group.

When comparing differences in the length of stay for the 3 groups before statistical adjustments, differences were observed; however, after removing the confounding factors related to treatment (age and CAD) and the outcome (age and the number of comorbid conditions), there were no statistically significant differences in the length of stay for the 3 groups. Average length of stay was 131 hours for direct-to-surgery patients, 142 hours for TTE-only patients, and 141 hours for cardiac consult patients.

The use of different anesthetic agents was compared for patients in the 3 groups. The majority of patients in the study (87.7%) were given propofol, and there were no differences after stratifying by ejection fraction (Table 3).

Discussion

The GHFMC was created to reduce surgical delays for hip fracture. Medical optimization was considered a primary, modifiable factor given that surgeons were reluctant to proceed without a cardiac consult. To address this gap, the committee recommended a preoperative TTE for patients with low or unknown functional status. This threshold provides a quick and easy method for stratifying patients who previously required risk stratification by a cardiologist, which often resulted in surgery delays.

In their recommendations for implementation of hip fracture quality improvement projects, the Geriatric Fracture Center emphasizes the importance of multidisciplinary physician leadership along with standardization of approach across patients.¹² This recommendation is supported by increasing evidence that orthogeriatric collaborations are associated with decreased mortality and length of stay.¹³ The GHFMC and subsequent interventions reflect this approach, allowing for collaboration to identify cross-disciplinary procedural barriers to care. In our institution, addressing identified procedural barriers to care was associated with a reduction in the average time to surgery from 51 hours to 25.3 hours.

Multiple approaches have been attempted to decrease presurgical time in hip fracture patients in various settings. Prehospital interventions, such as providing ambulances with checklists and ability to bypass the ED, have not been shown to decrease time to surgery for hip fracture patients, though similar strategies have been successful in other conditions, such as stroke.^14,15 In-hospital procedures, such as implementation of a hip fracture protocol and reduction of preoperative interventions, have more consistently been found to decrease time to surgery and in-hospital mortality.^16,17 However, reduced delays have not been found universally. Luttrell and Nana found that preoperative TTE resulted in approximately 30.8-hour delays from the ED to OR, compared to patients who did not receive a preoperative TTE.¹⁸ However, in that study hospitalists used TTE at their own discretion, and there may have been confounding factors contributing to delays. When used as part of a protocol targeting patients with poor or unknown functional capacity, we believe that preoperative TTE results in modest surgical delays yet provides clinically useful information about each patient.

ACC/AHA preoperative guidelines were updated after we implemented our intervention and now recommend that patients with poor or unknown functional capacity in whom stress testing will not influence care proceed to surgery “according to guideline-directed medical care.”¹¹ While routine use of preoperative evaluation of left ventricular function is not recommended, assessing left ventricular function may be reasonable for patients with heart failure with a change in clinical status. Guidelines also recommend that patients with clinically suspected valvular stenosis undergo preoperative echocardiography.¹¹

Limitations

This study has several limitations. First, due to resource limitations, a substantial period of time elapsed between implementation of the new protocol and the analysis of the data set. That is, the hip fracture protocol assessed in this paper occurred from January 2010 through April 2014, and final analysis of the data set occurred in April 2020. This limitation precludes our ability to formally assess any pre- or post-protocol changes in patient outcomes. Second, randomization was not used to create groups that were balanced in differing health characteristics (ie, patients with noncardiac-related surgeries, patients in different age groups); however, the use of inverse probability treatment regression analysis was a way to statistically address these between-group differences. Moreover, this study is limited by the factors that were measured; unmeasured factors cannot be accounted for. Third, health care providers working at the hospital during this time were aware of the goal to decrease presurgical time, possibly creating exaggerated effects compared to a blinded trial. Finally, although this intervention is likely translatable to other centers, these results represent the experiences of a single level 1 trauma center and may not be replicable elsewhere.

Conclusion

Preoperative TTE in lieu of cardiac consultation has several advantages. First, it requires interdepartmental collaboration for implementation, but can be implemented through a single hospital or hospital system. Unlike prehospital interventions, preoperative urgent TTE for patients with low functional capacity does not require the support of emergency medical technicians, ambulance services, or other hospitals in the region. Second, while costs are associated with TTE, they are offset by a reduction in expensive consultations with specialists, surgical delays, and longer lengths of stay. Third, despite likely increased ED-to-OR times compared to no intervention, urgent TTE decreases time to surgery compared with cardiology consultation. Prior to the GHFMC, the ED-to-OR time at our institution was 51 hours. In contrast, the mean time following the GHFMC-led protocol was less than half that, at 25.3 hours (SD, 19.1 hours). In fact, nearly two-thirds (65.2%) of the patients evaluated in this study underwent surgery within 24 hours of admission. This improvement in presurgical time was attributed, in part, to the implementation of preoperative TTE over cardiology consultations.

Acknowledgments: The authors thank Jenny Williams, RN, who was instrumental in obtaining the data set for analysis, and Shauna Ayres, MPH, from the OhioHealth Research Institute, who provided writing and technical assistance.

Corresponding author: Robert Skully, MD, OhioHealth Family Medicine Grant, 290 East Town St., Columbus, OH 43215; robert.skully@ohiohealth.com.

Funding: This work was supported by the OhioHealth Summer Research Externship Program.

Financial disclosures: None.

From Dignity Health Methodist Hospital of Sacramento Family Medicine Residency Program, Sacramento, CA (Dr. Oldach); Nationwide Children’s Hospital, Columbus, OH (Dr. Irwin); OhioHealth Research Institute, Columbus, OH (Dr. Pershing); Department of Clinical Transformation, OhioHealth, Columbus, OH (Dr. Zigmont and Dr. Gascon); and Department of Geriatrics, OhioHealth, Columbus, OH (Dr. Skully).

Abstract

Objective: An interdisciplinary committee was formed to identify factors contributing to surgical delays in urgent hip fracture repair at an urban, level 1 trauma center, with the goal of reducing preoperative time to less than 24 hours. Surgical optimization was identified as a primary, modifiable factor, as surgeons were reluctant to clear patients for surgery without cardiac consultation. Preoperative transthoracic echocardiogram (TTE) was recommended as a safe alternative to cardiac consultation in most patients.

Methods: A retrospective review was conducted for patients who underwent urgent hip fracture repair between January 2010 and April 2014 (n = 316). Time to medical optimization, time to surgery, hospital length of stay, and anesthesia induction were compared for 3 patient groups of interest: those who received (1) neither TTE nor cardiology consultation (ie, direct to surgery); (2) a preoperative TTE; or (3) preoperative cardiac consultation.

Results: There were significant between-group differences in medical optimization time (P = 0.001) and mean time to surgery (P < 0.001) when comparing the 3 groups of interest. Patients in the preoperative cardiac consult group had the longest times, followed by the TTE and direct-to-surgery groups. There were no differences in the type of induction agent used across treatment groups when stratifying by ejection fraction.

Conclusion: Preoperative TTE allows for decreased preoperative time compared to a cardiology consultation. It provides an easily implemented inter-departmental, intra-institutional intervention to decrease preoperative time in patients presenting with hip fractures.

Keywords: surgical delay; preoperative risk stratification; process improvement.

Hip fractures are common, expensive, and associated with poor outcomes.^1,2 Ample literature suggests that morbidity, mortality, and cost of care may be reduced by minimizing surgical delays.^3-5 While individual reports indicate mixed evidence, in a 2010 meta-analysis, surgery within 72 hours was associated with significant reductions in pneumonia and pressure sores, as well as a 19% reduction in all-cause mortality through 1 year.⁶ Additional reviews suggest evidence of improved patient outcomes (pain, length of stay, non-union, and/or mortality) when surgery occurs early, within 12 to 72 hours after injury.^4,6,7 Regardless of the definition of “early surgery” used, surgical delay remains a challenge, often due to organizational factors, including admission day of the week and hospital staffing, and patient characteristics, such as comorbidities, echocardiographic findings, age, and insurance status.^7-9

Among factors that contribute to surgical delays, the need for preoperative cardiovascular risk stratification is significantly modifiable.¹⁰ The American College of Cardiology (ACC)/American Heart Association (AHA) Task Force risk stratification framework for preoperative cardiac testing assists clinicians in determining surgical urgency, active cardiac conditions, cardiovascular risk factors, and functional capacity of each patient, and is well established for low- or intermediate-risk patients.¹¹ Specifically, metabolic equivalents (METs) measurements are used to identify medically stable patients with good or excellent functional capacity versus poor or unknown functional status. Patients with ≥ 4 METs may proceed to surgery without further testing; patients with < 4 METs may either proceed with planned surgery or undergo additional testing. Patients with a perceived increased risk profile who require urgent or semi-urgent hip fracture repair may be confounded by disagreement about required preoperative cardiac testing.

At OhioHealth Grant Medical Center (GMC), an urban, level 1 trauma center, the consideration of further preoperative noninvasive testing frequently contributed to surgical delays. In 2009, hip fracture patients arriving to the emergency department (ED) waited an average of 51 hours before being transferred to the operating room (OR) for surgery. Presuming prompt surgery is both desirable and feasible, the Grant Hip Fracture Management Committee (GHFMC) was developed in order to expedite surgeries in hip fracture patients. The GHFMC recommended a preoperative hip fracture protocol, and the outcomes from protocol implementation are described in this article.

Methods

This study was approved by the OhioHealth Institutional Review Board, with a waiver of the informed consent requirement. Medical records from patients treated at GMC during the time period between January 2010 and April 2014 (ie, following implementation of GHFMC recommendations) were retrospectively reviewed to identify the extent to which the use of preoperative transthoracic echocardiography (TTE) reduced average time to surgery and total length of stay, compared to cardiac consultation. This chart review included 316 participants and was used to identify primary induction agent utilized, time to medical optimization, time to surgery, and total length of hospital stay.

Intervention

The GHFMC conducted a 9-month quality improvement project to decrease ED-to-OR time to less than 24 hours for hip fracture patients. The multidisciplinary committee consisted of physicians from orthopedic surgery, anesthesia, hospital medicine, and geriatrics, along with key administrators and nurse outcomes managers. While there is lack of complete clarity surrounding optimal surgical timing, the committee decided that surgery within 24 hours would be beneficial for the majority of patients and therefore was considered a prudent goal.

Based on identified barriers that contributed to surgical delays, several process improvement strategies were implemented, including admitting patients to the hospitalist service, engaging the orthopedic trauma team, and implementing pre- and postoperative protocols and order sets (eg, ED and pain management order sets). Specific emphasis was placed on establishing guidelines for determining medical optimization. In the absence of established guidelines, medical optimization was determined at the discretion of the attending physician. The necessity of preoperative cardiac assessment was based, in part, on physician concerns about determining safe anesthesia protocols and hemodynamically managing patients who may have occult heart disease, specifically those patients with low functional capacity (< 4 METs) and/or inability to accurately communicate their medical history.

Many hip fractures result from a fall, and it may be unclear whether the fall causing a fracture was purely mechanical or indicative of a distinct acute or chronic illness. As a result, many patients received cardiac consultations, with or without pharmacologic stress testing, adding another 24 to 36 hours to preoperative time. As invasive preoperative cardiac procedures generally result in surgical delays without improving outcomes,¹¹ the committee recommended that clinicians reserve preoperative cardiac consultation for patients with active cardiac conditions.

In lieu of cardiac consultation, the committee suggested preoperative TTE. While use of TTE has not been shown to improve preoperative risk stratification in routine noncardiac surgeries, it has been shown to provide clinically useful information in patients at high risk for cardiac complications.¹¹ There was consensus for incorporating preoperative TTE for several reasons: (1) the patients with hip fractures were not “routine,” and often did not have a reliable medical history; (2) a large percentage of patients had cardiac risk factors; (3) patients with undiagnosed aortic stenosis, severe left ventricular dysfunction, or severe pulmonary hypertension would likely have altered intraoperative fluid management; and (4) in supplanting cardiac consultations, TTE would likely expedite patients’ ED-to-OR times. Therefore, the GHFMC created a recommendation of ordering urgent TTE for patients who were unable to exercise at ≥ 4 METs but needed urgent hip fracture surgery.

In order to evaluate the success of the new protocol, the ED-to-OR times were calculated for a cohort of patients who underwent surgery for hip fracture following algorithm implementation.

Participants

A chart review was conducted for patients admitted to GMC between January 2010 and April 2014 for operative treatment of a hip fracture. Exclusion criteria included lack of radiologist-diagnosed hip fracture, periprosthetic hip fracture, or multiple traumas. Electronic patient charts were reviewed by investigators (KI and BO) using a standardized, electronic abstraction form for 3 groups of patients who (1) proceeded directly to planned surgery without TTE or cardiac consultation (direct-to-surgery group); (2) received preoperative TTE but not a cardiac consultation (TTE-only group); or (3) received preoperative cardiac consultation (cardiac consult group).

Measures

Demographics, comorbid conditions, MET score, anesthesia protocol, and in-hospital morbidity and mortality were extracted from medical charts. Medical optimization time was determined by the latest time stamp of 1 of the following: time that the final consulting specialist stated that the patient was stable for surgery; time that the hospitalist described the patient as being ready for surgery; time that the TTE report was certified by the reading cardiologist; or time that the hospitalist described the outcome of completed preoperative risk stratification. Time elapsed prior to medical optimization, surgery, and discharge were calculated using differences between the patient’s arrival date and time at the ED, first recorded time of medical optimization, surgical start time (from the surgical report), and discharge time, respectively.

To assess whether the TTE protocol may have affected anesthesia selection, the induction agent (etomidate or propofol) was abstracted from anesthesia reports and stratified by the ejection fraction of each patient: very low (≤ 35%), low (36%–50%), or normal (> 50%). Patients without an echocardiogram report were assumed to have a normal ejection fraction for this analysis.

Analysis

Descriptive statistics were produced using mean and standard deviation (SD) for continuous variables and frequency and percentage for categorical variables. To determine whether statistically significant differences existed between the 3 groups, the Kruskal-Wallis test was used to compare skewed continuous variables, and Pearson’s chi-square test was used to compare categorical variables. Due to differences in baseline patient characteristics across the 3 treatment groups, inverse probability weights were used to adjust for group differences (using a multinomial logit treatment model) while comparing differences in outcome variables. This modeling strategy does not rely on any assumptions for the distribution of the outcome variable. Covariates were considered for inclusion in the treatment or outcome model if they were significantly associated (P < 0.05) with the group variable. Additionally, anesthetic agent (etomidate or propofol) was compared across the treatment groups after stratifying by ejection fraction to identify whether any differences existed in anesthesia regimen. Patients who were prescribed more than 1 anesthetic agent (n = 2) or an agent that was not of interest were removed from the analysis (n = 13). Stata (version 14) was used for analysis. All other missing data with respect to the tested variables were omitted in the analysis for that variable. Any disagreements about abstraction were resolved through consensus between the investigators.

Results

A total of 316 cases met inclusion criteria, including 108 direct-to-surgery patients, 143 preoperative TTE patients, and 65 cardiac consult patients. Patient demographics and preoperative characteristics are shown in Table 1. The average age for all patients was 76.5 years of age (SD, 12.89; IQR, 34-97); however, direct-to-surgery patients were significantly (P < 0.001) younger (71.2 years; SD, 14.2; interquartile range [IQR], 34-95 years) than TTE-only patients (79.0 years; SD, 11.5; IQR, 35-97 years) and cardiac consult patients (79.57 years; SD, 10.63; IQR, 49-97 years). The majority of patients were female (69.9%) and experienced a fall prior to admission (94%). Almost three-fourths of patients had 1 or more cardiac risk factors (73.7%), including history of congestive heart failure (CHF; 19%), coronary artery disease (CAD; 26.3%), chronic obstructive pulmonary disease (COPD; 19.3%), or aortic stenosis (AS; 3.5%). Due to between-group differences in these comorbid conditions, confounding factors were adjusted for in subsequent analyses.

As shown in Table 2, before adjustment for confounding factors, there were significant between-group differences in medical optimization time for patients in all 3 groups. After adjustment for treatment differences using age and number of comorbid diseases, and medical optimization time differences using age and COPD, fewer between-group differences were statistically significant. Patients who received a cardiac consult had an 18.44-hour longer medical optimization time compared to patients who went directly to surgery (29.136 vs 10.696 hours; P = 0.001). Optimization remained approximately 5 hours longer for the TTE-only group than for the direct-to-surgery group; however, this difference was not significant (P = 0.075).

When comparing differences in ED-to-OR time for the 3 groups after adjusting the probability of treatment for age and the number of comorbid conditions, and adjusting the probability of ED-to-OR time for age, COPD, and CHF, significant differences remained in ED-to-OR times across all groups. Specifically, patients in the direct-to-surgery group experienced the shortest time (mean, 20.64 hours), compared to patients in the TTE-only group (mean, 26.32; P = 0.04) or patients in the cardiac consult group (mean, 36.08; P < 0.001). TTE-only patients had a longer time of 5.68 hours, compared to the direct-to-surgery group, and patients in the preoperative cardiac consult group were on average 15.44 hours longer than the direct-to-surgery group.

When comparing differences in the length of stay for the 3 groups before statistical adjustments, differences were observed; however, after removing the confounding factors related to treatment (age and CAD) and the outcome (age and the number of comorbid conditions), there were no statistically significant differences in the length of stay for the 3 groups. Average length of stay was 131 hours for direct-to-surgery patients, 142 hours for TTE-only patients, and 141 hours for cardiac consult patients.

The use of different anesthetic agents was compared for patients in the 3 groups. The majority of patients in the study (87.7%) were given propofol, and there were no differences after stratifying by ejection fraction (Table 3).

Discussion

The GHFMC was created to reduce surgical delays for hip fracture. Medical optimization was considered a primary, modifiable factor given that surgeons were reluctant to proceed without a cardiac consult. To address this gap, the committee recommended a preoperative TTE for patients with low or unknown functional status. This threshold provides a quick and easy method for stratifying patients who previously required risk stratification by a cardiologist, which often resulted in surgery delays.

In their recommendations for implementation of hip fracture quality improvement projects, the Geriatric Fracture Center emphasizes the importance of multidisciplinary physician leadership along with standardization of approach across patients.¹² This recommendation is supported by increasing evidence that orthogeriatric collaborations are associated with decreased mortality and length of stay.¹³ The GHFMC and subsequent interventions reflect this approach, allowing for collaboration to identify cross-disciplinary procedural barriers to care. In our institution, addressing identified procedural barriers to care was associated with a reduction in the average time to surgery from 51 hours to 25.3 hours.

Multiple approaches have been attempted to decrease presurgical time in hip fracture patients in various settings. Prehospital interventions, such as providing ambulances with checklists and ability to bypass the ED, have not been shown to decrease time to surgery for hip fracture patients, though similar strategies have been successful in other conditions, such as stroke.^14,15 In-hospital procedures, such as implementation of a hip fracture protocol and reduction of preoperative interventions, have more consistently been found to decrease time to surgery and in-hospital mortality.^16,17 However, reduced delays have not been found universally. Luttrell and Nana found that preoperative TTE resulted in approximately 30.8-hour delays from the ED to OR, compared to patients who did not receive a preoperative TTE.¹⁸ However, in that study hospitalists used TTE at their own discretion, and there may have been confounding factors contributing to delays. When used as part of a protocol targeting patients with poor or unknown functional capacity, we believe that preoperative TTE results in modest surgical delays yet provides clinically useful information about each patient.

ACC/AHA preoperative guidelines were updated after we implemented our intervention and now recommend that patients with poor or unknown functional capacity in whom stress testing will not influence care proceed to surgery “according to guideline-directed medical care.”¹¹ While routine use of preoperative evaluation of left ventricular function is not recommended, assessing left ventricular function may be reasonable for patients with heart failure with a change in clinical status. Guidelines also recommend that patients with clinically suspected valvular stenosis undergo preoperative echocardiography.¹¹

Limitations

This study has several limitations. First, due to resource limitations, a substantial period of time elapsed between implementation of the new protocol and the analysis of the data set. That is, the hip fracture protocol assessed in this paper occurred from January 2010 through April 2014, and final analysis of the data set occurred in April 2020. This limitation precludes our ability to formally assess any pre- or post-protocol changes in patient outcomes. Second, randomization was not used to create groups that were balanced in differing health characteristics (ie, patients with noncardiac-related surgeries, patients in different age groups); however, the use of inverse probability treatment regression analysis was a way to statistically address these between-group differences. Moreover, this study is limited by the factors that were measured; unmeasured factors cannot be accounted for. Third, health care providers working at the hospital during this time were aware of the goal to decrease presurgical time, possibly creating exaggerated effects compared to a blinded trial. Finally, although this intervention is likely translatable to other centers, these results represent the experiences of a single level 1 trauma center and may not be replicable elsewhere.

Conclusion

Preoperative TTE in lieu of cardiac consultation has several advantages. First, it requires interdepartmental collaboration for implementation, but can be implemented through a single hospital or hospital system. Unlike prehospital interventions, preoperative urgent TTE for patients with low functional capacity does not require the support of emergency medical technicians, ambulance services, or other hospitals in the region. Second, while costs are associated with TTE, they are offset by a reduction in expensive consultations with specialists, surgical delays, and longer lengths of stay. Third, despite likely increased ED-to-OR times compared to no intervention, urgent TTE decreases time to surgery compared with cardiology consultation. Prior to the GHFMC, the ED-to-OR time at our institution was 51 hours. In contrast, the mean time following the GHFMC-led protocol was less than half that, at 25.3 hours (SD, 19.1 hours). In fact, nearly two-thirds (65.2%) of the patients evaluated in this study underwent surgery within 24 hours of admission. This improvement in presurgical time was attributed, in part, to the implementation of preoperative TTE over cardiology consultations.

Acknowledgments: The authors thank Jenny Williams, RN, who was instrumental in obtaining the data set for analysis, and Shauna Ayres, MPH, from the OhioHealth Research Institute, who provided writing and technical assistance.

Corresponding author: Robert Skully, MD, OhioHealth Family Medicine Grant, 290 East Town St., Columbus, OH 43215; robert.skully@ohiohealth.com.

Funding: This work was supported by the OhioHealth Summer Research Externship Program.

Financial disclosures: None.