Thirty-six million people are hospitalized annually in the United States,¹ and a significant proportion of these patients are rehospitalized within 30 days.² Gaps in hospital care are many and well documented, including high rates of adverse events, hospital-acquired conditions, and suboptimal care transitions.^3-5 Despite significant efforts to improve the care of hospitalized patients and some incremental improvement in the safety of hospital care, hospital care remains suboptimal.^6-9 Importantly, hospitalization remains a challenging and vulnerable time for patients and caregivers.

Despite research efforts to improve hospital care, there remains very little data regarding what patients, caregivers, and other stakeholders believe are the most important priorities for improving hospital care, experiences, and outcomes. Small studies described in brief reports provide limited insights into what aspects of hospital care are most important to patients and to their families.^10-13 These small studies suggest that communication and the comfort of caregivers and of patient family members are important priorities, as are the provision of adequate sleeping arrangements, food choices, and psychosocial support. However, the limited nature of these studies precludes the possibility of larger conclusions regarding patient priorities.^10-13

The evolution of patient-centered care has led to increasing efforts to engage, and partner, with patients, caregivers, and other stakeholders to obtain their input on healthcare, research, and improvement efforts.¹⁴ The guiding principle of this engagement is that patients and their caregivers are uniquely positioned to share their lived experiences of care and that their involvement ensures their voices are represented.^15-17 Therefore to obtain greater insight into priority areas from the perspectives of patients, caregivers, and other healthcare stakeholders, we undertook a systematic engagement process to create a patient-partnered and stakeholder-­partnered research agenda for improving the care of hospitalized adult patients.

Guiding Frameworks for Study Methods

We used two established, validated methods to guide our collaborative, inclusive, and consultative approach to patient and stakeholder engagement and research prioritization:

• The Patient-Centered Outcomes Research Institute (PCORI) standards for formulating patient-centered research questions,¹⁸ which includes methods for stakeholder engagement that ensures the representativeness of engaged groups and dissemination of study results.¹⁸
• The James Lind Alliance (JLA) approach to “priority setting partnerships,” through which patients, caregivers, and clinicians partner to identify and prioritize unanswered questions.¹⁹

The Improving Hospital Outcomes through Patient Engagement (i-HOPE) study included eight stepwise phases to formulate and prioritize a set of patient-centered research questions to improve the care and experiences of hospitalized patients and their families.²⁰ Our process is described below and summarized in Table 1.

Phases of Question Development

Phase 1: Steering Committee Formation

Nine clinical researchers, nine patients and/or caregivers, and two administrators from eight academic and community hospitals from across the United States formed a steering committee to participate in teleconferences every other week to manage all stages of the project including design, implementation, and dissemination. At the time of the project conceptualization, the researchers were a subgroup of the Society of Hospital Medicine Research Committee.²¹ Patient partners on the steering committee were identified from local patient and family advisory councils (PFACs) of the researchers’ institutions. Patients partners had previously participated in research or improvement initiatives with their hospitalist partners. Patient partners received stipends throughout the project in recognition of their participation and expertise. Included in the committee was a representative from the Society of Hospital Medicine (SHM)—our supporting and dissemination partner.

Phase 2: Stakeholder Identification

We created a list of potential stakeholder organizations to participate in the study based on the following:

• Organizations with which SHM has worked on initiatives related to the care of hospitalized adult patients
• Organizations with which steering committee members had worked
• Internet searches of organizations participating in similar PCORI-funded projects and of other professional societies that represented patients or providers who work in hospital or post-acute care settings
• Suggestions from stakeholders identified through the first two approaches as described above

We intended to have a broad representation of stakeholders to ensure diverse perspectives were included in the study. Stakeholder organizations included patient advocacy groups, providers, researchers, payers, policy makers and funding agencies.

Phase 3: Stakeholder Engagement and Awareness Training

Representatives from 39 stakeholder organizations who agreed to participate in the study were further orientated to the study rationale and methods via a series of interactive online webinars. This included reminding organziations that everyone’s input and perspective were valued and that we had a flat organization structure that ensured all stakeholders were equal.

Phase 4: Survey Development and Administration

We chose a survey approach to solicit input on identifying gaps in patient care and to generate research questions. The steering committee developed an online survey collaboratively with stakeholder organization representatives. We used survey pretesting with patient and researcher members from the steering committee. The goal of pretesting was to ensure accessibility and comprehension for all potential respondents, particularly patients and caregivers. The final survey asked respondents to record up to three questions that they thought would improve the care of hospitalized adult patients and their families. The specific wording of the survey is shown in the Figure and the entire survey in Appendix Document 1.

We chose three questions because that is the number of entries per participant that is recommended by JLA; it also minimizes responder burden.¹⁹ We asked respondents to identify the stakeholder group they represented (eg, patient, caregiver, healthcare provider, researcher) and for providers to identify where they primarily worked (eg, acute care hospital, post-acute care, advocacy group).

Survey Administration. We administered the survey electronically using Research Electronic Data Capture (REDCap), a secure web-based application used for collecting research data.²² Stakeholders were asked to disseminate the survey broadly using whatever methods that they felt was appropriate to their leadership or members.

Phase 5: Initial Question Categorization Using Qualitative Content Analysis

Six members of the steering committee independently performed qualitative content analysis to categorize all submitted questions.^23,24 This analytic approach identifies, analyzes, and reports patterns within the data.^23,24 We hypothesized that some of the submitted questions would relate to already-­known problems with hospitalization. Therefore the steering committee developed an a priori codebook of 48 categories using common systems-based issues and diseases related to the care of hospitalized patients based on the hospitalist core competency topics developed by hospitalists and the SHM Education Committee,²⁵ personal and clinical knowledge and experience related to the care of hospitalized adult patients, and published literature on the topic. These a priori categories and their definitions are shown in Appendix Document 2 and were the basis for our initial theory-driven (deductive) approach to data analysis.²³

Once coding began, we identified 32 new and additional categories based on our review of the submitted questions, and these were the basis of our data-driven (inductive) approach to analysis.²³ All proposed new codes and definitions were discussed with and approved by the entire steering committee prior to being added to the codebook (Appendix Document 2).

While coding categories were mutually exclusive, multiple codes could be attributed to a question depending on the content and meaning of a question. To ensure methodological rigor, reviewers met regularly via teleconference or communicated via email throughout the analysis to iteratively refine and define coding categories. All questions were reviewed independently, and then discussed, by at least two members of the analysis team. Any coding disparities were discussed and resolved by negotiated consensus.²⁶ Analysis was conducted using Dedoose V8.0.35 (Sociocultural Research Consultants, Los Angeles, California).

Phase 6: Initial Question Identification Using Quantitative Content Analysis

Following thematic categorization, all steering committee members then reviewed each category to identify and quantify the most commonly submitted questions.²⁷ A question was determined to be a commonly submitted question when it appeared at least 10 times.

Phase 7: Interim Priority Setting

We sent the list of the most commonly submitted questions (Appendix Document 3) to stakeholder organizations and patient partner networks for review and evaluation. Each organization was asked to engage with their constituents and leaders to collectively decide on which of these questions resonated and was most important. These preferences would then be used during the in-person meeting (Phase 8). We did not provide stakeholder organizations with information about how many times each question was submitted by respondents because we felt this could potentially bias their decision-making processes such that true importance and relevance would not obtained.

Phase 8: In-person Meeting for Final Question Prioritization and Refinement

Representatives from all 39 participating stakeholder organizations were invited to participate in a 2-day, in-person meeting to create a final prioritized list of questions to be used to guide patient-centered research seeking to improve the care of hospitalized adult patients and their caregivers. This meeting was attended by 43 stakeholders (26 stakeholder organization representatives and 17 steering committee members) from 37 unique stakeholder organizations. To facilitate the inclusiveness and to ensure a consensus-driven process, we used nominal group technique (NGT) to allow all of the meeting participants to discuss the list of prioritized questions in small groups.²⁸ NGT allows participants to comprehend each other’s point of view to ensure no perpsectives are excluded.²⁸ The NGT was followed by two rounds of individual voting. Stakeholders were then asked to frame their discussions and their votes based on the perspectives of their organizations or PFACs that they represent. The voting process required participants to make choices regarding the relative importance of all of the questions, which therefore makes the resulting list a true prioritized list. In the first round of voting, participants voted for up to five questions for inclusion on the prioritized list. Based on the distribution of votes, where one vote equals one point, each of the 36 questions was then ranked in order of the assigned points. The rank-ordering process resulted in a natural cut point or delineated point, resulting in the 11 questions considered to be the highest prioritized questions. Following this, a second round of voting took place with the same parameters as the first round and allowed us to rank order questions by order of importance and priority. Finally, during small and large group discussions, the original text of each question was edited, refined, and reformatted into questions that could drive a research agenda.

Ethical Oversight

This study was reviewed by the Institutional Review Board of the University of Texas Health Science Center at San Antonio and deemed not to be human subject research (UT Health San Antonio IRB Protocol Number: HSC20170058N).

In total, 499 respondents from 39 unique stakeholder organizations responded to our survey. Respondents self-identified into multiple categorizes resulting in 267 healthcare providers, 244 patients and caregivers, and 63 researchers. Characteristics of respondents to the survey are shown in Table 2.

An overview of study results is shown in Table 1. Respondents submitted a total of 782 questions related to improving the care of hospitalized patients. These questions were categorized during thematic analysis into 70 distinct categories—52 that were health system related and 18 that were disease specific (Appendix 2). The most frequently used health system–related categories were related to discharge care transitions, medications, patient understanding, and patient-family-care team communication (Appendix 2).

From these categories, 36 questions met our criteria for “commonly identified,” ie, submitted at least 10 times (Appendix Document 3). Notably, these 36 questions were derived from 67 different coding categories, of which 24 (36%) were a priori (theory-driven) categories²³ created by the Steering Committee before analysis began and 43 (64%) categories were created as a result of this study’s stakeholder-engaged process and a data-driven approach²³ to analysis (Appendix Document 3). These groups of questions were then presented during the 2-day, in-person meeting and reduced to a final 11 questions that were identified in rank order as top priorities (Table 3). The questions considered highest priority related to ensuring shared treatment and goals of care decision making, improving hospital discharge handoff to other care facilities and providers, and reducing the confusion related to education on medications, conditions, hospital care, and discharge.

Using a dynamic and collaborative stakeholder engagement process, we identified 11 questions prioritized in order of importance by patients, caregivers, and other healthcare stakeholders to improve the care of hospitalized adult patients. While some of the topics identified are already well-known topics in need of research and improvement, our findings frame these topics according to the perspectives of patients, caregivers, and stakeholders. This unique perspective adds a level of richness and nuance that provides insight into how to better address these topics and ultimately inform research and quality improvement efforts.

The question considered to be the highest priority area for future research and improvement surmised how it may be possible to implement interventions that engage patients in shared decision making. Shared decision making involves patients and their care team working together to make decisions about treatment, and other aspects of care based on sound clinical evidence that balances the risks and outcomes with patient preferences and values. Although considered critically important,²⁹ a recent evaluation of shared decision making practices in over 250 inpatient encounters identified significant gaps in physicians’ abilities to perform key elements of a shared decision making approach and reinforced the need to identify what strategies can best promote widespread shared decision making.³⁰ While there has been considerable effort to faciliate shared decision making in practice, there remains mixed evidence regarding the sustainability and impact of tools seeking to support shared decision making, such as decision aids, question prompt lists, and coaches.³¹ This suggests that new approaches to shared decision making may be required and likely explains why this question was rated as a top priority by stakeholders in the current study.

Respondents frequently framed their questions in terms of their lived experiences, providing stories and scenarios to illustrate the importance of the questions they submitted. This personal framing highlighted to us the need to think about improving care delivery from the end-user perspective. For example, respondents framed questions about care transitions not with regard to early appointments, instructions, or medication lists, but rather in terms of whom to call with questions or how best to reach their physician, nurse, or other identified provider. These perspectives suggest that strategies and approaches to improvement that start with patient and caregiver experiences, such as design thinking,³² may be important to continued efforts to improve hospital care. Additionally, the focus on the interpersonal aspects of care delivery highlights the need to focus on the patient-provider relationship and communication.

Questions submitted by respondents demonstrated a stark difference between “patient education” and “patient understanding,” which suggests that being provided with education or education materials regarding care did not necessarily lead to a clear patient understanding. The potential for lack of understanding was particularly prominent in the context of care plan development and during times of care transition—topics that were encompassed in 9 out of 11 of our prioritized research questions. This may suggest that approaches that improve the ability for healthcare providers to deliver information may not be sufficient to meet the needs of patients and caregivers. Rather, partnering to develop a shared understanding—whether about prognosis, medications, hospital, or discharge care plans—is critical. Improved communication practices are not an endpoint for information delivery, but rather a starting point leading to a shared understanding.

Several of the priority areas identified in our study reflect the immensely complex intersections among patients, caregivers, clinicians, and the healthcare delivery system. Addressing these gaps in order to reach the goal of ideal hospital care and an improved patient experience will likely require coordinated approaches and strong involvement and buy-in from multiple stakeholders including the voices of patients and caregivers. Creating patient-centered and stakeholder-driven research has been an increasing priority nationally.³³ Yet to realize this, we must continue to understand the foundations and best practices of authentic stakeholder engagement so that it can be achieved in practice.³⁴ We intend for this prioritized list of questions to galvanize funders, researchers, clinicians, professional societies, and patient and caregiver advocacy groups to work together to address these topics through the creation of new research evidence or the sustainable implementation of existing evidence.

Our findings provide a foundation for stakeholder groups to work in partnership to find research and improvement solutions to the problems identified. Our efforts demonstrate the value and importance of a systematic and broad engagement process to ensure that the voices of patients, caregivers, and other healthcare stakeholders are included in guiding hospital research and quality improvement efforts. This is highlighted by the fact our results of prioritized category areas for research were largely only uncovered following the creation of coding categories during the analysis process and were not captured using a priori catgeories that were expected by the steering committee.

The strengths of this study include our attempts to systematically identify and engage a wide range of perspectives in hospital medicine, including perspectives from patients and their caregivers. There are also acknowledged limitations in our study. While we included patients and PFACs from across the country, the opinions of the people we included may not be representative of all patients. Similarly, the perspectives of the other participants may not have completely represented their stakeholder organizations. While we attempted to include a broad range of organizations, there may be other relevant groups who were not represented in our sample.

In summary, our findings provide direction for the multiple stakeholders involved in improving hospital care. The results will allow the research community to focus on questions that are most important to patients, caregivers, and other stakeholders, reframing them in ways that are more relevant to patients’ lived experiences and that reflect the complexity of the issues. Our findings can also be used by healthcare providers and delivery organizations to target local improvement efforts. We hope that patients and caregivers will use our results to advocate for research and improvement in areas that matter the most to them. We hope that policy makers and funding agencies use our results to promote work in these areas and drive a national conversation about how to most effectively improve hospital care.

The Society of Hospital Medicine (SHM) provided additional administrative, logistical, and technical support.

The authors would like to thank all patients, caregivers, and stakeholders who completed the survey. The authors also would like to acknowledge the organizations and individuals who participated in this study (see Appendix Document 4 for full list). At SHM, the authors would like to specifically thank Claudia Stahl, Jenna Goldstein, Kevin Vuernick, Dr Brad Sharpe, and Dr Larry Wellikson for their support.

The statements presented in this publication are solely the responsibility of the authors and do not necessarily represent the views of the Department of Veterans Affairs, Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors, or Methodology Committee.

Thirty-six million people are hospitalized annually in the United States,¹ and a significant proportion of these patients are rehospitalized within 30 days.² Gaps in hospital care are many and well documented, including high rates of adverse events, hospital-acquired conditions, and suboptimal care transitions.^3-5 Despite significant efforts to improve the care of hospitalized patients and some incremental improvement in the safety of hospital care, hospital care remains suboptimal.^6-9 Importantly, hospitalization remains a challenging and vulnerable time for patients and caregivers.

Despite research efforts to improve hospital care, there remains very little data regarding what patients, caregivers, and other stakeholders believe are the most important priorities for improving hospital care, experiences, and outcomes. Small studies described in brief reports provide limited insights into what aspects of hospital care are most important to patients and to their families.^10-13 These small studies suggest that communication and the comfort of caregivers and of patient family members are important priorities, as are the provision of adequate sleeping arrangements, food choices, and psychosocial support. However, the limited nature of these studies precludes the possibility of larger conclusions regarding patient priorities.^10-13

The evolution of patient-centered care has led to increasing efforts to engage, and partner, with patients, caregivers, and other stakeholders to obtain their input on healthcare, research, and improvement efforts.¹⁴ The guiding principle of this engagement is that patients and their caregivers are uniquely positioned to share their lived experiences of care and that their involvement ensures their voices are represented.^15-17 Therefore to obtain greater insight into priority areas from the perspectives of patients, caregivers, and other healthcare stakeholders, we undertook a systematic engagement process to create a patient-partnered and stakeholder-­partnered research agenda for improving the care of hospitalized adult patients.

Guiding Frameworks for Study Methods

We used two established, validated methods to guide our collaborative, inclusive, and consultative approach to patient and stakeholder engagement and research prioritization:

• The Patient-Centered Outcomes Research Institute (PCORI) standards for formulating patient-centered research questions,¹⁸ which includes methods for stakeholder engagement that ensures the representativeness of engaged groups and dissemination of study results.¹⁸
• The James Lind Alliance (JLA) approach to “priority setting partnerships,” through which patients, caregivers, and clinicians partner to identify and prioritize unanswered questions.¹⁹

The Improving Hospital Outcomes through Patient Engagement (i-HOPE) study included eight stepwise phases to formulate and prioritize a set of patient-centered research questions to improve the care and experiences of hospitalized patients and their families.²⁰ Our process is described below and summarized in Table 1.

Phases of Question Development

Phase 1: Steering Committee Formation

Nine clinical researchers, nine patients and/or caregivers, and two administrators from eight academic and community hospitals from across the United States formed a steering committee to participate in teleconferences every other week to manage all stages of the project including design, implementation, and dissemination. At the time of the project conceptualization, the researchers were a subgroup of the Society of Hospital Medicine Research Committee.²¹ Patient partners on the steering committee were identified from local patient and family advisory councils (PFACs) of the researchers’ institutions. Patients partners had previously participated in research or improvement initiatives with their hospitalist partners. Patient partners received stipends throughout the project in recognition of their participation and expertise. Included in the committee was a representative from the Society of Hospital Medicine (SHM)—our supporting and dissemination partner.

Phase 2: Stakeholder Identification

We created a list of potential stakeholder organizations to participate in the study based on the following:

• Organizations with which SHM has worked on initiatives related to the care of hospitalized adult patients
• Organizations with which steering committee members had worked
• Internet searches of organizations participating in similar PCORI-funded projects and of other professional societies that represented patients or providers who work in hospital or post-acute care settings
• Suggestions from stakeholders identified through the first two approaches as described above

We intended to have a broad representation of stakeholders to ensure diverse perspectives were included in the study. Stakeholder organizations included patient advocacy groups, providers, researchers, payers, policy makers and funding agencies.

Phase 3: Stakeholder Engagement and Awareness Training

Representatives from 39 stakeholder organizations who agreed to participate in the study were further orientated to the study rationale and methods via a series of interactive online webinars. This included reminding organziations that everyone’s input and perspective were valued and that we had a flat organization structure that ensured all stakeholders were equal.

Phase 4: Survey Development and Administration

We chose a survey approach to solicit input on identifying gaps in patient care and to generate research questions. The steering committee developed an online survey collaboratively with stakeholder organization representatives. We used survey pretesting with patient and researcher members from the steering committee. The goal of pretesting was to ensure accessibility and comprehension for all potential respondents, particularly patients and caregivers. The final survey asked respondents to record up to three questions that they thought would improve the care of hospitalized adult patients and their families. The specific wording of the survey is shown in the Figure and the entire survey in Appendix Document 1.

We chose three questions because that is the number of entries per participant that is recommended by JLA; it also minimizes responder burden.¹⁹ We asked respondents to identify the stakeholder group they represented (eg, patient, caregiver, healthcare provider, researcher) and for providers to identify where they primarily worked (eg, acute care hospital, post-acute care, advocacy group).

Survey Administration. We administered the survey electronically using Research Electronic Data Capture (REDCap), a secure web-based application used for collecting research data.²² Stakeholders were asked to disseminate the survey broadly using whatever methods that they felt was appropriate to their leadership or members.

Phase 5: Initial Question Categorization Using Qualitative Content Analysis

Six members of the steering committee independently performed qualitative content analysis to categorize all submitted questions.^23,24 This analytic approach identifies, analyzes, and reports patterns within the data.^23,24 We hypothesized that some of the submitted questions would relate to already-­known problems with hospitalization. Therefore the steering committee developed an a priori codebook of 48 categories using common systems-based issues and diseases related to the care of hospitalized patients based on the hospitalist core competency topics developed by hospitalists and the SHM Education Committee,²⁵ personal and clinical knowledge and experience related to the care of hospitalized adult patients, and published literature on the topic. These a priori categories and their definitions are shown in Appendix Document 2 and were the basis for our initial theory-driven (deductive) approach to data analysis.²³

Once coding began, we identified 32 new and additional categories based on our review of the submitted questions, and these were the basis of our data-driven (inductive) approach to analysis.²³ All proposed new codes and definitions were discussed with and approved by the entire steering committee prior to being added to the codebook (Appendix Document 2).

While coding categories were mutually exclusive, multiple codes could be attributed to a question depending on the content and meaning of a question. To ensure methodological rigor, reviewers met regularly via teleconference or communicated via email throughout the analysis to iteratively refine and define coding categories. All questions were reviewed independently, and then discussed, by at least two members of the analysis team. Any coding disparities were discussed and resolved by negotiated consensus.²⁶ Analysis was conducted using Dedoose V8.0.35 (Sociocultural Research Consultants, Los Angeles, California).

Phase 6: Initial Question Identification Using Quantitative Content Analysis

Following thematic categorization, all steering committee members then reviewed each category to identify and quantify the most commonly submitted questions.²⁷ A question was determined to be a commonly submitted question when it appeared at least 10 times.

Phase 7: Interim Priority Setting

We sent the list of the most commonly submitted questions (Appendix Document 3) to stakeholder organizations and patient partner networks for review and evaluation. Each organization was asked to engage with their constituents and leaders to collectively decide on which of these questions resonated and was most important. These preferences would then be used during the in-person meeting (Phase 8). We did not provide stakeholder organizations with information about how many times each question was submitted by respondents because we felt this could potentially bias their decision-making processes such that true importance and relevance would not obtained.

Phase 8: In-person Meeting for Final Question Prioritization and Refinement

Representatives from all 39 participating stakeholder organizations were invited to participate in a 2-day, in-person meeting to create a final prioritized list of questions to be used to guide patient-centered research seeking to improve the care of hospitalized adult patients and their caregivers. This meeting was attended by 43 stakeholders (26 stakeholder organization representatives and 17 steering committee members) from 37 unique stakeholder organizations. To facilitate the inclusiveness and to ensure a consensus-driven process, we used nominal group technique (NGT) to allow all of the meeting participants to discuss the list of prioritized questions in small groups.²⁸ NGT allows participants to comprehend each other’s point of view to ensure no perpsectives are excluded.²⁸ The NGT was followed by two rounds of individual voting. Stakeholders were then asked to frame their discussions and their votes based on the perspectives of their organizations or PFACs that they represent. The voting process required participants to make choices regarding the relative importance of all of the questions, which therefore makes the resulting list a true prioritized list. In the first round of voting, participants voted for up to five questions for inclusion on the prioritized list. Based on the distribution of votes, where one vote equals one point, each of the 36 questions was then ranked in order of the assigned points. The rank-ordering process resulted in a natural cut point or delineated point, resulting in the 11 questions considered to be the highest prioritized questions. Following this, a second round of voting took place with the same parameters as the first round and allowed us to rank order questions by order of importance and priority. Finally, during small and large group discussions, the original text of each question was edited, refined, and reformatted into questions that could drive a research agenda.

Ethical Oversight

This study was reviewed by the Institutional Review Board of the University of Texas Health Science Center at San Antonio and deemed not to be human subject research (UT Health San Antonio IRB Protocol Number: HSC20170058N).

In total, 499 respondents from 39 unique stakeholder organizations responded to our survey. Respondents self-identified into multiple categorizes resulting in 267 healthcare providers, 244 patients and caregivers, and 63 researchers. Characteristics of respondents to the survey are shown in Table 2.

An overview of study results is shown in Table 1. Respondents submitted a total of 782 questions related to improving the care of hospitalized patients. These questions were categorized during thematic analysis into 70 distinct categories—52 that were health system related and 18 that were disease specific (Appendix 2). The most frequently used health system–related categories were related to discharge care transitions, medications, patient understanding, and patient-family-care team communication (Appendix 2).

From these categories, 36 questions met our criteria for “commonly identified,” ie, submitted at least 10 times (Appendix Document 3). Notably, these 36 questions were derived from 67 different coding categories, of which 24 (36%) were a priori (theory-driven) categories²³ created by the Steering Committee before analysis began and 43 (64%) categories were created as a result of this study’s stakeholder-engaged process and a data-driven approach²³ to analysis (Appendix Document 3). These groups of questions were then presented during the 2-day, in-person meeting and reduced to a final 11 questions that were identified in rank order as top priorities (Table 3). The questions considered highest priority related to ensuring shared treatment and goals of care decision making, improving hospital discharge handoff to other care facilities and providers, and reducing the confusion related to education on medications, conditions, hospital care, and discharge.

Using a dynamic and collaborative stakeholder engagement process, we identified 11 questions prioritized in order of importance by patients, caregivers, and other healthcare stakeholders to improve the care of hospitalized adult patients. While some of the topics identified are already well-known topics in need of research and improvement, our findings frame these topics according to the perspectives of patients, caregivers, and stakeholders. This unique perspective adds a level of richness and nuance that provides insight into how to better address these topics and ultimately inform research and quality improvement efforts.

The question considered to be the highest priority area for future research and improvement surmised how it may be possible to implement interventions that engage patients in shared decision making. Shared decision making involves patients and their care team working together to make decisions about treatment, and other aspects of care based on sound clinical evidence that balances the risks and outcomes with patient preferences and values. Although considered critically important,²⁹ a recent evaluation of shared decision making practices in over 250 inpatient encounters identified significant gaps in physicians’ abilities to perform key elements of a shared decision making approach and reinforced the need to identify what strategies can best promote widespread shared decision making.³⁰ While there has been considerable effort to faciliate shared decision making in practice, there remains mixed evidence regarding the sustainability and impact of tools seeking to support shared decision making, such as decision aids, question prompt lists, and coaches.³¹ This suggests that new approaches to shared decision making may be required and likely explains why this question was rated as a top priority by stakeholders in the current study.

Respondents frequently framed their questions in terms of their lived experiences, providing stories and scenarios to illustrate the importance of the questions they submitted. This personal framing highlighted to us the need to think about improving care delivery from the end-user perspective. For example, respondents framed questions about care transitions not with regard to early appointments, instructions, or medication lists, but rather in terms of whom to call with questions or how best to reach their physician, nurse, or other identified provider. These perspectives suggest that strategies and approaches to improvement that start with patient and caregiver experiences, such as design thinking,³² may be important to continued efforts to improve hospital care. Additionally, the focus on the interpersonal aspects of care delivery highlights the need to focus on the patient-provider relationship and communication.

Questions submitted by respondents demonstrated a stark difference between “patient education” and “patient understanding,” which suggests that being provided with education or education materials regarding care did not necessarily lead to a clear patient understanding. The potential for lack of understanding was particularly prominent in the context of care plan development and during times of care transition—topics that were encompassed in 9 out of 11 of our prioritized research questions. This may suggest that approaches that improve the ability for healthcare providers to deliver information may not be sufficient to meet the needs of patients and caregivers. Rather, partnering to develop a shared understanding—whether about prognosis, medications, hospital, or discharge care plans—is critical. Improved communication practices are not an endpoint for information delivery, but rather a starting point leading to a shared understanding.

Several of the priority areas identified in our study reflect the immensely complex intersections among patients, caregivers, clinicians, and the healthcare delivery system. Addressing these gaps in order to reach the goal of ideal hospital care and an improved patient experience will likely require coordinated approaches and strong involvement and buy-in from multiple stakeholders including the voices of patients and caregivers. Creating patient-centered and stakeholder-driven research has been an increasing priority nationally.³³ Yet to realize this, we must continue to understand the foundations and best practices of authentic stakeholder engagement so that it can be achieved in practice.³⁴ We intend for this prioritized list of questions to galvanize funders, researchers, clinicians, professional societies, and patient and caregiver advocacy groups to work together to address these topics through the creation of new research evidence or the sustainable implementation of existing evidence.

Our findings provide a foundation for stakeholder groups to work in partnership to find research and improvement solutions to the problems identified. Our efforts demonstrate the value and importance of a systematic and broad engagement process to ensure that the voices of patients, caregivers, and other healthcare stakeholders are included in guiding hospital research and quality improvement efforts. This is highlighted by the fact our results of prioritized category areas for research were largely only uncovered following the creation of coding categories during the analysis process and were not captured using a priori catgeories that were expected by the steering committee.

The strengths of this study include our attempts to systematically identify and engage a wide range of perspectives in hospital medicine, including perspectives from patients and their caregivers. There are also acknowledged limitations in our study. While we included patients and PFACs from across the country, the opinions of the people we included may not be representative of all patients. Similarly, the perspectives of the other participants may not have completely represented their stakeholder organizations. While we attempted to include a broad range of organizations, there may be other relevant groups who were not represented in our sample.

In summary, our findings provide direction for the multiple stakeholders involved in improving hospital care. The results will allow the research community to focus on questions that are most important to patients, caregivers, and other stakeholders, reframing them in ways that are more relevant to patients’ lived experiences and that reflect the complexity of the issues. Our findings can also be used by healthcare providers and delivery organizations to target local improvement efforts. We hope that patients and caregivers will use our results to advocate for research and improvement in areas that matter the most to them. We hope that policy makers and funding agencies use our results to promote work in these areas and drive a national conversation about how to most effectively improve hospital care.

The Society of Hospital Medicine (SHM) provided additional administrative, logistical, and technical support.

The authors would like to thank all patients, caregivers, and stakeholders who completed the survey. The authors also would like to acknowledge the organizations and individuals who participated in this study (see Appendix Document 4 for full list). At SHM, the authors would like to specifically thank Claudia Stahl, Jenna Goldstein, Kevin Vuernick, Dr Brad Sharpe, and Dr Larry Wellikson for their support.

The statements presented in this publication are solely the responsibility of the authors and do not necessarily represent the views of the Department of Veterans Affairs, Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors, or Methodology Committee.

Thirty-six million people are hospitalized annually in the United States,¹ and a significant proportion of these patients are rehospitalized within 30 days.² Gaps in hospital care are many and well documented, including high rates of adverse events, hospital-acquired conditions, and suboptimal care transitions.^3-5 Despite significant efforts to improve the care of hospitalized patients and some incremental improvement in the safety of hospital care, hospital care remains suboptimal.^6-9 Importantly, hospitalization remains a challenging and vulnerable time for patients and caregivers.

Despite research efforts to improve hospital care, there remains very little data regarding what patients, caregivers, and other stakeholders believe are the most important priorities for improving hospital care, experiences, and outcomes. Small studies described in brief reports provide limited insights into what aspects of hospital care are most important to patients and to their families.^10-13 These small studies suggest that communication and the comfort of caregivers and of patient family members are important priorities, as are the provision of adequate sleeping arrangements, food choices, and psychosocial support. However, the limited nature of these studies precludes the possibility of larger conclusions regarding patient priorities.^10-13

The evolution of patient-centered care has led to increasing efforts to engage, and partner, with patients, caregivers, and other stakeholders to obtain their input on healthcare, research, and improvement efforts.¹⁴ The guiding principle of this engagement is that patients and their caregivers are uniquely positioned to share their lived experiences of care and that their involvement ensures their voices are represented.^15-17 Therefore to obtain greater insight into priority areas from the perspectives of patients, caregivers, and other healthcare stakeholders, we undertook a systematic engagement process to create a patient-partnered and stakeholder-­partnered research agenda for improving the care of hospitalized adult patients.

Guiding Frameworks for Study Methods

We used two established, validated methods to guide our collaborative, inclusive, and consultative approach to patient and stakeholder engagement and research prioritization:

• The Patient-Centered Outcomes Research Institute (PCORI) standards for formulating patient-centered research questions,¹⁸ which includes methods for stakeholder engagement that ensures the representativeness of engaged groups and dissemination of study results.¹⁸
• The James Lind Alliance (JLA) approach to “priority setting partnerships,” through which patients, caregivers, and clinicians partner to identify and prioritize unanswered questions.¹⁹

The Improving Hospital Outcomes through Patient Engagement (i-HOPE) study included eight stepwise phases to formulate and prioritize a set of patient-centered research questions to improve the care and experiences of hospitalized patients and their families.²⁰ Our process is described below and summarized in Table 1.

Phases of Question Development

Phase 1: Steering Committee Formation

Nine clinical researchers, nine patients and/or caregivers, and two administrators from eight academic and community hospitals from across the United States formed a steering committee to participate in teleconferences every other week to manage all stages of the project including design, implementation, and dissemination. At the time of the project conceptualization, the researchers were a subgroup of the Society of Hospital Medicine Research Committee.²¹ Patient partners on the steering committee were identified from local patient and family advisory councils (PFACs) of the researchers’ institutions. Patients partners had previously participated in research or improvement initiatives with their hospitalist partners. Patient partners received stipends throughout the project in recognition of their participation and expertise. Included in the committee was a representative from the Society of Hospital Medicine (SHM)—our supporting and dissemination partner.

Phase 2: Stakeholder Identification

We created a list of potential stakeholder organizations to participate in the study based on the following:

• Organizations with which SHM has worked on initiatives related to the care of hospitalized adult patients
• Organizations with which steering committee members had worked
• Internet searches of organizations participating in similar PCORI-funded projects and of other professional societies that represented patients or providers who work in hospital or post-acute care settings
• Suggestions from stakeholders identified through the first two approaches as described above

We intended to have a broad representation of stakeholders to ensure diverse perspectives were included in the study. Stakeholder organizations included patient advocacy groups, providers, researchers, payers, policy makers and funding agencies.

Phase 3: Stakeholder Engagement and Awareness Training

Representatives from 39 stakeholder organizations who agreed to participate in the study were further orientated to the study rationale and methods via a series of interactive online webinars. This included reminding organziations that everyone’s input and perspective were valued and that we had a flat organization structure that ensured all stakeholders were equal.

Phase 4: Survey Development and Administration

We chose a survey approach to solicit input on identifying gaps in patient care and to generate research questions. The steering committee developed an online survey collaboratively with stakeholder organization representatives. We used survey pretesting with patient and researcher members from the steering committee. The goal of pretesting was to ensure accessibility and comprehension for all potential respondents, particularly patients and caregivers. The final survey asked respondents to record up to three questions that they thought would improve the care of hospitalized adult patients and their families. The specific wording of the survey is shown in the Figure and the entire survey in Appendix Document 1.

We chose three questions because that is the number of entries per participant that is recommended by JLA; it also minimizes responder burden.¹⁹ We asked respondents to identify the stakeholder group they represented (eg, patient, caregiver, healthcare provider, researcher) and for providers to identify where they primarily worked (eg, acute care hospital, post-acute care, advocacy group).

Survey Administration. We administered the survey electronically using Research Electronic Data Capture (REDCap), a secure web-based application used for collecting research data.²² Stakeholders were asked to disseminate the survey broadly using whatever methods that they felt was appropriate to their leadership or members.

Phase 5: Initial Question Categorization Using Qualitative Content Analysis

Six members of the steering committee independently performed qualitative content analysis to categorize all submitted questions.^23,24 This analytic approach identifies, analyzes, and reports patterns within the data.^23,24 We hypothesized that some of the submitted questions would relate to already-­known problems with hospitalization. Therefore the steering committee developed an a priori codebook of 48 categories using common systems-based issues and diseases related to the care of hospitalized patients based on the hospitalist core competency topics developed by hospitalists and the SHM Education Committee,²⁵ personal and clinical knowledge and experience related to the care of hospitalized adult patients, and published literature on the topic. These a priori categories and their definitions are shown in Appendix Document 2 and were the basis for our initial theory-driven (deductive) approach to data analysis.²³

Once coding began, we identified 32 new and additional categories based on our review of the submitted questions, and these were the basis of our data-driven (inductive) approach to analysis.²³ All proposed new codes and definitions were discussed with and approved by the entire steering committee prior to being added to the codebook (Appendix Document 2).

While coding categories were mutually exclusive, multiple codes could be attributed to a question depending on the content and meaning of a question. To ensure methodological rigor, reviewers met regularly via teleconference or communicated via email throughout the analysis to iteratively refine and define coding categories. All questions were reviewed independently, and then discussed, by at least two members of the analysis team. Any coding disparities were discussed and resolved by negotiated consensus.²⁶ Analysis was conducted using Dedoose V8.0.35 (Sociocultural Research Consultants, Los Angeles, California).

Phase 6: Initial Question Identification Using Quantitative Content Analysis

Following thematic categorization, all steering committee members then reviewed each category to identify and quantify the most commonly submitted questions.²⁷ A question was determined to be a commonly submitted question when it appeared at least 10 times.

Phase 7: Interim Priority Setting

We sent the list of the most commonly submitted questions (Appendix Document 3) to stakeholder organizations and patient partner networks for review and evaluation. Each organization was asked to engage with their constituents and leaders to collectively decide on which of these questions resonated and was most important. These preferences would then be used during the in-person meeting (Phase 8). We did not provide stakeholder organizations with information about how many times each question was submitted by respondents because we felt this could potentially bias their decision-making processes such that true importance and relevance would not obtained.

Phase 8: In-person Meeting for Final Question Prioritization and Refinement

Representatives from all 39 participating stakeholder organizations were invited to participate in a 2-day, in-person meeting to create a final prioritized list of questions to be used to guide patient-centered research seeking to improve the care of hospitalized adult patients and their caregivers. This meeting was attended by 43 stakeholders (26 stakeholder organization representatives and 17 steering committee members) from 37 unique stakeholder organizations. To facilitate the inclusiveness and to ensure a consensus-driven process, we used nominal group technique (NGT) to allow all of the meeting participants to discuss the list of prioritized questions in small groups.²⁸ NGT allows participants to comprehend each other’s point of view to ensure no perpsectives are excluded.²⁸ The NGT was followed by two rounds of individual voting. Stakeholders were then asked to frame their discussions and their votes based on the perspectives of their organizations or PFACs that they represent. The voting process required participants to make choices regarding the relative importance of all of the questions, which therefore makes the resulting list a true prioritized list. In the first round of voting, participants voted for up to five questions for inclusion on the prioritized list. Based on the distribution of votes, where one vote equals one point, each of the 36 questions was then ranked in order of the assigned points. The rank-ordering process resulted in a natural cut point or delineated point, resulting in the 11 questions considered to be the highest prioritized questions. Following this, a second round of voting took place with the same parameters as the first round and allowed us to rank order questions by order of importance and priority. Finally, during small and large group discussions, the original text of each question was edited, refined, and reformatted into questions that could drive a research agenda.

Ethical Oversight

This study was reviewed by the Institutional Review Board of the University of Texas Health Science Center at San Antonio and deemed not to be human subject research (UT Health San Antonio IRB Protocol Number: HSC20170058N).

In total, 499 respondents from 39 unique stakeholder organizations responded to our survey. Respondents self-identified into multiple categorizes resulting in 267 healthcare providers, 244 patients and caregivers, and 63 researchers. Characteristics of respondents to the survey are shown in Table 2.

An overview of study results is shown in Table 1. Respondents submitted a total of 782 questions related to improving the care of hospitalized patients. These questions were categorized during thematic analysis into 70 distinct categories—52 that were health system related and 18 that were disease specific (Appendix 2). The most frequently used health system–related categories were related to discharge care transitions, medications, patient understanding, and patient-family-care team communication (Appendix 2).

From these categories, 36 questions met our criteria for “commonly identified,” ie, submitted at least 10 times (Appendix Document 3). Notably, these 36 questions were derived from 67 different coding categories, of which 24 (36%) were a priori (theory-driven) categories²³ created by the Steering Committee before analysis began and 43 (64%) categories were created as a result of this study’s stakeholder-engaged process and a data-driven approach²³ to analysis (Appendix Document 3). These groups of questions were then presented during the 2-day, in-person meeting and reduced to a final 11 questions that were identified in rank order as top priorities (Table 3). The questions considered highest priority related to ensuring shared treatment and goals of care decision making, improving hospital discharge handoff to other care facilities and providers, and reducing the confusion related to education on medications, conditions, hospital care, and discharge.

Using a dynamic and collaborative stakeholder engagement process, we identified 11 questions prioritized in order of importance by patients, caregivers, and other healthcare stakeholders to improve the care of hospitalized adult patients. While some of the topics identified are already well-known topics in need of research and improvement, our findings frame these topics according to the perspectives of patients, caregivers, and stakeholders. This unique perspective adds a level of richness and nuance that provides insight into how to better address these topics and ultimately inform research and quality improvement efforts.

The question considered to be the highest priority area for future research and improvement surmised how it may be possible to implement interventions that engage patients in shared decision making. Shared decision making involves patients and their care team working together to make decisions about treatment, and other aspects of care based on sound clinical evidence that balances the risks and outcomes with patient preferences and values. Although considered critically important,²⁹ a recent evaluation of shared decision making practices in over 250 inpatient encounters identified significant gaps in physicians’ abilities to perform key elements of a shared decision making approach and reinforced the need to identify what strategies can best promote widespread shared decision making.³⁰ While there has been considerable effort to faciliate shared decision making in practice, there remains mixed evidence regarding the sustainability and impact of tools seeking to support shared decision making, such as decision aids, question prompt lists, and coaches.³¹ This suggests that new approaches to shared decision making may be required and likely explains why this question was rated as a top priority by stakeholders in the current study.

Respondents frequently framed their questions in terms of their lived experiences, providing stories and scenarios to illustrate the importance of the questions they submitted. This personal framing highlighted to us the need to think about improving care delivery from the end-user perspective. For example, respondents framed questions about care transitions not with regard to early appointments, instructions, or medication lists, but rather in terms of whom to call with questions or how best to reach their physician, nurse, or other identified provider. These perspectives suggest that strategies and approaches to improvement that start with patient and caregiver experiences, such as design thinking,³² may be important to continued efforts to improve hospital care. Additionally, the focus on the interpersonal aspects of care delivery highlights the need to focus on the patient-provider relationship and communication.

Questions submitted by respondents demonstrated a stark difference between “patient education” and “patient understanding,” which suggests that being provided with education or education materials regarding care did not necessarily lead to a clear patient understanding. The potential for lack of understanding was particularly prominent in the context of care plan development and during times of care transition—topics that were encompassed in 9 out of 11 of our prioritized research questions. This may suggest that approaches that improve the ability for healthcare providers to deliver information may not be sufficient to meet the needs of patients and caregivers. Rather, partnering to develop a shared understanding—whether about prognosis, medications, hospital, or discharge care plans—is critical. Improved communication practices are not an endpoint for information delivery, but rather a starting point leading to a shared understanding.

Several of the priority areas identified in our study reflect the immensely complex intersections among patients, caregivers, clinicians, and the healthcare delivery system. Addressing these gaps in order to reach the goal of ideal hospital care and an improved patient experience will likely require coordinated approaches and strong involvement and buy-in from multiple stakeholders including the voices of patients and caregivers. Creating patient-centered and stakeholder-driven research has been an increasing priority nationally.³³ Yet to realize this, we must continue to understand the foundations and best practices of authentic stakeholder engagement so that it can be achieved in practice.³⁴ We intend for this prioritized list of questions to galvanize funders, researchers, clinicians, professional societies, and patient and caregiver advocacy groups to work together to address these topics through the creation of new research evidence or the sustainable implementation of existing evidence.

Our findings provide a foundation for stakeholder groups to work in partnership to find research and improvement solutions to the problems identified. Our efforts demonstrate the value and importance of a systematic and broad engagement process to ensure that the voices of patients, caregivers, and other healthcare stakeholders are included in guiding hospital research and quality improvement efforts. This is highlighted by the fact our results of prioritized category areas for research were largely only uncovered following the creation of coding categories during the analysis process and were not captured using a priori catgeories that were expected by the steering committee.

The strengths of this study include our attempts to systematically identify and engage a wide range of perspectives in hospital medicine, including perspectives from patients and their caregivers. There are also acknowledged limitations in our study. While we included patients and PFACs from across the country, the opinions of the people we included may not be representative of all patients. Similarly, the perspectives of the other participants may not have completely represented their stakeholder organizations. While we attempted to include a broad range of organizations, there may be other relevant groups who were not represented in our sample.

In summary, our findings provide direction for the multiple stakeholders involved in improving hospital care. The results will allow the research community to focus on questions that are most important to patients, caregivers, and other stakeholders, reframing them in ways that are more relevant to patients’ lived experiences and that reflect the complexity of the issues. Our findings can also be used by healthcare providers and delivery organizations to target local improvement efforts. We hope that patients and caregivers will use our results to advocate for research and improvement in areas that matter the most to them. We hope that policy makers and funding agencies use our results to promote work in these areas and drive a national conversation about how to most effectively improve hospital care.

The Society of Hospital Medicine (SHM) provided additional administrative, logistical, and technical support.

The authors would like to thank all patients, caregivers, and stakeholders who completed the survey. The authors also would like to acknowledge the organizations and individuals who participated in this study (see Appendix Document 4 for full list). At SHM, the authors would like to specifically thank Claudia Stahl, Jenna Goldstein, Kevin Vuernick, Dr Brad Sharpe, and Dr Larry Wellikson for their support.

The statements presented in this publication are solely the responsibility of the authors and do not necessarily represent the views of the Department of Veterans Affairs, Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors, or Methodology Committee.

Geographic cohorting (GCh, also known as “localization” or “regionalization”) refers to the practice wherein hospitalists are assigned to a single inpatient unit. Its adoption is increasing and in 2017, 30% of surveyed United States hospital medicine group leaders reported that their clinicians rounded on 1-2 units daily.¹ As a component of intervention bundles, GCh is associated with reductions in mortality, length of stay, and costs.^2,3

However, details on how GCh affects the hospitalist workday are unknown. Most time-motion studies of inpatient clinicians have reported the experiences of physicians in training with few specifically evaluating the workflow of attending hospitalists.⁴ Three studies of the attending hospitalist’s workday that were performed a decade ago excluded teams with learners, had patient loads as low as 9.4 per day, and did not differentiate between GCh and non-GCh models.^5-7

The objective of this observational study was to describe and compare the workday of GCh and non-GCh hospitalists by using automated geographical-tracking methods supplemented by in-person observations.

Setting and Participants

This work was conducted at a large academic center in the Midwestern US which adopted GCh in 2012. During the study, hospitalists staffed 11 GCh and four non-GCh teams. GCh teams aim to maintain ≥80% of their patients on their assigned unit and conduct interprofessional huddles on weekdays.³ Some units specialize in the care of specific populations (eg, patients with oncologic diagnoses), while others serve as general medical or surgical units. Non-GCh teams are assigned patients without regard to location. Resident housestaff are assigned only to GCh teams and residents and advanced practice providers (APPs) are never assigned to the same team. Based on team members, this yielded five distinct team types: GCh-hospitalist, GCh-hospitalist with APP, GCh-hospitalist with resident, non-GCh-hospitalist, and non-GCh-hospitalist with APP. Hospitalists provided verbal consent to participate. The protocol was reviewed and approved by the Indiana University Institutional Review Board. Two complementary observation modalities were used. Locator badges were used to quantify direct and indirect time unobtrusively over long periods. In-person observations were conducted to examine the workday in greater detail. Data were collected between October 2017 and May 2018.

Observations by Locator Badges

Our institution uses a system designed by Hill-Rom^® (Cary, North Carolina) to facilitate staff communication. Staff wear the I-Badge^® Locator Badge, which emits an infra-red signal.⁸ Centrally located receivers tabulate time spent by the badge wearer in each location (Appendix Figure 1). Each hospitalist was given a badge to wear at work for a minimum of six weeks, after which the I-Badge^® data were downloaded.

Schedules detailing each team’s members and assigned units (if cohorted) were retrieved. For each observed day, the hospitalist was linked to his or her team type and unit. Team lists were retrieved to ascertain patient load at the start of the day. Data sources were merged to categorize observations.

Observation Categories for Locator Badge Data

The I-Badge^® data provided details of how much time the hospitalist spent in each location (eg, nursing station, hallways, patient rooms). All observations in patient rooms were considered “direct care” while all other locations were categorized as “Indirect Care”. Observations were also categorized by the intensity of care provided on that unit, which included the Emergency Department (ED), Progressive Care Units (PCU), Medical-Surgical + PCU units (for units having a mixture of Medical-Surgical and PCU beds), and Medical-Surgical units.

In-person Observations

Four research assistants (RAs) were trained until interrater reliability using task times achieved an intraclass correlation coefficient of 0.98. Task categories included direct care (all time with patients), indirect care (computer interactions, communication), professional development, and travel and personal time. Interruptions were defined as “an unplanned and unscheduled task, causing a discontinuation, a noticeable break, or task switch behavior”.⁹ “Electronic interruptions” were caused by pagers or phones whereas in-person interruptions were “face-to-face” interruptions. When at least two tasks were performed simultaneously, it was considered multitasking. A data collection form created in REDCap was accessed on computer tablets or smartphones¹⁰ (Appendix Table 1). To limit each observation period to five hours, two RAs were scheduled each day. Observations were continued until the hospitalist reported that work activities were complete or until 5 pm.

Statistical Analysis

Due to the nested structure of the locator badge data, multilevel models that permit predictors to vary at more than one level were used.¹¹ The distribution of the duration of direct care observations was log normal for which the parameters were estimated using generalized linear mixed models (GLMM). The GLMM estimates were converted using a nonlinear transformation to predict the mean duration of interactions. The GLMM estimates were then used to predict time allocations for hospitalists with various workloads and contexts. The five team types were captured in a single categorical variable.

Univariate three-level models predicting minutes spent in direct care were tested for each predictor. Predictors, described below, were selected due to their hypothesized relation to time spent in direct patient care, or to account statistically for differences among teams due to the observational nature of the study.¹² Predictors were: Level 3, hospitalist characteristics (years since medical school, age, gender, international graduate, years at current hospital); Level 2, work day characteristics (number of units visited, number of patients visited, team type, weekday); and Level 1, individual observation characteristics (intensity of care on unit, number of visits to the same patient room per day). Predictors that were significantly related to the duration of direct care at P value <.05 and whose inclusion resulted in better model fit based on likelihood ratio tests were retained in a multivariate model. Additionally, a logistic regression model with random effects was tested to determine whether hospitalists working in GCh vs non-GCh teams (including teams with APPs and residents) made more than one visit to the same patient in a day. For duration of direct care encounters, the amount of variation explained (intraclass correlation) at the hospitalist level was .05, and at the day level was .03.

For total daily indirect care, a similar modeling process was used. A log normal distribution was used because the data was right-skewed and contained positive values. The restricted maximum likelihood method was used to calculate final estimates for models. Least square mean values for independent variables were subjected to backward transformation for interpretation. Post hoc pairwise comparisons between team types were conducted using Tukey–Kramer tests for direct and indirect care time. Analyses were conducted using SAS software version 9.4 (Cary, North Carolina).

The in-person observations were summarized using descriptive statistics. Exploratory analyses were performed using t-tests and Fisher’s exact tests to compare continuous and categorical variables respectively.

Locator Badge Observations

Participants

The 17 hospitalists had a mean (SD) age of 38 years (6.4); 10 (59%) were male, 7 (41%) were international medical graduates, and 10 (59%) had worked at the hospital ≥5 years. The duration of observation was <45 days for 7 hospitalists, 46-55 days for 4, and >55 days for 6, yielding observations for 666 hospitalist workdays. The mean time since medical school graduation was 13 years. Seven hospitalists were observed only in the GCh model, one was observed only in the non-GCh model, and nine were observed in both.

Team Characteristics

On average, non-GCh teams visited more units per day than GCh teams. Teams with APPs had higher patient loads (Table 1).

Time Observed in Direct and Indirect Care

In total, 10,522 observations were recorded in providing direct care. The average duration of a direct care encounter ranged from 4.1 to 5.8 minutes. The ratio of indirect to direct time ranged from 2.7 to 3.7 (Table 2).

The number of times that a hospitalist visited the same patient room in one day ranged from 1 to 9. Most (84%) of the patient rooms were visited once per day. The odds that a GCh hospitalist would visit a patient more than once per day were 1.8 times higher (95% CI: 1.37, 2.34; P < .0001) than for a non-GCh hospitalist (data not shown).

Predictors Associated with Time Expenditure

Predictors significantly associated with both the duration of direct care encounters and total daily indirect care time included team type and patient count. Predicted time in direct care encounters was highest for the GCh-hospitalist team (9.5 minutes) and lowest for the GCh-hospitalist with residents team (7 minutes). Predicted total indirect care time was highest for the GCh-hospitalist with APP team (160 minutes) while the lowest expenditure in indirect care time was predicted for the non-GCh-hospitalist team (102 minutes). Increasing patient load from 10 to 20 was predicted to decrease the duration of a direct care encounter by one minute (14%) and increase the total indirect care time by a larger amount (39 min, 24%).

The duration of direct care encounters was also inversely related with years since medical school and number of visits made to same patient room. Finally, acuity of care was associated with the duration of direct care encounters with the longest predicted encounters in the ED (9.4 minutes). Physician gender and age, international graduation, years at current hospital, weekday, and the number of units visited in a day were neither associated with direct care time at P value < .05 nor improved model fit and therefore were not retained in the final model (Table 3).

In-person Observations

Four hospitalists cohorted to general medical units and four non-GCh hospitalists were observed for one day each, yielding a total of 3,032 minutes of data. These hospitalists were on teams without residents or APPs. On average, GCh hospitalists had 78% of their patients on their assigned unit, rounded on fewer units (3 vs 6) and had two more patients at the start of the day than non-GCh hospitalists (14 vs 12). Age and gender distribution of the GCh and non-GCh hospitalists were similar.

As a percentage of total observed time, GCh hospitalists were noted to spend a larger proportion of the workday in computer interactions vs non-GCh hospitalists (56% vs 39%; P = .005). The proportion of time in other activities or locations was not statistically different between GCh and non-GCh hospitalists, including face-to-face communication (21% vs 15%), multitasking (18% vs 14%), time spent at the nursing station (58% vs 34%), direct care (15% vs 20%), and time traveling (4% vs 11%). The most frequently observed combination of multitasking was computer and phone use (59% of all multitasking) followed by computer use and face-to-face communication (17%; Appendix Figure 2).

The mean duration of an interruption was 1.3 minutes. More interruptions were observed in the GCh group than the non-GCh group (139 vs 102). Interruptions in the GCh group were face-to-face in 62% of instances and electronic in 25%. The remaining 13% were instances in which electronic and face-to-face interruptions occurred simultaneously. In the non-GCh group, 51% of interruptions were face-to-face; 47% were electronic; and 2% were simultaneous. GCh hospitalists were interrupted once every 14 minutes in the morning, with interruption frequency increasing to once every eight minutes in the afternoon. Non-GCh hospitalists were interrupted once every 13 minutes in the morning and saw interruption frequency decrease to once every 17 minutes in the afternoon. The task most frequently interrupted was computer use.

DISCUSSION

Previous investigations have studied the impact of cohorting on outcomes, including the facilitation of bedside rounding, adverse events, agreement between nurses and physicians on the plan of care, productivity, and the number of pages received.^13-16 Cohorting’s benefits are theorized to include increased hospitalist time with patients, while its downsides are perceived to include increased interruptions.^17,18 Neither has previously been evaluated by direct observation.

Our findings support cohorting’s association with increased hospitalist–patient time. While GCh hospitalists were observed spending 5% less time in direct care than non-GCh hospitalists by in-person observations, this difference did not achieve statistical significance and was unadjusted for hospitalist, patient load, team or patient characteristics. Using the larger badge dataset, the predicted values for time spent in direct care encounters were higher in cohorted teams. Pairwise comparisons consistently trended toward longer durations in cohorted vs noncohorted teams. The notable exception was in cohorted teams with residents, which had the shortest predicted patient visits; however, we did not have noncohorted teams with residents in our study, limiting interpretation. Additionally, the odds of repeat visits to a patient in a single day were almost twice as high in the cohorted vs noncohorted group. The magnitude of this gain, however, is estimated to be a modest 1.2 minutes for a hospitalist only team and 1.7 minutes for a hospitalist with APP team and may be insufficient to provide compassionate, patient-centered care.¹⁹

Furthermore, these gains may be eroded if patient loads are high: similar to a previous study, we found that the duration of each patient visit decreased by 14% when the load increased from 10 to 20 patients.⁶ The expected gains in efficiency from cohorting leads to an expectation that hospitalists can manage more patients, but such reflexive increases should be carefully considered.¹⁸

Similar to earlier investigations where hospitalists were found to spend 60 to 69% of the day in indirect care activities,^5,6 hospitalists in both cohorted and noncohorted models spent approximately three times more time in indirect than direct care. Cohorting was associated with increased indirect care time. This association was expected as interdisciplinary huddles and increased nursing and physician communication are both related to cohorting.^3,14 However, similar to previous reports, in-person observations revealed that the bulk of this indirect time was spent in computer interactions, rather than in interprofessional communication. Interactions with the electronic health record (EHR) consume between one-third to one-half of the day in inpatient settings.^20,21 While EHRs are intended to enhance safety, they also fulfill multiple, nonclinical purposes and increase time spent on documentation.^22,23 Nonclinical tasks may contribute to clinician burnout and detract from patient centeredness.²² Our findings suggest that cohorting may not offset the burden of these time-intensive EHR tasks. The larger expenditure of time spent in computer interactions observed in the GCh group may be partially explained both by the higher number of patients and the higher frequency of interruptions observed in this group; computer use was the task most frequently observed to be interrupted. While longer tasks are more likely to be interrupted, the interruption in turn further increases the time taken to complete the task.²⁴

The interruption rates we observed are concerning. The hospitalist workday emerges as cognitively intense. GCh hospitalists were noted to be interrupted as frequently as once every eight minutes, a rate more than double that of an earlier investigation and approaching that of ED physicians.^5,25,26 Interruptions and multitasking contribute to errors and a perception of increased workload and frustration for clinicians.^9,27-29 Although interruptions were pervasive, GCh hospitalists were interrupted more frequently, corroborating a national survey in which hospitalists perceived that cohorting increased face-to-face interruptions.³⁰ The prolonged availability of the cohorted hospitalist on the unit may require different strategies for promoting timely interactions while preserving uninterrupted work time. Our work, however, does not allow us to quantify appropriate and urgent interruptions that reflect improved teamwork and patient safety. Interruptions increase as patient loads increase.²⁵ The contribution to interruptions by the higher patient census on the GCh teams cannot be quantified in this work, but without attention to these details, potential benefits from GCh may be attenuated.

Previous work has delineated variables important in determining hospitalist workload,³¹ and our work contributes additional considerations. Hospitalist experience and resident presence on cohorted teams was associated with shorter patient visits, while ED encounters were predicted to be the most time intensive. Increasing numbers of units visited in a day was associated with more indirect time, while weekends were associated with a lower burden of indirect care. As expected, APP presence was associated with more time in indirect care as the hospitalist spends time in providing oversight. As noted, cohorting was associated with increases in both direct and indirect care time. These findings may help inform hospital medicine groups. Additionally, attention should be paid to the fact that while support for cohorting stems from investigations in which it was used as part of a bundle of interventions,^2,3 in practice, it is often implemented incompletely, with cohorted hospitalists dispersed over several units, or in isolation from other interventions.¹

Our work has several limitations. As a single-center investigation, our findings may not be generalizable to other institutions. Second, we did not evaluate clinical outcomes, clinician, patient or nursing satisfaction to assess the effect of cohorting. Third, we cannot comment on whether the observed interruptions were beneficial or detrimental. Finally, while we used statistical control for the measured imbalanced variables between groups, unmeasured confounding factors between team types including differences in patient populations, pathologies and severity of illness, or the unit’s work environment and processes may have affected results.

Our work underscores the importance of paying careful attention to specific components and monitoring for unintended consequences in a complex intervention such as cohorting to allow subsequent refinement. Further studies to assess the interplay between models of care, their impact on interruptions, multitasking, errors and clinician burnout may be necessary. Such investigations will be critical to support the evolution of hospital medicine that enables it to be the driver of excellence in care.

Acknowledgments

The authors thank the participating hospitalists, research assistants, Shelly Harrison, Joni Godfrey, Mark Luetkemeyer, Deanne Kashiwagi, Tammy Kemlage, Dustin Hertel and Adeel Zaidi for their enthusiasm and support. The authors also thank Ann Cottingham, Rich Frankel and Greg Sachs from the ASPIRE program for their guidance and vision. Dr. Weiner is Chief of Health Services Research and Development at the Richard L. Roudebush Veterans Affairs Medical Center in Indianapolis, Indiana.

Disclaimer

The views expressed in this article are those of the authors and do not necessarily represent the views of the U.S. Department of Veterans Affairs.

Geographic cohorting (GCh, also known as “localization” or “regionalization”) refers to the practice wherein hospitalists are assigned to a single inpatient unit. Its adoption is increasing and in 2017, 30% of surveyed United States hospital medicine group leaders reported that their clinicians rounded on 1-2 units daily.¹ As a component of intervention bundles, GCh is associated with reductions in mortality, length of stay, and costs.^2,3

However, details on how GCh affects the hospitalist workday are unknown. Most time-motion studies of inpatient clinicians have reported the experiences of physicians in training with few specifically evaluating the workflow of attending hospitalists.⁴ Three studies of the attending hospitalist’s workday that were performed a decade ago excluded teams with learners, had patient loads as low as 9.4 per day, and did not differentiate between GCh and non-GCh models.^5-7

The objective of this observational study was to describe and compare the workday of GCh and non-GCh hospitalists by using automated geographical-tracking methods supplemented by in-person observations.

Setting and Participants

This work was conducted at a large academic center in the Midwestern US which adopted GCh in 2012. During the study, hospitalists staffed 11 GCh and four non-GCh teams. GCh teams aim to maintain ≥80% of their patients on their assigned unit and conduct interprofessional huddles on weekdays.³ Some units specialize in the care of specific populations (eg, patients with oncologic diagnoses), while others serve as general medical or surgical units. Non-GCh teams are assigned patients without regard to location. Resident housestaff are assigned only to GCh teams and residents and advanced practice providers (APPs) are never assigned to the same team. Based on team members, this yielded five distinct team types: GCh-hospitalist, GCh-hospitalist with APP, GCh-hospitalist with resident, non-GCh-hospitalist, and non-GCh-hospitalist with APP. Hospitalists provided verbal consent to participate. The protocol was reviewed and approved by the Indiana University Institutional Review Board. Two complementary observation modalities were used. Locator badges were used to quantify direct and indirect time unobtrusively over long periods. In-person observations were conducted to examine the workday in greater detail. Data were collected between October 2017 and May 2018.

Observations by Locator Badges

Our institution uses a system designed by Hill-Rom^® (Cary, North Carolina) to facilitate staff communication. Staff wear the I-Badge^® Locator Badge, which emits an infra-red signal.⁸ Centrally located receivers tabulate time spent by the badge wearer in each location (Appendix Figure 1). Each hospitalist was given a badge to wear at work for a minimum of six weeks, after which the I-Badge^® data were downloaded.

Schedules detailing each team’s members and assigned units (if cohorted) were retrieved. For each observed day, the hospitalist was linked to his or her team type and unit. Team lists were retrieved to ascertain patient load at the start of the day. Data sources were merged to categorize observations.

Observation Categories for Locator Badge Data

The I-Badge^® data provided details of how much time the hospitalist spent in each location (eg, nursing station, hallways, patient rooms). All observations in patient rooms were considered “direct care” while all other locations were categorized as “Indirect Care”. Observations were also categorized by the intensity of care provided on that unit, which included the Emergency Department (ED), Progressive Care Units (PCU), Medical-Surgical + PCU units (for units having a mixture of Medical-Surgical and PCU beds), and Medical-Surgical units.

In-person Observations

Four research assistants (RAs) were trained until interrater reliability using task times achieved an intraclass correlation coefficient of 0.98. Task categories included direct care (all time with patients), indirect care (computer interactions, communication), professional development, and travel and personal time. Interruptions were defined as “an unplanned and unscheduled task, causing a discontinuation, a noticeable break, or task switch behavior”.⁹ “Electronic interruptions” were caused by pagers or phones whereas in-person interruptions were “face-to-face” interruptions. When at least two tasks were performed simultaneously, it was considered multitasking. A data collection form created in REDCap was accessed on computer tablets or smartphones¹⁰ (Appendix Table 1). To limit each observation period to five hours, two RAs were scheduled each day. Observations were continued until the hospitalist reported that work activities were complete or until 5 pm.

Statistical Analysis

Due to the nested structure of the locator badge data, multilevel models that permit predictors to vary at more than one level were used.¹¹ The distribution of the duration of direct care observations was log normal for which the parameters were estimated using generalized linear mixed models (GLMM). The GLMM estimates were converted using a nonlinear transformation to predict the mean duration of interactions. The GLMM estimates were then used to predict time allocations for hospitalists with various workloads and contexts. The five team types were captured in a single categorical variable.

Univariate three-level models predicting minutes spent in direct care were tested for each predictor. Predictors, described below, were selected due to their hypothesized relation to time spent in direct patient care, or to account statistically for differences among teams due to the observational nature of the study.¹² Predictors were: Level 3, hospitalist characteristics (years since medical school, age, gender, international graduate, years at current hospital); Level 2, work day characteristics (number of units visited, number of patients visited, team type, weekday); and Level 1, individual observation characteristics (intensity of care on unit, number of visits to the same patient room per day). Predictors that were significantly related to the duration of direct care at P value <.05 and whose inclusion resulted in better model fit based on likelihood ratio tests were retained in a multivariate model. Additionally, a logistic regression model with random effects was tested to determine whether hospitalists working in GCh vs non-GCh teams (including teams with APPs and residents) made more than one visit to the same patient in a day. For duration of direct care encounters, the amount of variation explained (intraclass correlation) at the hospitalist level was .05, and at the day level was .03.

For total daily indirect care, a similar modeling process was used. A log normal distribution was used because the data was right-skewed and contained positive values. The restricted maximum likelihood method was used to calculate final estimates for models. Least square mean values for independent variables were subjected to backward transformation for interpretation. Post hoc pairwise comparisons between team types were conducted using Tukey–Kramer tests for direct and indirect care time. Analyses were conducted using SAS software version 9.4 (Cary, North Carolina).

The in-person observations were summarized using descriptive statistics. Exploratory analyses were performed using t-tests and Fisher’s exact tests to compare continuous and categorical variables respectively.

Locator Badge Observations

Participants

The 17 hospitalists had a mean (SD) age of 38 years (6.4); 10 (59%) were male, 7 (41%) were international medical graduates, and 10 (59%) had worked at the hospital ≥5 years. The duration of observation was <45 days for 7 hospitalists, 46-55 days for 4, and >55 days for 6, yielding observations for 666 hospitalist workdays. The mean time since medical school graduation was 13 years. Seven hospitalists were observed only in the GCh model, one was observed only in the non-GCh model, and nine were observed in both.

Team Characteristics

On average, non-GCh teams visited more units per day than GCh teams. Teams with APPs had higher patient loads (Table 1).

Time Observed in Direct and Indirect Care

In total, 10,522 observations were recorded in providing direct care. The average duration of a direct care encounter ranged from 4.1 to 5.8 minutes. The ratio of indirect to direct time ranged from 2.7 to 3.7 (Table 2).

The number of times that a hospitalist visited the same patient room in one day ranged from 1 to 9. Most (84%) of the patient rooms were visited once per day. The odds that a GCh hospitalist would visit a patient more than once per day were 1.8 times higher (95% CI: 1.37, 2.34; P < .0001) than for a non-GCh hospitalist (data not shown).

Predictors Associated with Time Expenditure

Predictors significantly associated with both the duration of direct care encounters and total daily indirect care time included team type and patient count. Predicted time in direct care encounters was highest for the GCh-hospitalist team (9.5 minutes) and lowest for the GCh-hospitalist with residents team (7 minutes). Predicted total indirect care time was highest for the GCh-hospitalist with APP team (160 minutes) while the lowest expenditure in indirect care time was predicted for the non-GCh-hospitalist team (102 minutes). Increasing patient load from 10 to 20 was predicted to decrease the duration of a direct care encounter by one minute (14%) and increase the total indirect care time by a larger amount (39 min, 24%).

The duration of direct care encounters was also inversely related with years since medical school and number of visits made to same patient room. Finally, acuity of care was associated with the duration of direct care encounters with the longest predicted encounters in the ED (9.4 minutes). Physician gender and age, international graduation, years at current hospital, weekday, and the number of units visited in a day were neither associated with direct care time at P value < .05 nor improved model fit and therefore were not retained in the final model (Table 3).

In-person Observations

Four hospitalists cohorted to general medical units and four non-GCh hospitalists were observed for one day each, yielding a total of 3,032 minutes of data. These hospitalists were on teams without residents or APPs. On average, GCh hospitalists had 78% of their patients on their assigned unit, rounded on fewer units (3 vs 6) and had two more patients at the start of the day than non-GCh hospitalists (14 vs 12). Age and gender distribution of the GCh and non-GCh hospitalists were similar.

As a percentage of total observed time, GCh hospitalists were noted to spend a larger proportion of the workday in computer interactions vs non-GCh hospitalists (56% vs 39%; P = .005). The proportion of time in other activities or locations was not statistically different between GCh and non-GCh hospitalists, including face-to-face communication (21% vs 15%), multitasking (18% vs 14%), time spent at the nursing station (58% vs 34%), direct care (15% vs 20%), and time traveling (4% vs 11%). The most frequently observed combination of multitasking was computer and phone use (59% of all multitasking) followed by computer use and face-to-face communication (17%; Appendix Figure 2).

The mean duration of an interruption was 1.3 minutes. More interruptions were observed in the GCh group than the non-GCh group (139 vs 102). Interruptions in the GCh group were face-to-face in 62% of instances and electronic in 25%. The remaining 13% were instances in which electronic and face-to-face interruptions occurred simultaneously. In the non-GCh group, 51% of interruptions were face-to-face; 47% were electronic; and 2% were simultaneous. GCh hospitalists were interrupted once every 14 minutes in the morning, with interruption frequency increasing to once every eight minutes in the afternoon. Non-GCh hospitalists were interrupted once every 13 minutes in the morning and saw interruption frequency decrease to once every 17 minutes in the afternoon. The task most frequently interrupted was computer use.

DISCUSSION

Previous investigations have studied the impact of cohorting on outcomes, including the facilitation of bedside rounding, adverse events, agreement between nurses and physicians on the plan of care, productivity, and the number of pages received.^13-16 Cohorting’s benefits are theorized to include increased hospitalist time with patients, while its downsides are perceived to include increased interruptions.^17,18 Neither has previously been evaluated by direct observation.

Our findings support cohorting’s association with increased hospitalist–patient time. While GCh hospitalists were observed spending 5% less time in direct care than non-GCh hospitalists by in-person observations, this difference did not achieve statistical significance and was unadjusted for hospitalist, patient load, team or patient characteristics. Using the larger badge dataset, the predicted values for time spent in direct care encounters were higher in cohorted teams. Pairwise comparisons consistently trended toward longer durations in cohorted vs noncohorted teams. The notable exception was in cohorted teams with residents, which had the shortest predicted patient visits; however, we did not have noncohorted teams with residents in our study, limiting interpretation. Additionally, the odds of repeat visits to a patient in a single day were almost twice as high in the cohorted vs noncohorted group. The magnitude of this gain, however, is estimated to be a modest 1.2 minutes for a hospitalist only team and 1.7 minutes for a hospitalist with APP team and may be insufficient to provide compassionate, patient-centered care.¹⁹

Furthermore, these gains may be eroded if patient loads are high: similar to a previous study, we found that the duration of each patient visit decreased by 14% when the load increased from 10 to 20 patients.⁶ The expected gains in efficiency from cohorting leads to an expectation that hospitalists can manage more patients, but such reflexive increases should be carefully considered.¹⁸

Similar to earlier investigations where hospitalists were found to spend 60 to 69% of the day in indirect care activities,^5,6 hospitalists in both cohorted and noncohorted models spent approximately three times more time in indirect than direct care. Cohorting was associated with increased indirect care time. This association was expected as interdisciplinary huddles and increased nursing and physician communication are both related to cohorting.^3,14 However, similar to previous reports, in-person observations revealed that the bulk of this indirect time was spent in computer interactions, rather than in interprofessional communication. Interactions with the electronic health record (EHR) consume between one-third to one-half of the day in inpatient settings.^20,21 While EHRs are intended to enhance safety, they also fulfill multiple, nonclinical purposes and increase time spent on documentation.^22,23 Nonclinical tasks may contribute to clinician burnout and detract from patient centeredness.²² Our findings suggest that cohorting may not offset the burden of these time-intensive EHR tasks. The larger expenditure of time spent in computer interactions observed in the GCh group may be partially explained both by the higher number of patients and the higher frequency of interruptions observed in this group; computer use was the task most frequently observed to be interrupted. While longer tasks are more likely to be interrupted, the interruption in turn further increases the time taken to complete the task.²⁴

The interruption rates we observed are concerning. The hospitalist workday emerges as cognitively intense. GCh hospitalists were noted to be interrupted as frequently as once every eight minutes, a rate more than double that of an earlier investigation and approaching that of ED physicians.^5,25,26 Interruptions and multitasking contribute to errors and a perception of increased workload and frustration for clinicians.^9,27-29 Although interruptions were pervasive, GCh hospitalists were interrupted more frequently, corroborating a national survey in which hospitalists perceived that cohorting increased face-to-face interruptions.³⁰ The prolonged availability of the cohorted hospitalist on the unit may require different strategies for promoting timely interactions while preserving uninterrupted work time. Our work, however, does not allow us to quantify appropriate and urgent interruptions that reflect improved teamwork and patient safety. Interruptions increase as patient loads increase.²⁵ The contribution to interruptions by the higher patient census on the GCh teams cannot be quantified in this work, but without attention to these details, potential benefits from GCh may be attenuated.

Previous work has delineated variables important in determining hospitalist workload,³¹ and our work contributes additional considerations. Hospitalist experience and resident presence on cohorted teams was associated with shorter patient visits, while ED encounters were predicted to be the most time intensive. Increasing numbers of units visited in a day was associated with more indirect time, while weekends were associated with a lower burden of indirect care. As expected, APP presence was associated with more time in indirect care as the hospitalist spends time in providing oversight. As noted, cohorting was associated with increases in both direct and indirect care time. These findings may help inform hospital medicine groups. Additionally, attention should be paid to the fact that while support for cohorting stems from investigations in which it was used as part of a bundle of interventions,^2,3 in practice, it is often implemented incompletely, with cohorted hospitalists dispersed over several units, or in isolation from other interventions.¹

Our work has several limitations. As a single-center investigation, our findings may not be generalizable to other institutions. Second, we did not evaluate clinical outcomes, clinician, patient or nursing satisfaction to assess the effect of cohorting. Third, we cannot comment on whether the observed interruptions were beneficial or detrimental. Finally, while we used statistical control for the measured imbalanced variables between groups, unmeasured confounding factors between team types including differences in patient populations, pathologies and severity of illness, or the unit’s work environment and processes may have affected results.

Our work underscores the importance of paying careful attention to specific components and monitoring for unintended consequences in a complex intervention such as cohorting to allow subsequent refinement. Further studies to assess the interplay between models of care, their impact on interruptions, multitasking, errors and clinician burnout may be necessary. Such investigations will be critical to support the evolution of hospital medicine that enables it to be the driver of excellence in care.

Acknowledgments

The authors thank the participating hospitalists, research assistants, Shelly Harrison, Joni Godfrey, Mark Luetkemeyer, Deanne Kashiwagi, Tammy Kemlage, Dustin Hertel and Adeel Zaidi for their enthusiasm and support. The authors also thank Ann Cottingham, Rich Frankel and Greg Sachs from the ASPIRE program for their guidance and vision. Dr. Weiner is Chief of Health Services Research and Development at the Richard L. Roudebush Veterans Affairs Medical Center in Indianapolis, Indiana.

Disclaimer

The views expressed in this article are those of the authors and do not necessarily represent the views of the U.S. Department of Veterans Affairs.

Geographic cohorting (GCh, also known as “localization” or “regionalization”) refers to the practice wherein hospitalists are assigned to a single inpatient unit. Its adoption is increasing and in 2017, 30% of surveyed United States hospital medicine group leaders reported that their clinicians rounded on 1-2 units daily.¹ As a component of intervention bundles, GCh is associated with reductions in mortality, length of stay, and costs.^2,3

However, details on how GCh affects the hospitalist workday are unknown. Most time-motion studies of inpatient clinicians have reported the experiences of physicians in training with few specifically evaluating the workflow of attending hospitalists.⁴ Three studies of the attending hospitalist’s workday that were performed a decade ago excluded teams with learners, had patient loads as low as 9.4 per day, and did not differentiate between GCh and non-GCh models.^5-7

The objective of this observational study was to describe and compare the workday of GCh and non-GCh hospitalists by using automated geographical-tracking methods supplemented by in-person observations.

Setting and Participants

This work was conducted at a large academic center in the Midwestern US which adopted GCh in 2012. During the study, hospitalists staffed 11 GCh and four non-GCh teams. GCh teams aim to maintain ≥80% of their patients on their assigned unit and conduct interprofessional huddles on weekdays.³ Some units specialize in the care of specific populations (eg, patients with oncologic diagnoses), while others serve as general medical or surgical units. Non-GCh teams are assigned patients without regard to location. Resident housestaff are assigned only to GCh teams and residents and advanced practice providers (APPs) are never assigned to the same team. Based on team members, this yielded five distinct team types: GCh-hospitalist, GCh-hospitalist with APP, GCh-hospitalist with resident, non-GCh-hospitalist, and non-GCh-hospitalist with APP. Hospitalists provided verbal consent to participate. The protocol was reviewed and approved by the Indiana University Institutional Review Board. Two complementary observation modalities were used. Locator badges were used to quantify direct and indirect time unobtrusively over long periods. In-person observations were conducted to examine the workday in greater detail. Data were collected between October 2017 and May 2018.

Observations by Locator Badges

Our institution uses a system designed by Hill-Rom^® (Cary, North Carolina) to facilitate staff communication. Staff wear the I-Badge^® Locator Badge, which emits an infra-red signal.⁸ Centrally located receivers tabulate time spent by the badge wearer in each location (Appendix Figure 1). Each hospitalist was given a badge to wear at work for a minimum of six weeks, after which the I-Badge^® data were downloaded.

Schedules detailing each team’s members and assigned units (if cohorted) were retrieved. For each observed day, the hospitalist was linked to his or her team type and unit. Team lists were retrieved to ascertain patient load at the start of the day. Data sources were merged to categorize observations.

Observation Categories for Locator Badge Data

The I-Badge^® data provided details of how much time the hospitalist spent in each location (eg, nursing station, hallways, patient rooms). All observations in patient rooms were considered “direct care” while all other locations were categorized as “Indirect Care”. Observations were also categorized by the intensity of care provided on that unit, which included the Emergency Department (ED), Progressive Care Units (PCU), Medical-Surgical + PCU units (for units having a mixture of Medical-Surgical and PCU beds), and Medical-Surgical units.

In-person Observations

Four research assistants (RAs) were trained until interrater reliability using task times achieved an intraclass correlation coefficient of 0.98. Task categories included direct care (all time with patients), indirect care (computer interactions, communication), professional development, and travel and personal time. Interruptions were defined as “an unplanned and unscheduled task, causing a discontinuation, a noticeable break, or task switch behavior”.⁹ “Electronic interruptions” were caused by pagers or phones whereas in-person interruptions were “face-to-face” interruptions. When at least two tasks were performed simultaneously, it was considered multitasking. A data collection form created in REDCap was accessed on computer tablets or smartphones¹⁰ (Appendix Table 1). To limit each observation period to five hours, two RAs were scheduled each day. Observations were continued until the hospitalist reported that work activities were complete or until 5 pm.

Statistical Analysis

Due to the nested structure of the locator badge data, multilevel models that permit predictors to vary at more than one level were used.¹¹ The distribution of the duration of direct care observations was log normal for which the parameters were estimated using generalized linear mixed models (GLMM). The GLMM estimates were converted using a nonlinear transformation to predict the mean duration of interactions. The GLMM estimates were then used to predict time allocations for hospitalists with various workloads and contexts. The five team types were captured in a single categorical variable.

Univariate three-level models predicting minutes spent in direct care were tested for each predictor. Predictors, described below, were selected due to their hypothesized relation to time spent in direct patient care, or to account statistically for differences among teams due to the observational nature of the study.¹² Predictors were: Level 3, hospitalist characteristics (years since medical school, age, gender, international graduate, years at current hospital); Level 2, work day characteristics (number of units visited, number of patients visited, team type, weekday); and Level 1, individual observation characteristics (intensity of care on unit, number of visits to the same patient room per day). Predictors that were significantly related to the duration of direct care at P value <.05 and whose inclusion resulted in better model fit based on likelihood ratio tests were retained in a multivariate model. Additionally, a logistic regression model with random effects was tested to determine whether hospitalists working in GCh vs non-GCh teams (including teams with APPs and residents) made more than one visit to the same patient in a day. For duration of direct care encounters, the amount of variation explained (intraclass correlation) at the hospitalist level was .05, and at the day level was .03.

For total daily indirect care, a similar modeling process was used. A log normal distribution was used because the data was right-skewed and contained positive values. The restricted maximum likelihood method was used to calculate final estimates for models. Least square mean values for independent variables were subjected to backward transformation for interpretation. Post hoc pairwise comparisons between team types were conducted using Tukey–Kramer tests for direct and indirect care time. Analyses were conducted using SAS software version 9.4 (Cary, North Carolina).

The in-person observations were summarized using descriptive statistics. Exploratory analyses were performed using t-tests and Fisher’s exact tests to compare continuous and categorical variables respectively.

Locator Badge Observations

Participants

The 17 hospitalists had a mean (SD) age of 38 years (6.4); 10 (59%) were male, 7 (41%) were international medical graduates, and 10 (59%) had worked at the hospital ≥5 years. The duration of observation was <45 days for 7 hospitalists, 46-55 days for 4, and >55 days for 6, yielding observations for 666 hospitalist workdays. The mean time since medical school graduation was 13 years. Seven hospitalists were observed only in the GCh model, one was observed only in the non-GCh model, and nine were observed in both.

Team Characteristics

On average, non-GCh teams visited more units per day than GCh teams. Teams with APPs had higher patient loads (Table 1).

Time Observed in Direct and Indirect Care

In total, 10,522 observations were recorded in providing direct care. The average duration of a direct care encounter ranged from 4.1 to 5.8 minutes. The ratio of indirect to direct time ranged from 2.7 to 3.7 (Table 2).

The number of times that a hospitalist visited the same patient room in one day ranged from 1 to 9. Most (84%) of the patient rooms were visited once per day. The odds that a GCh hospitalist would visit a patient more than once per day were 1.8 times higher (95% CI: 1.37, 2.34; P < .0001) than for a non-GCh hospitalist (data not shown).

Predictors Associated with Time Expenditure

Predictors significantly associated with both the duration of direct care encounters and total daily indirect care time included team type and patient count. Predicted time in direct care encounters was highest for the GCh-hospitalist team (9.5 minutes) and lowest for the GCh-hospitalist with residents team (7 minutes). Predicted total indirect care time was highest for the GCh-hospitalist with APP team (160 minutes) while the lowest expenditure in indirect care time was predicted for the non-GCh-hospitalist team (102 minutes). Increasing patient load from 10 to 20 was predicted to decrease the duration of a direct care encounter by one minute (14%) and increase the total indirect care time by a larger amount (39 min, 24%).

The duration of direct care encounters was also inversely related with years since medical school and number of visits made to same patient room. Finally, acuity of care was associated with the duration of direct care encounters with the longest predicted encounters in the ED (9.4 minutes). Physician gender and age, international graduation, years at current hospital, weekday, and the number of units visited in a day were neither associated with direct care time at P value < .05 nor improved model fit and therefore were not retained in the final model (Table 3).

In-person Observations

Four hospitalists cohorted to general medical units and four non-GCh hospitalists were observed for one day each, yielding a total of 3,032 minutes of data. These hospitalists were on teams without residents or APPs. On average, GCh hospitalists had 78% of their patients on their assigned unit, rounded on fewer units (3 vs 6) and had two more patients at the start of the day than non-GCh hospitalists (14 vs 12). Age and gender distribution of the GCh and non-GCh hospitalists were similar.

As a percentage of total observed time, GCh hospitalists were noted to spend a larger proportion of the workday in computer interactions vs non-GCh hospitalists (56% vs 39%; P = .005). The proportion of time in other activities or locations was not statistically different between GCh and non-GCh hospitalists, including face-to-face communication (21% vs 15%), multitasking (18% vs 14%), time spent at the nursing station (58% vs 34%), direct care (15% vs 20%), and time traveling (4% vs 11%). The most frequently observed combination of multitasking was computer and phone use (59% of all multitasking) followed by computer use and face-to-face communication (17%; Appendix Figure 2).

The mean duration of an interruption was 1.3 minutes. More interruptions were observed in the GCh group than the non-GCh group (139 vs 102). Interruptions in the GCh group were face-to-face in 62% of instances and electronic in 25%. The remaining 13% were instances in which electronic and face-to-face interruptions occurred simultaneously. In the non-GCh group, 51% of interruptions were face-to-face; 47% were electronic; and 2% were simultaneous. GCh hospitalists were interrupted once every 14 minutes in the morning, with interruption frequency increasing to once every eight minutes in the afternoon. Non-GCh hospitalists were interrupted once every 13 minutes in the morning and saw interruption frequency decrease to once every 17 minutes in the afternoon. The task most frequently interrupted was computer use.

DISCUSSION

Previous investigations have studied the impact of cohorting on outcomes, including the facilitation of bedside rounding, adverse events, agreement between nurses and physicians on the plan of care, productivity, and the number of pages received.^13-16 Cohorting’s benefits are theorized to include increased hospitalist time with patients, while its downsides are perceived to include increased interruptions.^17,18 Neither has previously been evaluated by direct observation.

Our findings support cohorting’s association with increased hospitalist–patient time. While GCh hospitalists were observed spending 5% less time in direct care than non-GCh hospitalists by in-person observations, this difference did not achieve statistical significance and was unadjusted for hospitalist, patient load, team or patient characteristics. Using the larger badge dataset, the predicted values for time spent in direct care encounters were higher in cohorted teams. Pairwise comparisons consistently trended toward longer durations in cohorted vs noncohorted teams. The notable exception was in cohorted teams with residents, which had the shortest predicted patient visits; however, we did not have noncohorted teams with residents in our study, limiting interpretation. Additionally, the odds of repeat visits to a patient in a single day were almost twice as high in the cohorted vs noncohorted group. The magnitude of this gain, however, is estimated to be a modest 1.2 minutes for a hospitalist only team and 1.7 minutes for a hospitalist with APP team and may be insufficient to provide compassionate, patient-centered care.¹⁹

Furthermore, these gains may be eroded if patient loads are high: similar to a previous study, we found that the duration of each patient visit decreased by 14% when the load increased from 10 to 20 patients.⁶ The expected gains in efficiency from cohorting leads to an expectation that hospitalists can manage more patients, but such reflexive increases should be carefully considered.¹⁸

Similar to earlier investigations where hospitalists were found to spend 60 to 69% of the day in indirect care activities,^5,6 hospitalists in both cohorted and noncohorted models spent approximately three times more time in indirect than direct care. Cohorting was associated with increased indirect care time. This association was expected as interdisciplinary huddles and increased nursing and physician communication are both related to cohorting.^3,14 However, similar to previous reports, in-person observations revealed that the bulk of this indirect time was spent in computer interactions, rather than in interprofessional communication. Interactions with the electronic health record (EHR) consume between one-third to one-half of the day in inpatient settings.^20,21 While EHRs are intended to enhance safety, they also fulfill multiple, nonclinical purposes and increase time spent on documentation.^22,23 Nonclinical tasks may contribute to clinician burnout and detract from patient centeredness.²² Our findings suggest that cohorting may not offset the burden of these time-intensive EHR tasks. The larger expenditure of time spent in computer interactions observed in the GCh group may be partially explained both by the higher number of patients and the higher frequency of interruptions observed in this group; computer use was the task most frequently observed to be interrupted. While longer tasks are more likely to be interrupted, the interruption in turn further increases the time taken to complete the task.²⁴

The interruption rates we observed are concerning. The hospitalist workday emerges as cognitively intense. GCh hospitalists were noted to be interrupted as frequently as once every eight minutes, a rate more than double that of an earlier investigation and approaching that of ED physicians.^5,25,26 Interruptions and multitasking contribute to errors and a perception of increased workload and frustration for clinicians.^9,27-29 Although interruptions were pervasive, GCh hospitalists were interrupted more frequently, corroborating a national survey in which hospitalists perceived that cohorting increased face-to-face interruptions.³⁰ The prolonged availability of the cohorted hospitalist on the unit may require different strategies for promoting timely interactions while preserving uninterrupted work time. Our work, however, does not allow us to quantify appropriate and urgent interruptions that reflect improved teamwork and patient safety. Interruptions increase as patient loads increase.²⁵ The contribution to interruptions by the higher patient census on the GCh teams cannot be quantified in this work, but without attention to these details, potential benefits from GCh may be attenuated.

Previous work has delineated variables important in determining hospitalist workload,³¹ and our work contributes additional considerations. Hospitalist experience and resident presence on cohorted teams was associated with shorter patient visits, while ED encounters were predicted to be the most time intensive. Increasing numbers of units visited in a day was associated with more indirect time, while weekends were associated with a lower burden of indirect care. As expected, APP presence was associated with more time in indirect care as the hospitalist spends time in providing oversight. As noted, cohorting was associated with increases in both direct and indirect care time. These findings may help inform hospital medicine groups. Additionally, attention should be paid to the fact that while support for cohorting stems from investigations in which it was used as part of a bundle of interventions,^2,3 in practice, it is often implemented incompletely, with cohorted hospitalists dispersed over several units, or in isolation from other interventions.¹

Our work has several limitations. As a single-center investigation, our findings may not be generalizable to other institutions. Second, we did not evaluate clinical outcomes, clinician, patient or nursing satisfaction to assess the effect of cohorting. Third, we cannot comment on whether the observed interruptions were beneficial or detrimental. Finally, while we used statistical control for the measured imbalanced variables between groups, unmeasured confounding factors between team types including differences in patient populations, pathologies and severity of illness, or the unit’s work environment and processes may have affected results.

Our work underscores the importance of paying careful attention to specific components and monitoring for unintended consequences in a complex intervention such as cohorting to allow subsequent refinement. Further studies to assess the interplay between models of care, their impact on interruptions, multitasking, errors and clinician burnout may be necessary. Such investigations will be critical to support the evolution of hospital medicine that enables it to be the driver of excellence in care.

Acknowledgments

The authors thank the participating hospitalists, research assistants, Shelly Harrison, Joni Godfrey, Mark Luetkemeyer, Deanne Kashiwagi, Tammy Kemlage, Dustin Hertel and Adeel Zaidi for their enthusiasm and support. The authors also thank Ann Cottingham, Rich Frankel and Greg Sachs from the ASPIRE program for their guidance and vision. Dr. Weiner is Chief of Health Services Research and Development at the Richard L. Roudebush Veterans Affairs Medical Center in Indianapolis, Indiana.

Disclaimer

The views expressed in this article are those of the authors and do not necessarily represent the views of the U.S. Department of Veterans Affairs.

Healthcare delivery in rural America faces unique, growing challenges related to health and emergency care access.¹ Telemedicine approaches have the potential to increase rural hospitals’ ability to deliver efficient emergency care and reduce clinician shortages.² While initial evidence of telemedicine success exists, more quality research is needed to understand telemedicine patient and care team experiences,³ especially with real-time, clinician-initiated video conferencing in critical access hospital (CAH) emergency departments (ED). Some experience studies exist,⁴ but results are primarily quantitative⁵ and lack the nuanced qualitative depth needed to understand topics such as satisfaction and communication.⁶ Additionally, few explore combined patient and care team perspectives.⁵ The lack of breadth and depth makes it difficult to provide actionable recommendations for improvements and affects the feasibility of continuing this work and improving telemedicine care quality. To address these gaps, we evaluated a real-time, clinician-­initiated video conferencing program with overnight clinicians servicing ED patients in three Midwestern care system CAHs. This evaluation assessed patient and care team (nurse and clinician) experience with telemedicine using quantitative and qualitative survey data analysis.

Because this evaluation was designed to measure and improve program quality in a single healthcare system, it was deemed non-human subjects research by the organization’s institutional review board. This brief report follows telemedicine reporting guidelines.⁷

Setting and Telemedicine Program

This program, designed to reduce the need for on-call hospitalist clinicians to be onsite at CAHs overnight, was implemented in a large Midwestern nonprofit integrated healthcare system with three rural CAHs (combined capacity for 75 inpatient admissions, with full-time onsite ED clinicians and nurses, as well as on-call hospitalist clinicians) and a large metropolitan tertiary-care hospital. All adult patients presenting to CAH EDs between 6 pm and 8 am were evaluated, as usual, by an onsite ED clinician. If the admitting ED clinician and charge nurse determined that admission was appropriate, patients were signed out to remote hospitalist clinicians and roomed by onsite nurses. Nurses facilitated live audio-video telemedicine “history and physical” visits with remote clinicians via telemedicine carts (AmericanWell C750, Boston, Massachusetts, and ThinkLabs One Electronic Stethoscope, Centennial, Colorado). Already-­hospitalized patients, as well as patients admitted to a remote clinician, were cared for by the remote clinician and onsite nurse for the remainder of the night, which eliminated the need for local on-call clinicians. The onsite ED clinician responded to emergencies of already-hospitalized patients, but often consulted with remote clinicians to assist virtually with necessary orders and documentation. Remote clinicians were located at the metropolitan tertiary care hospital or home work stations.

Following a pilot period, the full-scale program was implemented in September 2017 and included 14 remote clinicians and 60 onsite nurses.

Survey Administration and Design

A postimplementation survey was designed to explore patient and care team experience with telemedicine. Patients who received a telemedicine visit between September 2017 and April 2018 were mailed a paper survey. Nonresponders were called by professional interviewers affiliated with the healthcare system. All participating clinicians (N = 14, all MDs) and nurses (N = 60, all RNs) were emailed an online care team survey with phone-in option. Care team nonresponders were sent up to two reminder emails.

Surveys captured the following five constructs: communication, workflow integration, telemedicine technology, quality of care, and general satisfaction. Existing questionnaires were used where possible; additional items were designed with clinical experts following survey design best practices.⁸ Patient-perceived communication was assessed via three Consumer Assessment of Healthcare Providers and Systems Outpatient and Ambulatory Surgery Survey items.⁹ Five additional program-developed patient survey items included satisfaction with clinician-nurse communication, satisfaction with technology, telemedicine quality of care overall and in comparison with traditional care, and whether or not patients would recommend telemedicine (Table). Four open-ended questions asked patients about improvement opportunities and general satisfaction.

Care team surveys included two items regarding ability to effectively communicate, two about satisfaction with workflow integration, one about technical problems, two about quality of care, and one about general satisfaction. Open-ended questions gathered further information and recommendations to improve communication, workflow integration, technology issues, and general satisfaction.

Analysis

Closed-ended items were dichotomized (satisfied yes/no); descriptive statistics (frequencies/percents) are presented to quantify patient and care team experience. Quantitative analyses were conducted in SAS software version 9.4 (SAS Institute, Cary, North Carolina). Open-ended responses were coded separately for patient and care team experience, following qualitative content analysis best practices.¹⁰ A lead coder read all responses, created a coding framework of identified themes, and coded individual responses. A second coder independently coded responses using the same framework. Interrater reliability was calculated for each major theme using percent agreement and prevalence- and bias-adjusted k (PABAK) statistic. A single representative quote was selected and lightly edited for each subtheme to deepen understanding and provide respondent voice.¹¹

Of eligible patients mailed a survey (N = 408), 3% self-reported as ineligible, and 54% completed the survey. This is a maximum response rate (response rate 6) according to the American Association for Public Opinion Research.¹² Patients were 67 years old on average (SD = 15), they were primarily white (97%), and 54% were female. All clinicians and 63% of nurses completed the survey.¹² Clinicians and nurses were 29% and 95% female, respectively.

Quantitative results (Table) show generally positive experience across patient and care team respondents. Over 90% were satisfied with all measures of communication. Care teams had high satisfaction with admissions processes and reported telemedicine improved cross-coverage. Patient-reported technology experience was positive but was less positive from the care team perspective. Care teams reported lower absolute quality of care than did patients but were more likely to perceive telemedicine as high quality, compared with traditional care. Most patients, clinicians, and nurses would recommend telemedicine.

Qualitatively, four major themes were identified in open-­ended responses with high interrater reliability (PABAK ranging from 0.92 to 0.98 in patient responses and 0.88 to 0.95 in care team responses) and aligned with the quantitative survey constructs: clinician-nurse communication, clinician-patient communication, workflow integration, and telemedicine technology. Patients reported satisfaction with communication with remote clinicians:

“[The clinician] was extremely attentive to me and what was going on. She was articulate and clear. I understood what was going to happen.” –Patient

Care teams suggested concrete improvement opportunities:

“I’d prefer to have some time with nursing staff both before and (sometimes) after the patient encounter.” –Clinician

“Since we cannot hear what [the clinicians] are hearing with the stethoscope, it’s nice when they tell us when to move it to the next spot.” –Nurse

Clinicians and nurses gave favorable responses regarding workflow integration, though time (both admissions wait time and session duration) was a reported opportunity:

“It would be helpful if we could speed up the time from admit request to screen time.” –Clinician

“When the [clinicians] get swamped, they’re hard to get a hold of, and admissions can take a long time. They may have too much on their plates dealing with several locations.” –Nurse

Technology issues—internet connection, stethoscope, sound, and screen or camera—were mentioned by patients and care teams, though technology was reviewed favorably overall by most patients:

“I was fascinated by the technology. Visiting someone over a television was impressive. ... The picture, the sound clarity, and the connection itself was flawless.” –Patient

Some patients commented that telemedicine was the best option given the situation, but still preferred an in-person doctor:

“If a doctor wasn’t available, telemedicine is better than nothing.” –Patient

Nurses who would not recommend telemedicine noted the need for personal connection:

“[I] still prefer [an] in-person MD for more personal contact. The older patients often state they wish the doctor would come and see them.” –Nurse

Patients who would not recommend telemedicine also desired personal connection:

“I would sooner talk to a person than a machine.” –Patient

A few clinicians noted the connection with patients would be improved if they knew about others in the room:

“It’d be nice if everyone in the room was introduced. Sometimes people are sitting out of view of the camera and I don’t realize they’re there until later.” –Clinician

These results make important contributions to understanding and improving the telemedicine experience in rural emergency hospital medicine. While the predominantly white patient respondent population limits generalizability, these demographics are representative of the overall population of the participating hospitals. A strength of this evaluation is its contemporaneous consideration of patient and care team experience with both quantitative and rich, qualitative analysis. Patients and care teams alike thought overnight telemedicine was better than the status quo. While our quality of care findings align with some previous literature,¹³ care teams in the current analysis overwhelmingly would recommend telemedicine, whereas some clinicians in prior work would not recommend telemedicine.¹⁴

In terms of communication, in line with existing literature, some patients still preferred in-person visits,¹⁵ a view also shared by some care team members. Workflow and technology barriers were raised, corroborating existing work,¹³ but actionable solutions (eg, adding care team–only time before visits or verbalizing when to move stethoscopes) were also identified.

Embedding patient and care team experience surveys and sharing results is critical in advancing telemedicine. Findings from this evaluation strengthen the case for payer reimbursement of telemedicine in rural acute care. Continued work to improve, test, and publish findings on patient and care team experience with telemedicine is critical to providing quality services in often-underserved communities.

The authors would like to acknowledge the contributions of Ann Werner in identifying the patient survey sample, Brian Barklind in identifying source data for the analysis, and both Brian Barklind and Rachael Rivard for conducting the analyses and summarizing results. We would also like to thank Kelly Logue for her involvement in conceptualizing the telemedicine evaluation described here, as well as Larisa Polynskaya for her help preparing the manuscript for publication, and the care teams and patients who provided valuable input.

Healthcare delivery in rural America faces unique, growing challenges related to health and emergency care access.¹ Telemedicine approaches have the potential to increase rural hospitals’ ability to deliver efficient emergency care and reduce clinician shortages.² While initial evidence of telemedicine success exists, more quality research is needed to understand telemedicine patient and care team experiences,³ especially with real-time, clinician-initiated video conferencing in critical access hospital (CAH) emergency departments (ED). Some experience studies exist,⁴ but results are primarily quantitative⁵ and lack the nuanced qualitative depth needed to understand topics such as satisfaction and communication.⁶ Additionally, few explore combined patient and care team perspectives.⁵ The lack of breadth and depth makes it difficult to provide actionable recommendations for improvements and affects the feasibility of continuing this work and improving telemedicine care quality. To address these gaps, we evaluated a real-time, clinician-­initiated video conferencing program with overnight clinicians servicing ED patients in three Midwestern care system CAHs. This evaluation assessed patient and care team (nurse and clinician) experience with telemedicine using quantitative and qualitative survey data analysis.

Because this evaluation was designed to measure and improve program quality in a single healthcare system, it was deemed non-human subjects research by the organization’s institutional review board. This brief report follows telemedicine reporting guidelines.⁷

Setting and Telemedicine Program

This program, designed to reduce the need for on-call hospitalist clinicians to be onsite at CAHs overnight, was implemented in a large Midwestern nonprofit integrated healthcare system with three rural CAHs (combined capacity for 75 inpatient admissions, with full-time onsite ED clinicians and nurses, as well as on-call hospitalist clinicians) and a large metropolitan tertiary-care hospital. All adult patients presenting to CAH EDs between 6 pm and 8 am were evaluated, as usual, by an onsite ED clinician. If the admitting ED clinician and charge nurse determined that admission was appropriate, patients were signed out to remote hospitalist clinicians and roomed by onsite nurses. Nurses facilitated live audio-video telemedicine “history and physical” visits with remote clinicians via telemedicine carts (AmericanWell C750, Boston, Massachusetts, and ThinkLabs One Electronic Stethoscope, Centennial, Colorado). Already-­hospitalized patients, as well as patients admitted to a remote clinician, were cared for by the remote clinician and onsite nurse for the remainder of the night, which eliminated the need for local on-call clinicians. The onsite ED clinician responded to emergencies of already-hospitalized patients, but often consulted with remote clinicians to assist virtually with necessary orders and documentation. Remote clinicians were located at the metropolitan tertiary care hospital or home work stations.

Following a pilot period, the full-scale program was implemented in September 2017 and included 14 remote clinicians and 60 onsite nurses.

Survey Administration and Design

A postimplementation survey was designed to explore patient and care team experience with telemedicine. Patients who received a telemedicine visit between September 2017 and April 2018 were mailed a paper survey. Nonresponders were called by professional interviewers affiliated with the healthcare system. All participating clinicians (N = 14, all MDs) and nurses (N = 60, all RNs) were emailed an online care team survey with phone-in option. Care team nonresponders were sent up to two reminder emails.

Surveys captured the following five constructs: communication, workflow integration, telemedicine technology, quality of care, and general satisfaction. Existing questionnaires were used where possible; additional items were designed with clinical experts following survey design best practices.⁸ Patient-perceived communication was assessed via three Consumer Assessment of Healthcare Providers and Systems Outpatient and Ambulatory Surgery Survey items.⁹ Five additional program-developed patient survey items included satisfaction with clinician-nurse communication, satisfaction with technology, telemedicine quality of care overall and in comparison with traditional care, and whether or not patients would recommend telemedicine (Table). Four open-ended questions asked patients about improvement opportunities and general satisfaction.

Care team surveys included two items regarding ability to effectively communicate, two about satisfaction with workflow integration, one about technical problems, two about quality of care, and one about general satisfaction. Open-ended questions gathered further information and recommendations to improve communication, workflow integration, technology issues, and general satisfaction.

Analysis

Closed-ended items were dichotomized (satisfied yes/no); descriptive statistics (frequencies/percents) are presented to quantify patient and care team experience. Quantitative analyses were conducted in SAS software version 9.4 (SAS Institute, Cary, North Carolina). Open-ended responses were coded separately for patient and care team experience, following qualitative content analysis best practices.¹⁰ A lead coder read all responses, created a coding framework of identified themes, and coded individual responses. A second coder independently coded responses using the same framework. Interrater reliability was calculated for each major theme using percent agreement and prevalence- and bias-adjusted k (PABAK) statistic. A single representative quote was selected and lightly edited for each subtheme to deepen understanding and provide respondent voice.¹¹

Of eligible patients mailed a survey (N = 408), 3% self-reported as ineligible, and 54% completed the survey. This is a maximum response rate (response rate 6) according to the American Association for Public Opinion Research.¹² Patients were 67 years old on average (SD = 15), they were primarily white (97%), and 54% were female. All clinicians and 63% of nurses completed the survey.¹² Clinicians and nurses were 29% and 95% female, respectively.

Quantitative results (Table) show generally positive experience across patient and care team respondents. Over 90% were satisfied with all measures of communication. Care teams had high satisfaction with admissions processes and reported telemedicine improved cross-coverage. Patient-reported technology experience was positive but was less positive from the care team perspective. Care teams reported lower absolute quality of care than did patients but were more likely to perceive telemedicine as high quality, compared with traditional care. Most patients, clinicians, and nurses would recommend telemedicine.

Qualitatively, four major themes were identified in open-­ended responses with high interrater reliability (PABAK ranging from 0.92 to 0.98 in patient responses and 0.88 to 0.95 in care team responses) and aligned with the quantitative survey constructs: clinician-nurse communication, clinician-patient communication, workflow integration, and telemedicine technology. Patients reported satisfaction with communication with remote clinicians:

“[The clinician] was extremely attentive to me and what was going on. She was articulate and clear. I understood what was going to happen.” –Patient

Care teams suggested concrete improvement opportunities:

“I’d prefer to have some time with nursing staff both before and (sometimes) after the patient encounter.” –Clinician

“Since we cannot hear what [the clinicians] are hearing with the stethoscope, it’s nice when they tell us when to move it to the next spot.” –Nurse

Clinicians and nurses gave favorable responses regarding workflow integration, though time (both admissions wait time and session duration) was a reported opportunity:

“It would be helpful if we could speed up the time from admit request to screen time.” –Clinician

“When the [clinicians] get swamped, they’re hard to get a hold of, and admissions can take a long time. They may have too much on their plates dealing with several locations.” –Nurse

Technology issues—internet connection, stethoscope, sound, and screen or camera—were mentioned by patients and care teams, though technology was reviewed favorably overall by most patients:

“I was fascinated by the technology. Visiting someone over a television was impressive. ... The picture, the sound clarity, and the connection itself was flawless.” –Patient

Some patients commented that telemedicine was the best option given the situation, but still preferred an in-person doctor:

“If a doctor wasn’t available, telemedicine is better than nothing.” –Patient

Nurses who would not recommend telemedicine noted the need for personal connection:

“[I] still prefer [an] in-person MD for more personal contact. The older patients often state they wish the doctor would come and see them.” –Nurse

Patients who would not recommend telemedicine also desired personal connection:

“I would sooner talk to a person than a machine.” –Patient

A few clinicians noted the connection with patients would be improved if they knew about others in the room:

“It’d be nice if everyone in the room was introduced. Sometimes people are sitting out of view of the camera and I don’t realize they’re there until later.” –Clinician

These results make important contributions to understanding and improving the telemedicine experience in rural emergency hospital medicine. While the predominantly white patient respondent population limits generalizability, these demographics are representative of the overall population of the participating hospitals. A strength of this evaluation is its contemporaneous consideration of patient and care team experience with both quantitative and rich, qualitative analysis. Patients and care teams alike thought overnight telemedicine was better than the status quo. While our quality of care findings align with some previous literature,¹³ care teams in the current analysis overwhelmingly would recommend telemedicine, whereas some clinicians in prior work would not recommend telemedicine.¹⁴

In terms of communication, in line with existing literature, some patients still preferred in-person visits,¹⁵ a view also shared by some care team members. Workflow and technology barriers were raised, corroborating existing work,¹³ but actionable solutions (eg, adding care team–only time before visits or verbalizing when to move stethoscopes) were also identified.

Embedding patient and care team experience surveys and sharing results is critical in advancing telemedicine. Findings from this evaluation strengthen the case for payer reimbursement of telemedicine in rural acute care. Continued work to improve, test, and publish findings on patient and care team experience with telemedicine is critical to providing quality services in often-underserved communities.

The authors would like to acknowledge the contributions of Ann Werner in identifying the patient survey sample, Brian Barklind in identifying source data for the analysis, and both Brian Barklind and Rachael Rivard for conducting the analyses and summarizing results. We would also like to thank Kelly Logue for her involvement in conceptualizing the telemedicine evaluation described here, as well as Larisa Polynskaya for her help preparing the manuscript for publication, and the care teams and patients who provided valuable input.

Healthcare delivery in rural America faces unique, growing challenges related to health and emergency care access.¹ Telemedicine approaches have the potential to increase rural hospitals’ ability to deliver efficient emergency care and reduce clinician shortages.² While initial evidence of telemedicine success exists, more quality research is needed to understand telemedicine patient and care team experiences,³ especially with real-time, clinician-initiated video conferencing in critical access hospital (CAH) emergency departments (ED). Some experience studies exist,⁴ but results are primarily quantitative⁵ and lack the nuanced qualitative depth needed to understand topics such as satisfaction and communication.⁶ Additionally, few explore combined patient and care team perspectives.⁵ The lack of breadth and depth makes it difficult to provide actionable recommendations for improvements and affects the feasibility of continuing this work and improving telemedicine care quality. To address these gaps, we evaluated a real-time, clinician-­initiated video conferencing program with overnight clinicians servicing ED patients in three Midwestern care system CAHs. This evaluation assessed patient and care team (nurse and clinician) experience with telemedicine using quantitative and qualitative survey data analysis.

Because this evaluation was designed to measure and improve program quality in a single healthcare system, it was deemed non-human subjects research by the organization’s institutional review board. This brief report follows telemedicine reporting guidelines.⁷

Setting and Telemedicine Program

This program, designed to reduce the need for on-call hospitalist clinicians to be onsite at CAHs overnight, was implemented in a large Midwestern nonprofit integrated healthcare system with three rural CAHs (combined capacity for 75 inpatient admissions, with full-time onsite ED clinicians and nurses, as well as on-call hospitalist clinicians) and a large metropolitan tertiary-care hospital. All adult patients presenting to CAH EDs between 6 pm and 8 am were evaluated, as usual, by an onsite ED clinician. If the admitting ED clinician and charge nurse determined that admission was appropriate, patients were signed out to remote hospitalist clinicians and roomed by onsite nurses. Nurses facilitated live audio-video telemedicine “history and physical” visits with remote clinicians via telemedicine carts (AmericanWell C750, Boston, Massachusetts, and ThinkLabs One Electronic Stethoscope, Centennial, Colorado). Already-­hospitalized patients, as well as patients admitted to a remote clinician, were cared for by the remote clinician and onsite nurse for the remainder of the night, which eliminated the need for local on-call clinicians. The onsite ED clinician responded to emergencies of already-hospitalized patients, but often consulted with remote clinicians to assist virtually with necessary orders and documentation. Remote clinicians were located at the metropolitan tertiary care hospital or home work stations.

Following a pilot period, the full-scale program was implemented in September 2017 and included 14 remote clinicians and 60 onsite nurses.

Survey Administration and Design

A postimplementation survey was designed to explore patient and care team experience with telemedicine. Patients who received a telemedicine visit between September 2017 and April 2018 were mailed a paper survey. Nonresponders were called by professional interviewers affiliated with the healthcare system. All participating clinicians (N = 14, all MDs) and nurses (N = 60, all RNs) were emailed an online care team survey with phone-in option. Care team nonresponders were sent up to two reminder emails.

Surveys captured the following five constructs: communication, workflow integration, telemedicine technology, quality of care, and general satisfaction. Existing questionnaires were used where possible; additional items were designed with clinical experts following survey design best practices.⁸ Patient-perceived communication was assessed via three Consumer Assessment of Healthcare Providers and Systems Outpatient and Ambulatory Surgery Survey items.⁹ Five additional program-developed patient survey items included satisfaction with clinician-nurse communication, satisfaction with technology, telemedicine quality of care overall and in comparison with traditional care, and whether or not patients would recommend telemedicine (Table). Four open-ended questions asked patients about improvement opportunities and general satisfaction.

Care team surveys included two items regarding ability to effectively communicate, two about satisfaction with workflow integration, one about technical problems, two about quality of care, and one about general satisfaction. Open-ended questions gathered further information and recommendations to improve communication, workflow integration, technology issues, and general satisfaction.

Analysis

Closed-ended items were dichotomized (satisfied yes/no); descriptive statistics (frequencies/percents) are presented to quantify patient and care team experience. Quantitative analyses were conducted in SAS software version 9.4 (SAS Institute, Cary, North Carolina). Open-ended responses were coded separately for patient and care team experience, following qualitative content analysis best practices.¹⁰ A lead coder read all responses, created a coding framework of identified themes, and coded individual responses. A second coder independently coded responses using the same framework. Interrater reliability was calculated for each major theme using percent agreement and prevalence- and bias-adjusted k (PABAK) statistic. A single representative quote was selected and lightly edited for each subtheme to deepen understanding and provide respondent voice.¹¹

Of eligible patients mailed a survey (N = 408), 3% self-reported as ineligible, and 54% completed the survey. This is a maximum response rate (response rate 6) according to the American Association for Public Opinion Research.¹² Patients were 67 years old on average (SD = 15), they were primarily white (97%), and 54% were female. All clinicians and 63% of nurses completed the survey.¹² Clinicians and nurses were 29% and 95% female, respectively.

Quantitative results (Table) show generally positive experience across patient and care team respondents. Over 90% were satisfied with all measures of communication. Care teams had high satisfaction with admissions processes and reported telemedicine improved cross-coverage. Patient-reported technology experience was positive but was less positive from the care team perspective. Care teams reported lower absolute quality of care than did patients but were more likely to perceive telemedicine as high quality, compared with traditional care. Most patients, clinicians, and nurses would recommend telemedicine.

Qualitatively, four major themes were identified in open-­ended responses with high interrater reliability (PABAK ranging from 0.92 to 0.98 in patient responses and 0.88 to 0.95 in care team responses) and aligned with the quantitative survey constructs: clinician-nurse communication, clinician-patient communication, workflow integration, and telemedicine technology. Patients reported satisfaction with communication with remote clinicians:

“[The clinician] was extremely attentive to me and what was going on. She was articulate and clear. I understood what was going to happen.” –Patient

Care teams suggested concrete improvement opportunities:

“I’d prefer to have some time with nursing staff both before and (sometimes) after the patient encounter.” –Clinician

“Since we cannot hear what [the clinicians] are hearing with the stethoscope, it’s nice when they tell us when to move it to the next spot.” –Nurse

Clinicians and nurses gave favorable responses regarding workflow integration, though time (both admissions wait time and session duration) was a reported opportunity:

“It would be helpful if we could speed up the time from admit request to screen time.” –Clinician

“When the [clinicians] get swamped, they’re hard to get a hold of, and admissions can take a long time. They may have too much on their plates dealing with several locations.” –Nurse

Technology issues—internet connection, stethoscope, sound, and screen or camera—were mentioned by patients and care teams, though technology was reviewed favorably overall by most patients:

“I was fascinated by the technology. Visiting someone over a television was impressive. ... The picture, the sound clarity, and the connection itself was flawless.” –Patient

Some patients commented that telemedicine was the best option given the situation, but still preferred an in-person doctor:

“If a doctor wasn’t available, telemedicine is better than nothing.” –Patient

Nurses who would not recommend telemedicine noted the need for personal connection:

“[I] still prefer [an] in-person MD for more personal contact. The older patients often state they wish the doctor would come and see them.” –Nurse

Patients who would not recommend telemedicine also desired personal connection:

“I would sooner talk to a person than a machine.” –Patient

A few clinicians noted the connection with patients would be improved if they knew about others in the room:

“It’d be nice if everyone in the room was introduced. Sometimes people are sitting out of view of the camera and I don’t realize they’re there until later.” –Clinician

These results make important contributions to understanding and improving the telemedicine experience in rural emergency hospital medicine. While the predominantly white patient respondent population limits generalizability, these demographics are representative of the overall population of the participating hospitals. A strength of this evaluation is its contemporaneous consideration of patient and care team experience with both quantitative and rich, qualitative analysis. Patients and care teams alike thought overnight telemedicine was better than the status quo. While our quality of care findings align with some previous literature,¹³ care teams in the current analysis overwhelmingly would recommend telemedicine, whereas some clinicians in prior work would not recommend telemedicine.¹⁴

In terms of communication, in line with existing literature, some patients still preferred in-person visits,¹⁵ a view also shared by some care team members. Workflow and technology barriers were raised, corroborating existing work,¹³ but actionable solutions (eg, adding care team–only time before visits or verbalizing when to move stethoscopes) were also identified.

Embedding patient and care team experience surveys and sharing results is critical in advancing telemedicine. Findings from this evaluation strengthen the case for payer reimbursement of telemedicine in rural acute care. Continued work to improve, test, and publish findings on patient and care team experience with telemedicine is critical to providing quality services in often-underserved communities.

The authors would like to acknowledge the contributions of Ann Werner in identifying the patient survey sample, Brian Barklind in identifying source data for the analysis, and both Brian Barklind and Rachael Rivard for conducting the analyses and summarizing results. We would also like to thank Kelly Logue for her involvement in conceptualizing the telemedicine evaluation described here, as well as Larisa Polynskaya for her help preparing the manuscript for publication, and the care teams and patients who provided valuable input.

Sleep disturbance is common in hospitals, and both the quality and quantity of sleep are negatively affected in hospitalized patients.¹ Sleep disturbances in hospitals are associated with hyperglycemia,² delirium,³ lower patient satisfaction,⁴ and increased risk of readmission.⁵

A significant proportion of hospitalized patients receive sleep medications (ie, hypnotic medication) despite limited evidence.^6,7 Sleep medications have adverse effects including falls, fractures, cognitive impairment, and delirium.⁸ Commonly used nonbenzodiazepine sleep medications (eg, zopiclone) are perceived as safer but may have similar risks.⁹

Melatonin is increasingly used to treat insomnia, although evidence for its efficacy and safety is lacking.¹⁰ While melatonin use doubled between 2007 and 2012 in the United States,¹¹ previous hospital-based studies have not included melatonin.⁷ It is not known if increased melatonin use in the hospital mitigates use of higher-risk medications. Melatonin preparations can also have quality issues, including deviations from labelled dosage and contamination with compounds such as serotonin,¹² and patients continuing melatonin after discharge could have adverse effects if switched to different preparations.

In this study, we aimed to determine temporal trends in melatonin use in hospitalized patients, and compare them with trends in use of other sleep medications.

We conducted the study at two urban academic hospitals with a total of 706 acute care beds in the same network in Toronto, Canada. This study was approved by the University Health Network’s research ethics board.

We abstracted pharmacy dispensing data on melatonin, zopiclone, and lorazepam from January 1, 2013, to December 31, 2018. We included oral medications dispensed to inpatient units or admitted patients in the emergency department (ED). Zopiclone is the most commonly used nonbenzodiazepine sleep medication in Canada.¹³ Lorazepam is the most commonly used benzodiazepine at our institution (97% of all oral benzodiazepine doses). While lorazepam is prescribed for many nonsleep indications, we included it to assess the impact of melatonin. We did not include antipsychotics or trazodone, which are rarely newly initiated for insomnia at our institution.

We abstracted the monthly number of doses dispensed by unit and hospital. We categorized units based on the primary patient population as either internal medicine, critical care, or other. Admitted patients in the ED were counted as “other” regardless of service. As the focus of our study was on internal medicine and critical care, we did not analyze by type of unit in the “other” group, which is heterogeneous.

Each medication-dispensing event was counted as one dose, regardless of the number or strength of tablets (eg, a patient dispensed two 3-mg tablets of melatonin, for a total of 6 mg, would be counted as a single dose). Most unused doses are credited back (ie, if a medication was refused and returned to pharmacy, it was not counted). Lorazepam and zopiclone are on hospital formulary, while melatonin is not. To order melatonin, clinicians must select “Nonformulary medication” in the electronic health record and manually enter medication name, dose, route, and frequency, as well as select a justification for use. The hospital supplies nonformulary medications such as melatonin to patients.

To account for changes in patient volumes, we standardized medication dispensing rates per 1,000 inpatient days. We discovered rare instances in which the monthly number of doses was a negative number because of pharmacy inventory accounting. This issue affected only 0.13% of observations and the magnitude was small (8 doses or fewer); in these cases, we assumed the number of doses was zero.

We used line charts to visualize changes in medication dispensing over time by medication and hospital. We compared rates of medications use between unit type and hospital with use of relative difference and rate difference. Statistical analysis was performed with R (The R Foundation for Statistical Computing, 2018) using lubridate (2011), dplyr (2018), ggplot2 (2016), fmsb (2019), and forcats (2018).

A total of 1,542,225 inpatient days were analyzed, of which 60.4% were at hospital A. Internal medicine accounted for 23.5% of inpatient days, critical care for 11.7%, and other units for 64.8%.

Overall Trends in Sleep Medication Use

There were 351,131 dispensed doses of study medications (13% melatonin, 43% lorazepam, and 44% zopiclone). Overall use of the three study medications per 1,000 inpatient days increased by 25.7% during the study.

Melatonin use increased by 71.3 doses per 1,000 inpatient days during 2013-2018, while zopiclone use decreased by 20.4 doses per 1,000 inpatient days (Table). Lorazepam use increased slightly by 3.5 doses per 1,000 inpatient days. All rate differences reported in the results are statistically significant (Appendix Table).

Unit Type Comparison

Melatonin use was highest in critical care and internal medicine (50.9 and 48.4 doses per 1,000 inpatient days, respectively), compared with that in other units (19.3 doses per 1,000 inpatient days). Among critical care units, melatonin use was highest in medical-surgical units (67.4 doses per 1,000 inpatient days) and lower in cardiac and cardiovascular surgery units (24.6 and 18.3 doses per 1,000 inpatient days respectively). Zopiclone use was highest in critical care and other units (117.9 and 112.2 doses per 1,000 inpatient days, respectively) and lowest in internal medicine (57.0 doses per 1,000 inpatient days).

Hospital Site Comparison

Overall melatonin use was 65.4% higher at hospital B than at hospital A (42.4 vs 21.5 doses per 1,000 inpatient days; Figure). Zopiclone use was 81.7% lower at hospital B (54.6 vs 130.0 doses per 1,000 inpatient days).

When similar units were compared between hospitals, the trends were similar. For example, among internal medicine units, melatonin use was 66.7% higher at hospital B than at hospital A (64.4 vs 32.3 doses per 1,000 inpatient days).

During this 6-year study period of sleep medication use at two academic hospitals, overall use of melatonin, zopiclone, and lorazepam increased by 25.7%. Melatonin increased from almost no use to more than 70 doses per 1,000 inpatient days. The increase in melatonin was not accompanied by a proportional decline in zopiclone, which only decreased by 20.4 doses per 1,000 inpatient days. Lorazepam use increased slightly. This suggests that melatonin is not simply being substituted for higher-risk sleep medications and is instead being given to patients who might not have received sleep medications otherwise.

There are a few potential explanations for the disproportionate increase in melatonin. Providers may be more liberal in prescribing melatonin for insomnia because of perceived greater safety, compared with other medications. Melatonin may also be prescribed for delirium, despite a lack of high-quality evidence.¹⁴ Interestingly, melatonin use has increased despite a paucity of evidence for its efficacy or safety in hospital.⁶ Considering the additional barriers that exist to ordering melatonin, a nonformulary medication at our institution, the magnitude of increase is even more striking.

Melatonin use was highest on internal medicine and critical care units. This may reflect patient differences (eg, older patients with more comorbid conditions might leave prescribers reluctant to use benzodiazepines), differences in the physical environment (eg, noise/lighting), differences in nursing practices (eg, intensity of monitoring or medication administration), or differences in prescribing.

Melatonin use was almost twice as high at hospital B as it was at hospital A. While the services differ at each hospital, the results were similar when comparing the same unit type (eg, internal medicine). Internal medicine units have similar (though not identical) patient populations and team structures at both hospitals, and residents rotate between hospitals. Attendings and nurses are based primarily at one hospital and their practice patterns might differ. Geriatricians have a stronger presence at hospital B. Higher zopiclone use at hospital A could explain lower melatonin use. Lastly, improvement initiatives may have contributed (eg, one unit at hospital B promoted melatonin in 2017).

Our study has potential limitations. We studied dispensed rather than administered medications; however, numbers of doses dispensed but not administered are expected to be low because most unused doses are accounted for. By studying dispensing data, we might have underestimated the number of prescriptions (eg, if a patient was prescribed but refused a medication, this would not be captured). Our study did not examine medications after hospital discharge, although medications started in a hospital are often continued at discharge.^7,15 Our study could not determine indications for medication prescribing, and melatonin and lorazepam are both used for nonsleep indications. Our study could not differentiate between continuation of home medications and new prescriptions. Finally, the results may not be generalizable to other settings.

In this 6-year study of sleep medication use at two academic hospitals, we found that overall use of melatonin, zopiclone, and lorazepam increased by 25%, predominantly because of markedly increased melatonin use. Given the current lack of high-quality evidence, further research on the use of melatonin in hospitalized patients is needed.

Sleep disturbance is common in hospitals, and both the quality and quantity of sleep are negatively affected in hospitalized patients.¹ Sleep disturbances in hospitals are associated with hyperglycemia,² delirium,³ lower patient satisfaction,⁴ and increased risk of readmission.⁵

A significant proportion of hospitalized patients receive sleep medications (ie, hypnotic medication) despite limited evidence.^6,7 Sleep medications have adverse effects including falls, fractures, cognitive impairment, and delirium.⁸ Commonly used nonbenzodiazepine sleep medications (eg, zopiclone) are perceived as safer but may have similar risks.⁹

Melatonin is increasingly used to treat insomnia, although evidence for its efficacy and safety is lacking.¹⁰ While melatonin use doubled between 2007 and 2012 in the United States,¹¹ previous hospital-based studies have not included melatonin.⁷ It is not known if increased melatonin use in the hospital mitigates use of higher-risk medications. Melatonin preparations can also have quality issues, including deviations from labelled dosage and contamination with compounds such as serotonin,¹² and patients continuing melatonin after discharge could have adverse effects if switched to different preparations.

In this study, we aimed to determine temporal trends in melatonin use in hospitalized patients, and compare them with trends in use of other sleep medications.

We conducted the study at two urban academic hospitals with a total of 706 acute care beds in the same network in Toronto, Canada. This study was approved by the University Health Network’s research ethics board.

We abstracted pharmacy dispensing data on melatonin, zopiclone, and lorazepam from January 1, 2013, to December 31, 2018. We included oral medications dispensed to inpatient units or admitted patients in the emergency department (ED). Zopiclone is the most commonly used nonbenzodiazepine sleep medication in Canada.¹³ Lorazepam is the most commonly used benzodiazepine at our institution (97% of all oral benzodiazepine doses). While lorazepam is prescribed for many nonsleep indications, we included it to assess the impact of melatonin. We did not include antipsychotics or trazodone, which are rarely newly initiated for insomnia at our institution.

We abstracted the monthly number of doses dispensed by unit and hospital. We categorized units based on the primary patient population as either internal medicine, critical care, or other. Admitted patients in the ED were counted as “other” regardless of service. As the focus of our study was on internal medicine and critical care, we did not analyze by type of unit in the “other” group, which is heterogeneous.

Each medication-dispensing event was counted as one dose, regardless of the number or strength of tablets (eg, a patient dispensed two 3-mg tablets of melatonin, for a total of 6 mg, would be counted as a single dose). Most unused doses are credited back (ie, if a medication was refused and returned to pharmacy, it was not counted). Lorazepam and zopiclone are on hospital formulary, while melatonin is not. To order melatonin, clinicians must select “Nonformulary medication” in the electronic health record and manually enter medication name, dose, route, and frequency, as well as select a justification for use. The hospital supplies nonformulary medications such as melatonin to patients.

To account for changes in patient volumes, we standardized medication dispensing rates per 1,000 inpatient days. We discovered rare instances in which the monthly number of doses was a negative number because of pharmacy inventory accounting. This issue affected only 0.13% of observations and the magnitude was small (8 doses or fewer); in these cases, we assumed the number of doses was zero.

We used line charts to visualize changes in medication dispensing over time by medication and hospital. We compared rates of medications use between unit type and hospital with use of relative difference and rate difference. Statistical analysis was performed with R (The R Foundation for Statistical Computing, 2018) using lubridate (2011), dplyr (2018), ggplot2 (2016), fmsb (2019), and forcats (2018).

A total of 1,542,225 inpatient days were analyzed, of which 60.4% were at hospital A. Internal medicine accounted for 23.5% of inpatient days, critical care for 11.7%, and other units for 64.8%.

Overall Trends in Sleep Medication Use

There were 351,131 dispensed doses of study medications (13% melatonin, 43% lorazepam, and 44% zopiclone). Overall use of the three study medications per 1,000 inpatient days increased by 25.7% during the study.

Melatonin use increased by 71.3 doses per 1,000 inpatient days during 2013-2018, while zopiclone use decreased by 20.4 doses per 1,000 inpatient days (Table). Lorazepam use increased slightly by 3.5 doses per 1,000 inpatient days. All rate differences reported in the results are statistically significant (Appendix Table).

Unit Type Comparison

Melatonin use was highest in critical care and internal medicine (50.9 and 48.4 doses per 1,000 inpatient days, respectively), compared with that in other units (19.3 doses per 1,000 inpatient days). Among critical care units, melatonin use was highest in medical-surgical units (67.4 doses per 1,000 inpatient days) and lower in cardiac and cardiovascular surgery units (24.6 and 18.3 doses per 1,000 inpatient days respectively). Zopiclone use was highest in critical care and other units (117.9 and 112.2 doses per 1,000 inpatient days, respectively) and lowest in internal medicine (57.0 doses per 1,000 inpatient days).

Hospital Site Comparison

Overall melatonin use was 65.4% higher at hospital B than at hospital A (42.4 vs 21.5 doses per 1,000 inpatient days; Figure). Zopiclone use was 81.7% lower at hospital B (54.6 vs 130.0 doses per 1,000 inpatient days).

When similar units were compared between hospitals, the trends were similar. For example, among internal medicine units, melatonin use was 66.7% higher at hospital B than at hospital A (64.4 vs 32.3 doses per 1,000 inpatient days).

During this 6-year study period of sleep medication use at two academic hospitals, overall use of melatonin, zopiclone, and lorazepam increased by 25.7%. Melatonin increased from almost no use to more than 70 doses per 1,000 inpatient days. The increase in melatonin was not accompanied by a proportional decline in zopiclone, which only decreased by 20.4 doses per 1,000 inpatient days. Lorazepam use increased slightly. This suggests that melatonin is not simply being substituted for higher-risk sleep medications and is instead being given to patients who might not have received sleep medications otherwise.

There are a few potential explanations for the disproportionate increase in melatonin. Providers may be more liberal in prescribing melatonin for insomnia because of perceived greater safety, compared with other medications. Melatonin may also be prescribed for delirium, despite a lack of high-quality evidence.¹⁴ Interestingly, melatonin use has increased despite a paucity of evidence for its efficacy or safety in hospital.⁶ Considering the additional barriers that exist to ordering melatonin, a nonformulary medication at our institution, the magnitude of increase is even more striking.

Melatonin use was highest on internal medicine and critical care units. This may reflect patient differences (eg, older patients with more comorbid conditions might leave prescribers reluctant to use benzodiazepines), differences in the physical environment (eg, noise/lighting), differences in nursing practices (eg, intensity of monitoring or medication administration), or differences in prescribing.

Melatonin use was almost twice as high at hospital B as it was at hospital A. While the services differ at each hospital, the results were similar when comparing the same unit type (eg, internal medicine). Internal medicine units have similar (though not identical) patient populations and team structures at both hospitals, and residents rotate between hospitals. Attendings and nurses are based primarily at one hospital and their practice patterns might differ. Geriatricians have a stronger presence at hospital B. Higher zopiclone use at hospital A could explain lower melatonin use. Lastly, improvement initiatives may have contributed (eg, one unit at hospital B promoted melatonin in 2017).

Our study has potential limitations. We studied dispensed rather than administered medications; however, numbers of doses dispensed but not administered are expected to be low because most unused doses are accounted for. By studying dispensing data, we might have underestimated the number of prescriptions (eg, if a patient was prescribed but refused a medication, this would not be captured). Our study did not examine medications after hospital discharge, although medications started in a hospital are often continued at discharge.^7,15 Our study could not determine indications for medication prescribing, and melatonin and lorazepam are both used for nonsleep indications. Our study could not differentiate between continuation of home medications and new prescriptions. Finally, the results may not be generalizable to other settings.

In this 6-year study of sleep medication use at two academic hospitals, we found that overall use of melatonin, zopiclone, and lorazepam increased by 25%, predominantly because of markedly increased melatonin use. Given the current lack of high-quality evidence, further research on the use of melatonin in hospitalized patients is needed.

Sleep disturbance is common in hospitals, and both the quality and quantity of sleep are negatively affected in hospitalized patients.¹ Sleep disturbances in hospitals are associated with hyperglycemia,² delirium,³ lower patient satisfaction,⁴ and increased risk of readmission.⁵

A significant proportion of hospitalized patients receive sleep medications (ie, hypnotic medication) despite limited evidence.^6,7 Sleep medications have adverse effects including falls, fractures, cognitive impairment, and delirium.⁸ Commonly used nonbenzodiazepine sleep medications (eg, zopiclone) are perceived as safer but may have similar risks.⁹

Melatonin is increasingly used to treat insomnia, although evidence for its efficacy and safety is lacking.¹⁰ While melatonin use doubled between 2007 and 2012 in the United States,¹¹ previous hospital-based studies have not included melatonin.⁷ It is not known if increased melatonin use in the hospital mitigates use of higher-risk medications. Melatonin preparations can also have quality issues, including deviations from labelled dosage and contamination with compounds such as serotonin,¹² and patients continuing melatonin after discharge could have adverse effects if switched to different preparations.

In this study, we aimed to determine temporal trends in melatonin use in hospitalized patients, and compare them with trends in use of other sleep medications.

We conducted the study at two urban academic hospitals with a total of 706 acute care beds in the same network in Toronto, Canada. This study was approved by the University Health Network’s research ethics board.

We abstracted pharmacy dispensing data on melatonin, zopiclone, and lorazepam from January 1, 2013, to December 31, 2018. We included oral medications dispensed to inpatient units or admitted patients in the emergency department (ED). Zopiclone is the most commonly used nonbenzodiazepine sleep medication in Canada.¹³ Lorazepam is the most commonly used benzodiazepine at our institution (97% of all oral benzodiazepine doses). While lorazepam is prescribed for many nonsleep indications, we included it to assess the impact of melatonin. We did not include antipsychotics or trazodone, which are rarely newly initiated for insomnia at our institution.

We abstracted the monthly number of doses dispensed by unit and hospital. We categorized units based on the primary patient population as either internal medicine, critical care, or other. Admitted patients in the ED were counted as “other” regardless of service. As the focus of our study was on internal medicine and critical care, we did not analyze by type of unit in the “other” group, which is heterogeneous.

Each medication-dispensing event was counted as one dose, regardless of the number or strength of tablets (eg, a patient dispensed two 3-mg tablets of melatonin, for a total of 6 mg, would be counted as a single dose). Most unused doses are credited back (ie, if a medication was refused and returned to pharmacy, it was not counted). Lorazepam and zopiclone are on hospital formulary, while melatonin is not. To order melatonin, clinicians must select “Nonformulary medication” in the electronic health record and manually enter medication name, dose, route, and frequency, as well as select a justification for use. The hospital supplies nonformulary medications such as melatonin to patients.

To account for changes in patient volumes, we standardized medication dispensing rates per 1,000 inpatient days. We discovered rare instances in which the monthly number of doses was a negative number because of pharmacy inventory accounting. This issue affected only 0.13% of observations and the magnitude was small (8 doses or fewer); in these cases, we assumed the number of doses was zero.

We used line charts to visualize changes in medication dispensing over time by medication and hospital. We compared rates of medications use between unit type and hospital with use of relative difference and rate difference. Statistical analysis was performed with R (The R Foundation for Statistical Computing, 2018) using lubridate (2011), dplyr (2018), ggplot2 (2016), fmsb (2019), and forcats (2018).

A total of 1,542,225 inpatient days were analyzed, of which 60.4% were at hospital A. Internal medicine accounted for 23.5% of inpatient days, critical care for 11.7%, and other units for 64.8%.

Overall Trends in Sleep Medication Use

There were 351,131 dispensed doses of study medications (13% melatonin, 43% lorazepam, and 44% zopiclone). Overall use of the three study medications per 1,000 inpatient days increased by 25.7% during the study.

Melatonin use increased by 71.3 doses per 1,000 inpatient days during 2013-2018, while zopiclone use decreased by 20.4 doses per 1,000 inpatient days (Table). Lorazepam use increased slightly by 3.5 doses per 1,000 inpatient days. All rate differences reported in the results are statistically significant (Appendix Table).

Unit Type Comparison

Melatonin use was highest in critical care and internal medicine (50.9 and 48.4 doses per 1,000 inpatient days, respectively), compared with that in other units (19.3 doses per 1,000 inpatient days). Among critical care units, melatonin use was highest in medical-surgical units (67.4 doses per 1,000 inpatient days) and lower in cardiac and cardiovascular surgery units (24.6 and 18.3 doses per 1,000 inpatient days respectively). Zopiclone use was highest in critical care and other units (117.9 and 112.2 doses per 1,000 inpatient days, respectively) and lowest in internal medicine (57.0 doses per 1,000 inpatient days).

Hospital Site Comparison

Overall melatonin use was 65.4% higher at hospital B than at hospital A (42.4 vs 21.5 doses per 1,000 inpatient days; Figure). Zopiclone use was 81.7% lower at hospital B (54.6 vs 130.0 doses per 1,000 inpatient days).

When similar units were compared between hospitals, the trends were similar. For example, among internal medicine units, melatonin use was 66.7% higher at hospital B than at hospital A (64.4 vs 32.3 doses per 1,000 inpatient days).

During this 6-year study period of sleep medication use at two academic hospitals, overall use of melatonin, zopiclone, and lorazepam increased by 25.7%. Melatonin increased from almost no use to more than 70 doses per 1,000 inpatient days. The increase in melatonin was not accompanied by a proportional decline in zopiclone, which only decreased by 20.4 doses per 1,000 inpatient days. Lorazepam use increased slightly. This suggests that melatonin is not simply being substituted for higher-risk sleep medications and is instead being given to patients who might not have received sleep medications otherwise.

There are a few potential explanations for the disproportionate increase in melatonin. Providers may be more liberal in prescribing melatonin for insomnia because of perceived greater safety, compared with other medications. Melatonin may also be prescribed for delirium, despite a lack of high-quality evidence.¹⁴ Interestingly, melatonin use has increased despite a paucity of evidence for its efficacy or safety in hospital.⁶ Considering the additional barriers that exist to ordering melatonin, a nonformulary medication at our institution, the magnitude of increase is even more striking.

Melatonin use was highest on internal medicine and critical care units. This may reflect patient differences (eg, older patients with more comorbid conditions might leave prescribers reluctant to use benzodiazepines), differences in the physical environment (eg, noise/lighting), differences in nursing practices (eg, intensity of monitoring or medication administration), or differences in prescribing.

Melatonin use was almost twice as high at hospital B as it was at hospital A. While the services differ at each hospital, the results were similar when comparing the same unit type (eg, internal medicine). Internal medicine units have similar (though not identical) patient populations and team structures at both hospitals, and residents rotate between hospitals. Attendings and nurses are based primarily at one hospital and their practice patterns might differ. Geriatricians have a stronger presence at hospital B. Higher zopiclone use at hospital A could explain lower melatonin use. Lastly, improvement initiatives may have contributed (eg, one unit at hospital B promoted melatonin in 2017).

Our study has potential limitations. We studied dispensed rather than administered medications; however, numbers of doses dispensed but not administered are expected to be low because most unused doses are accounted for. By studying dispensing data, we might have underestimated the number of prescriptions (eg, if a patient was prescribed but refused a medication, this would not be captured). Our study did not examine medications after hospital discharge, although medications started in a hospital are often continued at discharge.^7,15 Our study could not determine indications for medication prescribing, and melatonin and lorazepam are both used for nonsleep indications. Our study could not differentiate between continuation of home medications and new prescriptions. Finally, the results may not be generalizable to other settings.

In this 6-year study of sleep medication use at two academic hospitals, we found that overall use of melatonin, zopiclone, and lorazepam increased by 25%, predominantly because of markedly increased melatonin use. Given the current lack of high-quality evidence, further research on the use of melatonin in hospitalized patients is needed.

“Knowing yourself is the beginning of all wisdom.”—Aristotle

On the wards, your white coat and stethoscope signal your role as a healthcare provider. These external symbols of your work represent your expertise, experience, and commitment to service, and your patients look to these signals for comfort and reassurance. But when you are running a meeting, managing projects, or leading people in your organization, how do others know what you have to offer? Although signaling your values, skills, and intentions is as important in leadership as it is in the clinical setting, few clinicians spend time reflecting on how best to do this. Crafting a strong, consistent personal leadership brand can help.

As described by Norm Smallwood and Dave Ulrich, a personal leadership brand is the external projection of your strengths and interests, which demonstrates how you create value for others.^1,2 In other words, a personal leadership brand helps constituents, stakeholders, and potential partners understand what you offer as a leader. Having a brand keeps you on track as a leader and helps get you noticed for future opportunities by helping you shape and meet expectations in a way that is deliberate, dynamic, and authentic.

Building your personal leadership brand is an exercise in reflection. Leaders should challenge themselves to answer the following questions:

• What do I have to offer, and what do others appreciate about me?
• What are my values?
• Where am I trying to go?
• How does my path align with organizational goals?

The answers to these simple questions can help you create your personal leadership brand. First, reflect on what you want to be known for, your values, and how you are currently perceived. Then, identify the results you are aiming to produce, aligning them with your strengths and organizational goals. Write these down, and share your reflections with trusted peers and your mentoring team. Shape your thoughts into a personal vision statement with a focus on what you put out into the world to help you stay true to yourself while producing the desired results. For example, a vision statement for a gifted communicator with a background in quality improvement may be: “I will use my strong communication skills to address complex problems impacting our hospital to reduce cost and improve quality with the goal of building a career as a health system leader.” Finally, be authentic, and share your personal brand in an articulate and succinct way to help others understand your place in the structure and narrative of an organization.

Your personal leadership brand should not be static; rather, it is a process that should iterate over time. Ask for direct feedback from trusted advisers and allies at regular intervals. Investigate whether your organization offers a formal structure, such as a “360 Evaluation,” to get perspective on how your unique strengths, skills, and goals are perceived. Then, explore and clarify discrepancies between where you think you are and how others see you. Approaching these conversations with humility will keep you aligned with your values, which makes it easier for others to be invested in your development.

A strong personal leadership brand is a force multiplier, providing clarity within teams and helping align a leader’s assets and values with organizational goals. It is a solid external signal of what others can expect from your work and will help you focus on your strengths while identifying areas for growth. A personal leadership brand is formed through reflection and, at its core, its authenticity. In the words of Paracelsus, a Renaissance physician, astrologer, and alchemist, “Be not another, if you can be yourself.”³

“Knowing yourself is the beginning of all wisdom.”—Aristotle

On the wards, your white coat and stethoscope signal your role as a healthcare provider. These external symbols of your work represent your expertise, experience, and commitment to service, and your patients look to these signals for comfort and reassurance. But when you are running a meeting, managing projects, or leading people in your organization, how do others know what you have to offer? Although signaling your values, skills, and intentions is as important in leadership as it is in the clinical setting, few clinicians spend time reflecting on how best to do this. Crafting a strong, consistent personal leadership brand can help.

As described by Norm Smallwood and Dave Ulrich, a personal leadership brand is the external projection of your strengths and interests, which demonstrates how you create value for others.^1,2 In other words, a personal leadership brand helps constituents, stakeholders, and potential partners understand what you offer as a leader. Having a brand keeps you on track as a leader and helps get you noticed for future opportunities by helping you shape and meet expectations in a way that is deliberate, dynamic, and authentic.

Building your personal leadership brand is an exercise in reflection. Leaders should challenge themselves to answer the following questions:

• What do I have to offer, and what do others appreciate about me?
• What are my values?
• Where am I trying to go?
• How does my path align with organizational goals?

The answers to these simple questions can help you create your personal leadership brand. First, reflect on what you want to be known for, your values, and how you are currently perceived. Then, identify the results you are aiming to produce, aligning them with your strengths and organizational goals. Write these down, and share your reflections with trusted peers and your mentoring team. Shape your thoughts into a personal vision statement with a focus on what you put out into the world to help you stay true to yourself while producing the desired results. For example, a vision statement for a gifted communicator with a background in quality improvement may be: “I will use my strong communication skills to address complex problems impacting our hospital to reduce cost and improve quality with the goal of building a career as a health system leader.” Finally, be authentic, and share your personal brand in an articulate and succinct way to help others understand your place in the structure and narrative of an organization.

Your personal leadership brand should not be static; rather, it is a process that should iterate over time. Ask for direct feedback from trusted advisers and allies at regular intervals. Investigate whether your organization offers a formal structure, such as a “360 Evaluation,” to get perspective on how your unique strengths, skills, and goals are perceived. Then, explore and clarify discrepancies between where you think you are and how others see you. Approaching these conversations with humility will keep you aligned with your values, which makes it easier for others to be invested in your development.

A strong personal leadership brand is a force multiplier, providing clarity within teams and helping align a leader’s assets and values with organizational goals. It is a solid external signal of what others can expect from your work and will help you focus on your strengths while identifying areas for growth. A personal leadership brand is formed through reflection and, at its core, its authenticity. In the words of Paracelsus, a Renaissance physician, astrologer, and alchemist, “Be not another, if you can be yourself.”³

“Knowing yourself is the beginning of all wisdom.”—Aristotle

On the wards, your white coat and stethoscope signal your role as a healthcare provider. These external symbols of your work represent your expertise, experience, and commitment to service, and your patients look to these signals for comfort and reassurance. But when you are running a meeting, managing projects, or leading people in your organization, how do others know what you have to offer? Although signaling your values, skills, and intentions is as important in leadership as it is in the clinical setting, few clinicians spend time reflecting on how best to do this. Crafting a strong, consistent personal leadership brand can help.

As described by Norm Smallwood and Dave Ulrich, a personal leadership brand is the external projection of your strengths and interests, which demonstrates how you create value for others.^1,2 In other words, a personal leadership brand helps constituents, stakeholders, and potential partners understand what you offer as a leader. Having a brand keeps you on track as a leader and helps get you noticed for future opportunities by helping you shape and meet expectations in a way that is deliberate, dynamic, and authentic.

Building your personal leadership brand is an exercise in reflection. Leaders should challenge themselves to answer the following questions:

• What do I have to offer, and what do others appreciate about me?
• What are my values?
• Where am I trying to go?
• How does my path align with organizational goals?

The answers to these simple questions can help you create your personal leadership brand. First, reflect on what you want to be known for, your values, and how you are currently perceived. Then, identify the results you are aiming to produce, aligning them with your strengths and organizational goals. Write these down, and share your reflections with trusted peers and your mentoring team. Shape your thoughts into a personal vision statement with a focus on what you put out into the world to help you stay true to yourself while producing the desired results. For example, a vision statement for a gifted communicator with a background in quality improvement may be: “I will use my strong communication skills to address complex problems impacting our hospital to reduce cost and improve quality with the goal of building a career as a health system leader.” Finally, be authentic, and share your personal brand in an articulate and succinct way to help others understand your place in the structure and narrative of an organization.

Your personal leadership brand should not be static; rather, it is a process that should iterate over time. Ask for direct feedback from trusted advisers and allies at regular intervals. Investigate whether your organization offers a formal structure, such as a “360 Evaluation,” to get perspective on how your unique strengths, skills, and goals are perceived. Then, explore and clarify discrepancies between where you think you are and how others see you. Approaching these conversations with humility will keep you aligned with your values, which makes it easier for others to be invested in your development.

A strong personal leadership brand is a force multiplier, providing clarity within teams and helping align a leader’s assets and values with organizational goals. It is a solid external signal of what others can expect from your work and will help you focus on your strengths while identifying areas for growth. A personal leadership brand is formed through reflection and, at its core, its authenticity. In the words of Paracelsus, a Renaissance physician, astrologer, and alchemist, “Be not another, if you can be yourself.”³

COVID-19, the disease caused by the novel coronavirus SARS-CoV-2, was declared a pandemic on March 11, 2020. Although most patients (81%) develop mild illness, 14% develop severe illness, and 5% develop critical illness, including acute respiratory failure, septic shock, and multiorgan dysfunction.¹

Point-of-care ultrasound (POCUS), or bedside ultrasound performed by a clinician caring for the patient, is being used to support the diagnosis and serially monitor patients with COVID-19. We performed a literature search of electronically discoverable peer-reviewed publications on POCUS use in COVID-19 from December 1, 2019, to April 10, 2020. We review key POCUS applications that are most relevant to frontline providers in the care of COVID-19 patients.

Diagnosing COVID-19 disease by polymerase chain reaction is limited by availability of testing, delays in test positivity (mean 5.1 days), and high false-negative rate early in the course of the disease (sensitivity 81%).² Chest computed tomography (CT) scans are often requested during the initial evaluation of suspected COVID-19, but the American College of Radiology has recommend against the routine use of CT scans for diagnosing COVID-19.³

The diagnostic accuracy of lung ultrasound (LUS) has been shown to be similar to chest CT scans in patients presenting with respiratory complaints, such as dyspnea and hypoxemia, caused by non–COVID-19 pneumonia (sensitivity, 85%; specificity, 93%).⁴ Normal LUS findings correlate well with CT chest scans showing absence of typical ground glass opacities. This negative predictive value is very important.⁵ However, early in the course of COVID-19, similar to CT scans, LUS may be normal during the first 5 days or in patients with mild disease.² Unique advantages of LUS in COVID-19 include immediate availability of results, repeatability over time, and performance at the bedside, which avoids transportation of patients to radiology suites and disinfection of large imaging equipment.

LUS findings in COVID-19 include (a) an irregular, thickened pleural line, (b) B-lines in various patterns (discrete and confluent), (c) small subpleural consolidations, and (d) absence of pleural effusions (Figure). Bilateral, multifocal disease is common, while lobar alveolar consolidation is less common.^6,7 In addition to supporting the initial diagnosis, LUS is being used to serially monitor hospitalized COVID-19 patients. As lung interstitial fluid content increases, discrete B-lines become confluent, and the number of affected lung zones increases, which can guide decisions about escalation of care. LUS is often used to guide decisions about prone ventilation, extracorporeal membrane oxygenation, and weaning from mechanical ventilation in acute respiratory failure of non–COVID-19 patients,⁸ and these concepts are being applied to COVID-19 patients. During recovery, reappearance of A-lines can be seen, but normalization of the LUS pattern is gradual over several weeks based on our experience and one report.⁹ Multiple LUS protocols examining 6 to 12 lung zones have been published prior to the COVID-19 pandemic. We recommend continuing to use an institutional protocol and evaluating at least one to two rib interspaces on the anterior, lateral, and posterior chest wall.

Myriad cardiac complications have been described in COVID-19 – including acute coronary syndrome, myocarditis, cardiomyopathy with heart failure, and arrhythmias – secondary to increased cardiac stress from hypoxia, direct myocardial infection, or indirect injury from a hyperinflammatory response. Mortality is higher in patients with hypertension, diabetes, and coronary artery disease.^10,11 Cardiac POCUS is being used to evaluate COVID-19 patients when troponin and B-type natriuretic peptide (BNP) are elevated or when there are hemodynamic or electrocardiogram changes. Given the high incidence of venous thromboembolism (VTE) in COVID-19,¹² cardiac POCUS is being used to rapidly assess for right ventricular (RV) dysfunction and acute pulmonary hypertension.

The American Society of Echocardiography has recommended the use of cardiac POCUS by frontline providers for detection or characterization of preexisting cardiovascular disease, early identification of worsening cardiac function, serial monitoring and examination, and elucidation of cardiovascular pathologies associated with COVID-19.¹³ Sharing cardiac POCUS images in real time or through an image archive can reduce the need for consultative echocardiography, which ultimately reduces staff exposure, conserves personal protective equipment, and reduces need for decontamination of echocardiographic equipment.

The minimum cardiac POCUS views recommended in COVID-19 patients include the parasternal long-axis and short-axis views (midventricular level), either the apical or subcostal four-chamber view, and the subcostal long-axis view of the inferior vena cava.¹³ The goal of a cardiac POCUS exam is to qualitatively assess left ventricular (LV) systolic function, RV size and contractility, gross valvular and regional wall motion abnormalities, and pericardial effusion. In prone position ventilation, the swimmer’s position with one arm elevated above the shoulder may permit acquisition of apical views. Finally, integrated cardiopulmonary ultrasonography, including evaluation for deep vein thrombosis (DVT; see below), is ideal for proper characterization of underlying LV and RV function, volume status, and titration of vasopressor and inotropic support.

COVID-19 has been associated with a proinflammatory and hypercoagulable state with elevated d-dimer and higher-than-­expected incidence of VTE (27%) in critically ill patients.^12,14 Previous studies have demonstrated that frontline providers, including hospitalists, can detect lower extremity (LE) DVTs with high diagnostic accuracy using POCUS.¹⁵ Given the high incidence of DVTs despite prophylactic anticoagulation, some reports have suggested screening or serially monitoring for LE DVT in hospitalized COVID-19 patients.¹⁶ In patients with suspected pulmonary embolism (PE), POCUS can rapidly detect venous thrombosis that justifies prompt initiation of anticoagulation (eg, finding DVT or clot-in-transit), supportive findings of PE (eg, acute RV dysfunction, pulmonary infarcts), or alternative diagnoses (eg, bacterial pneumonia). However, it is important to recognize POCUS cannot definitively rule out PE. Additionally, subpleural consolidations are common in COVID-19 patients and could be caused by either infection or infarction. The American Society of Hematology has endorsed the use of POCUS, LE compression ultrasonography, and echocardiography in COVID-19 patients with suspected PE when availability of CT pulmonary angiography or ventilation-perfusion lung scans is limited.¹⁴

A POCUS exam for LE DVT consists of two-dimensional venous compression alone and yields results similar to formal vascular studies in both critically ill and noncritically ill patients. Because proximal LE thrombi have the highest risk of embolization, evaluation of the common femoral vein, femoral vein, and popliteal vein is most important.¹⁵ Either inability to compress a vein completely with wall-to-wall apposition or visualization of echogenic thrombus within the vein is diagnostic of DVT. Acute thrombi are gelatinous and may appear anechoic, while subacute or chronic thrombi are echogenic, but all veins with a DVT will not compress completely.

Ultrasound guidance for central venous catheter (CVC) insertion has been shown to increase procedure success rates and decrease mechanical complications, primarily arterial puncture and pneumothorax. Similarly, higher success rates and fewer insertion attempts have been observed with ultrasound-guided peripheral intravenous line and arterial line placement.¹⁷ Ultrasound-­guided PIV placement can reduce referrals for midlines and peripherally inserted central catheters in hospitalized patients.¹⁸

In COVID-19 patients, use of ultrasound guidance for vascular access has distinct advantages. First, given the high incidence of DVT in COVID-19 patients,¹² POCUS allows preprocedural evaluation of the target vessel for thrombosis, as well as anatomic variations and stenosis. Second, visualizing the needle tip and guidewire within the target vein prior to dilation nearly eliminates the risk of arterial puncture and inadvertent arterial dilation, which is particularly important in COVID-19 patients receiving high-dose prophylactic or therapeutic anticoagulation. Third, when inserting internal jugular and subclavian CVCs, visualization of normal lung sliding before and after the procedure safely rules out pneumothorax. However, if lung sliding is not seen before the procedure, it cannot be used to rule out pneumothorax afterward. Additionally, visualizing absence of the catheter tip in the right atrium and presence of a rapid atrial swirl sign within 2 seconds of briskly injecting 10 mL of saline confirms catheter tip placement near the superior vena cava/right atrial junction, which can eliminate the need for a postprocedure chest radiograph.¹⁷

POCUS can be used to rapidly confirm endotracheal tube (ETT) placement, which can reduce reliance on postintubation chest radiographs. A meta-analysis of prospective and randomized trials showed transtracheal ultrasonography had high sensitivity (98.7%) and specificity (97.1%) for confirming tracheal placement of ETTs.¹⁹ Confirming endotracheal intubation involves two steps: First, a linear transducer is placed transversely over the suprasternal notch to visualize the ETT passing through the trachea, and not the esophagus, during insertion. Second, after the ETT cuff has been inflated, bilateral lung sliding should be seen in sync with the respiratory cycle if the ETT is in the trachea. Absent lung sliding, but preserved lung pulse, on the anterior hemithorax is likely caused by main stem bronchial intubation, and withdrawing the ETT until bilateral lung sliding is seen confirms tracheal placement. Additionally, the following steps are recommended to reduce the risk of exposure to healthcare workers: minimizing use of bag-valve-mask ventilation, performing rapid sequence intubation using video laryngoscopy, and connecting the ETT to the ventilator immediately.

Important considerations when selecting an ultrasound machine for use in COVID-19 patients include image quality, portability, functionality, and ease of disinfection. Advantages of handheld devices include portability and ease of disinfection, whereas cart-based systems generally have better image quality and functionality. To minimize the risk of cross contamination, an ultrasound machine should be dedicated exclusively for use on patients with confirmed COVID-19 and not shared with patients with suspected COVID-19.²⁰ To minimize exposure to COVID-19 patients, frontline providers should perform POCUS exams only when findings may change management, and timing of the exam and views acquired should be selected deliberately.

Ultrasound machine disinfection should be integrated into routine donning and doffing procedures. When possible, both handheld and cart-based machines should be draped with protective covers during aerosol-generating procedures. Single use ultrasound gel packets are recommended in order to decrease the risk of nosocomial infection.²⁰ After every use of an ultrasound machine on intact skin or for percutaneous procedures, low-level disinfection should be performed with an Environmental Protection Agency–recommended product that is effective against coronavirus.

Some ultrasound manufacturers have added teleultrasound software that allows remote training of novice POCUS users and remote guidance in actual patient care. Teleultrasound can be utilized to share images in real time with consultants or expert providers.

POCUS is uniquely poised to improve patient care during the COVID-19 pandemic. POCUS can be used to support the diagnosis of COVID-19 patients and monitor patients with confirmed disease. Common POCUS applications used in COVID-19 patients include evaluation of the lungs, heart, and deep veins, as well as performance of bedside procedures. Ultrasound machine portability and disinfection are important considerations in COVID-19 patients.

COVID-19, the disease caused by the novel coronavirus SARS-CoV-2, was declared a pandemic on March 11, 2020. Although most patients (81%) develop mild illness, 14% develop severe illness, and 5% develop critical illness, including acute respiratory failure, septic shock, and multiorgan dysfunction.¹

Point-of-care ultrasound (POCUS), or bedside ultrasound performed by a clinician caring for the patient, is being used to support the diagnosis and serially monitor patients with COVID-19. We performed a literature search of electronically discoverable peer-reviewed publications on POCUS use in COVID-19 from December 1, 2019, to April 10, 2020. We review key POCUS applications that are most relevant to frontline providers in the care of COVID-19 patients.

Diagnosing COVID-19 disease by polymerase chain reaction is limited by availability of testing, delays in test positivity (mean 5.1 days), and high false-negative rate early in the course of the disease (sensitivity 81%).² Chest computed tomography (CT) scans are often requested during the initial evaluation of suspected COVID-19, but the American College of Radiology has recommend against the routine use of CT scans for diagnosing COVID-19.³

The diagnostic accuracy of lung ultrasound (LUS) has been shown to be similar to chest CT scans in patients presenting with respiratory complaints, such as dyspnea and hypoxemia, caused by non–COVID-19 pneumonia (sensitivity, 85%; specificity, 93%).⁴ Normal LUS findings correlate well with CT chest scans showing absence of typical ground glass opacities. This negative predictive value is very important.⁵ However, early in the course of COVID-19, similar to CT scans, LUS may be normal during the first 5 days or in patients with mild disease.² Unique advantages of LUS in COVID-19 include immediate availability of results, repeatability over time, and performance at the bedside, which avoids transportation of patients to radiology suites and disinfection of large imaging equipment.

LUS findings in COVID-19 include (a) an irregular, thickened pleural line, (b) B-lines in various patterns (discrete and confluent), (c) small subpleural consolidations, and (d) absence of pleural effusions (Figure). Bilateral, multifocal disease is common, while lobar alveolar consolidation is less common.^6,7 In addition to supporting the initial diagnosis, LUS is being used to serially monitor hospitalized COVID-19 patients. As lung interstitial fluid content increases, discrete B-lines become confluent, and the number of affected lung zones increases, which can guide decisions about escalation of care. LUS is often used to guide decisions about prone ventilation, extracorporeal membrane oxygenation, and weaning from mechanical ventilation in acute respiratory failure of non–COVID-19 patients,⁸ and these concepts are being applied to COVID-19 patients. During recovery, reappearance of A-lines can be seen, but normalization of the LUS pattern is gradual over several weeks based on our experience and one report.⁹ Multiple LUS protocols examining 6 to 12 lung zones have been published prior to the COVID-19 pandemic. We recommend continuing to use an institutional protocol and evaluating at least one to two rib interspaces on the anterior, lateral, and posterior chest wall.

Myriad cardiac complications have been described in COVID-19 – including acute coronary syndrome, myocarditis, cardiomyopathy with heart failure, and arrhythmias – secondary to increased cardiac stress from hypoxia, direct myocardial infection, or indirect injury from a hyperinflammatory response. Mortality is higher in patients with hypertension, diabetes, and coronary artery disease.^10,11 Cardiac POCUS is being used to evaluate COVID-19 patients when troponin and B-type natriuretic peptide (BNP) are elevated or when there are hemodynamic or electrocardiogram changes. Given the high incidence of venous thromboembolism (VTE) in COVID-19,¹² cardiac POCUS is being used to rapidly assess for right ventricular (RV) dysfunction and acute pulmonary hypertension.

The American Society of Echocardiography has recommended the use of cardiac POCUS by frontline providers for detection or characterization of preexisting cardiovascular disease, early identification of worsening cardiac function, serial monitoring and examination, and elucidation of cardiovascular pathologies associated with COVID-19.¹³ Sharing cardiac POCUS images in real time or through an image archive can reduce the need for consultative echocardiography, which ultimately reduces staff exposure, conserves personal protective equipment, and reduces need for decontamination of echocardiographic equipment.

The minimum cardiac POCUS views recommended in COVID-19 patients include the parasternal long-axis and short-axis views (midventricular level), either the apical or subcostal four-chamber view, and the subcostal long-axis view of the inferior vena cava.¹³ The goal of a cardiac POCUS exam is to qualitatively assess left ventricular (LV) systolic function, RV size and contractility, gross valvular and regional wall motion abnormalities, and pericardial effusion. In prone position ventilation, the swimmer’s position with one arm elevated above the shoulder may permit acquisition of apical views. Finally, integrated cardiopulmonary ultrasonography, including evaluation for deep vein thrombosis (DVT; see below), is ideal for proper characterization of underlying LV and RV function, volume status, and titration of vasopressor and inotropic support.

COVID-19 has been associated with a proinflammatory and hypercoagulable state with elevated d-dimer and higher-than-­expected incidence of VTE (27%) in critically ill patients.^12,14 Previous studies have demonstrated that frontline providers, including hospitalists, can detect lower extremity (LE) DVTs with high diagnostic accuracy using POCUS.¹⁵ Given the high incidence of DVTs despite prophylactic anticoagulation, some reports have suggested screening or serially monitoring for LE DVT in hospitalized COVID-19 patients.¹⁶ In patients with suspected pulmonary embolism (PE), POCUS can rapidly detect venous thrombosis that justifies prompt initiation of anticoagulation (eg, finding DVT or clot-in-transit), supportive findings of PE (eg, acute RV dysfunction, pulmonary infarcts), or alternative diagnoses (eg, bacterial pneumonia). However, it is important to recognize POCUS cannot definitively rule out PE. Additionally, subpleural consolidations are common in COVID-19 patients and could be caused by either infection or infarction. The American Society of Hematology has endorsed the use of POCUS, LE compression ultrasonography, and echocardiography in COVID-19 patients with suspected PE when availability of CT pulmonary angiography or ventilation-perfusion lung scans is limited.¹⁴

A POCUS exam for LE DVT consists of two-dimensional venous compression alone and yields results similar to formal vascular studies in both critically ill and noncritically ill patients. Because proximal LE thrombi have the highest risk of embolization, evaluation of the common femoral vein, femoral vein, and popliteal vein is most important.¹⁵ Either inability to compress a vein completely with wall-to-wall apposition or visualization of echogenic thrombus within the vein is diagnostic of DVT. Acute thrombi are gelatinous and may appear anechoic, while subacute or chronic thrombi are echogenic, but all veins with a DVT will not compress completely.

Ultrasound guidance for central venous catheter (CVC) insertion has been shown to increase procedure success rates and decrease mechanical complications, primarily arterial puncture and pneumothorax. Similarly, higher success rates and fewer insertion attempts have been observed with ultrasound-guided peripheral intravenous line and arterial line placement.¹⁷ Ultrasound-­guided PIV placement can reduce referrals for midlines and peripherally inserted central catheters in hospitalized patients.¹⁸

In COVID-19 patients, use of ultrasound guidance for vascular access has distinct advantages. First, given the high incidence of DVT in COVID-19 patients,¹² POCUS allows preprocedural evaluation of the target vessel for thrombosis, as well as anatomic variations and stenosis. Second, visualizing the needle tip and guidewire within the target vein prior to dilation nearly eliminates the risk of arterial puncture and inadvertent arterial dilation, which is particularly important in COVID-19 patients receiving high-dose prophylactic or therapeutic anticoagulation. Third, when inserting internal jugular and subclavian CVCs, visualization of normal lung sliding before and after the procedure safely rules out pneumothorax. However, if lung sliding is not seen before the procedure, it cannot be used to rule out pneumothorax afterward. Additionally, visualizing absence of the catheter tip in the right atrium and presence of a rapid atrial swirl sign within 2 seconds of briskly injecting 10 mL of saline confirms catheter tip placement near the superior vena cava/right atrial junction, which can eliminate the need for a postprocedure chest radiograph.¹⁷

POCUS can be used to rapidly confirm endotracheal tube (ETT) placement, which can reduce reliance on postintubation chest radiographs. A meta-analysis of prospective and randomized trials showed transtracheal ultrasonography had high sensitivity (98.7%) and specificity (97.1%) for confirming tracheal placement of ETTs.¹⁹ Confirming endotracheal intubation involves two steps: First, a linear transducer is placed transversely over the suprasternal notch to visualize the ETT passing through the trachea, and not the esophagus, during insertion. Second, after the ETT cuff has been inflated, bilateral lung sliding should be seen in sync with the respiratory cycle if the ETT is in the trachea. Absent lung sliding, but preserved lung pulse, on the anterior hemithorax is likely caused by main stem bronchial intubation, and withdrawing the ETT until bilateral lung sliding is seen confirms tracheal placement. Additionally, the following steps are recommended to reduce the risk of exposure to healthcare workers: minimizing use of bag-valve-mask ventilation, performing rapid sequence intubation using video laryngoscopy, and connecting the ETT to the ventilator immediately.

Important considerations when selecting an ultrasound machine for use in COVID-19 patients include image quality, portability, functionality, and ease of disinfection. Advantages of handheld devices include portability and ease of disinfection, whereas cart-based systems generally have better image quality and functionality. To minimize the risk of cross contamination, an ultrasound machine should be dedicated exclusively for use on patients with confirmed COVID-19 and not shared with patients with suspected COVID-19.²⁰ To minimize exposure to COVID-19 patients, frontline providers should perform POCUS exams only when findings may change management, and timing of the exam and views acquired should be selected deliberately.

Ultrasound machine disinfection should be integrated into routine donning and doffing procedures. When possible, both handheld and cart-based machines should be draped with protective covers during aerosol-generating procedures. Single use ultrasound gel packets are recommended in order to decrease the risk of nosocomial infection.²⁰ After every use of an ultrasound machine on intact skin or for percutaneous procedures, low-level disinfection should be performed with an Environmental Protection Agency–recommended product that is effective against coronavirus.

Some ultrasound manufacturers have added teleultrasound software that allows remote training of novice POCUS users and remote guidance in actual patient care. Teleultrasound can be utilized to share images in real time with consultants or expert providers.

POCUS is uniquely poised to improve patient care during the COVID-19 pandemic. POCUS can be used to support the diagnosis of COVID-19 patients and monitor patients with confirmed disease. Common POCUS applications used in COVID-19 patients include evaluation of the lungs, heart, and deep veins, as well as performance of bedside procedures. Ultrasound machine portability and disinfection are important considerations in COVID-19 patients.

COVID-19, the disease caused by the novel coronavirus SARS-CoV-2, was declared a pandemic on March 11, 2020. Although most patients (81%) develop mild illness, 14% develop severe illness, and 5% develop critical illness, including acute respiratory failure, septic shock, and multiorgan dysfunction.¹

Point-of-care ultrasound (POCUS), or bedside ultrasound performed by a clinician caring for the patient, is being used to support the diagnosis and serially monitor patients with COVID-19. We performed a literature search of electronically discoverable peer-reviewed publications on POCUS use in COVID-19 from December 1, 2019, to April 10, 2020. We review key POCUS applications that are most relevant to frontline providers in the care of COVID-19 patients.

Diagnosing COVID-19 disease by polymerase chain reaction is limited by availability of testing, delays in test positivity (mean 5.1 days), and high false-negative rate early in the course of the disease (sensitivity 81%).² Chest computed tomography (CT) scans are often requested during the initial evaluation of suspected COVID-19, but the American College of Radiology has recommend against the routine use of CT scans for diagnosing COVID-19.³

The diagnostic accuracy of lung ultrasound (LUS) has been shown to be similar to chest CT scans in patients presenting with respiratory complaints, such as dyspnea and hypoxemia, caused by non–COVID-19 pneumonia (sensitivity, 85%; specificity, 93%).⁴ Normal LUS findings correlate well with CT chest scans showing absence of typical ground glass opacities. This negative predictive value is very important.⁵ However, early in the course of COVID-19, similar to CT scans, LUS may be normal during the first 5 days or in patients with mild disease.² Unique advantages of LUS in COVID-19 include immediate availability of results, repeatability over time, and performance at the bedside, which avoids transportation of patients to radiology suites and disinfection of large imaging equipment.

LUS findings in COVID-19 include (a) an irregular, thickened pleural line, (b) B-lines in various patterns (discrete and confluent), (c) small subpleural consolidations, and (d) absence of pleural effusions (Figure). Bilateral, multifocal disease is common, while lobar alveolar consolidation is less common.^6,7 In addition to supporting the initial diagnosis, LUS is being used to serially monitor hospitalized COVID-19 patients. As lung interstitial fluid content increases, discrete B-lines become confluent, and the number of affected lung zones increases, which can guide decisions about escalation of care. LUS is often used to guide decisions about prone ventilation, extracorporeal membrane oxygenation, and weaning from mechanical ventilation in acute respiratory failure of non–COVID-19 patients,⁸ and these concepts are being applied to COVID-19 patients. During recovery, reappearance of A-lines can be seen, but normalization of the LUS pattern is gradual over several weeks based on our experience and one report.⁹ Multiple LUS protocols examining 6 to 12 lung zones have been published prior to the COVID-19 pandemic. We recommend continuing to use an institutional protocol and evaluating at least one to two rib interspaces on the anterior, lateral, and posterior chest wall.

Myriad cardiac complications have been described in COVID-19 – including acute coronary syndrome, myocarditis, cardiomyopathy with heart failure, and arrhythmias – secondary to increased cardiac stress from hypoxia, direct myocardial infection, or indirect injury from a hyperinflammatory response. Mortality is higher in patients with hypertension, diabetes, and coronary artery disease.^10,11 Cardiac POCUS is being used to evaluate COVID-19 patients when troponin and B-type natriuretic peptide (BNP) are elevated or when there are hemodynamic or electrocardiogram changes. Given the high incidence of venous thromboembolism (VTE) in COVID-19,¹² cardiac POCUS is being used to rapidly assess for right ventricular (RV) dysfunction and acute pulmonary hypertension.

The American Society of Echocardiography has recommended the use of cardiac POCUS by frontline providers for detection or characterization of preexisting cardiovascular disease, early identification of worsening cardiac function, serial monitoring and examination, and elucidation of cardiovascular pathologies associated with COVID-19.¹³ Sharing cardiac POCUS images in real time or through an image archive can reduce the need for consultative echocardiography, which ultimately reduces staff exposure, conserves personal protective equipment, and reduces need for decontamination of echocardiographic equipment.

The minimum cardiac POCUS views recommended in COVID-19 patients include the parasternal long-axis and short-axis views (midventricular level), either the apical or subcostal four-chamber view, and the subcostal long-axis view of the inferior vena cava.¹³ The goal of a cardiac POCUS exam is to qualitatively assess left ventricular (LV) systolic function, RV size and contractility, gross valvular and regional wall motion abnormalities, and pericardial effusion. In prone position ventilation, the swimmer’s position with one arm elevated above the shoulder may permit acquisition of apical views. Finally, integrated cardiopulmonary ultrasonography, including evaluation for deep vein thrombosis (DVT; see below), is ideal for proper characterization of underlying LV and RV function, volume status, and titration of vasopressor and inotropic support.

COVID-19 has been associated with a proinflammatory and hypercoagulable state with elevated d-dimer and higher-than-­expected incidence of VTE (27%) in critically ill patients.^12,14 Previous studies have demonstrated that frontline providers, including hospitalists, can detect lower extremity (LE) DVTs with high diagnostic accuracy using POCUS.¹⁵ Given the high incidence of DVTs despite prophylactic anticoagulation, some reports have suggested screening or serially monitoring for LE DVT in hospitalized COVID-19 patients.¹⁶ In patients with suspected pulmonary embolism (PE), POCUS can rapidly detect venous thrombosis that justifies prompt initiation of anticoagulation (eg, finding DVT or clot-in-transit), supportive findings of PE (eg, acute RV dysfunction, pulmonary infarcts), or alternative diagnoses (eg, bacterial pneumonia). However, it is important to recognize POCUS cannot definitively rule out PE. Additionally, subpleural consolidations are common in COVID-19 patients and could be caused by either infection or infarction. The American Society of Hematology has endorsed the use of POCUS, LE compression ultrasonography, and echocardiography in COVID-19 patients with suspected PE when availability of CT pulmonary angiography or ventilation-perfusion lung scans is limited.¹⁴

A POCUS exam for LE DVT consists of two-dimensional venous compression alone and yields results similar to formal vascular studies in both critically ill and noncritically ill patients. Because proximal LE thrombi have the highest risk of embolization, evaluation of the common femoral vein, femoral vein, and popliteal vein is most important.¹⁵ Either inability to compress a vein completely with wall-to-wall apposition or visualization of echogenic thrombus within the vein is diagnostic of DVT. Acute thrombi are gelatinous and may appear anechoic, while subacute or chronic thrombi are echogenic, but all veins with a DVT will not compress completely.

Ultrasound guidance for central venous catheter (CVC) insertion has been shown to increase procedure success rates and decrease mechanical complications, primarily arterial puncture and pneumothorax. Similarly, higher success rates and fewer insertion attempts have been observed with ultrasound-guided peripheral intravenous line and arterial line placement.¹⁷ Ultrasound-­guided PIV placement can reduce referrals for midlines and peripherally inserted central catheters in hospitalized patients.¹⁸

In COVID-19 patients, use of ultrasound guidance for vascular access has distinct advantages. First, given the high incidence of DVT in COVID-19 patients,¹² POCUS allows preprocedural evaluation of the target vessel for thrombosis, as well as anatomic variations and stenosis. Second, visualizing the needle tip and guidewire within the target vein prior to dilation nearly eliminates the risk of arterial puncture and inadvertent arterial dilation, which is particularly important in COVID-19 patients receiving high-dose prophylactic or therapeutic anticoagulation. Third, when inserting internal jugular and subclavian CVCs, visualization of normal lung sliding before and after the procedure safely rules out pneumothorax. However, if lung sliding is not seen before the procedure, it cannot be used to rule out pneumothorax afterward. Additionally, visualizing absence of the catheter tip in the right atrium and presence of a rapid atrial swirl sign within 2 seconds of briskly injecting 10 mL of saline confirms catheter tip placement near the superior vena cava/right atrial junction, which can eliminate the need for a postprocedure chest radiograph.¹⁷

POCUS can be used to rapidly confirm endotracheal tube (ETT) placement, which can reduce reliance on postintubation chest radiographs. A meta-analysis of prospective and randomized trials showed transtracheal ultrasonography had high sensitivity (98.7%) and specificity (97.1%) for confirming tracheal placement of ETTs.¹⁹ Confirming endotracheal intubation involves two steps: First, a linear transducer is placed transversely over the suprasternal notch to visualize the ETT passing through the trachea, and not the esophagus, during insertion. Second, after the ETT cuff has been inflated, bilateral lung sliding should be seen in sync with the respiratory cycle if the ETT is in the trachea. Absent lung sliding, but preserved lung pulse, on the anterior hemithorax is likely caused by main stem bronchial intubation, and withdrawing the ETT until bilateral lung sliding is seen confirms tracheal placement. Additionally, the following steps are recommended to reduce the risk of exposure to healthcare workers: minimizing use of bag-valve-mask ventilation, performing rapid sequence intubation using video laryngoscopy, and connecting the ETT to the ventilator immediately.

Important considerations when selecting an ultrasound machine for use in COVID-19 patients include image quality, portability, functionality, and ease of disinfection. Advantages of handheld devices include portability and ease of disinfection, whereas cart-based systems generally have better image quality and functionality. To minimize the risk of cross contamination, an ultrasound machine should be dedicated exclusively for use on patients with confirmed COVID-19 and not shared with patients with suspected COVID-19.²⁰ To minimize exposure to COVID-19 patients, frontline providers should perform POCUS exams only when findings may change management, and timing of the exam and views acquired should be selected deliberately.

Ultrasound machine disinfection should be integrated into routine donning and doffing procedures. When possible, both handheld and cart-based machines should be draped with protective covers during aerosol-generating procedures. Single use ultrasound gel packets are recommended in order to decrease the risk of nosocomial infection.²⁰ After every use of an ultrasound machine on intact skin or for percutaneous procedures, low-level disinfection should be performed with an Environmental Protection Agency–recommended product that is effective against coronavirus.

Some ultrasound manufacturers have added teleultrasound software that allows remote training of novice POCUS users and remote guidance in actual patient care. Teleultrasound can be utilized to share images in real time with consultants or expert providers.

POCUS is uniquely poised to improve patient care during the COVID-19 pandemic. POCUS can be used to support the diagnosis of COVID-19 patients and monitor patients with confirmed disease. Common POCUS applications used in COVID-19 patients include evaluation of the lungs, heart, and deep veins, as well as performance of bedside procedures. Ultrasound machine portability and disinfection are important considerations in COVID-19 patients.

Voluntary accountable care organizations (ACOs) and bundled payments have concurrently become cornerstone strategies in Medicare’s shift from volume-based fee-for-service toward value-based payment.

Physician practice and hospital participation in Medicare’s largest ACO model, the Medicare Shared Savings Program (MSSP),¹ grew to include 561 organizations in 2018. Under MSSP, participants assume financial accountability for the global quality and costs of care for defined populations of Medicare fee-for-service patients. ACOs that manage to maintain or improve quality while achieving savings (ie, containing costs below a predefined population-wide spending benchmark) are eligible to receive a portion of the difference back from Medicare in the form of “shared savings”.

Similarly, hospital participation in Medicare’s bundled payment programs has grown over time. Most notably, more than 700 participants enrolled in the recently concluded Bundled Payments for Care Improvement (BPCI) initiative,² Medicare’s largest bundled payment program over the past five years.³ Under BPCI, participants assumed financial accountability for the quality and costs of care for all Medicare patients triggering a qualifying “episode of care”. Participants that limit episode spending below a predefined benchmark without compromising quality were eligible for financial incentives.

As both ACOs and bundled payments grow in prominence and scale, they may increasingly overlap if patients attributed to ACOs receive care at bundled payment hospitals. Overlap could create synergies by increasing incentives to address shared processes (eg, discharge planning) or outcomes (eg, readmissions).⁴ An ACO focus on reducing hospital admissions could complement bundled payment efforts to increase hospital efficiency.

Conversely, Medicare’s approach to allocating savings and losses can penalize ACOs or bundled payment participants.³ For example, when a patient included in an MSSP ACO population receives episodic care at a hospital participating in BPCI, the historical costs of care for the hospital and the episode type, not the actual costs of care for that specific patient and his/her episode, are counted in the performance of the ACO. In other words, in these cases, the performance of the MSSP ACO is dependent on the historical spending at BPCI hospitals—despite it being out of ACO’s control and having little to do with the actual care its patients receive at BPCI hospitals—and MSSP ACOs cannot benefit from improvements over time. Therefore, MSSP ACOs may be functionally penalized if patients receive care at historically high-cost BPCI hospitals regardless of whether they have considerably improved the value of care delivered. As a corollary, Medicare rules involve a “claw back” stipulation in which savings are recouped from hospitals that participate in both BPCI and MSSP, effectively discouraging participation in both payment models.

Although these dynamics are complex, they highlight an intuitive point that has gained increasing awareness,⁵ ie, policymakers must understand the magnitude of overlap to evaluate the urgency in coordinating between the payment models. Our objective was to describe the extent of overlap and the characteristics of patients affected by it.

We used 100% institutional Medicare claims, MSSP beneficiary attribution, and BPCI hospital data to identify fee-for-service beneficiaries attributed to MSSP and/or receiving care at BPCI hospitals for its 48 included episodes from the start of BPCI in 2013 quarter 4 through 2016 quarter 4.

We examined the trends in the number of episodes across the following three groups: MSSP-attributed patients hospitalized at BPCI hospitals for an episode included in BPCI (Overlap), MSSP-attributed patients hospitalized for that episode at non-BPCI hospitals (MSSP-only), and non-MSSP-attributed patients hospitalized at BPCI hospitals for a BPCI episode (BPCI-only). We used Medicare and United States Census Bureau data to compare groups with respect to sociodemographic (eg, age, sex, residence in a low-income area),⁶ clinical (eg, Elixhauser comorbidity index),⁷ and prior utilization (eg, skilled nursing facility discharge) characteristics.

Categorical and continuous variables were compared using logistic regression and one-way analysis of variance, respectively. Analyses were performed using Stata (StataCorp, College Station, Texas), version 15.0. Statistical tests were 2-tailed and significant at α = 0.05. This study was approved by the institutional review board at the University of Pennsylvania.

The number of MSSP ACOs increased from 220 in 2013 to 432 in 2016. The number of BPCI hospitals increased from 9 to 389 over this period, peaking at 413 hospitals in 2015. Over our study period, a total of 243,392, 2,824,898, and 702,864 episodes occurred in the Overlap, ACO-only, and BPCI-only groups, respectively (Table). Among episodes, patients in the Overlap group generally showed lower severity than those in other groups, although the differences were small. The BPCI-only, MSSP-only, and Overlap groups also exhibited small differences with respect to other characteristics such as the proportion of patients with Medicare/Medicaid dual-eligibility (15% of individual vs 16% and 12%, respectively) and prior use of skilled nursing facilities (33% vs 34% vs 31%, respectively) and acute care hospitals (45% vs 41% vs 39%, respectively) (P < .001 for all).

The overall overlap facing MSSP patients (overlap as a proportion of all MSSP patients) increased from 0.3% at the end of 2013 to 10% at the end of 2016, whereas over the same period, overlap facing bundled payment patients (overlap as a proportion of all bundled payment patients) increased from 11.9% to 27% (Appendix Figure). Overlap facing MSSP ACOs varied according to episode type, ranging from 3% for both acute myocardial infarction and chronic obstructive pulmonary disease episodes to 18% for automatic implantable cardiac defibrillator episodes at the end of 2016. Similarly, overlap facing bundled payment patients varied from 21% for spinal fusion episodes to 32% for lower extremity joint replacement and automatic implantable cardiac defibrillator episodes.

To our knowledge, this is the first study to describe the sizable and growing overlap facing ACOs with attributed patients who receive care at bundled payment hospitals, as well as bundled payment hospitals that treat patients attributed to ACOs.

The major implication of our findings is that policymakers must address and anticipate forthcoming payment model overlap as a key policy priority. Given the emphasis on ACOs and bundled payments as payment models—for example, Medicare continues to implement both nationwide via the Next Generation ACO model⁸ and the recently launched BPCI-Advanced program⁹—policymakers urgently need insights about the extent of payment model overlap. In that context, it is notable that although we have evaluated MSSP and BPCI as flagship programs, true overlap may actually be greater once other programs are considered.

Several factors may underlie the differences in the magnitude of overlap facing bundled payment versus ACO patients. The models differ in how they identify relevant patient populations, with patients falling under bundled payments via hospitalization for certain episode types but patients falling under ACOs via attribution based on the plurality of primary care services. Furthermore, BPCI participation lagged behind MSSP participation in time, while also occurring disproportionately in areas with existing MSSP ACOs.

Given these findings, understanding the implications of overlap should be a priority for future research and policy strategies. Potential policy considerations should include revising cost accounting processes so that when ACO-attributed patients receive episodic care at bundled payment hospitals, actual rather than historical hospital costs are counted toward ACO cost performance. To encourage hospitals to assume more accountability over outcomes—the ostensible overarching goal of value-based payment reform—Medicare could elect not to recoup savings from hospitals in both payment models. Although such changes require careful accounting to protect Medicare from financial losses as it forgoes some savings achieved through payment reforms, this may be worthwhile if hospital engagement in both models yields synergies.

Importantly, any policy changes made to address program overlap would need to accommodate ongoing changes in ACO, bundled payments, and other payment programs. For example, Medicare overhauled MSSP in December 2018. Compared to the earlier rules, in which ACOs could avoid downside financial risk altogether via “upside only” arrangements for up to six years, new MSSP rules require all participants to assume downside risk after several years of participation. Separately, forthcoming payment reforms such as direct contracting¹⁰ may draw clinicians and hospitals previously not participating in either Medicare fee-for-service or value-based payment models into payment reform. These factors may affect overlap in unpredictable ways (eg, they may increase the overlap by increasing the number of patients whose care is covered by different payment models or they may decrease overlap by raising the financial stakes of payment reforms to a degree that organizations drop out altogether).

This study has limitations. First, generalizability is limited by the fact that our analysis did not include bundled payment episodes assigned to physician group participants in BPCI or hospitals in mandatory joint replacement bundles under the Medicare Comprehensive Care for Joint Replacement model.¹¹ Second, although this study provides the first description of overlap between ACO and bundled payment programs, it was descriptive in nature. Future research is needed to evaluate the impact of overlap on clinical, quality, and cost outcomes. This is particularly important because although we observed only small differences in patient characteristics among MSSP-only, BPCI-only, and Overlap groups, characteristics could change differentially over time. Payment reforms must be carefully monitored for potentially unintended consequences that could arise from differential changes in patient characteristics (eg, cherry-picking behavior that is disadvantageous to vulnerable individuals).

Nonetheless, this study underscores the importance and extent of overlap and the urgency to consider policy measures to coordinate between the payment models.

Acknowledgments

The authors thank research assistance from Sandra Vanderslice who did not receive any compensation for her work. This research was supported in part by The Commonwealth Fund. Rachel Werner was supported in part by K24-AG047908 from the NIA.

Voluntary accountable care organizations (ACOs) and bundled payments have concurrently become cornerstone strategies in Medicare’s shift from volume-based fee-for-service toward value-based payment.

Physician practice and hospital participation in Medicare’s largest ACO model, the Medicare Shared Savings Program (MSSP),¹ grew to include 561 organizations in 2018. Under MSSP, participants assume financial accountability for the global quality and costs of care for defined populations of Medicare fee-for-service patients. ACOs that manage to maintain or improve quality while achieving savings (ie, containing costs below a predefined population-wide spending benchmark) are eligible to receive a portion of the difference back from Medicare in the form of “shared savings”.

Similarly, hospital participation in Medicare’s bundled payment programs has grown over time. Most notably, more than 700 participants enrolled in the recently concluded Bundled Payments for Care Improvement (BPCI) initiative,² Medicare’s largest bundled payment program over the past five years.³ Under BPCI, participants assumed financial accountability for the quality and costs of care for all Medicare patients triggering a qualifying “episode of care”. Participants that limit episode spending below a predefined benchmark without compromising quality were eligible for financial incentives.

As both ACOs and bundled payments grow in prominence and scale, they may increasingly overlap if patients attributed to ACOs receive care at bundled payment hospitals. Overlap could create synergies by increasing incentives to address shared processes (eg, discharge planning) or outcomes (eg, readmissions).⁴ An ACO focus on reducing hospital admissions could complement bundled payment efforts to increase hospital efficiency.

Conversely, Medicare’s approach to allocating savings and losses can penalize ACOs or bundled payment participants.³ For example, when a patient included in an MSSP ACO population receives episodic care at a hospital participating in BPCI, the historical costs of care for the hospital and the episode type, not the actual costs of care for that specific patient and his/her episode, are counted in the performance of the ACO. In other words, in these cases, the performance of the MSSP ACO is dependent on the historical spending at BPCI hospitals—despite it being out of ACO’s control and having little to do with the actual care its patients receive at BPCI hospitals—and MSSP ACOs cannot benefit from improvements over time. Therefore, MSSP ACOs may be functionally penalized if patients receive care at historically high-cost BPCI hospitals regardless of whether they have considerably improved the value of care delivered. As a corollary, Medicare rules involve a “claw back” stipulation in which savings are recouped from hospitals that participate in both BPCI and MSSP, effectively discouraging participation in both payment models.

Although these dynamics are complex, they highlight an intuitive point that has gained increasing awareness,⁵ ie, policymakers must understand the magnitude of overlap to evaluate the urgency in coordinating between the payment models. Our objective was to describe the extent of overlap and the characteristics of patients affected by it.

We used 100% institutional Medicare claims, MSSP beneficiary attribution, and BPCI hospital data to identify fee-for-service beneficiaries attributed to MSSP and/or receiving care at BPCI hospitals for its 48 included episodes from the start of BPCI in 2013 quarter 4 through 2016 quarter 4.

We examined the trends in the number of episodes across the following three groups: MSSP-attributed patients hospitalized at BPCI hospitals for an episode included in BPCI (Overlap), MSSP-attributed patients hospitalized for that episode at non-BPCI hospitals (MSSP-only), and non-MSSP-attributed patients hospitalized at BPCI hospitals for a BPCI episode (BPCI-only). We used Medicare and United States Census Bureau data to compare groups with respect to sociodemographic (eg, age, sex, residence in a low-income area),⁶ clinical (eg, Elixhauser comorbidity index),⁷ and prior utilization (eg, skilled nursing facility discharge) characteristics.

Categorical and continuous variables were compared using logistic regression and one-way analysis of variance, respectively. Analyses were performed using Stata (StataCorp, College Station, Texas), version 15.0. Statistical tests were 2-tailed and significant at α = 0.05. This study was approved by the institutional review board at the University of Pennsylvania.

The number of MSSP ACOs increased from 220 in 2013 to 432 in 2016. The number of BPCI hospitals increased from 9 to 389 over this period, peaking at 413 hospitals in 2015. Over our study period, a total of 243,392, 2,824,898, and 702,864 episodes occurred in the Overlap, ACO-only, and BPCI-only groups, respectively (Table). Among episodes, patients in the Overlap group generally showed lower severity than those in other groups, although the differences were small. The BPCI-only, MSSP-only, and Overlap groups also exhibited small differences with respect to other characteristics such as the proportion of patients with Medicare/Medicaid dual-eligibility (15% of individual vs 16% and 12%, respectively) and prior use of skilled nursing facilities (33% vs 34% vs 31%, respectively) and acute care hospitals (45% vs 41% vs 39%, respectively) (P < .001 for all).

The overall overlap facing MSSP patients (overlap as a proportion of all MSSP patients) increased from 0.3% at the end of 2013 to 10% at the end of 2016, whereas over the same period, overlap facing bundled payment patients (overlap as a proportion of all bundled payment patients) increased from 11.9% to 27% (Appendix Figure). Overlap facing MSSP ACOs varied according to episode type, ranging from 3% for both acute myocardial infarction and chronic obstructive pulmonary disease episodes to 18% for automatic implantable cardiac defibrillator episodes at the end of 2016. Similarly, overlap facing bundled payment patients varied from 21% for spinal fusion episodes to 32% for lower extremity joint replacement and automatic implantable cardiac defibrillator episodes.

To our knowledge, this is the first study to describe the sizable and growing overlap facing ACOs with attributed patients who receive care at bundled payment hospitals, as well as bundled payment hospitals that treat patients attributed to ACOs.

The major implication of our findings is that policymakers must address and anticipate forthcoming payment model overlap as a key policy priority. Given the emphasis on ACOs and bundled payments as payment models—for example, Medicare continues to implement both nationwide via the Next Generation ACO model⁸ and the recently launched BPCI-Advanced program⁹—policymakers urgently need insights about the extent of payment model overlap. In that context, it is notable that although we have evaluated MSSP and BPCI as flagship programs, true overlap may actually be greater once other programs are considered.

Several factors may underlie the differences in the magnitude of overlap facing bundled payment versus ACO patients. The models differ in how they identify relevant patient populations, with patients falling under bundled payments via hospitalization for certain episode types but patients falling under ACOs via attribution based on the plurality of primary care services. Furthermore, BPCI participation lagged behind MSSP participation in time, while also occurring disproportionately in areas with existing MSSP ACOs.

Given these findings, understanding the implications of overlap should be a priority for future research and policy strategies. Potential policy considerations should include revising cost accounting processes so that when ACO-attributed patients receive episodic care at bundled payment hospitals, actual rather than historical hospital costs are counted toward ACO cost performance. To encourage hospitals to assume more accountability over outcomes—the ostensible overarching goal of value-based payment reform—Medicare could elect not to recoup savings from hospitals in both payment models. Although such changes require careful accounting to protect Medicare from financial losses as it forgoes some savings achieved through payment reforms, this may be worthwhile if hospital engagement in both models yields synergies.

Importantly, any policy changes made to address program overlap would need to accommodate ongoing changes in ACO, bundled payments, and other payment programs. For example, Medicare overhauled MSSP in December 2018. Compared to the earlier rules, in which ACOs could avoid downside financial risk altogether via “upside only” arrangements for up to six years, new MSSP rules require all participants to assume downside risk after several years of participation. Separately, forthcoming payment reforms such as direct contracting¹⁰ may draw clinicians and hospitals previously not participating in either Medicare fee-for-service or value-based payment models into payment reform. These factors may affect overlap in unpredictable ways (eg, they may increase the overlap by increasing the number of patients whose care is covered by different payment models or they may decrease overlap by raising the financial stakes of payment reforms to a degree that organizations drop out altogether).

This study has limitations. First, generalizability is limited by the fact that our analysis did not include bundled payment episodes assigned to physician group participants in BPCI or hospitals in mandatory joint replacement bundles under the Medicare Comprehensive Care for Joint Replacement model.¹¹ Second, although this study provides the first description of overlap between ACO and bundled payment programs, it was descriptive in nature. Future research is needed to evaluate the impact of overlap on clinical, quality, and cost outcomes. This is particularly important because although we observed only small differences in patient characteristics among MSSP-only, BPCI-only, and Overlap groups, characteristics could change differentially over time. Payment reforms must be carefully monitored for potentially unintended consequences that could arise from differential changes in patient characteristics (eg, cherry-picking behavior that is disadvantageous to vulnerable individuals).

Nonetheless, this study underscores the importance and extent of overlap and the urgency to consider policy measures to coordinate between the payment models.

Acknowledgments

The authors thank research assistance from Sandra Vanderslice who did not receive any compensation for her work. This research was supported in part by The Commonwealth Fund. Rachel Werner was supported in part by K24-AG047908 from the NIA.

Voluntary accountable care organizations (ACOs) and bundled payments have concurrently become cornerstone strategies in Medicare’s shift from volume-based fee-for-service toward value-based payment.

Physician practice and hospital participation in Medicare’s largest ACO model, the Medicare Shared Savings Program (MSSP),¹ grew to include 561 organizations in 2018. Under MSSP, participants assume financial accountability for the global quality and costs of care for defined populations of Medicare fee-for-service patients. ACOs that manage to maintain or improve quality while achieving savings (ie, containing costs below a predefined population-wide spending benchmark) are eligible to receive a portion of the difference back from Medicare in the form of “shared savings”.

Similarly, hospital participation in Medicare’s bundled payment programs has grown over time. Most notably, more than 700 participants enrolled in the recently concluded Bundled Payments for Care Improvement (BPCI) initiative,² Medicare’s largest bundled payment program over the past five years.³ Under BPCI, participants assumed financial accountability for the quality and costs of care for all Medicare patients triggering a qualifying “episode of care”. Participants that limit episode spending below a predefined benchmark without compromising quality were eligible for financial incentives.

As both ACOs and bundled payments grow in prominence and scale, they may increasingly overlap if patients attributed to ACOs receive care at bundled payment hospitals. Overlap could create synergies by increasing incentives to address shared processes (eg, discharge planning) or outcomes (eg, readmissions).⁴ An ACO focus on reducing hospital admissions could complement bundled payment efforts to increase hospital efficiency.

Conversely, Medicare’s approach to allocating savings and losses can penalize ACOs or bundled payment participants.³ For example, when a patient included in an MSSP ACO population receives episodic care at a hospital participating in BPCI, the historical costs of care for the hospital and the episode type, not the actual costs of care for that specific patient and his/her episode, are counted in the performance of the ACO. In other words, in these cases, the performance of the MSSP ACO is dependent on the historical spending at BPCI hospitals—despite it being out of ACO’s control and having little to do with the actual care its patients receive at BPCI hospitals—and MSSP ACOs cannot benefit from improvements over time. Therefore, MSSP ACOs may be functionally penalized if patients receive care at historically high-cost BPCI hospitals regardless of whether they have considerably improved the value of care delivered. As a corollary, Medicare rules involve a “claw back” stipulation in which savings are recouped from hospitals that participate in both BPCI and MSSP, effectively discouraging participation in both payment models.

Although these dynamics are complex, they highlight an intuitive point that has gained increasing awareness,⁵ ie, policymakers must understand the magnitude of overlap to evaluate the urgency in coordinating between the payment models. Our objective was to describe the extent of overlap and the characteristics of patients affected by it.

We used 100% institutional Medicare claims, MSSP beneficiary attribution, and BPCI hospital data to identify fee-for-service beneficiaries attributed to MSSP and/or receiving care at BPCI hospitals for its 48 included episodes from the start of BPCI in 2013 quarter 4 through 2016 quarter 4.

We examined the trends in the number of episodes across the following three groups: MSSP-attributed patients hospitalized at BPCI hospitals for an episode included in BPCI (Overlap), MSSP-attributed patients hospitalized for that episode at non-BPCI hospitals (MSSP-only), and non-MSSP-attributed patients hospitalized at BPCI hospitals for a BPCI episode (BPCI-only). We used Medicare and United States Census Bureau data to compare groups with respect to sociodemographic (eg, age, sex, residence in a low-income area),⁶ clinical (eg, Elixhauser comorbidity index),⁷ and prior utilization (eg, skilled nursing facility discharge) characteristics.

Categorical and continuous variables were compared using logistic regression and one-way analysis of variance, respectively. Analyses were performed using Stata (StataCorp, College Station, Texas), version 15.0. Statistical tests were 2-tailed and significant at α = 0.05. This study was approved by the institutional review board at the University of Pennsylvania.

The number of MSSP ACOs increased from 220 in 2013 to 432 in 2016. The number of BPCI hospitals increased from 9 to 389 over this period, peaking at 413 hospitals in 2015. Over our study period, a total of 243,392, 2,824,898, and 702,864 episodes occurred in the Overlap, ACO-only, and BPCI-only groups, respectively (Table). Among episodes, patients in the Overlap group generally showed lower severity than those in other groups, although the differences were small. The BPCI-only, MSSP-only, and Overlap groups also exhibited small differences with respect to other characteristics such as the proportion of patients with Medicare/Medicaid dual-eligibility (15% of individual vs 16% and 12%, respectively) and prior use of skilled nursing facilities (33% vs 34% vs 31%, respectively) and acute care hospitals (45% vs 41% vs 39%, respectively) (P < .001 for all).

The overall overlap facing MSSP patients (overlap as a proportion of all MSSP patients) increased from 0.3% at the end of 2013 to 10% at the end of 2016, whereas over the same period, overlap facing bundled payment patients (overlap as a proportion of all bundled payment patients) increased from 11.9% to 27% (Appendix Figure). Overlap facing MSSP ACOs varied according to episode type, ranging from 3% for both acute myocardial infarction and chronic obstructive pulmonary disease episodes to 18% for automatic implantable cardiac defibrillator episodes at the end of 2016. Similarly, overlap facing bundled payment patients varied from 21% for spinal fusion episodes to 32% for lower extremity joint replacement and automatic implantable cardiac defibrillator episodes.

To our knowledge, this is the first study to describe the sizable and growing overlap facing ACOs with attributed patients who receive care at bundled payment hospitals, as well as bundled payment hospitals that treat patients attributed to ACOs.

The major implication of our findings is that policymakers must address and anticipate forthcoming payment model overlap as a key policy priority. Given the emphasis on ACOs and bundled payments as payment models—for example, Medicare continues to implement both nationwide via the Next Generation ACO model⁸ and the recently launched BPCI-Advanced program⁹—policymakers urgently need insights about the extent of payment model overlap. In that context, it is notable that although we have evaluated MSSP and BPCI as flagship programs, true overlap may actually be greater once other programs are considered.

Several factors may underlie the differences in the magnitude of overlap facing bundled payment versus ACO patients. The models differ in how they identify relevant patient populations, with patients falling under bundled payments via hospitalization for certain episode types but patients falling under ACOs via attribution based on the plurality of primary care services. Furthermore, BPCI participation lagged behind MSSP participation in time, while also occurring disproportionately in areas with existing MSSP ACOs.

Given these findings, understanding the implications of overlap should be a priority for future research and policy strategies. Potential policy considerations should include revising cost accounting processes so that when ACO-attributed patients receive episodic care at bundled payment hospitals, actual rather than historical hospital costs are counted toward ACO cost performance. To encourage hospitals to assume more accountability over outcomes—the ostensible overarching goal of value-based payment reform—Medicare could elect not to recoup savings from hospitals in both payment models. Although such changes require careful accounting to protect Medicare from financial losses as it forgoes some savings achieved through payment reforms, this may be worthwhile if hospital engagement in both models yields synergies.

Importantly, any policy changes made to address program overlap would need to accommodate ongoing changes in ACO, bundled payments, and other payment programs. For example, Medicare overhauled MSSP in December 2018. Compared to the earlier rules, in which ACOs could avoid downside financial risk altogether via “upside only” arrangements for up to six years, new MSSP rules require all participants to assume downside risk after several years of participation. Separately, forthcoming payment reforms such as direct contracting¹⁰ may draw clinicians and hospitals previously not participating in either Medicare fee-for-service or value-based payment models into payment reform. These factors may affect overlap in unpredictable ways (eg, they may increase the overlap by increasing the number of patients whose care is covered by different payment models or they may decrease overlap by raising the financial stakes of payment reforms to a degree that organizations drop out altogether).

This study has limitations. First, generalizability is limited by the fact that our analysis did not include bundled payment episodes assigned to physician group participants in BPCI or hospitals in mandatory joint replacement bundles under the Medicare Comprehensive Care for Joint Replacement model.¹¹ Second, although this study provides the first description of overlap between ACO and bundled payment programs, it was descriptive in nature. Future research is needed to evaluate the impact of overlap on clinical, quality, and cost outcomes. This is particularly important because although we observed only small differences in patient characteristics among MSSP-only, BPCI-only, and Overlap groups, characteristics could change differentially over time. Payment reforms must be carefully monitored for potentially unintended consequences that could arise from differential changes in patient characteristics (eg, cherry-picking behavior that is disadvantageous to vulnerable individuals).

Nonetheless, this study underscores the importance and extent of overlap and the urgency to consider policy measures to coordinate between the payment models.

Acknowledgments

The authors thank research assistance from Sandra Vanderslice who did not receive any compensation for her work. This research was supported in part by The Commonwealth Fund. Rachel Werner was supported in part by K24-AG047908 from the NIA.

With more than 3 million people diagnosed and more than 200,000 deaths worldwide at the time this article was written, coronavirus disease of 2019 (COVID-19) poses an unprecedented challenge to the public and to our healthcare system.¹ The United States has surpassed every other country in the total number of COVID-19 cases. Hospitals in hotspots are operating beyond capacity, while others prepare for a predicted surge of patients suffering from COVID-19. Now more than ever, clinicians need to prioritize limited time and resources wisely in this rapidly changing environment. Our most precious limited resource, healthcare workers (HCWs), bravely care for patients while trying to avoid acquiring the infection. With each test and treatment, clinicians must carefully consider harms and benefits, including exposing themselves and other HCWs to SARS-CoV-2, the virus causing this disease.

Delivering any healthcare service in which the potential harm exceeds benefit represents one form of overuse. In the era of COVID-19, the harmful consequences of overuse go beyond the patient to the healthcare team. For example, unnecessary chest computed tomography (CT) to help diagnose COVID-19 comes with the usual risks to the patient including radiation, but it may also reveal a suspicious nodule. That incidental finding can lead to downstream consequences, such as more imaging, blood work, and biopsy. In the current pandemic, however, that CT comes with more than just the usual risk. The initial unnecessary chest CT can risk exposing the transporter, the staff in the hallways and elevator en route, the radiology staff operating the CT scanner, and the maintenance staff who must clean the room and scanner afterward. Potential downstream harms to staff include exposure of the pulmonary and interventional radiology consultants, as well as the staff who perform repeat imaging after the biopsy. Evaluation of the nodule potentially prolongs the patient’s stay and exposes more staff. Clinicians must weigh the benefits and harms of each test and treatment carefully with consideration of both the patient and the staff involved. Moreover, it may turn out that the patient and staff without symptoms of COVID-19 may pose the most risk to one another.

Choosing Wisely® partnered with patients and clinician societies to develop a Top 5 recommendations list for eliminating unnecessary testing and treatment. Our multi-institutional group from the High Value Practice Academic Alliance proposed this Top 5 list of overuse practices in hospital medicine that can lead to harm of both patients and HCWs in the COVID-19 era (Table). The following recommendations apply to all patients with unsuspected, suspected, or confirmed SARS-CoV-2 infection in the hospital setting.

• Do not obtain nonurgent labs in separate blood draws if they can be batched together.

This recommendation expands on the original Society of Hospital Medicine Choosing Wisely recommendation: Don’t perform repetitive complete blood count and chemistry testing in the face of clinical and lab stability.² Aside from patient harms such as pain and hospital-acquired anemia, the risk of exposure to HCWs who perform phlebotomy (phlebotomists, nurses, and other clinicians), as well as staff who transport, handle, and process the bloodwork in the lab, must be minimized. Most prior interventions to eliminate unnecessary bloodwork focused on the number of lab tests,³ but some also aimed to batch nonurgent labs together to effectively reduce unnecessary needlesticks (“think twice, stick once”).⁴ This concept can be brought into this pandemic to provide safe and appropriate care for both patients and HCWs.

• Do not use bronchodilators unless there is active obstructive airway disease, and if needed, use metered dose inhalers instead of nebulizers.

We do not recommend using bronchodilators to treat COVID-19 symptoms unless patients develop acute bronchospastic symptoms of their underlying obstructive airway disease.⁵ When needed, use metered dose inhalers (MDIs),⁶ if available, instead of nebulizers because the latter potentiates aerosolization that could lead to higher risk of spreading the infection. The risk extends to respiratory technicians and nurses who administer the nebulizer, as well as other HCWs who enter the room during or after administration. The Centers for Disease Control and Prevention (CDC) considers nebulized bronchodilator therapy a “high-risk” exposure for HCWs not wearing the proper personal protectvie equipment.⁷ Moreover, MDI therapy produces equivalent outcomes to nebulized treatments for patients who are not critically ill.⁶ Unfortunately, the supply of MDIs during this crisis has not kept up with the increased demand.⁸

There are no clear guidelines for reuse of MDIs in COVID-19; however, options include labeling patients’ MDIs to use for hospitalization and discharge or labeling an MDI for use during hospitalization and then disinfecting for reuse. For safety reasons, MDIs of COVID-19 patients should be reused only for other patients with COVID-19.⁸

• Do not use posteroanterior and lateral chest X-ray as initial imaging. Use a portable chest X-ray instead.

The CDC does not currently recommend diagnosing COVID-19 by chest X-ray (CXR).⁷ When used appropriately, CXR can provide information to support a COVID-19 diagnosis and rule out other etiologies that cause respiratory symptoms.⁹ Posteroanterior (PA) and lateral CXR are more sensitive than portable CXR for detecting pleural effusions, and lateral CXR is needed to examine structures along the axis of the body. Portable CXR also may cause the heart to appear magnified and the mediastinum widened, the diaphragm to appear higher, and vascular shadows to be obscured.¹⁰ The improved ability to detect these subtle differences should be weighed against the increased risk to HCWs required to perform PA and lateral CXR. A portable CXR exposes a relatively smaller number of staff who come to the bedside versus the larger number of people exposed in transporting the patient out of the room and into the hallway, elevator, and the radiology suite for a PA and lateral CXR.

• Avoid in-person evaluations in favor of virtual communication unless necessary.

To minimize HCW exposure to COVID-19 and optimize infection control, the CDC recommends the use of telemedicine when possible.⁷ Telemedicine refers to the use of technology to support clinical care across some distance, which includes video visits and remote clinical monitoring. At the time of writing, the Centers for Medicare & Medicaid Services had waived the rural site of care requirement for Medicare beneficiaries, granted 49 Medicaid waivers to states to enhance flexibility, and (at least temporarily) added inpatient care to the list of reimbursed telemedicine services.¹¹ Funding for expanded coverage under Medicare is included in the recent Coronavirus Preparedness and Response Supplemental Appropriations Act.¹² These federal changes open the door for commercial payers and state Medicaid programs to further boost telemedicine through reimbursement parity to in-person visits and other coverage policies. Hospitalists can ride this momentum and learn from ambulatory colleagues to harness the power of telemedicine and minimize unnecessary face-to-face interactions with patients who are suspected or confirmed to have COVID-19.¹³ Even if providers have to enter the patient’s room, telemedicine may still allow for large virtual family meetings despite strict visitor restrictions and physical distance with loved ones. If in-person visits are necessary, only one designated person should enter the patient’s room instead of the entire team.

• Do not delay goals of care conversations for hospitalized patients who are unlikely to benefit from life-sustaining treatments.

The COVID-19 pandemic amplifies the need for early goals of care discussions. Mortality rates range higher with acute respiratory distress syndrome from COVID-19, compared with other etiologies, and is associated with extended intensive care unit stays.¹⁴ The harms extend beyond the patient and families to our HCWs through psychological distress and heightened exposure from aerosolization during resuscitation. Advance care planning should center on the values and preferences of the patient. Rather than asking if the patient or family would want certain treatments, it is crucial for clinicians to be direct in making do-not-resuscitate recommendations if deemed futile care.¹⁵ This practice is well within legal confines and is distinct from withdrawal or withholding of life-sustaining resources.¹⁵

HCWs providing inpatient care during this pandemic remain among the highest risk for contracting the infection. As of April 9, 2020, nearly 9,300 HCWs in the United States have contracted COVID-19.¹⁶ One thing remains clear: If we want to protect our patients, we must start by protecting our HCWs. We must think critically to evaluate the potential harms to our extended healthcare teams and strive further to eliminate overuse from our care.

The authors represent members of the High Value Practice Academic Alliance. The High Value Practice Academic Alliance is a consortium of academic medical centers in the United States and Canada working to advance high-value healthcare through collaborative quality improvement, research, and education. Additional information is available at http://www.hvpaa.org.

With more than 3 million people diagnosed and more than 200,000 deaths worldwide at the time this article was written, coronavirus disease of 2019 (COVID-19) poses an unprecedented challenge to the public and to our healthcare system.¹ The United States has surpassed every other country in the total number of COVID-19 cases. Hospitals in hotspots are operating beyond capacity, while others prepare for a predicted surge of patients suffering from COVID-19. Now more than ever, clinicians need to prioritize limited time and resources wisely in this rapidly changing environment. Our most precious limited resource, healthcare workers (HCWs), bravely care for patients while trying to avoid acquiring the infection. With each test and treatment, clinicians must carefully consider harms and benefits, including exposing themselves and other HCWs to SARS-CoV-2, the virus causing this disease.

Delivering any healthcare service in which the potential harm exceeds benefit represents one form of overuse. In the era of COVID-19, the harmful consequences of overuse go beyond the patient to the healthcare team. For example, unnecessary chest computed tomography (CT) to help diagnose COVID-19 comes with the usual risks to the patient including radiation, but it may also reveal a suspicious nodule. That incidental finding can lead to downstream consequences, such as more imaging, blood work, and biopsy. In the current pandemic, however, that CT comes with more than just the usual risk. The initial unnecessary chest CT can risk exposing the transporter, the staff in the hallways and elevator en route, the radiology staff operating the CT scanner, and the maintenance staff who must clean the room and scanner afterward. Potential downstream harms to staff include exposure of the pulmonary and interventional radiology consultants, as well as the staff who perform repeat imaging after the biopsy. Evaluation of the nodule potentially prolongs the patient’s stay and exposes more staff. Clinicians must weigh the benefits and harms of each test and treatment carefully with consideration of both the patient and the staff involved. Moreover, it may turn out that the patient and staff without symptoms of COVID-19 may pose the most risk to one another.

Choosing Wisely® partnered with patients and clinician societies to develop a Top 5 recommendations list for eliminating unnecessary testing and treatment. Our multi-institutional group from the High Value Practice Academic Alliance proposed this Top 5 list of overuse practices in hospital medicine that can lead to harm of both patients and HCWs in the COVID-19 era (Table). The following recommendations apply to all patients with unsuspected, suspected, or confirmed SARS-CoV-2 infection in the hospital setting.

• Do not obtain nonurgent labs in separate blood draws if they can be batched together.

This recommendation expands on the original Society of Hospital Medicine Choosing Wisely recommendation: Don’t perform repetitive complete blood count and chemistry testing in the face of clinical and lab stability.² Aside from patient harms such as pain and hospital-acquired anemia, the risk of exposure to HCWs who perform phlebotomy (phlebotomists, nurses, and other clinicians), as well as staff who transport, handle, and process the bloodwork in the lab, must be minimized. Most prior interventions to eliminate unnecessary bloodwork focused on the number of lab tests,³ but some also aimed to batch nonurgent labs together to effectively reduce unnecessary needlesticks (“think twice, stick once”).⁴ This concept can be brought into this pandemic to provide safe and appropriate care for both patients and HCWs.

• Do not use bronchodilators unless there is active obstructive airway disease, and if needed, use metered dose inhalers instead of nebulizers.

We do not recommend using bronchodilators to treat COVID-19 symptoms unless patients develop acute bronchospastic symptoms of their underlying obstructive airway disease.⁵ When needed, use metered dose inhalers (MDIs),⁶ if available, instead of nebulizers because the latter potentiates aerosolization that could lead to higher risk of spreading the infection. The risk extends to respiratory technicians and nurses who administer the nebulizer, as well as other HCWs who enter the room during or after administration. The Centers for Disease Control and Prevention (CDC) considers nebulized bronchodilator therapy a “high-risk” exposure for HCWs not wearing the proper personal protectvie equipment.⁷ Moreover, MDI therapy produces equivalent outcomes to nebulized treatments for patients who are not critically ill.⁶ Unfortunately, the supply of MDIs during this crisis has not kept up with the increased demand.⁸

There are no clear guidelines for reuse of MDIs in COVID-19; however, options include labeling patients’ MDIs to use for hospitalization and discharge or labeling an MDI for use during hospitalization and then disinfecting for reuse. For safety reasons, MDIs of COVID-19 patients should be reused only for other patients with COVID-19.⁸

• Do not use posteroanterior and lateral chest X-ray as initial imaging. Use a portable chest X-ray instead.

The CDC does not currently recommend diagnosing COVID-19 by chest X-ray (CXR).⁷ When used appropriately, CXR can provide information to support a COVID-19 diagnosis and rule out other etiologies that cause respiratory symptoms.⁹ Posteroanterior (PA) and lateral CXR are more sensitive than portable CXR for detecting pleural effusions, and lateral CXR is needed to examine structures along the axis of the body. Portable CXR also may cause the heart to appear magnified and the mediastinum widened, the diaphragm to appear higher, and vascular shadows to be obscured.¹⁰ The improved ability to detect these subtle differences should be weighed against the increased risk to HCWs required to perform PA and lateral CXR. A portable CXR exposes a relatively smaller number of staff who come to the bedside versus the larger number of people exposed in transporting the patient out of the room and into the hallway, elevator, and the radiology suite for a PA and lateral CXR.

• Avoid in-person evaluations in favor of virtual communication unless necessary.

To minimize HCW exposure to COVID-19 and optimize infection control, the CDC recommends the use of telemedicine when possible.⁷ Telemedicine refers to the use of technology to support clinical care across some distance, which includes video visits and remote clinical monitoring. At the time of writing, the Centers for Medicare & Medicaid Services had waived the rural site of care requirement for Medicare beneficiaries, granted 49 Medicaid waivers to states to enhance flexibility, and (at least temporarily) added inpatient care to the list of reimbursed telemedicine services.¹¹ Funding for expanded coverage under Medicare is included in the recent Coronavirus Preparedness and Response Supplemental Appropriations Act.¹² These federal changes open the door for commercial payers and state Medicaid programs to further boost telemedicine through reimbursement parity to in-person visits and other coverage policies. Hospitalists can ride this momentum and learn from ambulatory colleagues to harness the power of telemedicine and minimize unnecessary face-to-face interactions with patients who are suspected or confirmed to have COVID-19.¹³ Even if providers have to enter the patient’s room, telemedicine may still allow for large virtual family meetings despite strict visitor restrictions and physical distance with loved ones. If in-person visits are necessary, only one designated person should enter the patient’s room instead of the entire team.

• Do not delay goals of care conversations for hospitalized patients who are unlikely to benefit from life-sustaining treatments.

The COVID-19 pandemic amplifies the need for early goals of care discussions. Mortality rates range higher with acute respiratory distress syndrome from COVID-19, compared with other etiologies, and is associated with extended intensive care unit stays.¹⁴ The harms extend beyond the patient and families to our HCWs through psychological distress and heightened exposure from aerosolization during resuscitation. Advance care planning should center on the values and preferences of the patient. Rather than asking if the patient or family would want certain treatments, it is crucial for clinicians to be direct in making do-not-resuscitate recommendations if deemed futile care.¹⁵ This practice is well within legal confines and is distinct from withdrawal or withholding of life-sustaining resources.¹⁵

HCWs providing inpatient care during this pandemic remain among the highest risk for contracting the infection. As of April 9, 2020, nearly 9,300 HCWs in the United States have contracted COVID-19.¹⁶ One thing remains clear: If we want to protect our patients, we must start by protecting our HCWs. We must think critically to evaluate the potential harms to our extended healthcare teams and strive further to eliminate overuse from our care.

The authors represent members of the High Value Practice Academic Alliance. The High Value Practice Academic Alliance is a consortium of academic medical centers in the United States and Canada working to advance high-value healthcare through collaborative quality improvement, research, and education. Additional information is available at http://www.hvpaa.org.

With more than 3 million people diagnosed and more than 200,000 deaths worldwide at the time this article was written, coronavirus disease of 2019 (COVID-19) poses an unprecedented challenge to the public and to our healthcare system.¹ The United States has surpassed every other country in the total number of COVID-19 cases. Hospitals in hotspots are operating beyond capacity, while others prepare for a predicted surge of patients suffering from COVID-19. Now more than ever, clinicians need to prioritize limited time and resources wisely in this rapidly changing environment. Our most precious limited resource, healthcare workers (HCWs), bravely care for patients while trying to avoid acquiring the infection. With each test and treatment, clinicians must carefully consider harms and benefits, including exposing themselves and other HCWs to SARS-CoV-2, the virus causing this disease.

Delivering any healthcare service in which the potential harm exceeds benefit represents one form of overuse. In the era of COVID-19, the harmful consequences of overuse go beyond the patient to the healthcare team. For example, unnecessary chest computed tomography (CT) to help diagnose COVID-19 comes with the usual risks to the patient including radiation, but it may also reveal a suspicious nodule. That incidental finding can lead to downstream consequences, such as more imaging, blood work, and biopsy. In the current pandemic, however, that CT comes with more than just the usual risk. The initial unnecessary chest CT can risk exposing the transporter, the staff in the hallways and elevator en route, the radiology staff operating the CT scanner, and the maintenance staff who must clean the room and scanner afterward. Potential downstream harms to staff include exposure of the pulmonary and interventional radiology consultants, as well as the staff who perform repeat imaging after the biopsy. Evaluation of the nodule potentially prolongs the patient’s stay and exposes more staff. Clinicians must weigh the benefits and harms of each test and treatment carefully with consideration of both the patient and the staff involved. Moreover, it may turn out that the patient and staff without symptoms of COVID-19 may pose the most risk to one another.

Choosing Wisely® partnered with patients and clinician societies to develop a Top 5 recommendations list for eliminating unnecessary testing and treatment. Our multi-institutional group from the High Value Practice Academic Alliance proposed this Top 5 list of overuse practices in hospital medicine that can lead to harm of both patients and HCWs in the COVID-19 era (Table). The following recommendations apply to all patients with unsuspected, suspected, or confirmed SARS-CoV-2 infection in the hospital setting.

• Do not obtain nonurgent labs in separate blood draws if they can be batched together.

This recommendation expands on the original Society of Hospital Medicine Choosing Wisely recommendation: Don’t perform repetitive complete blood count and chemistry testing in the face of clinical and lab stability.² Aside from patient harms such as pain and hospital-acquired anemia, the risk of exposure to HCWs who perform phlebotomy (phlebotomists, nurses, and other clinicians), as well as staff who transport, handle, and process the bloodwork in the lab, must be minimized. Most prior interventions to eliminate unnecessary bloodwork focused on the number of lab tests,³ but some also aimed to batch nonurgent labs together to effectively reduce unnecessary needlesticks (“think twice, stick once”).⁴ This concept can be brought into this pandemic to provide safe and appropriate care for both patients and HCWs.

• Do not use bronchodilators unless there is active obstructive airway disease, and if needed, use metered dose inhalers instead of nebulizers.

We do not recommend using bronchodilators to treat COVID-19 symptoms unless patients develop acute bronchospastic symptoms of their underlying obstructive airway disease.⁵ When needed, use metered dose inhalers (MDIs),⁶ if available, instead of nebulizers because the latter potentiates aerosolization that could lead to higher risk of spreading the infection. The risk extends to respiratory technicians and nurses who administer the nebulizer, as well as other HCWs who enter the room during or after administration. The Centers for Disease Control and Prevention (CDC) considers nebulized bronchodilator therapy a “high-risk” exposure for HCWs not wearing the proper personal protectvie equipment.⁷ Moreover, MDI therapy produces equivalent outcomes to nebulized treatments for patients who are not critically ill.⁶ Unfortunately, the supply of MDIs during this crisis has not kept up with the increased demand.⁸

There are no clear guidelines for reuse of MDIs in COVID-19; however, options include labeling patients’ MDIs to use for hospitalization and discharge or labeling an MDI for use during hospitalization and then disinfecting for reuse. For safety reasons, MDIs of COVID-19 patients should be reused only for other patients with COVID-19.⁸

• Do not use posteroanterior and lateral chest X-ray as initial imaging. Use a portable chest X-ray instead.

The CDC does not currently recommend diagnosing COVID-19 by chest X-ray (CXR).⁷ When used appropriately, CXR can provide information to support a COVID-19 diagnosis and rule out other etiologies that cause respiratory symptoms.⁹ Posteroanterior (PA) and lateral CXR are more sensitive than portable CXR for detecting pleural effusions, and lateral CXR is needed to examine structures along the axis of the body. Portable CXR also may cause the heart to appear magnified and the mediastinum widened, the diaphragm to appear higher, and vascular shadows to be obscured.¹⁰ The improved ability to detect these subtle differences should be weighed against the increased risk to HCWs required to perform PA and lateral CXR. A portable CXR exposes a relatively smaller number of staff who come to the bedside versus the larger number of people exposed in transporting the patient out of the room and into the hallway, elevator, and the radiology suite for a PA and lateral CXR.

• Avoid in-person evaluations in favor of virtual communication unless necessary.

To minimize HCW exposure to COVID-19 and optimize infection control, the CDC recommends the use of telemedicine when possible.⁷ Telemedicine refers to the use of technology to support clinical care across some distance, which includes video visits and remote clinical monitoring. At the time of writing, the Centers for Medicare & Medicaid Services had waived the rural site of care requirement for Medicare beneficiaries, granted 49 Medicaid waivers to states to enhance flexibility, and (at least temporarily) added inpatient care to the list of reimbursed telemedicine services.¹¹ Funding for expanded coverage under Medicare is included in the recent Coronavirus Preparedness and Response Supplemental Appropriations Act.¹² These federal changes open the door for commercial payers and state Medicaid programs to further boost telemedicine through reimbursement parity to in-person visits and other coverage policies. Hospitalists can ride this momentum and learn from ambulatory colleagues to harness the power of telemedicine and minimize unnecessary face-to-face interactions with patients who are suspected or confirmed to have COVID-19.¹³ Even if providers have to enter the patient’s room, telemedicine may still allow for large virtual family meetings despite strict visitor restrictions and physical distance with loved ones. If in-person visits are necessary, only one designated person should enter the patient’s room instead of the entire team.

• Do not delay goals of care conversations for hospitalized patients who are unlikely to benefit from life-sustaining treatments.

The COVID-19 pandemic amplifies the need for early goals of care discussions. Mortality rates range higher with acute respiratory distress syndrome from COVID-19, compared with other etiologies, and is associated with extended intensive care unit stays.¹⁴ The harms extend beyond the patient and families to our HCWs through psychological distress and heightened exposure from aerosolization during resuscitation. Advance care planning should center on the values and preferences of the patient. Rather than asking if the patient or family would want certain treatments, it is crucial for clinicians to be direct in making do-not-resuscitate recommendations if deemed futile care.¹⁵ This practice is well within legal confines and is distinct from withdrawal or withholding of life-sustaining resources.¹⁵

HCWs providing inpatient care during this pandemic remain among the highest risk for contracting the infection. As of April 9, 2020, nearly 9,300 HCWs in the United States have contracted COVID-19.¹⁶ One thing remains clear: If we want to protect our patients, we must start by protecting our HCWs. We must think critically to evaluate the potential harms to our extended healthcare teams and strive further to eliminate overuse from our care.

The authors represent members of the High Value Practice Academic Alliance. The High Value Practice Academic Alliance is a consortium of academic medical centers in the United States and Canada working to advance high-value healthcare through collaborative quality improvement, research, and education. Additional information is available at http://www.hvpaa.org.

As the death toll from the coronavirus disease 2019 (COVID-19) pandemic rapidly increases, the need to make a timely and accurate diagnosis has never been greater. Even before the pandemic, diagnostic errors (ie, missed, delayed, and incorrect diagnoses) had been one of the leading contributors to harm in health care.¹ The COVID-19 pandemic is likely to increase the risk of such errors for several reasons. The disease itself is new and knowledge of its clinical manifestations is still evolving. Both physical and psychological safety of clinicians and health system capacity are compromised and can affect clinical decision-making.² Situational factors such as staffing shortages and workarounds are more common, and clinicians in certain geographic areas are experiencing epic levels of stress, fatigue, and burnout. Finally, decisions in busy, chaotic and time-pressured healthcare systems with disrupted and/or newly designed care processes will be error prone.¹

Based on emerging literature and collaborative discussions across the globe, we propose a new typology of diagnostic errors of concern in the COVID-19 era (Table). These errors span the entire continuum of care and have both systems-based and cognitive origins. While some errors arise from previously described clinical reasoning fallacies, others are unique to the pandemic. We provide a user-friendly nomenclature while describing eight types of diagnostic errors and highlight mitigation strategies to reduce potential preventable harm caused by those errors.

The classic COVID-19 presentation of a febrile respiratory illness warrants confirmatory testing, but testing may not be available or produce a false-negative result, leading to an error we termed “Classic.” In the United States, efforts to develop and implement testing protocols are still evolving. There is wide local and regional variation in type and availability of tests, as well as accessibility of information regarding test performance characteristics or diagnostic yield.³ Test results that are false negatives or testing that is not performed can lead to delayed diagnosis of the disease, as well as continued spread.

Testing is similarly relevant when patients present with unusual or nonrespiratory symptoms. Both predominantly olfactory⁴ and gastrointestinal manifestations⁵ have now been described, and mysterious new associations, such as multisystem inflammatory syndromes, continue to emerge. A failure to recognize atypical presentations and associations, either because of testing problems or knowledge gaps, could lead to overlooking underlying COVID-19 diagnosis, an error we termed “Anomalous.”

Another error emerging in the pandemic is mislabeling patients who do not have COVID-19 as having the disease, particularly those with respiratory symptoms. This usually occurs in absence of testing in an overwhelmed health system with limited capacity to test or treat (eg, clinicians just assume it must be COVID-19 when the test is not available). This type of labeling error, called “Anchor,” introduces the risk of missing other respiratory infections such as bacterial sinusitis and pneumonia, as well as nonrespiratory conditions.

In patients with known COVID-19, a second underlying or concurrent condition may be missed, an error we termed “Secondary.” For instance, reports of coagulopathy-related pulmonary embolism⁶ and strokes in young patients with minimal symptoms⁷ have emerged just recently. Respiratory compromise may be mistakenly attributed to COVID-19 rather than looking for a new source of worsening, such as pulmonary embolism. Similarly, clinicians may not recognize subtle stroke symptoms in patients who were otherwise feeling well at home. Such cognitive errors will likely increase as it becomes harder for clinicians or health systems to keep up with new knowledge.

Collateral effects of the COVID-19 pandemic are also emerging. For instance, patients with symptoms of new acute conditions may be unwilling to visit acute care for evaluation because of infection risk, an error we termed “Acute Collateral.” Concerns are already being raised that patients with acute myocardial infarction⁸ and stroke⁹ are not coming in for evaluation. Similarly, there may be delays in diagnosis of important ambulatory conditions, including cancer,¹⁰ when appointments or elective procedures are canceled (“Chronic Collateral”). In the United Kingdom, referrals under the 2-week wait system–in which suspected cancer patients referred by general practitioners are seen within 2-weeks–fell by 70% over March to April, 2020.

Diagnosis of non–COVID-19 patients coming into the hospital may also be affected because of the understandably heightened state of attention to COVID-19 patients, capacity, and staffing issues, an error we termed “Strain.” Physicians, including surgeons, pediatricians, and radiologists, have been “redeployed” into acute care medical specialties. Cognitive errors increase when clinicians in new roles face unfamiliar situations and disease manifestations. Although these clinicians may be highly experienced previously, they may have insufficient skills and experience in their new roles and may not feel comfortable asking for guidance.¹¹

Lastly, clinicians are increasingly using intermediary mechanisms, such as PPE and telemedicine technologies, to interact with patients. This is new for both parties and could introduce new types of errors, which we termed “Unintended.” Furthermore, interactions mediated via telemedicine technologies or PPE, as well as PPE conservation measures such as reduced room entries and e-consultation, may reduce the ability of even well-trained clinicians to take effective histories, perform physical exams, and monitor symptoms. In fact, infection-prevention isolation has been shown to put patients at risk of preventable adverse events in hospitalized patients.¹²

There are many strategies that health systems could deploy to try to minimize these eight types of diagnostic errors. We organize mitigation strategies using the Safer Dx framework, which proposes sociotechnical approaches (ie, both technology and other systems-based approaches) to reduce diagnostic error.¹³

Technology for Cognitive Support

Up-to-date electronic decision support is needed to optimize test interpretation. Technology can also help scale and facilitate rapid adoption of standardized safety practices and protocols to address emerging risks areas. For instance, there are early efforts to create, implement, and disseminate smart algorithms to predict risks of non–COVID-19 diagnoses such as venous thromboembolism, patient transfer protocols on how best to reduce the burden at overstressed hospitals, protocols to triage rescheduling of elective procedures based on potential risk as determined from data in the electronic health record, new rules for creating outreach to patients who have missed appointments to prevent delays in their evaluation and diagnosis, and triage protocols and follow-up systems to optimize telemedicine.¹⁴

Optimized Workflow and Communication

When in-person contact is limited, specific practices (eg, providing patients with iPads, use of reflective listening, and use of optimal nonverbal communication strategies such as eye-contact) can still facilitate comprehensive discussions with patients and families about symptoms and encourage them to speak up if and when they have concerns.¹⁵ For patients reached through telemedicine, follow-up appointments and surveys should be done to ensure that symptoms and concerns have been addressed. For clinicians working in new clinical areas unfamiliar to them (eg, surgeons on medical floors, hospitalists in ICUs), buddy systems can pair these clinicians with more experienced clinicians to make it easier for them to ask for help. Visual aids, decision support, and reliable error-prevention resources can also be helpful.¹⁶

People-Focused Interventions

Some clinicians are used to practicing solo, but this is the time to start “diagnostic huddles” for discussion of challenging cases with symptoms that are unusual or not improving as expected or for determining whether anything has been missed. In addition to encouraging patients to use reliable digital tools for self-triage, outreach to patients and the public must also advise them (with the help of public health authorities and the media) to seek medical assistance for certain important conditions such as acute myocardial infarction and stroke.

Organizational Strategies

Fundamental safety strategies must be ensured. First, it is critical to have a strong safety culture in which staff feel empowered to speak up, ask questions or ask for help, and report concerns without fear of repercussions or judgement. Culture can take years to develop, but due to rapidly changing circumstances in a crisis, there are ways for healthcare leaders to create changes more quickly. In addition to having daily huddles, leaders should be visible and communicate clearly about the behaviors and norms they are supporting. In particular, frequent leadership rounding (either virtually or in person)—during which leaders ask questions and encourage discussions of concerns in a supportive way—can foster the kind of culture that is needed. All organizations should implement peer support, counseling, limits on hours worked, and other support strategies for all clinicians to minimize the fatigue, stress, and anxiety that can impair cognitive function.¹⁷

Organizations must also be able to identify these errors to help understand root causes and prioritize interventions.¹⁸ For example, streamlined reporting systems that use apps and hotlines could be developed quickly to ensure that clinicians and patients/families can easily report these errors. Electronic triggers can help detect specific situations indicative of error or delay (eg, patient not on precautions gets switched to precautions during a hospitalization; absence of follow-up on abnormal tests).¹⁹

Learning systems—both within and across hospitals—should continue to share diagnostic challenges, the most up-to-date information, and best practices/protocols, and identify opportunities for improvement together. Many hospitals are having virtual grand rounds, journals are rapidly sharing new information via open access, regional and national cross-organizational and multidisciplinary learning networks of various groups have emerged (such as networks of oncologists, infectious disease specialists, and hospitalists), and new and transparent communication channels have developed between state and local health departments, government leaders, health systems, and the public. These forums should discuss emerging knowledge on diagnosis and strategies for risk reduction, many of which will unfold over the next few months.

State/Federal Policies and Regulations

While there is progress, additional challenges with accessibility, accuracy and performance of testing should be addressed at a national level. Guidance is needed on which asymptomatic people should be tested, both within and outside hospitals. Standardized metrics should be developed to monitor diagnostic performance and outcomes and evaluate how COVID-19 diagnosis errors affect different demographics. For instance, black and Hispanic individuals are disproportionately represented in COVID-19 cases and deaths, so metrics could be further stratified by race and ethnicity to ensure that we can understand and eliminate inequities, such as lack of access to care or testing.²⁰

Clinicians must be provided with both cognitive and system support so they can do what they do best—diagnose and treat patients and save lives. Intermittent epidemic spikes based on location and season, including a potentially bigger spike of cases later this year, are now projected. Risks and recommendations discussed herein should therefore be rapidly shared to help redesign and strengthen the work system and protect patients from preventable diagnosis-related harm.

The views expressed in this article do not represent the views of the U.S. Department of Veterans Affairs or the United States government.

As the death toll from the coronavirus disease 2019 (COVID-19) pandemic rapidly increases, the need to make a timely and accurate diagnosis has never been greater. Even before the pandemic, diagnostic errors (ie, missed, delayed, and incorrect diagnoses) had been one of the leading contributors to harm in health care.¹ The COVID-19 pandemic is likely to increase the risk of such errors for several reasons. The disease itself is new and knowledge of its clinical manifestations is still evolving. Both physical and psychological safety of clinicians and health system capacity are compromised and can affect clinical decision-making.² Situational factors such as staffing shortages and workarounds are more common, and clinicians in certain geographic areas are experiencing epic levels of stress, fatigue, and burnout. Finally, decisions in busy, chaotic and time-pressured healthcare systems with disrupted and/or newly designed care processes will be error prone.¹

Based on emerging literature and collaborative discussions across the globe, we propose a new typology of diagnostic errors of concern in the COVID-19 era (Table). These errors span the entire continuum of care and have both systems-based and cognitive origins. While some errors arise from previously described clinical reasoning fallacies, others are unique to the pandemic. We provide a user-friendly nomenclature while describing eight types of diagnostic errors and highlight mitigation strategies to reduce potential preventable harm caused by those errors.

The classic COVID-19 presentation of a febrile respiratory illness warrants confirmatory testing, but testing may not be available or produce a false-negative result, leading to an error we termed “Classic.” In the United States, efforts to develop and implement testing protocols are still evolving. There is wide local and regional variation in type and availability of tests, as well as accessibility of information regarding test performance characteristics or diagnostic yield.³ Test results that are false negatives or testing that is not performed can lead to delayed diagnosis of the disease, as well as continued spread.

Testing is similarly relevant when patients present with unusual or nonrespiratory symptoms. Both predominantly olfactory⁴ and gastrointestinal manifestations⁵ have now been described, and mysterious new associations, such as multisystem inflammatory syndromes, continue to emerge. A failure to recognize atypical presentations and associations, either because of testing problems or knowledge gaps, could lead to overlooking underlying COVID-19 diagnosis, an error we termed “Anomalous.”

Another error emerging in the pandemic is mislabeling patients who do not have COVID-19 as having the disease, particularly those with respiratory symptoms. This usually occurs in absence of testing in an overwhelmed health system with limited capacity to test or treat (eg, clinicians just assume it must be COVID-19 when the test is not available). This type of labeling error, called “Anchor,” introduces the risk of missing other respiratory infections such as bacterial sinusitis and pneumonia, as well as nonrespiratory conditions.

In patients with known COVID-19, a second underlying or concurrent condition may be missed, an error we termed “Secondary.” For instance, reports of coagulopathy-related pulmonary embolism⁶ and strokes in young patients with minimal symptoms⁷ have emerged just recently. Respiratory compromise may be mistakenly attributed to COVID-19 rather than looking for a new source of worsening, such as pulmonary embolism. Similarly, clinicians may not recognize subtle stroke symptoms in patients who were otherwise feeling well at home. Such cognitive errors will likely increase as it becomes harder for clinicians or health systems to keep up with new knowledge.

Collateral effects of the COVID-19 pandemic are also emerging. For instance, patients with symptoms of new acute conditions may be unwilling to visit acute care for evaluation because of infection risk, an error we termed “Acute Collateral.” Concerns are already being raised that patients with acute myocardial infarction⁸ and stroke⁹ are not coming in for evaluation. Similarly, there may be delays in diagnosis of important ambulatory conditions, including cancer,¹⁰ when appointments or elective procedures are canceled (“Chronic Collateral”). In the United Kingdom, referrals under the 2-week wait system–in which suspected cancer patients referred by general practitioners are seen within 2-weeks–fell by 70% over March to April, 2020.

Diagnosis of non–COVID-19 patients coming into the hospital may also be affected because of the understandably heightened state of attention to COVID-19 patients, capacity, and staffing issues, an error we termed “Strain.” Physicians, including surgeons, pediatricians, and radiologists, have been “redeployed” into acute care medical specialties. Cognitive errors increase when clinicians in new roles face unfamiliar situations and disease manifestations. Although these clinicians may be highly experienced previously, they may have insufficient skills and experience in their new roles and may not feel comfortable asking for guidance.¹¹

Lastly, clinicians are increasingly using intermediary mechanisms, such as PPE and telemedicine technologies, to interact with patients. This is new for both parties and could introduce new types of errors, which we termed “Unintended.” Furthermore, interactions mediated via telemedicine technologies or PPE, as well as PPE conservation measures such as reduced room entries and e-consultation, may reduce the ability of even well-trained clinicians to take effective histories, perform physical exams, and monitor symptoms. In fact, infection-prevention isolation has been shown to put patients at risk of preventable adverse events in hospitalized patients.¹²

There are many strategies that health systems could deploy to try to minimize these eight types of diagnostic errors. We organize mitigation strategies using the Safer Dx framework, which proposes sociotechnical approaches (ie, both technology and other systems-based approaches) to reduce diagnostic error.¹³

Technology for Cognitive Support

Up-to-date electronic decision support is needed to optimize test interpretation. Technology can also help scale and facilitate rapid adoption of standardized safety practices and protocols to address emerging risks areas. For instance, there are early efforts to create, implement, and disseminate smart algorithms to predict risks of non–COVID-19 diagnoses such as venous thromboembolism, patient transfer protocols on how best to reduce the burden at overstressed hospitals, protocols to triage rescheduling of elective procedures based on potential risk as determined from data in the electronic health record, new rules for creating outreach to patients who have missed appointments to prevent delays in their evaluation and diagnosis, and triage protocols and follow-up systems to optimize telemedicine.¹⁴

Optimized Workflow and Communication

When in-person contact is limited, specific practices (eg, providing patients with iPads, use of reflective listening, and use of optimal nonverbal communication strategies such as eye-contact) can still facilitate comprehensive discussions with patients and families about symptoms and encourage them to speak up if and when they have concerns.¹⁵ For patients reached through telemedicine, follow-up appointments and surveys should be done to ensure that symptoms and concerns have been addressed. For clinicians working in new clinical areas unfamiliar to them (eg, surgeons on medical floors, hospitalists in ICUs), buddy systems can pair these clinicians with more experienced clinicians to make it easier for them to ask for help. Visual aids, decision support, and reliable error-prevention resources can also be helpful.¹⁶

People-Focused Interventions

Some clinicians are used to practicing solo, but this is the time to start “diagnostic huddles” for discussion of challenging cases with symptoms that are unusual or not improving as expected or for determining whether anything has been missed. In addition to encouraging patients to use reliable digital tools for self-triage, outreach to patients and the public must also advise them (with the help of public health authorities and the media) to seek medical assistance for certain important conditions such as acute myocardial infarction and stroke.

Organizational Strategies

Fundamental safety strategies must be ensured. First, it is critical to have a strong safety culture in which staff feel empowered to speak up, ask questions or ask for help, and report concerns without fear of repercussions or judgement. Culture can take years to develop, but due to rapidly changing circumstances in a crisis, there are ways for healthcare leaders to create changes more quickly. In addition to having daily huddles, leaders should be visible and communicate clearly about the behaviors and norms they are supporting. In particular, frequent leadership rounding (either virtually or in person)—during which leaders ask questions and encourage discussions of concerns in a supportive way—can foster the kind of culture that is needed. All organizations should implement peer support, counseling, limits on hours worked, and other support strategies for all clinicians to minimize the fatigue, stress, and anxiety that can impair cognitive function.¹⁷

Organizations must also be able to identify these errors to help understand root causes and prioritize interventions.¹⁸ For example, streamlined reporting systems that use apps and hotlines could be developed quickly to ensure that clinicians and patients/families can easily report these errors. Electronic triggers can help detect specific situations indicative of error or delay (eg, patient not on precautions gets switched to precautions during a hospitalization; absence of follow-up on abnormal tests).¹⁹

Learning systems—both within and across hospitals—should continue to share diagnostic challenges, the most up-to-date information, and best practices/protocols, and identify opportunities for improvement together. Many hospitals are having virtual grand rounds, journals are rapidly sharing new information via open access, regional and national cross-organizational and multidisciplinary learning networks of various groups have emerged (such as networks of oncologists, infectious disease specialists, and hospitalists), and new and transparent communication channels have developed between state and local health departments, government leaders, health systems, and the public. These forums should discuss emerging knowledge on diagnosis and strategies for risk reduction, many of which will unfold over the next few months.

State/Federal Policies and Regulations

While there is progress, additional challenges with accessibility, accuracy and performance of testing should be addressed at a national level. Guidance is needed on which asymptomatic people should be tested, both within and outside hospitals. Standardized metrics should be developed to monitor diagnostic performance and outcomes and evaluate how COVID-19 diagnosis errors affect different demographics. For instance, black and Hispanic individuals are disproportionately represented in COVID-19 cases and deaths, so metrics could be further stratified by race and ethnicity to ensure that we can understand and eliminate inequities, such as lack of access to care or testing.²⁰

Clinicians must be provided with both cognitive and system support so they can do what they do best—diagnose and treat patients and save lives. Intermittent epidemic spikes based on location and season, including a potentially bigger spike of cases later this year, are now projected. Risks and recommendations discussed herein should therefore be rapidly shared to help redesign and strengthen the work system and protect patients from preventable diagnosis-related harm.

The views expressed in this article do not represent the views of the U.S. Department of Veterans Affairs or the United States government.

As the death toll from the coronavirus disease 2019 (COVID-19) pandemic rapidly increases, the need to make a timely and accurate diagnosis has never been greater. Even before the pandemic, diagnostic errors (ie, missed, delayed, and incorrect diagnoses) had been one of the leading contributors to harm in health care.¹ The COVID-19 pandemic is likely to increase the risk of such errors for several reasons. The disease itself is new and knowledge of its clinical manifestations is still evolving. Both physical and psychological safety of clinicians and health system capacity are compromised and can affect clinical decision-making.² Situational factors such as staffing shortages and workarounds are more common, and clinicians in certain geographic areas are experiencing epic levels of stress, fatigue, and burnout. Finally, decisions in busy, chaotic and time-pressured healthcare systems with disrupted and/or newly designed care processes will be error prone.¹

Based on emerging literature and collaborative discussions across the globe, we propose a new typology of diagnostic errors of concern in the COVID-19 era (Table). These errors span the entire continuum of care and have both systems-based and cognitive origins. While some errors arise from previously described clinical reasoning fallacies, others are unique to the pandemic. We provide a user-friendly nomenclature while describing eight types of diagnostic errors and highlight mitigation strategies to reduce potential preventable harm caused by those errors.

The classic COVID-19 presentation of a febrile respiratory illness warrants confirmatory testing, but testing may not be available or produce a false-negative result, leading to an error we termed “Classic.” In the United States, efforts to develop and implement testing protocols are still evolving. There is wide local and regional variation in type and availability of tests, as well as accessibility of information regarding test performance characteristics or diagnostic yield.³ Test results that are false negatives or testing that is not performed can lead to delayed diagnosis of the disease, as well as continued spread.

Testing is similarly relevant when patients present with unusual or nonrespiratory symptoms. Both predominantly olfactory⁴ and gastrointestinal manifestations⁵ have now been described, and mysterious new associations, such as multisystem inflammatory syndromes, continue to emerge. A failure to recognize atypical presentations and associations, either because of testing problems or knowledge gaps, could lead to overlooking underlying COVID-19 diagnosis, an error we termed “Anomalous.”

Another error emerging in the pandemic is mislabeling patients who do not have COVID-19 as having the disease, particularly those with respiratory symptoms. This usually occurs in absence of testing in an overwhelmed health system with limited capacity to test or treat (eg, clinicians just assume it must be COVID-19 when the test is not available). This type of labeling error, called “Anchor,” introduces the risk of missing other respiratory infections such as bacterial sinusitis and pneumonia, as well as nonrespiratory conditions.

In patients with known COVID-19, a second underlying or concurrent condition may be missed, an error we termed “Secondary.” For instance, reports of coagulopathy-related pulmonary embolism⁶ and strokes in young patients with minimal symptoms⁷ have emerged just recently. Respiratory compromise may be mistakenly attributed to COVID-19 rather than looking for a new source of worsening, such as pulmonary embolism. Similarly, clinicians may not recognize subtle stroke symptoms in patients who were otherwise feeling well at home. Such cognitive errors will likely increase as it becomes harder for clinicians or health systems to keep up with new knowledge.

Collateral effects of the COVID-19 pandemic are also emerging. For instance, patients with symptoms of new acute conditions may be unwilling to visit acute care for evaluation because of infection risk, an error we termed “Acute Collateral.” Concerns are already being raised that patients with acute myocardial infarction⁸ and stroke⁹ are not coming in for evaluation. Similarly, there may be delays in diagnosis of important ambulatory conditions, including cancer,¹⁰ when appointments or elective procedures are canceled (“Chronic Collateral”). In the United Kingdom, referrals under the 2-week wait system–in which suspected cancer patients referred by general practitioners are seen within 2-weeks–fell by 70% over March to April, 2020.

Diagnosis of non–COVID-19 patients coming into the hospital may also be affected because of the understandably heightened state of attention to COVID-19 patients, capacity, and staffing issues, an error we termed “Strain.” Physicians, including surgeons, pediatricians, and radiologists, have been “redeployed” into acute care medical specialties. Cognitive errors increase when clinicians in new roles face unfamiliar situations and disease manifestations. Although these clinicians may be highly experienced previously, they may have insufficient skills and experience in their new roles and may not feel comfortable asking for guidance.¹¹

Lastly, clinicians are increasingly using intermediary mechanisms, such as PPE and telemedicine technologies, to interact with patients. This is new for both parties and could introduce new types of errors, which we termed “Unintended.” Furthermore, interactions mediated via telemedicine technologies or PPE, as well as PPE conservation measures such as reduced room entries and e-consultation, may reduce the ability of even well-trained clinicians to take effective histories, perform physical exams, and monitor symptoms. In fact, infection-prevention isolation has been shown to put patients at risk of preventable adverse events in hospitalized patients.¹²

There are many strategies that health systems could deploy to try to minimize these eight types of diagnostic errors. We organize mitigation strategies using the Safer Dx framework, which proposes sociotechnical approaches (ie, both technology and other systems-based approaches) to reduce diagnostic error.¹³

Technology for Cognitive Support

Up-to-date electronic decision support is needed to optimize test interpretation. Technology can also help scale and facilitate rapid adoption of standardized safety practices and protocols to address emerging risks areas. For instance, there are early efforts to create, implement, and disseminate smart algorithms to predict risks of non–COVID-19 diagnoses such as venous thromboembolism, patient transfer protocols on how best to reduce the burden at overstressed hospitals, protocols to triage rescheduling of elective procedures based on potential risk as determined from data in the electronic health record, new rules for creating outreach to patients who have missed appointments to prevent delays in their evaluation and diagnosis, and triage protocols and follow-up systems to optimize telemedicine.¹⁴

Optimized Workflow and Communication

When in-person contact is limited, specific practices (eg, providing patients with iPads, use of reflective listening, and use of optimal nonverbal communication strategies such as eye-contact) can still facilitate comprehensive discussions with patients and families about symptoms and encourage them to speak up if and when they have concerns.¹⁵ For patients reached through telemedicine, follow-up appointments and surveys should be done to ensure that symptoms and concerns have been addressed. For clinicians working in new clinical areas unfamiliar to them (eg, surgeons on medical floors, hospitalists in ICUs), buddy systems can pair these clinicians with more experienced clinicians to make it easier for them to ask for help. Visual aids, decision support, and reliable error-prevention resources can also be helpful.¹⁶

People-Focused Interventions

Some clinicians are used to practicing solo, but this is the time to start “diagnostic huddles” for discussion of challenging cases with symptoms that are unusual or not improving as expected or for determining whether anything has been missed. In addition to encouraging patients to use reliable digital tools for self-triage, outreach to patients and the public must also advise them (with the help of public health authorities and the media) to seek medical assistance for certain important conditions such as acute myocardial infarction and stroke.

Organizational Strategies

Fundamental safety strategies must be ensured. First, it is critical to have a strong safety culture in which staff feel empowered to speak up, ask questions or ask for help, and report concerns without fear of repercussions or judgement. Culture can take years to develop, but due to rapidly changing circumstances in a crisis, there are ways for healthcare leaders to create changes more quickly. In addition to having daily huddles, leaders should be visible and communicate clearly about the behaviors and norms they are supporting. In particular, frequent leadership rounding (either virtually or in person)—during which leaders ask questions and encourage discussions of concerns in a supportive way—can foster the kind of culture that is needed. All organizations should implement peer support, counseling, limits on hours worked, and other support strategies for all clinicians to minimize the fatigue, stress, and anxiety that can impair cognitive function.¹⁷

Organizations must also be able to identify these errors to help understand root causes and prioritize interventions.¹⁸ For example, streamlined reporting systems that use apps and hotlines could be developed quickly to ensure that clinicians and patients/families can easily report these errors. Electronic triggers can help detect specific situations indicative of error or delay (eg, patient not on precautions gets switched to precautions during a hospitalization; absence of follow-up on abnormal tests).¹⁹

Learning systems—both within and across hospitals—should continue to share diagnostic challenges, the most up-to-date information, and best practices/protocols, and identify opportunities for improvement together. Many hospitals are having virtual grand rounds, journals are rapidly sharing new information via open access, regional and national cross-organizational and multidisciplinary learning networks of various groups have emerged (such as networks of oncologists, infectious disease specialists, and hospitalists), and new and transparent communication channels have developed between state and local health departments, government leaders, health systems, and the public. These forums should discuss emerging knowledge on diagnosis and strategies for risk reduction, many of which will unfold over the next few months.

State/Federal Policies and Regulations

While there is progress, additional challenges with accessibility, accuracy and performance of testing should be addressed at a national level. Guidance is needed on which asymptomatic people should be tested, both within and outside hospitals. Standardized metrics should be developed to monitor diagnostic performance and outcomes and evaluate how COVID-19 diagnosis errors affect different demographics. For instance, black and Hispanic individuals are disproportionately represented in COVID-19 cases and deaths, so metrics could be further stratified by race and ethnicity to ensure that we can understand and eliminate inequities, such as lack of access to care or testing.²⁰

Clinicians must be provided with both cognitive and system support so they can do what they do best—diagnose and treat patients and save lives. Intermittent epidemic spikes based on location and season, including a potentially bigger spike of cases later this year, are now projected. Risks and recommendations discussed herein should therefore be rapidly shared to help redesign and strengthen the work system and protect patients from preventable diagnosis-related harm.

The views expressed in this article do not represent the views of the U.S. Department of Veterans Affairs or the United States government.

The coronavirus disease of 2019 (COVID-19) pandemic has strained the healthcare system by rapidly depleting multiple resources including hospital space, medications, ventilators, personal protective equipment (PPE), clinical revenue, and morale. One of the most essential at-risk resources is healthcare providers. Healthcare providers have been overwhelmed as hospital systems have experienced local surges in COVID-19 patients. Compounding this is the fact that providers are more likely to contract COVID-19, which could sideline portions of an already taxed workforce.

Multiple “surge” interventions have been planned or implemented to mitigate a current or anticipated dearth of physicians. Some institutions are reallocating subspecialists and surgeons to general ward and intensive care unit (ICU) roles, often with support from hospitalists and ICU physicians.¹ Others have used telemedicine to reduce personnel exposure and conserve PPE.² A novel and perhaps paradigm-shifting solution arose in March 2020 when several medical schools around the world announced they would graduate final year students early to allow them to join the workforce during the COVID-19 surge.^3-7 In the United States, fourth-year medical students at multiple institutions in cities such as New York, Boston, Phoenix, Tucson, Newark, Portland, and Bethesda were offered the opportunity to graduate in April rather than in May or June. The Liaison Committee on Medical Education stated that for students to graduate early, they must have already met all curricular requirements and be deemed ready by an evaluations and promotions committee.⁸ What these early graduates do with their “gap time” before residency is neither standardized nor prescribed. The Accreditation Council for Graduate Medical Education has discouraged individuals from joining their newly matched residency programs early.⁹ Some early graduates who wish to bolster the workforce have signed temporary training agreements with local healthcare systems to work for a 1- to 2-month period before moving on to their matched residency program. Some institutions have already been working with local and state officials to rapidly grant provisional temporary licenses for this purpose.¹⁰

Early medical school graduation in times of international crisis is not without precedent. When faced with physician shortages during World War II, the United States federal government urged medical colleges to graduate trainees in 3 years.¹¹ The national medical education milieu was different then, with standardized medical school training still crystalizing merely 30 years following the Flexner report. However, there was pressure from the federal government during World War II, whereas decisions around early graduation today are driven by institutional and local officials. While a few accelerated programs persist today, there has not been an urgent, unplanned early release of graduates to meet a public health need on such a large scale in recent history. The seasonal timing of the pandemic surge in the United States may have been a key factor in deciding to graduate students early. With a late winter and early spring peak, final year students are graduating only 2 to 3 months early. But what if another peak occurs in late summer or early fall, and some students are graduated even earlier? With which aspects of patient care would hospitalists trust these graduates, and with what level of supervision? Whether now or with a future COVID-19 peak, we describe how trust develops with learners and provide hospitalists with a framework for deliberate entrustment if and when they are asked to integrate early medical school graduates into their workforce.

The degree of supervision that is provided to a learner is linked to how much a supervisor trusts the learner, as well as the specific context. Trust has many forms, often depending on what type of information informs it. Presumptive trust is trust based on credentials, without any actual interaction with the learner.¹² Healthcare systems typically assume that medical school graduates are ready to perform intern-level tasks based on their medical degree. This presumptive trust may be bolstered by the assumption that a residency program director has vetted a learner’s credentials during the match process. On meeting a learner, we develop initial trust, which is based on first impressions and snap-judgment. Over time, presumptive and initial trust can be replaced by grounded trust, or trust based on demonstrated performance after prolonged experience with a learner. Under normal circumstances, supervisors use observations of learner performance in the clinical environment to develop grounded trust. With early graduates, especially those who sign temporary work agreements, the usual progression of trust may be compressed. Hospitalists may have less presumptive trust because these students graduated early and little time to develop grounded trust before integrating new graduates into patient care. How should hospitalists navigate supervision in this setting?

Missing a few months at the end of medical school likely does not significantly affect competence and, therefore, should not affect presumptive trust. The value of the fourth year of medical school has been questioned because, after fulfilling graduation requirements, students often spend significant amounts of time interviewing, traveling, taking electives with lighter workloads, or exploring nonclinical interests late in the year.¹³ More intense “subintern” rotations, which are important for the residency application process, occur earlier in the academic year. It is therefore reasonable to presume that most students graduating in April are not less prepared than those graduating in June.

Additionally, there is significant interlearner variability in rates of competence attainment.¹⁴ This means that there is no magic point in time at which students are fully ready for resident-­level responsibilities. Some students are likely competent to be interns without a fourth year at all, while others are still facing challenges in their development at the end of medical school. As Englander and Carraccio wrote, “The notion that every medical student across the nation has somehow achieved all the competencies necessary to start residency training on July 1 of their graduation year is magical thinking.”¹⁵ Since there is no universal, time-based finish line for competence, we should not be thrown by a slight change in the arbitrary line currently drawn in June. Whether students graduate in April or June, it remains true that some will be more ready than others.

With compressed timelines, hospitalists may default to initial trust, relying heavily on first impressions to determine how much supervision an early graduate requires. For example, a graduate who is extroverted, assertive, and articulate may give off an air of confidence, which could entice a supervising hospitalist to give a “longer leash” with higher-risk patient care tasks. It is easy to fall prey to the “confidence equals competence” heuristic, but this has been shown to be unreliable.¹⁶ Initial trust is influenced by both social biases (eg, gender, race, age) and cognitive biases (eg, halo effect) that have little or nothing to do with the actual abilities of learners. While initial trust and accompanying biases often develop unconsciously, it is important to reflect on how unfounded first impressions can influence trust and supervision decisions.

Hospitalists must be deliberate with entrustment decisions, especially in a pandemic environment. There are useful guides for making these decisions that can be used in a point-of-care manner.¹⁷ First, it is important to acknowledge that entrustment is based in part on the perceived trustworthiness of a person. Kennedy and colleagues have described four components of trustworthiness, all of which can be assessed by hospitalists in the moment of care delivery: (1) knowledge and skill (Does the trainee possess the requisite knowledge and skill to perform the task?), (2) conscientiousness (Does the trainee follow through on tasks? Are they thorough and dependable?), (3) discernment (Does the trainee recognize personal limitations and seek help when needed?), and (4) truthfulness (Does the trainee tell the truth?).¹⁷

Entrustment decisions also depend on the specific task being observed (eg, high risk vs low risk) and context (eg, severity of illness of the patient, acuity of the setting).¹⁸ Trust is linked with perceived risk and benefits.¹⁹ More entrustment (less supervision) may be given when perceived risk is low, such as prescribing acetaminophen on a stable patient or taking an initial history. Less entrustment (more supervision) may be given when perceived risk is high, such as with managing septic shock or inserting a central venous catheter. However, the duress of the COVID-19 pandemic may tilt the risk/benefit balance toward less-than-usual supervision if an early graduate is the only provider available for some higher-risk tasks. This underscores the importance of direct observation leading to grounded trust with progressively higher-risk tasks as dictated by the local pandemic environment.

As much as possible, trust should be determined based on direct observation, not fallible first impressions or inference. Supervisors often use inference when assuming that performance on one task reflects performance on others. For example, if learners are observed to be competent when interpreting electrocardiograms, one might infer they also know how to manage tachyarrhythmias. If they can manage tachyarrhythmias, one might infer they also know how to manage acute coronary syndrome. These inferences are not the way to build grounded trust because competence is task and context dependent.

Direct observation can include watching patient interactions, being present for procedures, think-alouds during didactics, cognitive autopsies, reviewing notes, and informal conversations. Being deliberate with direct observation and entrustment decision-making can be challenging because of the high cognitive load of caring for sick and complex patients, maintaining proper PPE practices, and simultaneously assessing an early graduate’s performance. However, maintaining a level of supervision that is appropriate for trainee competence is paramount for patient safety. It may be valuable to identify tasks needing to be performed by early graduates and using focused simulation to generate a significant number of observations over a short period of time. Trust should be gained once competence is observed, not inferred or assumed. Instead of “trust, but verify,” we should “observe, then trust.”

There is a moral obligation to patients to avoid placing trainees in situations for which they are ill prepared based on their current abilities. We must balance the risk that exists both in leaving early graduates on the sidelines (overprotecting them as learners) and in asking them to perform tasks for which they are not prepared (overextending them as a workforce). Focusing on grounded trust derived from direct observation of performance while also balancing the risks and benefits inherent in the local pandemic context can help hospitalists calibrate supervision to a level that helps extend the workforce in a time of crisis while maintaining patient safety.

The coronavirus disease of 2019 (COVID-19) pandemic has strained the healthcare system by rapidly depleting multiple resources including hospital space, medications, ventilators, personal protective equipment (PPE), clinical revenue, and morale. One of the most essential at-risk resources is healthcare providers. Healthcare providers have been overwhelmed as hospital systems have experienced local surges in COVID-19 patients. Compounding this is the fact that providers are more likely to contract COVID-19, which could sideline portions of an already taxed workforce.

Multiple “surge” interventions have been planned or implemented to mitigate a current or anticipated dearth of physicians. Some institutions are reallocating subspecialists and surgeons to general ward and intensive care unit (ICU) roles, often with support from hospitalists and ICU physicians.¹ Others have used telemedicine to reduce personnel exposure and conserve PPE.² A novel and perhaps paradigm-shifting solution arose in March 2020 when several medical schools around the world announced they would graduate final year students early to allow them to join the workforce during the COVID-19 surge.^3-7 In the United States, fourth-year medical students at multiple institutions in cities such as New York, Boston, Phoenix, Tucson, Newark, Portland, and Bethesda were offered the opportunity to graduate in April rather than in May or June. The Liaison Committee on Medical Education stated that for students to graduate early, they must have already met all curricular requirements and be deemed ready by an evaluations and promotions committee.⁸ What these early graduates do with their “gap time” before residency is neither standardized nor prescribed. The Accreditation Council for Graduate Medical Education has discouraged individuals from joining their newly matched residency programs early.⁹ Some early graduates who wish to bolster the workforce have signed temporary training agreements with local healthcare systems to work for a 1- to 2-month period before moving on to their matched residency program. Some institutions have already been working with local and state officials to rapidly grant provisional temporary licenses for this purpose.¹⁰

Early medical school graduation in times of international crisis is not without precedent. When faced with physician shortages during World War II, the United States federal government urged medical colleges to graduate trainees in 3 years.¹¹ The national medical education milieu was different then, with standardized medical school training still crystalizing merely 30 years following the Flexner report. However, there was pressure from the federal government during World War II, whereas decisions around early graduation today are driven by institutional and local officials. While a few accelerated programs persist today, there has not been an urgent, unplanned early release of graduates to meet a public health need on such a large scale in recent history. The seasonal timing of the pandemic surge in the United States may have been a key factor in deciding to graduate students early. With a late winter and early spring peak, final year students are graduating only 2 to 3 months early. But what if another peak occurs in late summer or early fall, and some students are graduated even earlier? With which aspects of patient care would hospitalists trust these graduates, and with what level of supervision? Whether now or with a future COVID-19 peak, we describe how trust develops with learners and provide hospitalists with a framework for deliberate entrustment if and when they are asked to integrate early medical school graduates into their workforce.

The degree of supervision that is provided to a learner is linked to how much a supervisor trusts the learner, as well as the specific context. Trust has many forms, often depending on what type of information informs it. Presumptive trust is trust based on credentials, without any actual interaction with the learner.¹² Healthcare systems typically assume that medical school graduates are ready to perform intern-level tasks based on their medical degree. This presumptive trust may be bolstered by the assumption that a residency program director has vetted a learner’s credentials during the match process. On meeting a learner, we develop initial trust, which is based on first impressions and snap-judgment. Over time, presumptive and initial trust can be replaced by grounded trust, or trust based on demonstrated performance after prolonged experience with a learner. Under normal circumstances, supervisors use observations of learner performance in the clinical environment to develop grounded trust. With early graduates, especially those who sign temporary work agreements, the usual progression of trust may be compressed. Hospitalists may have less presumptive trust because these students graduated early and little time to develop grounded trust before integrating new graduates into patient care. How should hospitalists navigate supervision in this setting?

Missing a few months at the end of medical school likely does not significantly affect competence and, therefore, should not affect presumptive trust. The value of the fourth year of medical school has been questioned because, after fulfilling graduation requirements, students often spend significant amounts of time interviewing, traveling, taking electives with lighter workloads, or exploring nonclinical interests late in the year.¹³ More intense “subintern” rotations, which are important for the residency application process, occur earlier in the academic year. It is therefore reasonable to presume that most students graduating in April are not less prepared than those graduating in June.

Additionally, there is significant interlearner variability in rates of competence attainment.¹⁴ This means that there is no magic point in time at which students are fully ready for resident-­level responsibilities. Some students are likely competent to be interns without a fourth year at all, while others are still facing challenges in their development at the end of medical school. As Englander and Carraccio wrote, “The notion that every medical student across the nation has somehow achieved all the competencies necessary to start residency training on July 1 of their graduation year is magical thinking.”¹⁵ Since there is no universal, time-based finish line for competence, we should not be thrown by a slight change in the arbitrary line currently drawn in June. Whether students graduate in April or June, it remains true that some will be more ready than others.

With compressed timelines, hospitalists may default to initial trust, relying heavily on first impressions to determine how much supervision an early graduate requires. For example, a graduate who is extroverted, assertive, and articulate may give off an air of confidence, which could entice a supervising hospitalist to give a “longer leash” with higher-risk patient care tasks. It is easy to fall prey to the “confidence equals competence” heuristic, but this has been shown to be unreliable.¹⁶ Initial trust is influenced by both social biases (eg, gender, race, age) and cognitive biases (eg, halo effect) that have little or nothing to do with the actual abilities of learners. While initial trust and accompanying biases often develop unconsciously, it is important to reflect on how unfounded first impressions can influence trust and supervision decisions.

Hospitalists must be deliberate with entrustment decisions, especially in a pandemic environment. There are useful guides for making these decisions that can be used in a point-of-care manner.¹⁷ First, it is important to acknowledge that entrustment is based in part on the perceived trustworthiness of a person. Kennedy and colleagues have described four components of trustworthiness, all of which can be assessed by hospitalists in the moment of care delivery: (1) knowledge and skill (Does the trainee possess the requisite knowledge and skill to perform the task?), (2) conscientiousness (Does the trainee follow through on tasks? Are they thorough and dependable?), (3) discernment (Does the trainee recognize personal limitations and seek help when needed?), and (4) truthfulness (Does the trainee tell the truth?).¹⁷

Entrustment decisions also depend on the specific task being observed (eg, high risk vs low risk) and context (eg, severity of illness of the patient, acuity of the setting).¹⁸ Trust is linked with perceived risk and benefits.¹⁹ More entrustment (less supervision) may be given when perceived risk is low, such as prescribing acetaminophen on a stable patient or taking an initial history. Less entrustment (more supervision) may be given when perceived risk is high, such as with managing septic shock or inserting a central venous catheter. However, the duress of the COVID-19 pandemic may tilt the risk/benefit balance toward less-than-usual supervision if an early graduate is the only provider available for some higher-risk tasks. This underscores the importance of direct observation leading to grounded trust with progressively higher-risk tasks as dictated by the local pandemic environment.

As much as possible, trust should be determined based on direct observation, not fallible first impressions or inference. Supervisors often use inference when assuming that performance on one task reflects performance on others. For example, if learners are observed to be competent when interpreting electrocardiograms, one might infer they also know how to manage tachyarrhythmias. If they can manage tachyarrhythmias, one might infer they also know how to manage acute coronary syndrome. These inferences are not the way to build grounded trust because competence is task and context dependent.

Direct observation can include watching patient interactions, being present for procedures, think-alouds during didactics, cognitive autopsies, reviewing notes, and informal conversations. Being deliberate with direct observation and entrustment decision-making can be challenging because of the high cognitive load of caring for sick and complex patients, maintaining proper PPE practices, and simultaneously assessing an early graduate’s performance. However, maintaining a level of supervision that is appropriate for trainee competence is paramount for patient safety. It may be valuable to identify tasks needing to be performed by early graduates and using focused simulation to generate a significant number of observations over a short period of time. Trust should be gained once competence is observed, not inferred or assumed. Instead of “trust, but verify,” we should “observe, then trust.”

There is a moral obligation to patients to avoid placing trainees in situations for which they are ill prepared based on their current abilities. We must balance the risk that exists both in leaving early graduates on the sidelines (overprotecting them as learners) and in asking them to perform tasks for which they are not prepared (overextending them as a workforce). Focusing on grounded trust derived from direct observation of performance while also balancing the risks and benefits inherent in the local pandemic context can help hospitalists calibrate supervision to a level that helps extend the workforce in a time of crisis while maintaining patient safety.

The coronavirus disease of 2019 (COVID-19) pandemic has strained the healthcare system by rapidly depleting multiple resources including hospital space, medications, ventilators, personal protective equipment (PPE), clinical revenue, and morale. One of the most essential at-risk resources is healthcare providers. Healthcare providers have been overwhelmed as hospital systems have experienced local surges in COVID-19 patients. Compounding this is the fact that providers are more likely to contract COVID-19, which could sideline portions of an already taxed workforce.

Multiple “surge” interventions have been planned or implemented to mitigate a current or anticipated dearth of physicians. Some institutions are reallocating subspecialists and surgeons to general ward and intensive care unit (ICU) roles, often with support from hospitalists and ICU physicians.¹ Others have used telemedicine to reduce personnel exposure and conserve PPE.² A novel and perhaps paradigm-shifting solution arose in March 2020 when several medical schools around the world announced they would graduate final year students early to allow them to join the workforce during the COVID-19 surge.^3-7 In the United States, fourth-year medical students at multiple institutions in cities such as New York, Boston, Phoenix, Tucson, Newark, Portland, and Bethesda were offered the opportunity to graduate in April rather than in May or June. The Liaison Committee on Medical Education stated that for students to graduate early, they must have already met all curricular requirements and be deemed ready by an evaluations and promotions committee.⁸ What these early graduates do with their “gap time” before residency is neither standardized nor prescribed. The Accreditation Council for Graduate Medical Education has discouraged individuals from joining their newly matched residency programs early.⁹ Some early graduates who wish to bolster the workforce have signed temporary training agreements with local healthcare systems to work for a 1- to 2-month period before moving on to their matched residency program. Some institutions have already been working with local and state officials to rapidly grant provisional temporary licenses for this purpose.¹⁰

Early medical school graduation in times of international crisis is not without precedent. When faced with physician shortages during World War II, the United States federal government urged medical colleges to graduate trainees in 3 years.¹¹ The national medical education milieu was different then, with standardized medical school training still crystalizing merely 30 years following the Flexner report. However, there was pressure from the federal government during World War II, whereas decisions around early graduation today are driven by institutional and local officials. While a few accelerated programs persist today, there has not been an urgent, unplanned early release of graduates to meet a public health need on such a large scale in recent history. The seasonal timing of the pandemic surge in the United States may have been a key factor in deciding to graduate students early. With a late winter and early spring peak, final year students are graduating only 2 to 3 months early. But what if another peak occurs in late summer or early fall, and some students are graduated even earlier? With which aspects of patient care would hospitalists trust these graduates, and with what level of supervision? Whether now or with a future COVID-19 peak, we describe how trust develops with learners and provide hospitalists with a framework for deliberate entrustment if and when they are asked to integrate early medical school graduates into their workforce.

The degree of supervision that is provided to a learner is linked to how much a supervisor trusts the learner, as well as the specific context. Trust has many forms, often depending on what type of information informs it. Presumptive trust is trust based on credentials, without any actual interaction with the learner.¹² Healthcare systems typically assume that medical school graduates are ready to perform intern-level tasks based on their medical degree. This presumptive trust may be bolstered by the assumption that a residency program director has vetted a learner’s credentials during the match process. On meeting a learner, we develop initial trust, which is based on first impressions and snap-judgment. Over time, presumptive and initial trust can be replaced by grounded trust, or trust based on demonstrated performance after prolonged experience with a learner. Under normal circumstances, supervisors use observations of learner performance in the clinical environment to develop grounded trust. With early graduates, especially those who sign temporary work agreements, the usual progression of trust may be compressed. Hospitalists may have less presumptive trust because these students graduated early and little time to develop grounded trust before integrating new graduates into patient care. How should hospitalists navigate supervision in this setting?

Missing a few months at the end of medical school likely does not significantly affect competence and, therefore, should not affect presumptive trust. The value of the fourth year of medical school has been questioned because, after fulfilling graduation requirements, students often spend significant amounts of time interviewing, traveling, taking electives with lighter workloads, or exploring nonclinical interests late in the year.¹³ More intense “subintern” rotations, which are important for the residency application process, occur earlier in the academic year. It is therefore reasonable to presume that most students graduating in April are not less prepared than those graduating in June.

Additionally, there is significant interlearner variability in rates of competence attainment.¹⁴ This means that there is no magic point in time at which students are fully ready for resident-­level responsibilities. Some students are likely competent to be interns without a fourth year at all, while others are still facing challenges in their development at the end of medical school. As Englander and Carraccio wrote, “The notion that every medical student across the nation has somehow achieved all the competencies necessary to start residency training on July 1 of their graduation year is magical thinking.”¹⁵ Since there is no universal, time-based finish line for competence, we should not be thrown by a slight change in the arbitrary line currently drawn in June. Whether students graduate in April or June, it remains true that some will be more ready than others.

With compressed timelines, hospitalists may default to initial trust, relying heavily on first impressions to determine how much supervision an early graduate requires. For example, a graduate who is extroverted, assertive, and articulate may give off an air of confidence, which could entice a supervising hospitalist to give a “longer leash” with higher-risk patient care tasks. It is easy to fall prey to the “confidence equals competence” heuristic, but this has been shown to be unreliable.¹⁶ Initial trust is influenced by both social biases (eg, gender, race, age) and cognitive biases (eg, halo effect) that have little or nothing to do with the actual abilities of learners. While initial trust and accompanying biases often develop unconsciously, it is important to reflect on how unfounded first impressions can influence trust and supervision decisions.

Hospitalists must be deliberate with entrustment decisions, especially in a pandemic environment. There are useful guides for making these decisions that can be used in a point-of-care manner.¹⁷ First, it is important to acknowledge that entrustment is based in part on the perceived trustworthiness of a person. Kennedy and colleagues have described four components of trustworthiness, all of which can be assessed by hospitalists in the moment of care delivery: (1) knowledge and skill (Does the trainee possess the requisite knowledge and skill to perform the task?), (2) conscientiousness (Does the trainee follow through on tasks? Are they thorough and dependable?), (3) discernment (Does the trainee recognize personal limitations and seek help when needed?), and (4) truthfulness (Does the trainee tell the truth?).¹⁷

Entrustment decisions also depend on the specific task being observed (eg, high risk vs low risk) and context (eg, severity of illness of the patient, acuity of the setting).¹⁸ Trust is linked with perceived risk and benefits.¹⁹ More entrustment (less supervision) may be given when perceived risk is low, such as prescribing acetaminophen on a stable patient or taking an initial history. Less entrustment (more supervision) may be given when perceived risk is high, such as with managing septic shock or inserting a central venous catheter. However, the duress of the COVID-19 pandemic may tilt the risk/benefit balance toward less-than-usual supervision if an early graduate is the only provider available for some higher-risk tasks. This underscores the importance of direct observation leading to grounded trust with progressively higher-risk tasks as dictated by the local pandemic environment.

As much as possible, trust should be determined based on direct observation, not fallible first impressions or inference. Supervisors often use inference when assuming that performance on one task reflects performance on others. For example, if learners are observed to be competent when interpreting electrocardiograms, one might infer they also know how to manage tachyarrhythmias. If they can manage tachyarrhythmias, one might infer they also know how to manage acute coronary syndrome. These inferences are not the way to build grounded trust because competence is task and context dependent.

Direct observation can include watching patient interactions, being present for procedures, think-alouds during didactics, cognitive autopsies, reviewing notes, and informal conversations. Being deliberate with direct observation and entrustment decision-making can be challenging because of the high cognitive load of caring for sick and complex patients, maintaining proper PPE practices, and simultaneously assessing an early graduate’s performance. However, maintaining a level of supervision that is appropriate for trainee competence is paramount for patient safety. It may be valuable to identify tasks needing to be performed by early graduates and using focused simulation to generate a significant number of observations over a short period of time. Trust should be gained once competence is observed, not inferred or assumed. Instead of “trust, but verify,” we should “observe, then trust.”

There is a moral obligation to patients to avoid placing trainees in situations for which they are ill prepared based on their current abilities. We must balance the risk that exists both in leaving early graduates on the sidelines (overprotecting them as learners) and in asking them to perform tasks for which they are not prepared (overextending them as a workforce). Focusing on grounded trust derived from direct observation of performance while also balancing the risks and benefits inherent in the local pandemic context can help hospitalists calibrate supervision to a level that helps extend the workforce in a time of crisis while maintaining patient safety.