Charlie M Wray, DO, MS

For years, professional athletes have used simulation-based training (SBT), a combination of virtual and experiential learning that aims to optimize technical skills, teamwork, and communication.¹ In SBT, critical plays and skills are first watched on video or reviewed on a chalkboard, and then run in the presence of a coach who offers immediate feedback to the player. The hope is that the individual will then be able to perfectly execute that play or scenario when it is game time. While SBT is a developing tool in medical education—allowing learners to practice important clinical skills prior to practicing in the higher-stakes clinical environment—an important question remains: what training can go virtual and what needs to stay in person?

In this issue, Carter et al² present a single-site, telesimulation curriculum that addresses consult request and handoff communication using SBT. Due to the COVID-19 pandemic, the authors converted an in-person intern bootcamp into a virtual, Zoom^®-based workshop and compared assessments and evaluations to the previous year’s (2019) in-person bootcamp. Compared to the in-person class, the telesimulation-based cohort were equally or better trained in the consult request portion of the workshop. However, participants were significantly less likely to perform the assessed handoff skills optimally, with only a quarter (26%) appropriately prioritizing patients and less than half (49%) providing an appropriate amount of information in the patient summary. Additionally, postworkshop surveys found that SBT participants were more satisfied with their performance in both the consult request and handoff scenarios and felt more prepared (99% vs 91%) to perform handoffs in clinical practice compared to the previous year’s in-person cohort.

We focus on this work as it explores the role that SBT or virtual training could have in hospital communication and patient safety training. While previous work has highlighted that technical and procedural skills often lend themselves to in-person adaptation (eg, point-of-care ultrasound), this work suggests that nontechnical skills training could be adapted to the virtual environment. Hospitalists and internal medicine trainees perform a myriad of nontechnical activities, such as end-of-life discussions, obtaining informed consent, providing peer-to-peer feedback, and leading multidisciplinary teams. Activities like these, which require no hands-on interactions, may be well-suited for simulation or virtual-based training.³

However, we make this suggestion with some caution. In Carter et al’s study,² while we assumed that telesimulation would work for the handoff portion of the workshop, interestingly, the telesimulation-based cohort performed worse than the interns who participated in the previous year’s in-person training while simultaneously and paradoxically reporting that they felt more prepared. The authors offer several possible explanations, including alterations in the assessment checklist and a shift in the facilitators from peer observers to faculty hospitalists. We suspect that differences in the participants’ experiences prior to the bootcamp may also be at play. Given the onset of the pandemic during their final year in undergraduate training, many in this intern cohort were likely removed from their fourth-year clinical clerkships,⁴ taking from them pivotal opportunities to hone and refine this skill set prior to starting their graduate medical education.

As telesimulation and other virtual care educational opportunities continue to evolve, we must ensure that such training does not sacrifice quality for ease and satisfaction. As the authors’ findings show, simply replicating an in-person curriculum in a virtual environment does not ensure equivalence for all skill sets. We remain cautiously optimistic that as we adjust to a postpandemic world, more SBT and virtual-based educational interventions will allow medical trainees to be ready to perform come game time.

For years, professional athletes have used simulation-based training (SBT), a combination of virtual and experiential learning that aims to optimize technical skills, teamwork, and communication.¹ In SBT, critical plays and skills are first watched on video or reviewed on a chalkboard, and then run in the presence of a coach who offers immediate feedback to the player. The hope is that the individual will then be able to perfectly execute that play or scenario when it is game time. While SBT is a developing tool in medical education—allowing learners to practice important clinical skills prior to practicing in the higher-stakes clinical environment—an important question remains: what training can go virtual and what needs to stay in person?

In this issue, Carter et al² present a single-site, telesimulation curriculum that addresses consult request and handoff communication using SBT. Due to the COVID-19 pandemic, the authors converted an in-person intern bootcamp into a virtual, Zoom^®-based workshop and compared assessments and evaluations to the previous year’s (2019) in-person bootcamp. Compared to the in-person class, the telesimulation-based cohort were equally or better trained in the consult request portion of the workshop. However, participants were significantly less likely to perform the assessed handoff skills optimally, with only a quarter (26%) appropriately prioritizing patients and less than half (49%) providing an appropriate amount of information in the patient summary. Additionally, postworkshop surveys found that SBT participants were more satisfied with their performance in both the consult request and handoff scenarios and felt more prepared (99% vs 91%) to perform handoffs in clinical practice compared to the previous year’s in-person cohort.

We focus on this work as it explores the role that SBT or virtual training could have in hospital communication and patient safety training. While previous work has highlighted that technical and procedural skills often lend themselves to in-person adaptation (eg, point-of-care ultrasound), this work suggests that nontechnical skills training could be adapted to the virtual environment. Hospitalists and internal medicine trainees perform a myriad of nontechnical activities, such as end-of-life discussions, obtaining informed consent, providing peer-to-peer feedback, and leading multidisciplinary teams. Activities like these, which require no hands-on interactions, may be well-suited for simulation or virtual-based training.³

However, we make this suggestion with some caution. In Carter et al’s study,² while we assumed that telesimulation would work for the handoff portion of the workshop, interestingly, the telesimulation-based cohort performed worse than the interns who participated in the previous year’s in-person training while simultaneously and paradoxically reporting that they felt more prepared. The authors offer several possible explanations, including alterations in the assessment checklist and a shift in the facilitators from peer observers to faculty hospitalists. We suspect that differences in the participants’ experiences prior to the bootcamp may also be at play. Given the onset of the pandemic during their final year in undergraduate training, many in this intern cohort were likely removed from their fourth-year clinical clerkships,⁴ taking from them pivotal opportunities to hone and refine this skill set prior to starting their graduate medical education.

As telesimulation and other virtual care educational opportunities continue to evolve, we must ensure that such training does not sacrifice quality for ease and satisfaction. As the authors’ findings show, simply replicating an in-person curriculum in a virtual environment does not ensure equivalence for all skill sets. We remain cautiously optimistic that as we adjust to a postpandemic world, more SBT and virtual-based educational interventions will allow medical trainees to be ready to perform come game time.

For years, professional athletes have used simulation-based training (SBT), a combination of virtual and experiential learning that aims to optimize technical skills, teamwork, and communication.¹ In SBT, critical plays and skills are first watched on video or reviewed on a chalkboard, and then run in the presence of a coach who offers immediate feedback to the player. The hope is that the individual will then be able to perfectly execute that play or scenario when it is game time. While SBT is a developing tool in medical education—allowing learners to practice important clinical skills prior to practicing in the higher-stakes clinical environment—an important question remains: what training can go virtual and what needs to stay in person?

In this issue, Carter et al² present a single-site, telesimulation curriculum that addresses consult request and handoff communication using SBT. Due to the COVID-19 pandemic, the authors converted an in-person intern bootcamp into a virtual, Zoom^®-based workshop and compared assessments and evaluations to the previous year’s (2019) in-person bootcamp. Compared to the in-person class, the telesimulation-based cohort were equally or better trained in the consult request portion of the workshop. However, participants were significantly less likely to perform the assessed handoff skills optimally, with only a quarter (26%) appropriately prioritizing patients and less than half (49%) providing an appropriate amount of information in the patient summary. Additionally, postworkshop surveys found that SBT participants were more satisfied with their performance in both the consult request and handoff scenarios and felt more prepared (99% vs 91%) to perform handoffs in clinical practice compared to the previous year’s in-person cohort.

We focus on this work as it explores the role that SBT or virtual training could have in hospital communication and patient safety training. While previous work has highlighted that technical and procedural skills often lend themselves to in-person adaptation (eg, point-of-care ultrasound), this work suggests that nontechnical skills training could be adapted to the virtual environment. Hospitalists and internal medicine trainees perform a myriad of nontechnical activities, such as end-of-life discussions, obtaining informed consent, providing peer-to-peer feedback, and leading multidisciplinary teams. Activities like these, which require no hands-on interactions, may be well-suited for simulation or virtual-based training.³

However, we make this suggestion with some caution. In Carter et al’s study,² while we assumed that telesimulation would work for the handoff portion of the workshop, interestingly, the telesimulation-based cohort performed worse than the interns who participated in the previous year’s in-person training while simultaneously and paradoxically reporting that they felt more prepared. The authors offer several possible explanations, including alterations in the assessment checklist and a shift in the facilitators from peer observers to faculty hospitalists. We suspect that differences in the participants’ experiences prior to the bootcamp may also be at play. Given the onset of the pandemic during their final year in undergraduate training, many in this intern cohort were likely removed from their fourth-year clinical clerkships,⁴ taking from them pivotal opportunities to hone and refine this skill set prior to starting their graduate medical education.

As telesimulation and other virtual care educational opportunities continue to evolve, we must ensure that such training does not sacrifice quality for ease and satisfaction. As the authors’ findings show, simply replicating an in-person curriculum in a virtual environment does not ensure equivalence for all skill sets. We remain cautiously optimistic that as we adjust to a postpandemic world, more SBT and virtual-based educational interventions will allow medical trainees to be ready to perform come game time.

Horwitz and Detsky¹ provide readers with a personal, experientially based primer on how healthcare professionals can more effectively engage on Twitter. As experienced physicians, researchers, and active social media users, the authors outline pragmatic and specific recommendations on how to engage misinformation and add value to social media discourse. We applaud the authors for offering best-practice approaches that are valuable to newcomers as well as seasoned social media users. In highlighting that social media is merely a modern tool for engagement and discussion, the authors underscore the time-held idea that only when a tool is used effectively will it yield the desired outcome. As a medical journal that regularly uses social media as a tool for outreach and dissemination, we could not agree more with the authors’ assertion.

Since 2015, the Journal of Hospital Medicine (JHM) has used social media to engage its readership and extend the impact of the work published in its pages. Like Horwitz and Detsky, JHM has developed insights and experience in how medical journals, organizations, institutions, and other academic programs can use social media effectively. Because of our experience in this area, we are often asked how to build a successful and sustainable social media presence. Here, we share five primary lessons on how to use social media as a tool to disseminate, connect, and engage.

As the flagship journal for the field of hospital medicine, we seek to disseminate the ideas and research that will inform health policy, optimize healthcare delivery, and improve patient outcomes while also building and sustaining an online community for professional engagement and growth. Our social media goals provide direction on how to interact, allow us to focus attention on what is important, and motivate our growth in this area. Simply put, we believe that using social media without defined goals would be like sailing a ship without a rudder.

As your organization establishes its goals, it is important to consider with whom you want to connect. Knowing your audience will allow you to better tailor the content you deliver through social media. For instance, we understand that as a journal focused on hospital medicine, our audience consists of busy clinicians, researchers, and medical educators who are trying to efficiently gather the most up-to-date information in our field. Recognizing this, we produce (and make available for download) Visual Abstracts and publish them on Twitter to help our followers assimilate information from new studies quickly and easily.² Moreover, we recognize that our followers are interested in how to use social media in their professional lives and have published several articles in this topic area.^3-5

We have found that having multiple individuals on our social media team has led to greater creativity and thoughtfulness on how we engage our readership. Our teams span generations, clinical experience, institutions, and cultural backgrounds. This intentional approach has allowed for diversity in thoughts and opinions and has helped shape the JHM social media message. Additionally, we have not only formalized editorial roles through the creation of Digital Media Editor positions, but we have also created the JHM Digital Media Fellowship, a training program and development pipeline for those interested in cultivating organization-based social media experiences and skill sets.⁶

Many organizations believe that successful social media outreach means creating an account and posting content when convenient. Experience has taught us that daily postings and regular engagement will build your brand as a regular and reliable source of information for your followers. Additionally, while many academic journals and organizations only occasionally post material and rarely interact with their followers, we have found that engaging and facilitating conversations through our monthly Twitter discussion (#JHMChat) has established a community, created opportunities for professional networking, and further disseminated the work published in JHM.⁷ As an academic journal or organization entering this field, recognize the product for which people follow you and deliver that product on a consistent basis.

It will only be a matter of time before your organization makes a misstep on social media. Instead of hiding, we recommend stepping into that tension and owning the mistake. For example, we recently published an article that contained a culturally offensive term. As a journal, we reflected on our error and took concrete steps to correct it. Further, we shared our thoughts with our followers to ensure transparency.⁸ Moving forward, we have inserted specific stopgaps in our editorial review process to avoid such missteps in the future.

Although every organization will have different goals and reasons for engaging on social media, we believe these central tenets will help optimize the use of this platform. Although we have established specific objectives for our engagement on social media, we believe Horwitz and Detsky¹ put it best when they note that, at the end of the day, our ultimate goal is in “…promoting knowledge and science in a way that helps us all live healthier and happier lives."

Horwitz and Detsky¹ provide readers with a personal, experientially based primer on how healthcare professionals can more effectively engage on Twitter. As experienced physicians, researchers, and active social media users, the authors outline pragmatic and specific recommendations on how to engage misinformation and add value to social media discourse. We applaud the authors for offering best-practice approaches that are valuable to newcomers as well as seasoned social media users. In highlighting that social media is merely a modern tool for engagement and discussion, the authors underscore the time-held idea that only when a tool is used effectively will it yield the desired outcome. As a medical journal that regularly uses social media as a tool for outreach and dissemination, we could not agree more with the authors’ assertion.

Since 2015, the Journal of Hospital Medicine (JHM) has used social media to engage its readership and extend the impact of the work published in its pages. Like Horwitz and Detsky, JHM has developed insights and experience in how medical journals, organizations, institutions, and other academic programs can use social media effectively. Because of our experience in this area, we are often asked how to build a successful and sustainable social media presence. Here, we share five primary lessons on how to use social media as a tool to disseminate, connect, and engage.

As the flagship journal for the field of hospital medicine, we seek to disseminate the ideas and research that will inform health policy, optimize healthcare delivery, and improve patient outcomes while also building and sustaining an online community for professional engagement and growth. Our social media goals provide direction on how to interact, allow us to focus attention on what is important, and motivate our growth in this area. Simply put, we believe that using social media without defined goals would be like sailing a ship without a rudder.

As your organization establishes its goals, it is important to consider with whom you want to connect. Knowing your audience will allow you to better tailor the content you deliver through social media. For instance, we understand that as a journal focused on hospital medicine, our audience consists of busy clinicians, researchers, and medical educators who are trying to efficiently gather the most up-to-date information in our field. Recognizing this, we produce (and make available for download) Visual Abstracts and publish them on Twitter to help our followers assimilate information from new studies quickly and easily.² Moreover, we recognize that our followers are interested in how to use social media in their professional lives and have published several articles in this topic area.^3-5

We have found that having multiple individuals on our social media team has led to greater creativity and thoughtfulness on how we engage our readership. Our teams span generations, clinical experience, institutions, and cultural backgrounds. This intentional approach has allowed for diversity in thoughts and opinions and has helped shape the JHM social media message. Additionally, we have not only formalized editorial roles through the creation of Digital Media Editor positions, but we have also created the JHM Digital Media Fellowship, a training program and development pipeline for those interested in cultivating organization-based social media experiences and skill sets.⁶

Many organizations believe that successful social media outreach means creating an account and posting content when convenient. Experience has taught us that daily postings and regular engagement will build your brand as a regular and reliable source of information for your followers. Additionally, while many academic journals and organizations only occasionally post material and rarely interact with their followers, we have found that engaging and facilitating conversations through our monthly Twitter discussion (#JHMChat) has established a community, created opportunities for professional networking, and further disseminated the work published in JHM.⁷ As an academic journal or organization entering this field, recognize the product for which people follow you and deliver that product on a consistent basis.

It will only be a matter of time before your organization makes a misstep on social media. Instead of hiding, we recommend stepping into that tension and owning the mistake. For example, we recently published an article that contained a culturally offensive term. As a journal, we reflected on our error and took concrete steps to correct it. Further, we shared our thoughts with our followers to ensure transparency.⁸ Moving forward, we have inserted specific stopgaps in our editorial review process to avoid such missteps in the future.

Although every organization will have different goals and reasons for engaging on social media, we believe these central tenets will help optimize the use of this platform. Although we have established specific objectives for our engagement on social media, we believe Horwitz and Detsky¹ put it best when they note that, at the end of the day, our ultimate goal is in “…promoting knowledge and science in a way that helps us all live healthier and happier lives."

Horwitz and Detsky¹ provide readers with a personal, experientially based primer on how healthcare professionals can more effectively engage on Twitter. As experienced physicians, researchers, and active social media users, the authors outline pragmatic and specific recommendations on how to engage misinformation and add value to social media discourse. We applaud the authors for offering best-practice approaches that are valuable to newcomers as well as seasoned social media users. In highlighting that social media is merely a modern tool for engagement and discussion, the authors underscore the time-held idea that only when a tool is used effectively will it yield the desired outcome. As a medical journal that regularly uses social media as a tool for outreach and dissemination, we could not agree more with the authors’ assertion.

Since 2015, the Journal of Hospital Medicine (JHM) has used social media to engage its readership and extend the impact of the work published in its pages. Like Horwitz and Detsky, JHM has developed insights and experience in how medical journals, organizations, institutions, and other academic programs can use social media effectively. Because of our experience in this area, we are often asked how to build a successful and sustainable social media presence. Here, we share five primary lessons on how to use social media as a tool to disseminate, connect, and engage.

As the flagship journal for the field of hospital medicine, we seek to disseminate the ideas and research that will inform health policy, optimize healthcare delivery, and improve patient outcomes while also building and sustaining an online community for professional engagement and growth. Our social media goals provide direction on how to interact, allow us to focus attention on what is important, and motivate our growth in this area. Simply put, we believe that using social media without defined goals would be like sailing a ship without a rudder.

As your organization establishes its goals, it is important to consider with whom you want to connect. Knowing your audience will allow you to better tailor the content you deliver through social media. For instance, we understand that as a journal focused on hospital medicine, our audience consists of busy clinicians, researchers, and medical educators who are trying to efficiently gather the most up-to-date information in our field. Recognizing this, we produce (and make available for download) Visual Abstracts and publish them on Twitter to help our followers assimilate information from new studies quickly and easily.² Moreover, we recognize that our followers are interested in how to use social media in their professional lives and have published several articles in this topic area.^3-5

We have found that having multiple individuals on our social media team has led to greater creativity and thoughtfulness on how we engage our readership. Our teams span generations, clinical experience, institutions, and cultural backgrounds. This intentional approach has allowed for diversity in thoughts and opinions and has helped shape the JHM social media message. Additionally, we have not only formalized editorial roles through the creation of Digital Media Editor positions, but we have also created the JHM Digital Media Fellowship, a training program and development pipeline for those interested in cultivating organization-based social media experiences and skill sets.⁶

Many organizations believe that successful social media outreach means creating an account and posting content when convenient. Experience has taught us that daily postings and regular engagement will build your brand as a regular and reliable source of information for your followers. Additionally, while many academic journals and organizations only occasionally post material and rarely interact with their followers, we have found that engaging and facilitating conversations through our monthly Twitter discussion (#JHMChat) has established a community, created opportunities for professional networking, and further disseminated the work published in JHM.⁷ As an academic journal or organization entering this field, recognize the product for which people follow you and deliver that product on a consistent basis.

It will only be a matter of time before your organization makes a misstep on social media. Instead of hiding, we recommend stepping into that tension and owning the mistake. For example, we recently published an article that contained a culturally offensive term. As a journal, we reflected on our error and took concrete steps to correct it. Further, we shared our thoughts with our followers to ensure transparency.⁸ Moving forward, we have inserted specific stopgaps in our editorial review process to avoid such missteps in the future.

Although every organization will have different goals and reasons for engaging on social media, we believe these central tenets will help optimize the use of this platform. Although we have established specific objectives for our engagement on social media, we believe Horwitz and Detsky¹ put it best when they note that, at the end of the day, our ultimate goal is in “…promoting knowledge and science in a way that helps us all live healthier and happier lives."

Is this patient a good candidate? In medicine, we subconsciously answer this question for every clinical decision we make. Occasionally, though, a clinical scenario is so complex that it cannot or should not be answered by a single individual. One example is the decision on whether a patient should receive an organ transplant. In this situation, a multidisciplinary committee weighs the complex ethical, clinical, and financial implications of the decision before coming to a verdict. Together, team members discuss the risks and benefits of each patient’s candidacy and, in a united fashion, decide the best course of care. For hospitalists, a far more common question occurs every day and is similarly fraught with multifaceted implications: Is my patient a good candidate for a skilled nursing facility (SNF)? We often rely on a single individual to make the final call, but should we instead be leveraging the expertise of other care team members to assist with this decision?

In this issue, Boyle et al¹ describe the implementation of a multidisciplinary team consisting of physicians, case managers, social workers, physical and occupational therapists, and home-health representatives that reviewed all patients with an expected discharge to a SNF. Case managers or social workers began the process by referring eligible patients to the committee for review. If deemed appropriate, the committee discussed each case and reached a consensus recommendation as to whether a SNF was an appropriate discharge destination. The investigators used a matched, preintervention sample as a comparison group, with a primary outcome of total discharges to SNFs, and secondary outcomes consisting of readmissions, time to readmission, and median length of stay. The authors observed a 49.7% relative reduction in total SNF discharges (25.5% of preintervention patients discharged to a SNF vs 12.8% postintervention), as well as a 66.9% relative reduction in new SNF discharges. Despite the significant reduction in SNF utilization, no differences were noted in readmissions, time to readmission, or readmission length of stay.

While this study was performed during the COVID-19 pandemic, several characteristics make its findings applicable beyond this period. First, the structure and workflow of the team are extensively detailed and make the intervention easily generalizable to most hospitals. Second, while not specifically examined, the outcome of SNF reduction likely corresponds to an increase in the patient’s time at home—an important patient-centered target for most posthospitalization plans.² Finally, the intervention used existing infrastructure and individuals, and did not require new resources to improve patient care, which increases the feasibility of implementation at other institutions.

These findings also reveal potential overutilization of SNFs in the discharge process. On average, a typical SNF stay costs the health system more than $11,000.³ A simple intervention could lead to substantial savings for individuals and the healthcare system. With a nearly 50% reduction in SNF use, understanding why patients who were eligible to go home were ultimately discharged to a SNF will be a crucial question to answer. Are there barriers to patient or family education? Is there a perceived safety difference between a SNF and home for nonskilled nursing needs? Additionally, care should be taken to ensure that decreases in SNF utilization do not disproportionately affect certain populations. Further work should assess the performance of similar models in a non-COVID era and among multiple institutions to verify potential scalability and generalizability.

Like organ transplant committees, Boyle et al’s multidisciplinary approach to reduce SNF discharges had to include thoughtful and intentional decisions. Perhaps it is time we use this same model to transplant patients back into their homes as safely and efficiently as possible.

Is this patient a good candidate? In medicine, we subconsciously answer this question for every clinical decision we make. Occasionally, though, a clinical scenario is so complex that it cannot or should not be answered by a single individual. One example is the decision on whether a patient should receive an organ transplant. In this situation, a multidisciplinary committee weighs the complex ethical, clinical, and financial implications of the decision before coming to a verdict. Together, team members discuss the risks and benefits of each patient’s candidacy and, in a united fashion, decide the best course of care. For hospitalists, a far more common question occurs every day and is similarly fraught with multifaceted implications: Is my patient a good candidate for a skilled nursing facility (SNF)? We often rely on a single individual to make the final call, but should we instead be leveraging the expertise of other care team members to assist with this decision?

In this issue, Boyle et al¹ describe the implementation of a multidisciplinary team consisting of physicians, case managers, social workers, physical and occupational therapists, and home-health representatives that reviewed all patients with an expected discharge to a SNF. Case managers or social workers began the process by referring eligible patients to the committee for review. If deemed appropriate, the committee discussed each case and reached a consensus recommendation as to whether a SNF was an appropriate discharge destination. The investigators used a matched, preintervention sample as a comparison group, with a primary outcome of total discharges to SNFs, and secondary outcomes consisting of readmissions, time to readmission, and median length of stay. The authors observed a 49.7% relative reduction in total SNF discharges (25.5% of preintervention patients discharged to a SNF vs 12.8% postintervention), as well as a 66.9% relative reduction in new SNF discharges. Despite the significant reduction in SNF utilization, no differences were noted in readmissions, time to readmission, or readmission length of stay.

While this study was performed during the COVID-19 pandemic, several characteristics make its findings applicable beyond this period. First, the structure and workflow of the team are extensively detailed and make the intervention easily generalizable to most hospitals. Second, while not specifically examined, the outcome of SNF reduction likely corresponds to an increase in the patient’s time at home—an important patient-centered target for most posthospitalization plans.² Finally, the intervention used existing infrastructure and individuals, and did not require new resources to improve patient care, which increases the feasibility of implementation at other institutions.

These findings also reveal potential overutilization of SNFs in the discharge process. On average, a typical SNF stay costs the health system more than $11,000.³ A simple intervention could lead to substantial savings for individuals and the healthcare system. With a nearly 50% reduction in SNF use, understanding why patients who were eligible to go home were ultimately discharged to a SNF will be a crucial question to answer. Are there barriers to patient or family education? Is there a perceived safety difference between a SNF and home for nonskilled nursing needs? Additionally, care should be taken to ensure that decreases in SNF utilization do not disproportionately affect certain populations. Further work should assess the performance of similar models in a non-COVID era and among multiple institutions to verify potential scalability and generalizability.

Like organ transplant committees, Boyle et al’s multidisciplinary approach to reduce SNF discharges had to include thoughtful and intentional decisions. Perhaps it is time we use this same model to transplant patients back into their homes as safely and efficiently as possible.

Is this patient a good candidate? In medicine, we subconsciously answer this question for every clinical decision we make. Occasionally, though, a clinical scenario is so complex that it cannot or should not be answered by a single individual. One example is the decision on whether a patient should receive an organ transplant. In this situation, a multidisciplinary committee weighs the complex ethical, clinical, and financial implications of the decision before coming to a verdict. Together, team members discuss the risks and benefits of each patient’s candidacy and, in a united fashion, decide the best course of care. For hospitalists, a far more common question occurs every day and is similarly fraught with multifaceted implications: Is my patient a good candidate for a skilled nursing facility (SNF)? We often rely on a single individual to make the final call, but should we instead be leveraging the expertise of other care team members to assist with this decision?

In this issue, Boyle et al¹ describe the implementation of a multidisciplinary team consisting of physicians, case managers, social workers, physical and occupational therapists, and home-health representatives that reviewed all patients with an expected discharge to a SNF. Case managers or social workers began the process by referring eligible patients to the committee for review. If deemed appropriate, the committee discussed each case and reached a consensus recommendation as to whether a SNF was an appropriate discharge destination. The investigators used a matched, preintervention sample as a comparison group, with a primary outcome of total discharges to SNFs, and secondary outcomes consisting of readmissions, time to readmission, and median length of stay. The authors observed a 49.7% relative reduction in total SNF discharges (25.5% of preintervention patients discharged to a SNF vs 12.8% postintervention), as well as a 66.9% relative reduction in new SNF discharges. Despite the significant reduction in SNF utilization, no differences were noted in readmissions, time to readmission, or readmission length of stay.

While this study was performed during the COVID-19 pandemic, several characteristics make its findings applicable beyond this period. First, the structure and workflow of the team are extensively detailed and make the intervention easily generalizable to most hospitals. Second, while not specifically examined, the outcome of SNF reduction likely corresponds to an increase in the patient’s time at home—an important patient-centered target for most posthospitalization plans.² Finally, the intervention used existing infrastructure and individuals, and did not require new resources to improve patient care, which increases the feasibility of implementation at other institutions.

These findings also reveal potential overutilization of SNFs in the discharge process. On average, a typical SNF stay costs the health system more than $11,000.³ A simple intervention could lead to substantial savings for individuals and the healthcare system. With a nearly 50% reduction in SNF use, understanding why patients who were eligible to go home were ultimately discharged to a SNF will be a crucial question to answer. Are there barriers to patient or family education? Is there a perceived safety difference between a SNF and home for nonskilled nursing needs? Additionally, care should be taken to ensure that decreases in SNF utilization do not disproportionately affect certain populations. Further work should assess the performance of similar models in a non-COVID era and among multiple institutions to verify potential scalability and generalizability.

Like organ transplant committees, Boyle et al’s multidisciplinary approach to reduce SNF discharges had to include thoughtful and intentional decisions. Perhaps it is time we use this same model to transplant patients back into their homes as safely and efficiently as possible.

We recently published an article in our Leadership & Professional Development series titled “Tribalism: The Good, the Bad, and the Future.” Despite pre- and post-acceptance manuscript review and discussion by a diverse and thoughtful team of editors, we did not appreciate how particular language in this article would be hurtful to some communities. We also promoted the article using the hashtag “tribalism” in a journal tweet. Shortly after we posted the tweet, several readers on social media reached out with constructive feedback on the prejudicial nature of this terminology. Within hours of receiving this feedback, our editorial team met to better understand our error, and we made the decision to immediately retract the manuscript. We also deleted the tweet and issued an apology referencing a screenshot of the original tweet.^1,2 We have republished the original article with appropriate language.³ Tweets promoting the new article will incorporate this new language.

From this experience, we learned that the words “tribe” and “tribalism” have no consistent meaning, are associated with negative historical and cultural assumptions, and can promote misleading stereotypes.⁴ The term “tribe” became popular as a colonial construct to describe forms of social organization considered ”uncivilized” or ”primitive.“⁵ In using the term “tribe” to describe members of medical communities, we ignored the complex and dynamic identities of Native American, African, and other Indigenous Peoples and the history of their oppression.

The intent of the original article was to highlight how being part of a distinct medical discipline, such as hospital medicine or emergency medicine, conferred benefits, such as shared identity and social support structure, and caution how this group identity could also lead to nonconstructive partisan behaviors that might not best serve our patients. We recognize that other words more accurately convey our intent and do not cause harm. We used “tribe” when we meant “group,” “discipline,” or “specialty.” We used “tribalism” when we meant “siloed” or “factional.”

This misstep underscores how, even with the best intentions and diverse teams, microaggressions can happen. We accept responsibility for this mistake, and we will continue to do the work of respecting and advocating for all members of our community. To minimize the likelihood of future errors, we are developing a systematic process to identify language within manuscripts accepted for publication that may be racist, sexist, ableist, homophobic, or otherwise harmful. As we embrace a growth mindset, we vow to remain transparent, responsive, and welcoming of feedback. We are grateful to our readers for helping us learn.

We recently published an article in our Leadership & Professional Development series titled “Tribalism: The Good, the Bad, and the Future.” Despite pre- and post-acceptance manuscript review and discussion by a diverse and thoughtful team of editors, we did not appreciate how particular language in this article would be hurtful to some communities. We also promoted the article using the hashtag “tribalism” in a journal tweet. Shortly after we posted the tweet, several readers on social media reached out with constructive feedback on the prejudicial nature of this terminology. Within hours of receiving this feedback, our editorial team met to better understand our error, and we made the decision to immediately retract the manuscript. We also deleted the tweet and issued an apology referencing a screenshot of the original tweet.^1,2 We have republished the original article with appropriate language.³ Tweets promoting the new article will incorporate this new language.

From this experience, we learned that the words “tribe” and “tribalism” have no consistent meaning, are associated with negative historical and cultural assumptions, and can promote misleading stereotypes.⁴ The term “tribe” became popular as a colonial construct to describe forms of social organization considered ”uncivilized” or ”primitive.“⁵ In using the term “tribe” to describe members of medical communities, we ignored the complex and dynamic identities of Native American, African, and other Indigenous Peoples and the history of their oppression.

The intent of the original article was to highlight how being part of a distinct medical discipline, such as hospital medicine or emergency medicine, conferred benefits, such as shared identity and social support structure, and caution how this group identity could also lead to nonconstructive partisan behaviors that might not best serve our patients. We recognize that other words more accurately convey our intent and do not cause harm. We used “tribe” when we meant “group,” “discipline,” or “specialty.” We used “tribalism” when we meant “siloed” or “factional.”

This misstep underscores how, even with the best intentions and diverse teams, microaggressions can happen. We accept responsibility for this mistake, and we will continue to do the work of respecting and advocating for all members of our community. To minimize the likelihood of future errors, we are developing a systematic process to identify language within manuscripts accepted for publication that may be racist, sexist, ableist, homophobic, or otherwise harmful. As we embrace a growth mindset, we vow to remain transparent, responsive, and welcoming of feedback. We are grateful to our readers for helping us learn.

We recently published an article in our Leadership & Professional Development series titled “Tribalism: The Good, the Bad, and the Future.” Despite pre- and post-acceptance manuscript review and discussion by a diverse and thoughtful team of editors, we did not appreciate how particular language in this article would be hurtful to some communities. We also promoted the article using the hashtag “tribalism” in a journal tweet. Shortly after we posted the tweet, several readers on social media reached out with constructive feedback on the prejudicial nature of this terminology. Within hours of receiving this feedback, our editorial team met to better understand our error, and we made the decision to immediately retract the manuscript. We also deleted the tweet and issued an apology referencing a screenshot of the original tweet.^1,2 We have republished the original article with appropriate language.³ Tweets promoting the new article will incorporate this new language.

From this experience, we learned that the words “tribe” and “tribalism” have no consistent meaning, are associated with negative historical and cultural assumptions, and can promote misleading stereotypes.⁴ The term “tribe” became popular as a colonial construct to describe forms of social organization considered ”uncivilized” or ”primitive.“⁵ In using the term “tribe” to describe members of medical communities, we ignored the complex and dynamic identities of Native American, African, and other Indigenous Peoples and the history of their oppression.

The intent of the original article was to highlight how being part of a distinct medical discipline, such as hospital medicine or emergency medicine, conferred benefits, such as shared identity and social support structure, and caution how this group identity could also lead to nonconstructive partisan behaviors that might not best serve our patients. We recognize that other words more accurately convey our intent and do not cause harm. We used “tribe” when we meant “group,” “discipline,” or “specialty.” We used “tribalism” when we meant “siloed” or “factional.”

This misstep underscores how, even with the best intentions and diverse teams, microaggressions can happen. We accept responsibility for this mistake, and we will continue to do the work of respecting and advocating for all members of our community. To minimize the likelihood of future errors, we are developing a systematic process to identify language within manuscripts accepted for publication that may be racist, sexist, ableist, homophobic, or otherwise harmful. As we embrace a growth mindset, we vow to remain transparent, responsive, and welcoming of feedback. We are grateful to our readers for helping us learn.

Professional upheavals caused by the coronavirus disease 2019 (COVID-19) pandemic have affected the academic productivity of many physicians. This is due in part to rapid changes in clinical care and medical education: physician-researchers have been redeployed to frontline clinical care; clinician-educators have been forced to rapidly transition in-person curricula to virtual platforms; and primary care physicians and subspecialists have been forced to transition to telehealth-based practices. In addition to these changes in clinical and educational responsibilities, the COVID-19 pandemic has substantially altered the personal lives of physicians. During the height of the pandemic, clinicians simultaneously wrestled with a lack of available childcare, unexpected home-schooling responsibilities, decreased income, and many other COVID-19-related stresses.¹ Additionally, the ever-present “second pandemic” of structural racism, persistent health disparities, and racial inequity has further increased the personal and professional demands facing academic faculty.²

In particular, the pandemic has placed personal and professional pressure on female and minority faculty members. In spite of these pressures, however, the academic promotions process still requires rigid accounting of scholarly productivity. As the focus of academic practices has shifted to support clinical care during the pandemic, scholarly productivity has suffered for clinicians on the frontline. As a result, academic clinical faculty have expressed significant stress and concerns about failing to meet benchmarks for promotion (eg, publications, curricula development, national presentations). To counter these shifts (and the inherent inequity that they create for female clinicians and for men and women who are Black, Indigenous, and/or of color), academic institutions should not only recognize the effects the COVID-19 pandemic has had on faculty, but also adopt immediate solutions to more equitably account for such disruptions to academic portfolios. In this paper, we explore populations whose career trajectories are most at-risk and propose a framework to capture novel and nontraditional contributions while also acknowledging the rapid changes the COVID-19 pandemic has brought to academic medicine.

Even before the COVID-19 pandemic, physician mothers, underrepresented racial/ethnic minority groups, and junior faculty were most at-risk for career disruptions. The closure of daycare facilities and schools and shift to online learning resulting from the pandemic, along with the common challenges of parenting, have taken a significant toll on the lives of working parents. Because women tend to carry a disproportionate share of childcare and household responsibilities, these changes have inequitably leveraged themselves as a “mommy tax” on working women.^3,4

As underrepresented medicine faculty (particularly Black, Hispanic, Latino, and Native American clinicians) comprise only 8% of the academic medical workforce,they currently face a variety of personal and professional challenges.⁵ This is especially true for Black and Latinx physicians who have been experiencing an increased COVID-19 burden in their communities, while concurrently fighting entrenched structural racism and police violence. In academia, these challenges have worsened because of the “minority tax”—the toll of often uncompensated extra responsibilities (time or money) placed on minority faculty in the name of achieving diversity. The unintended consequences of these responsibilities result in having fewer mentors,⁶ caring for underserved populations,⁷ and performing more clinical care⁸ than non-underrepresented minority faculty. Because minority faculty are unlikely to be in leadership positions, it is reasonable to conclude they have been shouldering heavier clinical obligations and facing greater career disruption of scholarly work due to the COVID-19 pandemic.

Junior faculty (eg, instructors and assistant professors) also remain professionally vulnerable during the COVID-19 pandemic. Because junior faculty are often more clinically focused and less likely to hold leadership positions than senior faculty, they are more likely to have assumed frontline clinical positions, which come at the expense of academic work. Junior faculty are also at a critical building phase in their academic career—a time when they benefit from the opportunity to share their scholarly work and network at conferences. Unfortunately, many conferences have been canceled or moved to a virtual platform. Given that some institutions may be freezing academic funding for conferences due to budgetary shortfalls from the pandemic, junior faculty may be particularly at risk if they are not able to present their work. In addition, junior faculty often face disproportionate struggles at home, trying to balance demands of work and caring for young children. Considering the unique needs of each of these groups, it is especially important to consider intersectionality, or the compounded issues for individuals who exist in multiple disproportionately affected groups (eg, a Black female junior faculty member who is also a mother).

The typical format of a professional curriculum vitae (CV) at most academic institutions does not allow one to document potential disruptions or novel contributions, including those that occurred during the COVID-19 pandemic. As a group of academic clinicians, educators, and researchers whose careers have been affected by the pandemic, we created a COVID-19 CV matrix, a potential framework to serve as a supplement for faculty. In this matrix, faculty members may document their contributions, disruptions that affected their work, and caregiving responsibilities during this time period, while also providing a rubric for promotions and tenure committees to equitably evaluate the pandemic period on an academic CV. Our COVID-19 CV matrix consists of six domains: (1) clinical care, (2) research, (3) education, (4) service, (5) advocacy/media, and (6) social media. These domains encompass traditional and nontraditional contributions made by healthcare professionals during the pandemic (Table). This matrix broadens the ability of both faculty and institutions to determine the actual impact of individuals during the pandemic.

arora10280120e_t1.jpg

ACCOUNT FOR YOUR (NEW) IMPACT

Throughout the COVID-19 pandemic, academic faculty have been innovative, contributing in novel ways not routinely captured by promotions committees—eg, the digital health researcher who now directs the telemedicine response for their institution and the health disparities researcher who now leads daily webinar sessions on structural racism to medical students. Other novel contributions include advancing COVID-19 innovations and engaging in media and community advocacy (eg, organizing large-scale donations of equipment and funds to support organizations in need). While such nontraditional contributions may not have been readily captured or thought “CV worthy” in the past, faculty should now account for them. More importantly, promotions committees need to recognize that these pivots or alterations in career paths are not signals of professional failure, but rather evidence of a shifting landscape and the respective response of the individual. Furthermore, because these pivots often help fulfill an institutional mission, they are impactful.

ACKNOWLEDGE THE DISRUPTION

It is important for promotions and tenure committees to recognize the impact and disruption COVID-19 has had on traditional academic work, acknowledging the time and energy required for a faculty member to make needed work adjustments. This enables a leader to better assess how a faculty member’s academic portfolio has been affected. For example, researchers have had to halt studies, medical educators have had to redevelop and transition curricula to virtual platforms, and physicians have had to discontinue clinician quality improvement initiatives due to competing hospital priorities. Faculty members who document such unintentional alterations in their academic career path can explain to their institution how they have continued to positively influence their field and the community during the pandemic. This approach is analogous to the current model of accounting for clinical time when judging faculty members’ contributions in scholarly achievement.

The COVID-19 CV matrix has the potential to be annotated to explain the burden of one’s personal situation, which is often “invisible” in the professional environment. For example, many physicians have had to assume additional childcare responsibilities, tend to sick family members, friends, and even themselves. It is also possible that a faculty member has a partner who is also an essential worker, one who had to self-isolate due to COVID-19 exposure or illness, or who has been working overtime due to high patient volumes.

INSTITUTIONAL RESPONSE

How can institutions respond to the altered academic landscape caused by the COVID-19 pandemic? Promotions committees typically have two main tools at their disposal: adjusting the tenure clock or the benchmarks. Extending the period of time available to qualify for tenure is commonplace in the “publish-or-perish” academic tracks of university research professors. Clock adjustments are typically granted to faculty following the birth of a child or for other specific family- or health-related hardships, in accordance with the Family and Medical Leave Act. Unfortunately, tenure-clock extensions for female faculty members can exacerbate gender inequity: Data on tenure-clock extensions show a higher rate of tenure granted to male faculty compared to female faculty.⁹ For this reason, it is also important to explore adjustments or modifications to benchmark criteria. This could be accomplished by broadening the criteria for promotion, recognizing that impact occurs in many forms, thereby enabling meeting a benchmark. It can also occur by examining the trajectory of an individual within a promotion pathway before it was disrupted to determine impact. To avoid exacerbating social and gender inequities within academia, institutions should use these professional levers and create new ones to provide parity and equality across the promotional playing field. While the CV matrix openly acknowledges the disruptions and tangents the COVID-19 pandemic has had on academic careers, it remains important for academic institutions to recognize these disruptions and innovate the manner in which they acknowledge scholarly contributions.

While academic rigidity and known social taxes (minority and mommy taxes) are particularly problematic in the current climate, these issues have always been at play in evaluating academic success. Improved documentation of novel contributions, disruptions, caregiving, and other challenges can enable more holistic and timely professional advancement for all faculty, regardless of their sex, race, ethnicity, or social background. Ultimately, we hope this framework initiates further conversations among academic institutions on how to define productivity in an age where journal impact factor or number of publications is not the fullest measure of one’s impact in their field.

Professional upheavals caused by the coronavirus disease 2019 (COVID-19) pandemic have affected the academic productivity of many physicians. This is due in part to rapid changes in clinical care and medical education: physician-researchers have been redeployed to frontline clinical care; clinician-educators have been forced to rapidly transition in-person curricula to virtual platforms; and primary care physicians and subspecialists have been forced to transition to telehealth-based practices. In addition to these changes in clinical and educational responsibilities, the COVID-19 pandemic has substantially altered the personal lives of physicians. During the height of the pandemic, clinicians simultaneously wrestled with a lack of available childcare, unexpected home-schooling responsibilities, decreased income, and many other COVID-19-related stresses.¹ Additionally, the ever-present “second pandemic” of structural racism, persistent health disparities, and racial inequity has further increased the personal and professional demands facing academic faculty.²

In particular, the pandemic has placed personal and professional pressure on female and minority faculty members. In spite of these pressures, however, the academic promotions process still requires rigid accounting of scholarly productivity. As the focus of academic practices has shifted to support clinical care during the pandemic, scholarly productivity has suffered for clinicians on the frontline. As a result, academic clinical faculty have expressed significant stress and concerns about failing to meet benchmarks for promotion (eg, publications, curricula development, national presentations). To counter these shifts (and the inherent inequity that they create for female clinicians and for men and women who are Black, Indigenous, and/or of color), academic institutions should not only recognize the effects the COVID-19 pandemic has had on faculty, but also adopt immediate solutions to more equitably account for such disruptions to academic portfolios. In this paper, we explore populations whose career trajectories are most at-risk and propose a framework to capture novel and nontraditional contributions while also acknowledging the rapid changes the COVID-19 pandemic has brought to academic medicine.

Even before the COVID-19 pandemic, physician mothers, underrepresented racial/ethnic minority groups, and junior faculty were most at-risk for career disruptions. The closure of daycare facilities and schools and shift to online learning resulting from the pandemic, along with the common challenges of parenting, have taken a significant toll on the lives of working parents. Because women tend to carry a disproportionate share of childcare and household responsibilities, these changes have inequitably leveraged themselves as a “mommy tax” on working women.^3,4

As underrepresented medicine faculty (particularly Black, Hispanic, Latino, and Native American clinicians) comprise only 8% of the academic medical workforce,they currently face a variety of personal and professional challenges.⁵ This is especially true for Black and Latinx physicians who have been experiencing an increased COVID-19 burden in their communities, while concurrently fighting entrenched structural racism and police violence. In academia, these challenges have worsened because of the “minority tax”—the toll of often uncompensated extra responsibilities (time or money) placed on minority faculty in the name of achieving diversity. The unintended consequences of these responsibilities result in having fewer mentors,⁶ caring for underserved populations,⁷ and performing more clinical care⁸ than non-underrepresented minority faculty. Because minority faculty are unlikely to be in leadership positions, it is reasonable to conclude they have been shouldering heavier clinical obligations and facing greater career disruption of scholarly work due to the COVID-19 pandemic.

Junior faculty (eg, instructors and assistant professors) also remain professionally vulnerable during the COVID-19 pandemic. Because junior faculty are often more clinically focused and less likely to hold leadership positions than senior faculty, they are more likely to have assumed frontline clinical positions, which come at the expense of academic work. Junior faculty are also at a critical building phase in their academic career—a time when they benefit from the opportunity to share their scholarly work and network at conferences. Unfortunately, many conferences have been canceled or moved to a virtual platform. Given that some institutions may be freezing academic funding for conferences due to budgetary shortfalls from the pandemic, junior faculty may be particularly at risk if they are not able to present their work. In addition, junior faculty often face disproportionate struggles at home, trying to balance demands of work and caring for young children. Considering the unique needs of each of these groups, it is especially important to consider intersectionality, or the compounded issues for individuals who exist in multiple disproportionately affected groups (eg, a Black female junior faculty member who is also a mother).

The typical format of a professional curriculum vitae (CV) at most academic institutions does not allow one to document potential disruptions or novel contributions, including those that occurred during the COVID-19 pandemic. As a group of academic clinicians, educators, and researchers whose careers have been affected by the pandemic, we created a COVID-19 CV matrix, a potential framework to serve as a supplement for faculty. In this matrix, faculty members may document their contributions, disruptions that affected their work, and caregiving responsibilities during this time period, while also providing a rubric for promotions and tenure committees to equitably evaluate the pandemic period on an academic CV. Our COVID-19 CV matrix consists of six domains: (1) clinical care, (2) research, (3) education, (4) service, (5) advocacy/media, and (6) social media. These domains encompass traditional and nontraditional contributions made by healthcare professionals during the pandemic (Table). This matrix broadens the ability of both faculty and institutions to determine the actual impact of individuals during the pandemic.

arora10280120e_t1.jpg

ACCOUNT FOR YOUR (NEW) IMPACT

Throughout the COVID-19 pandemic, academic faculty have been innovative, contributing in novel ways not routinely captured by promotions committees—eg, the digital health researcher who now directs the telemedicine response for their institution and the health disparities researcher who now leads daily webinar sessions on structural racism to medical students. Other novel contributions include advancing COVID-19 innovations and engaging in media and community advocacy (eg, organizing large-scale donations of equipment and funds to support organizations in need). While such nontraditional contributions may not have been readily captured or thought “CV worthy” in the past, faculty should now account for them. More importantly, promotions committees need to recognize that these pivots or alterations in career paths are not signals of professional failure, but rather evidence of a shifting landscape and the respective response of the individual. Furthermore, because these pivots often help fulfill an institutional mission, they are impactful.

ACKNOWLEDGE THE DISRUPTION

It is important for promotions and tenure committees to recognize the impact and disruption COVID-19 has had on traditional academic work, acknowledging the time and energy required for a faculty member to make needed work adjustments. This enables a leader to better assess how a faculty member’s academic portfolio has been affected. For example, researchers have had to halt studies, medical educators have had to redevelop and transition curricula to virtual platforms, and physicians have had to discontinue clinician quality improvement initiatives due to competing hospital priorities. Faculty members who document such unintentional alterations in their academic career path can explain to their institution how they have continued to positively influence their field and the community during the pandemic. This approach is analogous to the current model of accounting for clinical time when judging faculty members’ contributions in scholarly achievement.

The COVID-19 CV matrix has the potential to be annotated to explain the burden of one’s personal situation, which is often “invisible” in the professional environment. For example, many physicians have had to assume additional childcare responsibilities, tend to sick family members, friends, and even themselves. It is also possible that a faculty member has a partner who is also an essential worker, one who had to self-isolate due to COVID-19 exposure or illness, or who has been working overtime due to high patient volumes.

INSTITUTIONAL RESPONSE

How can institutions respond to the altered academic landscape caused by the COVID-19 pandemic? Promotions committees typically have two main tools at their disposal: adjusting the tenure clock or the benchmarks. Extending the period of time available to qualify for tenure is commonplace in the “publish-or-perish” academic tracks of university research professors. Clock adjustments are typically granted to faculty following the birth of a child or for other specific family- or health-related hardships, in accordance with the Family and Medical Leave Act. Unfortunately, tenure-clock extensions for female faculty members can exacerbate gender inequity: Data on tenure-clock extensions show a higher rate of tenure granted to male faculty compared to female faculty.⁹ For this reason, it is also important to explore adjustments or modifications to benchmark criteria. This could be accomplished by broadening the criteria for promotion, recognizing that impact occurs in many forms, thereby enabling meeting a benchmark. It can also occur by examining the trajectory of an individual within a promotion pathway before it was disrupted to determine impact. To avoid exacerbating social and gender inequities within academia, institutions should use these professional levers and create new ones to provide parity and equality across the promotional playing field. While the CV matrix openly acknowledges the disruptions and tangents the COVID-19 pandemic has had on academic careers, it remains important for academic institutions to recognize these disruptions and innovate the manner in which they acknowledge scholarly contributions.

While academic rigidity and known social taxes (minority and mommy taxes) are particularly problematic in the current climate, these issues have always been at play in evaluating academic success. Improved documentation of novel contributions, disruptions, caregiving, and other challenges can enable more holistic and timely professional advancement for all faculty, regardless of their sex, race, ethnicity, or social background. Ultimately, we hope this framework initiates further conversations among academic institutions on how to define productivity in an age where journal impact factor or number of publications is not the fullest measure of one’s impact in their field.

Professional upheavals caused by the coronavirus disease 2019 (COVID-19) pandemic have affected the academic productivity of many physicians. This is due in part to rapid changes in clinical care and medical education: physician-researchers have been redeployed to frontline clinical care; clinician-educators have been forced to rapidly transition in-person curricula to virtual platforms; and primary care physicians and subspecialists have been forced to transition to telehealth-based practices. In addition to these changes in clinical and educational responsibilities, the COVID-19 pandemic has substantially altered the personal lives of physicians. During the height of the pandemic, clinicians simultaneously wrestled with a lack of available childcare, unexpected home-schooling responsibilities, decreased income, and many other COVID-19-related stresses.¹ Additionally, the ever-present “second pandemic” of structural racism, persistent health disparities, and racial inequity has further increased the personal and professional demands facing academic faculty.²

In particular, the pandemic has placed personal and professional pressure on female and minority faculty members. In spite of these pressures, however, the academic promotions process still requires rigid accounting of scholarly productivity. As the focus of academic practices has shifted to support clinical care during the pandemic, scholarly productivity has suffered for clinicians on the frontline. As a result, academic clinical faculty have expressed significant stress and concerns about failing to meet benchmarks for promotion (eg, publications, curricula development, national presentations). To counter these shifts (and the inherent inequity that they create for female clinicians and for men and women who are Black, Indigenous, and/or of color), academic institutions should not only recognize the effects the COVID-19 pandemic has had on faculty, but also adopt immediate solutions to more equitably account for such disruptions to academic portfolios. In this paper, we explore populations whose career trajectories are most at-risk and propose a framework to capture novel and nontraditional contributions while also acknowledging the rapid changes the COVID-19 pandemic has brought to academic medicine.

Even before the COVID-19 pandemic, physician mothers, underrepresented racial/ethnic minority groups, and junior faculty were most at-risk for career disruptions. The closure of daycare facilities and schools and shift to online learning resulting from the pandemic, along with the common challenges of parenting, have taken a significant toll on the lives of working parents. Because women tend to carry a disproportionate share of childcare and household responsibilities, these changes have inequitably leveraged themselves as a “mommy tax” on working women.^3,4

As underrepresented medicine faculty (particularly Black, Hispanic, Latino, and Native American clinicians) comprise only 8% of the academic medical workforce,they currently face a variety of personal and professional challenges.⁵ This is especially true for Black and Latinx physicians who have been experiencing an increased COVID-19 burden in their communities, while concurrently fighting entrenched structural racism and police violence. In academia, these challenges have worsened because of the “minority tax”—the toll of often uncompensated extra responsibilities (time or money) placed on minority faculty in the name of achieving diversity. The unintended consequences of these responsibilities result in having fewer mentors,⁶ caring for underserved populations,⁷ and performing more clinical care⁸ than non-underrepresented minority faculty. Because minority faculty are unlikely to be in leadership positions, it is reasonable to conclude they have been shouldering heavier clinical obligations and facing greater career disruption of scholarly work due to the COVID-19 pandemic.

Junior faculty (eg, instructors and assistant professors) also remain professionally vulnerable during the COVID-19 pandemic. Because junior faculty are often more clinically focused and less likely to hold leadership positions than senior faculty, they are more likely to have assumed frontline clinical positions, which come at the expense of academic work. Junior faculty are also at a critical building phase in their academic career—a time when they benefit from the opportunity to share their scholarly work and network at conferences. Unfortunately, many conferences have been canceled or moved to a virtual platform. Given that some institutions may be freezing academic funding for conferences due to budgetary shortfalls from the pandemic, junior faculty may be particularly at risk if they are not able to present their work. In addition, junior faculty often face disproportionate struggles at home, trying to balance demands of work and caring for young children. Considering the unique needs of each of these groups, it is especially important to consider intersectionality, or the compounded issues for individuals who exist in multiple disproportionately affected groups (eg, a Black female junior faculty member who is also a mother).

The typical format of a professional curriculum vitae (CV) at most academic institutions does not allow one to document potential disruptions or novel contributions, including those that occurred during the COVID-19 pandemic. As a group of academic clinicians, educators, and researchers whose careers have been affected by the pandemic, we created a COVID-19 CV matrix, a potential framework to serve as a supplement for faculty. In this matrix, faculty members may document their contributions, disruptions that affected their work, and caregiving responsibilities during this time period, while also providing a rubric for promotions and tenure committees to equitably evaluate the pandemic period on an academic CV. Our COVID-19 CV matrix consists of six domains: (1) clinical care, (2) research, (3) education, (4) service, (5) advocacy/media, and (6) social media. These domains encompass traditional and nontraditional contributions made by healthcare professionals during the pandemic (Table). This matrix broadens the ability of both faculty and institutions to determine the actual impact of individuals during the pandemic.

arora10280120e_t1.jpg

ACCOUNT FOR YOUR (NEW) IMPACT

Throughout the COVID-19 pandemic, academic faculty have been innovative, contributing in novel ways not routinely captured by promotions committees—eg, the digital health researcher who now directs the telemedicine response for their institution and the health disparities researcher who now leads daily webinar sessions on structural racism to medical students. Other novel contributions include advancing COVID-19 innovations and engaging in media and community advocacy (eg, organizing large-scale donations of equipment and funds to support organizations in need). While such nontraditional contributions may not have been readily captured or thought “CV worthy” in the past, faculty should now account for them. More importantly, promotions committees need to recognize that these pivots or alterations in career paths are not signals of professional failure, but rather evidence of a shifting landscape and the respective response of the individual. Furthermore, because these pivots often help fulfill an institutional mission, they are impactful.

ACKNOWLEDGE THE DISRUPTION

It is important for promotions and tenure committees to recognize the impact and disruption COVID-19 has had on traditional academic work, acknowledging the time and energy required for a faculty member to make needed work adjustments. This enables a leader to better assess how a faculty member’s academic portfolio has been affected. For example, researchers have had to halt studies, medical educators have had to redevelop and transition curricula to virtual platforms, and physicians have had to discontinue clinician quality improvement initiatives due to competing hospital priorities. Faculty members who document such unintentional alterations in their academic career path can explain to their institution how they have continued to positively influence their field and the community during the pandemic. This approach is analogous to the current model of accounting for clinical time when judging faculty members’ contributions in scholarly achievement.

The COVID-19 CV matrix has the potential to be annotated to explain the burden of one’s personal situation, which is often “invisible” in the professional environment. For example, many physicians have had to assume additional childcare responsibilities, tend to sick family members, friends, and even themselves. It is also possible that a faculty member has a partner who is also an essential worker, one who had to self-isolate due to COVID-19 exposure or illness, or who has been working overtime due to high patient volumes.

INSTITUTIONAL RESPONSE

How can institutions respond to the altered academic landscape caused by the COVID-19 pandemic? Promotions committees typically have two main tools at their disposal: adjusting the tenure clock or the benchmarks. Extending the period of time available to qualify for tenure is commonplace in the “publish-or-perish” academic tracks of university research professors. Clock adjustments are typically granted to faculty following the birth of a child or for other specific family- or health-related hardships, in accordance with the Family and Medical Leave Act. Unfortunately, tenure-clock extensions for female faculty members can exacerbate gender inequity: Data on tenure-clock extensions show a higher rate of tenure granted to male faculty compared to female faculty.⁹ For this reason, it is also important to explore adjustments or modifications to benchmark criteria. This could be accomplished by broadening the criteria for promotion, recognizing that impact occurs in many forms, thereby enabling meeting a benchmark. It can also occur by examining the trajectory of an individual within a promotion pathway before it was disrupted to determine impact. To avoid exacerbating social and gender inequities within academia, institutions should use these professional levers and create new ones to provide parity and equality across the promotional playing field. While the CV matrix openly acknowledges the disruptions and tangents the COVID-19 pandemic has had on academic careers, it remains important for academic institutions to recognize these disruptions and innovate the manner in which they acknowledge scholarly contributions.

While academic rigidity and known social taxes (minority and mommy taxes) are particularly problematic in the current climate, these issues have always been at play in evaluating academic success. Improved documentation of novel contributions, disruptions, caregiving, and other challenges can enable more holistic and timely professional advancement for all faculty, regardless of their sex, race, ethnicity, or social background. Ultimately, we hope this framework initiates further conversations among academic institutions on how to define productivity in an age where journal impact factor or number of publications is not the fullest measure of one’s impact in their field.

“Our goals can only be reached through the vehicle of a plan. There is no other route to success.”

—Pablo Picasso

Whether it be a research manuscript, quality improvement (QI) initiative, or educational curriculum, busy clinicians often struggle getting projects past the idea stage. Barriers to completion, such as a busy clinical schedule or lack of experience and mentorship, are well known. Importantly, these projects serve as “academic currency” used for promotion and advancement and also create generalizable knowledge, which can help others improve clinical practice or operational processes. Those who are successful in completing their academic project frequently follow a well-structured path. Consider the following principles to get your idea across the finish line:

Find a blueprint. Among most academic projects, whether a research paper, QI project or new curriculum, an underlying formula is commonly applied. Before starting, do your background research. Is there a paper or method that resembles your desired approach? Is there a question or concept that caught your eye? Using a blueprint from existing evidence allows you to identify important structures, phrases, and terms to inform your manuscript. Once you have identified the blueprint, define your project and approach.

Find a mentor. While career mentorship is important for professional growth, you first need a project mentor. Being a project mentor is a smaller ask for a more senior colleague than being a career mentor, and it’s a great way to test-drive a potential long-term working relationship. Moreover, the successful completion of one project can potentially lead to further opportunities, and perhaps even a long-term career mentor.

Take initiative. In business, there is a common adage: “Never bring a problem to your boss without a proposed solution in hand.”¹ In academics, consider: “Never show up with an idea without bringing a proposal.” By bringing a defined proposal to the conversation, your inquiry is more likely to get a response because (a) it is not a blind-ask and (b) it creates a foundation to build on. This is analogous to an early learner presenting their assessment and plan in the clinical setting; you don’t stop at the diagnosis (your idea) without having a plan for how you want to manage it.

Get an accountability partner. Publicly committing to a goal increases the probability of accomplishing your task by 65%, while having an accountability partner increases that by 95%.² An accountability partner serves as a coach to help you accomplish a task. This individual can be a colleague, spouse, or friend and is typically not a part of the project. By leveraging peer pressure, you increase the odds of successfully completing your project.

Carve out dedicated time. The entrepreneur and author Jim Rohn once said, “Discipline is the bridge between goals and accomplishments.”³ To complete a project, you have to make the time to do the work. While many believe that successful writers sit and write for hours on end, many famous writers only wrote for a few hours at a time—but they did so consistently.⁴ Create your routine by setting aside consistent, defined time to work on your project. To extract the most value, select a time of the day in which you work best (eg, early morning). Then, set a timer for 30 minutes and write—or work.

Because you are a busy clinician with constant demands on your time, having the skillset to reliably turn an idea into “academic currency” is a necessity. Having a plan and following these principles will help you earn that academic coin.

“Our goals can only be reached through the vehicle of a plan. There is no other route to success.”

—Pablo Picasso

Whether it be a research manuscript, quality improvement (QI) initiative, or educational curriculum, busy clinicians often struggle getting projects past the idea stage. Barriers to completion, such as a busy clinical schedule or lack of experience and mentorship, are well known. Importantly, these projects serve as “academic currency” used for promotion and advancement and also create generalizable knowledge, which can help others improve clinical practice or operational processes. Those who are successful in completing their academic project frequently follow a well-structured path. Consider the following principles to get your idea across the finish line:

Find a blueprint. Among most academic projects, whether a research paper, QI project or new curriculum, an underlying formula is commonly applied. Before starting, do your background research. Is there a paper or method that resembles your desired approach? Is there a question or concept that caught your eye? Using a blueprint from existing evidence allows you to identify important structures, phrases, and terms to inform your manuscript. Once you have identified the blueprint, define your project and approach.

Find a mentor. While career mentorship is important for professional growth, you first need a project mentor. Being a project mentor is a smaller ask for a more senior colleague than being a career mentor, and it’s a great way to test-drive a potential long-term working relationship. Moreover, the successful completion of one project can potentially lead to further opportunities, and perhaps even a long-term career mentor.

Take initiative. In business, there is a common adage: “Never bring a problem to your boss without a proposed solution in hand.”¹ In academics, consider: “Never show up with an idea without bringing a proposal.” By bringing a defined proposal to the conversation, your inquiry is more likely to get a response because (a) it is not a blind-ask and (b) it creates a foundation to build on. This is analogous to an early learner presenting their assessment and plan in the clinical setting; you don’t stop at the diagnosis (your idea) without having a plan for how you want to manage it.

Get an accountability partner. Publicly committing to a goal increases the probability of accomplishing your task by 65%, while having an accountability partner increases that by 95%.² An accountability partner serves as a coach to help you accomplish a task. This individual can be a colleague, spouse, or friend and is typically not a part of the project. By leveraging peer pressure, you increase the odds of successfully completing your project.

Carve out dedicated time. The entrepreneur and author Jim Rohn once said, “Discipline is the bridge between goals and accomplishments.”³ To complete a project, you have to make the time to do the work. While many believe that successful writers sit and write for hours on end, many famous writers only wrote for a few hours at a time—but they did so consistently.⁴ Create your routine by setting aside consistent, defined time to work on your project. To extract the most value, select a time of the day in which you work best (eg, early morning). Then, set a timer for 30 minutes and write—or work.

Because you are a busy clinician with constant demands on your time, having the skillset to reliably turn an idea into “academic currency” is a necessity. Having a plan and following these principles will help you earn that academic coin.

“Our goals can only be reached through the vehicle of a plan. There is no other route to success.”

—Pablo Picasso

Whether it be a research manuscript, quality improvement (QI) initiative, or educational curriculum, busy clinicians often struggle getting projects past the idea stage. Barriers to completion, such as a busy clinical schedule or lack of experience and mentorship, are well known. Importantly, these projects serve as “academic currency” used for promotion and advancement and also create generalizable knowledge, which can help others improve clinical practice or operational processes. Those who are successful in completing their academic project frequently follow a well-structured path. Consider the following principles to get your idea across the finish line:

Find a blueprint. Among most academic projects, whether a research paper, QI project or new curriculum, an underlying formula is commonly applied. Before starting, do your background research. Is there a paper or method that resembles your desired approach? Is there a question or concept that caught your eye? Using a blueprint from existing evidence allows you to identify important structures, phrases, and terms to inform your manuscript. Once you have identified the blueprint, define your project and approach.

Find a mentor. While career mentorship is important for professional growth, you first need a project mentor. Being a project mentor is a smaller ask for a more senior colleague than being a career mentor, and it’s a great way to test-drive a potential long-term working relationship. Moreover, the successful completion of one project can potentially lead to further opportunities, and perhaps even a long-term career mentor.

Take initiative. In business, there is a common adage: “Never bring a problem to your boss without a proposed solution in hand.”¹ In academics, consider: “Never show up with an idea without bringing a proposal.” By bringing a defined proposal to the conversation, your inquiry is more likely to get a response because (a) it is not a blind-ask and (b) it creates a foundation to build on. This is analogous to an early learner presenting their assessment and plan in the clinical setting; you don’t stop at the diagnosis (your idea) without having a plan for how you want to manage it.

Get an accountability partner. Publicly committing to a goal increases the probability of accomplishing your task by 65%, while having an accountability partner increases that by 95%.² An accountability partner serves as a coach to help you accomplish a task. This individual can be a colleague, spouse, or friend and is typically not a part of the project. By leveraging peer pressure, you increase the odds of successfully completing your project.

Carve out dedicated time. The entrepreneur and author Jim Rohn once said, “Discipline is the bridge between goals and accomplishments.”³ To complete a project, you have to make the time to do the work. While many believe that successful writers sit and write for hours on end, many famous writers only wrote for a few hours at a time—but they did so consistently.⁴ Create your routine by setting aside consistent, defined time to work on your project. To extract the most value, select a time of the day in which you work best (eg, early morning). Then, set a timer for 30 minutes and write—or work.

Because you are a busy clinician with constant demands on your time, having the skillset to reliably turn an idea into “academic currency” is a necessity. Having a plan and following these principles will help you earn that academic coin.

The pandemic of coronavirus disease 2019 (COVID-19) is confronting the modern world like nothing else before. With over 20 million individuals expected to require hospitalization in the US, this health crisis may become a generation-defining moment for healthcare systems and the field of hospital medicine.¹ The specific challenges facing hospital medicine are comparable to running a sprint and a marathon—at the same time. For the sprint underway, hospitalists must learn to respond to a rapidly changing environment in which critical decisions are made within hours and days. At the same time, hospitalists need to plan for the marathon of increased clinical needs over the coming months, the possibility of burnout, and concerns about staff well-­being. Although runners typically focus on either the sprint or the marathon, healthcare systems and hospital medicine providers will need to simultaneously prepare for both types of races.

Over the past several weeks, hospital medicine leaders have been rapidly responding to an evolving crisis. Leaders and clinicians are quickly learning how to restructure clinical operations, negotiate the short supply of personal protective equipment (PPE), and manage delays in COVID-19 testing. In these areas, our hospitalist group has experienced a steep learning curve. In addition to the strategies outlined in the Table, we will share here our experiences and insights on managing and preparing for the COVID-19 pandemic.

garg02240420_t1.jpg

Communication Is Central

During the sprint, focused, regular communication is imperative to ameliorate anxiety and fear. A study of crisis communication after 9/11 found that, for employees, good communication from leadership was one of the most valued factors.² Communications experts also note that, in times of crisis, leaders have a special role in communication, specifically around demystifying the situation, providing hope, and maintaining transparency.³

Mental bandwidth may be limited in a stressful environment, so efforts should be taken to maximize the value of each communication. Information on hospital metrics should be provided regularly, including the number of COVID-19 cases, the status of clinical services and staffing, hospital capacity, and resource availability.⁴ Although the ubiquity and ease of email is convenient, recognize that providers are likely receiving email updates from multiple layers within your healthcare organization. To guard against losing important information, we use the same templated format for daily email updates with changes highlighted, which allows busy clinicians to digest pertinent information easily.⁵ Finally, consider having a single individual be responsible for collating COVID-19–related emails sent to your group. Although clinicians may want to share the most recent studies or their clinical experiences with a group email, instead have them send this information to a single individual who can organize these materials and share them on a regular basis.

To keep two-way communication channels open in a busy, asynchronous environment, consider having a centralized shared document in which providers can give real-time feedback to capture on-the-ground experiences or share questions they would like answered. Within our group, we found that centralizing our conversation in a shared document eliminated redundancy, focused our meetings, and kept everyone up to date. Additionally, regularly scheduled meetings may need to be adapted to a remote format (eg, Zoom, WebEx) as clinicians are asked to work from home when not on clinical service. Finally, recognize that virtual meetings require a different skill set than that required by in-person meetings, including reestablishment of social norms and technology preparation.⁶

Optimize Your Staffing

Hospital volumes could increase to as high as 270% of current hospital bed capacities during this pandemic.¹ This surge is further complicated by the effort involved in caring for these patients, given their increased medical complexity, the use of new protocols, and the extra time needed to update staff and family. As the workload intensifies, staffing models and operations will also need to adapt.

First, optimize your inpatient resources based on the changes your hospital system is making. For instance, as elective surgeries were cancelled, we dissolved our surgical comanagement and consult services to better accommodate our hospitals’ needs. Further, consider using advanced practice providers (eg, physician assistants and nurse practitioners) released from their clinical duties to help with inpatient care in the event of a surge. If your hospital has trainees (eg, residents or fellows), consider reassigning those whose rotations have been postponed to newly created inpatient teams; trainees often have strong institutional knowledge and understanding of hospital protocols and resources.

Second, use hospitalists for their most relevant skills. Hospitalists are pluripotent clinicians who are comfortable with high-­acuity patients and can fit into a myriad of clinical positions. The initial instinct at our institution was to mobilize hospitalists across all areas of increasing needs in the hospital (eg, screening clinics,⁷ advice phone lines for patients, or in the Emergency Department), but we quickly recognized that the hospitalist group is a finite resource. We focused our hospitalists’ clinical work on the expanding inpatient needs and allowed other outpatient or procedure-based specialties that have less inpatient experience to fill the broader institutional gaps.

Finally, consider long-term implications of staffing decisions. Leaders are making challenging coverage decisions that can affect the morale and autonomy of staff. Does backup staffing happen on a volunteer basis? Who fills the need—those with less clinical time or those with fewer personal obligations? When a staffing model is challenged and your group is making such decisions, engaged communication again becomes paramount.

Experts believe that we are only at the beginning of this crisis, one for which we don’t know what the end looks like or when it will come. With this in mind, hospital medicine leadership must plan for the long-term implications of the lengthy race ahead. Recognizing that morale, motivation, and burnout will be issues to deal with on the horizon, a focus on sustainability and wellness will become increasingly important as the marathon continues. To date, we’ve found the following principles to be helpful.

Delegate Responsibilities

Hospitals will not be able to survive COVID-19 through the efforts of single individuals. Instead, consider creating “operational champion” roles for frontline clinicians. These individuals can lead in specific areas (eg, PPE, updates on COVID-19 testing, discharge protocols) and act as conduits for information, updates, and resources for your group. At our institution, such operational meetings and activities take hours out of each day. By creating a breadth of leadership roles, our group has spread the operational workload while still allowing clinicians to care for patients, avoid burnout, and build autonomy and opportunities for both personal and professional growth. While for most institutions, these positions are temporary and not compensated with salary or time, the contribution to the group should be recognized both now and in the future.

Focus on Wellness

Providers are battling a laundry list of both clinical and personal stressors. The Centers for Disease Control and Prevention has already recognized that stress and mental health are going to be large hurdles for both patients and providers during this crisis.⁸ From the beginning, hospitalist leadership should be attuned to physician wellness and be aware that burnout, mental and physical exhaustion, and the possibility of contracting COVID-19 will be issues in the coming weeks and months. Volunteerism is built into the physician’s work ethic, but we must be mindful about its cost for long-term staffing demands. In addition, scarce medical resources add an additional moral strain for clinicians as they face tough allocation decisions, as we’ve seen with our Italian colleagues.⁹

As regular meetings around COVID-19 have become commonplace, we’ve made sure to set aside defined time for staff to discuss and reflect on their experiences. Doing so has allowed our clinicians to feel heard and to acknowledge the difficulties they are facing in their clinical duties. Leaders should also consider frequent check-ins with individual providers. At our institution, the first positive COVID-19 patient did not radically change any protocol that was in place, but a check-in with the hospitalist on service that day proved helpful for a debrief and processing opportunity. Individual conversations can help those on the front lines feel supported and remind them they are not operating alone in an anonymous vacuum.

Continue by celebrating small victories because this marathon is not going to end with an obvious finish line or a singular moment in which everyone can rejoice. A negative test, a patient with a good outcome, and a donation of PPE are all opportunities to celebrate. It may be what keeps us going when there is no end in sight. We have relied on these celebrations and moments of levity as an integral part of our regular group meetings.

At the end of this pandemic, just as we hope that our social distancing feels like an overreaction, we similarly hope that our sprint to build capacity ends up being unnecessary as well. As we wrote this Perspectives piece, uncertainty about the extent, length, and impact of this pandemic still existed. By the time it is published it may be that the sprint is over, and the marathon is beginning. Or, if our wildest hopes come true, there will be no marathon to run at all.

The pandemic of coronavirus disease 2019 (COVID-19) is confronting the modern world like nothing else before. With over 20 million individuals expected to require hospitalization in the US, this health crisis may become a generation-defining moment for healthcare systems and the field of hospital medicine.¹ The specific challenges facing hospital medicine are comparable to running a sprint and a marathon—at the same time. For the sprint underway, hospitalists must learn to respond to a rapidly changing environment in which critical decisions are made within hours and days. At the same time, hospitalists need to plan for the marathon of increased clinical needs over the coming months, the possibility of burnout, and concerns about staff well-­being. Although runners typically focus on either the sprint or the marathon, healthcare systems and hospital medicine providers will need to simultaneously prepare for both types of races.

Over the past several weeks, hospital medicine leaders have been rapidly responding to an evolving crisis. Leaders and clinicians are quickly learning how to restructure clinical operations, negotiate the short supply of personal protective equipment (PPE), and manage delays in COVID-19 testing. In these areas, our hospitalist group has experienced a steep learning curve. In addition to the strategies outlined in the Table, we will share here our experiences and insights on managing and preparing for the COVID-19 pandemic.

garg02240420_t1.jpg

Communication Is Central

During the sprint, focused, regular communication is imperative to ameliorate anxiety and fear. A study of crisis communication after 9/11 found that, for employees, good communication from leadership was one of the most valued factors.² Communications experts also note that, in times of crisis, leaders have a special role in communication, specifically around demystifying the situation, providing hope, and maintaining transparency.³

Mental bandwidth may be limited in a stressful environment, so efforts should be taken to maximize the value of each communication. Information on hospital metrics should be provided regularly, including the number of COVID-19 cases, the status of clinical services and staffing, hospital capacity, and resource availability.⁴ Although the ubiquity and ease of email is convenient, recognize that providers are likely receiving email updates from multiple layers within your healthcare organization. To guard against losing important information, we use the same templated format for daily email updates with changes highlighted, which allows busy clinicians to digest pertinent information easily.⁵ Finally, consider having a single individual be responsible for collating COVID-19–related emails sent to your group. Although clinicians may want to share the most recent studies or their clinical experiences with a group email, instead have them send this information to a single individual who can organize these materials and share them on a regular basis.

To keep two-way communication channels open in a busy, asynchronous environment, consider having a centralized shared document in which providers can give real-time feedback to capture on-the-ground experiences or share questions they would like answered. Within our group, we found that centralizing our conversation in a shared document eliminated redundancy, focused our meetings, and kept everyone up to date. Additionally, regularly scheduled meetings may need to be adapted to a remote format (eg, Zoom, WebEx) as clinicians are asked to work from home when not on clinical service. Finally, recognize that virtual meetings require a different skill set than that required by in-person meetings, including reestablishment of social norms and technology preparation.⁶

Optimize Your Staffing

Hospital volumes could increase to as high as 270% of current hospital bed capacities during this pandemic.¹ This surge is further complicated by the effort involved in caring for these patients, given their increased medical complexity, the use of new protocols, and the extra time needed to update staff and family. As the workload intensifies, staffing models and operations will also need to adapt.

First, optimize your inpatient resources based on the changes your hospital system is making. For instance, as elective surgeries were cancelled, we dissolved our surgical comanagement and consult services to better accommodate our hospitals’ needs. Further, consider using advanced practice providers (eg, physician assistants and nurse practitioners) released from their clinical duties to help with inpatient care in the event of a surge. If your hospital has trainees (eg, residents or fellows), consider reassigning those whose rotations have been postponed to newly created inpatient teams; trainees often have strong institutional knowledge and understanding of hospital protocols and resources.

Second, use hospitalists for their most relevant skills. Hospitalists are pluripotent clinicians who are comfortable with high-­acuity patients and can fit into a myriad of clinical positions. The initial instinct at our institution was to mobilize hospitalists across all areas of increasing needs in the hospital (eg, screening clinics,⁷ advice phone lines for patients, or in the Emergency Department), but we quickly recognized that the hospitalist group is a finite resource. We focused our hospitalists’ clinical work on the expanding inpatient needs and allowed other outpatient or procedure-based specialties that have less inpatient experience to fill the broader institutional gaps.

Finally, consider long-term implications of staffing decisions. Leaders are making challenging coverage decisions that can affect the morale and autonomy of staff. Does backup staffing happen on a volunteer basis? Who fills the need—those with less clinical time or those with fewer personal obligations? When a staffing model is challenged and your group is making such decisions, engaged communication again becomes paramount.

Experts believe that we are only at the beginning of this crisis, one for which we don’t know what the end looks like or when it will come. With this in mind, hospital medicine leadership must plan for the long-term implications of the lengthy race ahead. Recognizing that morale, motivation, and burnout will be issues to deal with on the horizon, a focus on sustainability and wellness will become increasingly important as the marathon continues. To date, we’ve found the following principles to be helpful.

Delegate Responsibilities

Hospitals will not be able to survive COVID-19 through the efforts of single individuals. Instead, consider creating “operational champion” roles for frontline clinicians. These individuals can lead in specific areas (eg, PPE, updates on COVID-19 testing, discharge protocols) and act as conduits for information, updates, and resources for your group. At our institution, such operational meetings and activities take hours out of each day. By creating a breadth of leadership roles, our group has spread the operational workload while still allowing clinicians to care for patients, avoid burnout, and build autonomy and opportunities for both personal and professional growth. While for most institutions, these positions are temporary and not compensated with salary or time, the contribution to the group should be recognized both now and in the future.

Focus on Wellness

Providers are battling a laundry list of both clinical and personal stressors. The Centers for Disease Control and Prevention has already recognized that stress and mental health are going to be large hurdles for both patients and providers during this crisis.⁸ From the beginning, hospitalist leadership should be attuned to physician wellness and be aware that burnout, mental and physical exhaustion, and the possibility of contracting COVID-19 will be issues in the coming weeks and months. Volunteerism is built into the physician’s work ethic, but we must be mindful about its cost for long-term staffing demands. In addition, scarce medical resources add an additional moral strain for clinicians as they face tough allocation decisions, as we’ve seen with our Italian colleagues.⁹

As regular meetings around COVID-19 have become commonplace, we’ve made sure to set aside defined time for staff to discuss and reflect on their experiences. Doing so has allowed our clinicians to feel heard and to acknowledge the difficulties they are facing in their clinical duties. Leaders should also consider frequent check-ins with individual providers. At our institution, the first positive COVID-19 patient did not radically change any protocol that was in place, but a check-in with the hospitalist on service that day proved helpful for a debrief and processing opportunity. Individual conversations can help those on the front lines feel supported and remind them they are not operating alone in an anonymous vacuum.

Continue by celebrating small victories because this marathon is not going to end with an obvious finish line or a singular moment in which everyone can rejoice. A negative test, a patient with a good outcome, and a donation of PPE are all opportunities to celebrate. It may be what keeps us going when there is no end in sight. We have relied on these celebrations and moments of levity as an integral part of our regular group meetings.

At the end of this pandemic, just as we hope that our social distancing feels like an overreaction, we similarly hope that our sprint to build capacity ends up being unnecessary as well. As we wrote this Perspectives piece, uncertainty about the extent, length, and impact of this pandemic still existed. By the time it is published it may be that the sprint is over, and the marathon is beginning. Or, if our wildest hopes come true, there will be no marathon to run at all.

The pandemic of coronavirus disease 2019 (COVID-19) is confronting the modern world like nothing else before. With over 20 million individuals expected to require hospitalization in the US, this health crisis may become a generation-defining moment for healthcare systems and the field of hospital medicine.¹ The specific challenges facing hospital medicine are comparable to running a sprint and a marathon—at the same time. For the sprint underway, hospitalists must learn to respond to a rapidly changing environment in which critical decisions are made within hours and days. At the same time, hospitalists need to plan for the marathon of increased clinical needs over the coming months, the possibility of burnout, and concerns about staff well-­being. Although runners typically focus on either the sprint or the marathon, healthcare systems and hospital medicine providers will need to simultaneously prepare for both types of races.

Over the past several weeks, hospital medicine leaders have been rapidly responding to an evolving crisis. Leaders and clinicians are quickly learning how to restructure clinical operations, negotiate the short supply of personal protective equipment (PPE), and manage delays in COVID-19 testing. In these areas, our hospitalist group has experienced a steep learning curve. In addition to the strategies outlined in the Table, we will share here our experiences and insights on managing and preparing for the COVID-19 pandemic.

garg02240420_t1.jpg

Communication Is Central

During the sprint, focused, regular communication is imperative to ameliorate anxiety and fear. A study of crisis communication after 9/11 found that, for employees, good communication from leadership was one of the most valued factors.² Communications experts also note that, in times of crisis, leaders have a special role in communication, specifically around demystifying the situation, providing hope, and maintaining transparency.³

Mental bandwidth may be limited in a stressful environment, so efforts should be taken to maximize the value of each communication. Information on hospital metrics should be provided regularly, including the number of COVID-19 cases, the status of clinical services and staffing, hospital capacity, and resource availability.⁴ Although the ubiquity and ease of email is convenient, recognize that providers are likely receiving email updates from multiple layers within your healthcare organization. To guard against losing important information, we use the same templated format for daily email updates with changes highlighted, which allows busy clinicians to digest pertinent information easily.⁵ Finally, consider having a single individual be responsible for collating COVID-19–related emails sent to your group. Although clinicians may want to share the most recent studies or their clinical experiences with a group email, instead have them send this information to a single individual who can organize these materials and share them on a regular basis.

To keep two-way communication channels open in a busy, asynchronous environment, consider having a centralized shared document in which providers can give real-time feedback to capture on-the-ground experiences or share questions they would like answered. Within our group, we found that centralizing our conversation in a shared document eliminated redundancy, focused our meetings, and kept everyone up to date. Additionally, regularly scheduled meetings may need to be adapted to a remote format (eg, Zoom, WebEx) as clinicians are asked to work from home when not on clinical service. Finally, recognize that virtual meetings require a different skill set than that required by in-person meetings, including reestablishment of social norms and technology preparation.⁶

Optimize Your Staffing

Hospital volumes could increase to as high as 270% of current hospital bed capacities during this pandemic.¹ This surge is further complicated by the effort involved in caring for these patients, given their increased medical complexity, the use of new protocols, and the extra time needed to update staff and family. As the workload intensifies, staffing models and operations will also need to adapt.

First, optimize your inpatient resources based on the changes your hospital system is making. For instance, as elective surgeries were cancelled, we dissolved our surgical comanagement and consult services to better accommodate our hospitals’ needs. Further, consider using advanced practice providers (eg, physician assistants and nurse practitioners) released from their clinical duties to help with inpatient care in the event of a surge. If your hospital has trainees (eg, residents or fellows), consider reassigning those whose rotations have been postponed to newly created inpatient teams; trainees often have strong institutional knowledge and understanding of hospital protocols and resources.

Second, use hospitalists for their most relevant skills. Hospitalists are pluripotent clinicians who are comfortable with high-­acuity patients and can fit into a myriad of clinical positions. The initial instinct at our institution was to mobilize hospitalists across all areas of increasing needs in the hospital (eg, screening clinics,⁷ advice phone lines for patients, or in the Emergency Department), but we quickly recognized that the hospitalist group is a finite resource. We focused our hospitalists’ clinical work on the expanding inpatient needs and allowed other outpatient or procedure-based specialties that have less inpatient experience to fill the broader institutional gaps.

Finally, consider long-term implications of staffing decisions. Leaders are making challenging coverage decisions that can affect the morale and autonomy of staff. Does backup staffing happen on a volunteer basis? Who fills the need—those with less clinical time or those with fewer personal obligations? When a staffing model is challenged and your group is making such decisions, engaged communication again becomes paramount.

Experts believe that we are only at the beginning of this crisis, one for which we don’t know what the end looks like or when it will come. With this in mind, hospital medicine leadership must plan for the long-term implications of the lengthy race ahead. Recognizing that morale, motivation, and burnout will be issues to deal with on the horizon, a focus on sustainability and wellness will become increasingly important as the marathon continues. To date, we’ve found the following principles to be helpful.

Delegate Responsibilities

Hospitals will not be able to survive COVID-19 through the efforts of single individuals. Instead, consider creating “operational champion” roles for frontline clinicians. These individuals can lead in specific areas (eg, PPE, updates on COVID-19 testing, discharge protocols) and act as conduits for information, updates, and resources for your group. At our institution, such operational meetings and activities take hours out of each day. By creating a breadth of leadership roles, our group has spread the operational workload while still allowing clinicians to care for patients, avoid burnout, and build autonomy and opportunities for both personal and professional growth. While for most institutions, these positions are temporary and not compensated with salary or time, the contribution to the group should be recognized both now and in the future.

Focus on Wellness

Providers are battling a laundry list of both clinical and personal stressors. The Centers for Disease Control and Prevention has already recognized that stress and mental health are going to be large hurdles for both patients and providers during this crisis.⁸ From the beginning, hospitalist leadership should be attuned to physician wellness and be aware that burnout, mental and physical exhaustion, and the possibility of contracting COVID-19 will be issues in the coming weeks and months. Volunteerism is built into the physician’s work ethic, but we must be mindful about its cost for long-term staffing demands. In addition, scarce medical resources add an additional moral strain for clinicians as they face tough allocation decisions, as we’ve seen with our Italian colleagues.⁹

As regular meetings around COVID-19 have become commonplace, we’ve made sure to set aside defined time for staff to discuss and reflect on their experiences. Doing so has allowed our clinicians to feel heard and to acknowledge the difficulties they are facing in their clinical duties. Leaders should also consider frequent check-ins with individual providers. At our institution, the first positive COVID-19 patient did not radically change any protocol that was in place, but a check-in with the hospitalist on service that day proved helpful for a debrief and processing opportunity. Individual conversations can help those on the front lines feel supported and remind them they are not operating alone in an anonymous vacuum.

Continue by celebrating small victories because this marathon is not going to end with an obvious finish line or a singular moment in which everyone can rejoice. A negative test, a patient with a good outcome, and a donation of PPE are all opportunities to celebrate. It may be what keeps us going when there is no end in sight. We have relied on these celebrations and moments of levity as an integral part of our regular group meetings.

At the end of this pandemic, just as we hope that our social distancing feels like an overreaction, we similarly hope that our sprint to build capacity ends up being unnecessary as well. As we wrote this Perspectives piece, uncertainty about the extent, length, and impact of this pandemic still existed. By the time it is published it may be that the sprint is over, and the marathon is beginning. Or, if our wildest hopes come true, there will be no marathon to run at all.

Machine-learning is a type of artificial intelligence in which systems automatically learn and improve from experience without being explicitly programmed. Classification and Regression Tree (CART) analysis is a machine-learning algorithm that was developed to visually classify or segment populations into subgroups with similar characteristics and outcomes. CART analysis is a decision tree methodology that was initially developed in the 1960s for use in product marketing.¹ Since then, a number of health disciplines have used it to isolate patient subgroups from larger populations to guide clinical decision-making by better identifying those most likely to benefit.² The clinical utility of CART mirrors how most clinicians think, which is not in terms of coefficients (ie, regression output) but rather in terms of categories or classifications (eg, low vs high risk).

In this issue of the Journal of Hospital Medicine, Young and colleagues use classification trees to predict discharge placement (postacute care facility vs home) based on a patient’s hospital admission characteristics and mobility score. The resulting decision tree indicates that patients with the lowest mobility scores, as well as those 65 years and older, were most likely to be discharged to postacute care facilities.³ In this review, we orient the reader to the basics of CART analysis, discuss important intricacies, and weigh its pros, cons, and application as a statistical tool.

CART is a nonparametric (ie, makes no assumptions about data distribution) statistical tool that identifies subgroups within a population whose members share common characteristics as defined by the independent variables included in the model. CART analysis is unique in that it yields a visual output of the data in the form of a multisegmented structure that resembles the branches of a tree (Figure). CART analysis consists of four basic steps: (1) tree-building (including splitting criteria and estimation of classification error), (2) stopping the tree-building process, (3) tree “pruning,” and (4) tree selection.

wray06270318e_f1.jpg

In general, CART analysis begins with a single “node” or group, which contains the entire sample population. This is referred to as the “parent node.” The CART procedure simultaneously examines all available independent variables and selects one that results in two groups that are the most distinct with respect to the outcome variable of interest. In Young et al’s example, posthospital discharge placement is the outcome.³ This parent node then branches into two “child nodes” according to the independent variable that was selected. Within each of these “child nodes,” the tree-growing methodology recursively assesses each of the remaining independent variables to determine which will result in the best split according to the chosen splitting criterion.² Each subsequent “child node” will become a “parent node” to the two groups in which it splits. This process is repeated on the data in each subsequent “child node” and is stopped once a predefined stopping point is reached. Notably, while division into two groups is the most common application of CART modeling, there are models that can split data into more than two child nodes.

Since CART outcomes can be heavily dependent on the data being used (eg, electronic health records or administrative data), it is important to attempt to confirm results in a similar, but different, study cohort. Because obtaining separate data sources with similar cohorts can be difficult, many investigators using CART will utilize a “split sample approach” in which study data are split into separate training and validation sets.⁴ In the training set, which frequently comprises two-thirds of the available data, the algorithm is tested in exploratory analysis. Once the algorithm is defined and agreed upon, it is retested within a validation set, constructed from the remaining one-third of data. This approach, which Young et al utilize,³ allows for improved confidence and reduced risk of bias in the findings and allows for some degree of external validation. Further, the split sample approach supports more reliable measures of predictive accuracy: in Young et al’s case, the proportion of correctly classified patients discharged to a postacute care facility (sensitivity: 58%, 95% CI 49-68%) and the proportion of correctly classified patients discharged home (specificity: 84%, 95% CI 78-90%). Despite these advantages, the split sample approach is not universally used.

Classification Versus Regression Trees

While commonly grouped together, CARTs can be distinguished from one another based on the dependent, or outcome, variable. Categorical outcome variables require the use of a classification tree, while continuous outcomes utilize regression trees. Of note, the independent, or predictor, variables can be any combination of categorical or continuous variables. However, splitting at each node creates categorical output when using CART algorithms.

Splitting Criteria

The splitting of each node is based on reducing the degree of “impurity” (heterogeneity with respect to the outcome variable) within each node. For example, a node that has no impurity will have a zero error rate labeling its binary outcomes. While CART works well with categorical variables, continuous variables (eg, age) can also be assessed, though only with certain algorithms. Several different splitting criteria exist, each of which attempt to maximize the differences within each child node. While beyond the scope of this review, examples of popular splitting criteria are Gini, entropy, and minimum error.⁵

Stopping Rules

To manage the size of a tree, CART analysis allows for predefined stopping rules to minimize the extent of growth while also establishing a minimal degree of statistical difference between nodes that is considered meaningful. To accomplish this task, two stopping rules are often used. The first defines the minimum number of observations in child, or “terminal,” nodes. The second defines the maximum number of levels a tree may grow, thus allowing the investigator to decide the total number of predictor variables that can define a terminal node. While several other stopping rules exist, these are the most commonly utilized.

Pruning

To avoid missing important associations due to premature stoppage, investigators may use another mechanism to limit tree growth called “pruning.” For pruning, the first step is to grow a considerably large tree that includes many levels or nodes, possibly to the point where there are just a few observations per terminal node. Then, similar to the residual sum of squares in a regression, the investigator can calculate a misclassification cost (ie, goodness of fit) and select the tree with the smallest cost.² Of note, stopping rules and pruning can be used simultaneously.

Classification Error

Similar to other forms of statistical inference it remains important to understand the uncertainty within the inference. In regression modeling, for example, classification errors can be calculated using standard errors of the parameter estimates. In CART analysis, because random samples from a population may produce different trees, measures of variability can be more complicated. One strategy is to generate a tree from a test sample and then use the remaining data to calculate a measure of the misclassification cost (a measure of how much additional accuracy a split must add to the entire tree to warrant the additional complexity). Alternatively, a “k-fold cross-validation” can be performed in which the data is broken down into k subsets from which a tree is created using all data except for one of the subsets. The computed tree is then applied to the remaining subset to determine a misclassification cost. These classification costs are important as they also impact the stopping and pruning processes. Ultimately, a final tree, which best limits classification errors, is selected.

This method can be useful in multiple settings in which an investigator wants to characterize a subpopulation from a larger cohort. Adaptation of this could include, but is not limited to, risk stratification,⁶ diagnostics,⁷ and patient identification for medical interventions.⁸ Moreover, CART analysis has the added benefit of creating visually interpretable predictive models that can be utilized for front-line clinical decision making.^9,10

CART analysis has been shown to have several advantages over other commonly used modeling methods. First, it is a nonparametric model that can handle highly skewed data and does not require that the predictor, or predictors, takes on a predetermined form (allowing them to be constructed from the data). This is helpful as many clinical variables can have wide degrees of variance.

Unlike other modeling techniques, CART can identify higher-order interactions between multiple variables, meaning it can handle interactions that occur whenever one variable affects the nature of an interaction between two other variables. Further, CART can handle multiple correlated independent variables, something logistic regression models classically cannot do.

From a clinical standpoint, the “logic” of the visual-based CART output can be easier to interpret than the probabilistic output (eg, odds ratio) associated with logistic regression modeling, making it more practical, applicable, and easier for clinicians to adopt.^10,12 Finally, CART software is easy to use for those who do not have strong statistical backgrounds, and it is less resource intensive than other statistical methods.²

Despite these features, CART does have several disadvantages. First, due to the ease with which CART analysis can be performed, “data dredging” can be a significant concern. Its ideal use is with a priori consideration of independent variables.² Second, while CART is most beneficial in describing links and cutoffs between variables, it may not be useful for hypothesis testing.² Third, large data sets are needed to perform CART, especially if the investigator is using the split sample approach mentioned above.¹¹ Finally, while CART is the most utilized decision tree methodology, several other types of decision tree methods exist: C4.5, CRUISE, Quick, Unbiased, Efficient Statistical Trees, Chi-square-Automatic-Interaction-Detection, and others. Many of these allow for splitting into more than two groups and have other features that may be more advantageous to one’s analysis.¹³

Decision trees offer simple, interpretable results of multiple factors that can be easily applied to clinical scenarios. In this case, the authors specifically used classification tree analysis to take advantage of CART’s machine-learning ability to consider higher-order interactions to build their model—as they lacked a priori evidence to help guide them in traditional (ie, logistic regression) model construction. Furthermore, CART analysis created an output that logically and visually illustrates which combination of characteristics is most associated with discharge placement and can potentially be utilized to help facilitate discharge planning in future hospitalized patients. To sum up, this machine-learning methodology allowed the investigators to determine which variables taken together were the most suitable in predicting their outcome of interest and present these findings in a manner that busy clinicians can interpret and apply.

Machine-learning is a type of artificial intelligence in which systems automatically learn and improve from experience without being explicitly programmed. Classification and Regression Tree (CART) analysis is a machine-learning algorithm that was developed to visually classify or segment populations into subgroups with similar characteristics and outcomes. CART analysis is a decision tree methodology that was initially developed in the 1960s for use in product marketing.¹ Since then, a number of health disciplines have used it to isolate patient subgroups from larger populations to guide clinical decision-making by better identifying those most likely to benefit.² The clinical utility of CART mirrors how most clinicians think, which is not in terms of coefficients (ie, regression output) but rather in terms of categories or classifications (eg, low vs high risk).

In this issue of the Journal of Hospital Medicine, Young and colleagues use classification trees to predict discharge placement (postacute care facility vs home) based on a patient’s hospital admission characteristics and mobility score. The resulting decision tree indicates that patients with the lowest mobility scores, as well as those 65 years and older, were most likely to be discharged to postacute care facilities.³ In this review, we orient the reader to the basics of CART analysis, discuss important intricacies, and weigh its pros, cons, and application as a statistical tool.

CART is a nonparametric (ie, makes no assumptions about data distribution) statistical tool that identifies subgroups within a population whose members share common characteristics as defined by the independent variables included in the model. CART analysis is unique in that it yields a visual output of the data in the form of a multisegmented structure that resembles the branches of a tree (Figure). CART analysis consists of four basic steps: (1) tree-building (including splitting criteria and estimation of classification error), (2) stopping the tree-building process, (3) tree “pruning,” and (4) tree selection.

wray06270318e_f1.jpg

In general, CART analysis begins with a single “node” or group, which contains the entire sample population. This is referred to as the “parent node.” The CART procedure simultaneously examines all available independent variables and selects one that results in two groups that are the most distinct with respect to the outcome variable of interest. In Young et al’s example, posthospital discharge placement is the outcome.³ This parent node then branches into two “child nodes” according to the independent variable that was selected. Within each of these “child nodes,” the tree-growing methodology recursively assesses each of the remaining independent variables to determine which will result in the best split according to the chosen splitting criterion.² Each subsequent “child node” will become a “parent node” to the two groups in which it splits. This process is repeated on the data in each subsequent “child node” and is stopped once a predefined stopping point is reached. Notably, while division into two groups is the most common application of CART modeling, there are models that can split data into more than two child nodes.

Since CART outcomes can be heavily dependent on the data being used (eg, electronic health records or administrative data), it is important to attempt to confirm results in a similar, but different, study cohort. Because obtaining separate data sources with similar cohorts can be difficult, many investigators using CART will utilize a “split sample approach” in which study data are split into separate training and validation sets.⁴ In the training set, which frequently comprises two-thirds of the available data, the algorithm is tested in exploratory analysis. Once the algorithm is defined and agreed upon, it is retested within a validation set, constructed from the remaining one-third of data. This approach, which Young et al utilize,³ allows for improved confidence and reduced risk of bias in the findings and allows for some degree of external validation. Further, the split sample approach supports more reliable measures of predictive accuracy: in Young et al’s case, the proportion of correctly classified patients discharged to a postacute care facility (sensitivity: 58%, 95% CI 49-68%) and the proportion of correctly classified patients discharged home (specificity: 84%, 95% CI 78-90%). Despite these advantages, the split sample approach is not universally used.

Classification Versus Regression Trees

While commonly grouped together, CARTs can be distinguished from one another based on the dependent, or outcome, variable. Categorical outcome variables require the use of a classification tree, while continuous outcomes utilize regression trees. Of note, the independent, or predictor, variables can be any combination of categorical or continuous variables. However, splitting at each node creates categorical output when using CART algorithms.

Splitting Criteria

The splitting of each node is based on reducing the degree of “impurity” (heterogeneity with respect to the outcome variable) within each node. For example, a node that has no impurity will have a zero error rate labeling its binary outcomes. While CART works well with categorical variables, continuous variables (eg, age) can also be assessed, though only with certain algorithms. Several different splitting criteria exist, each of which attempt to maximize the differences within each child node. While beyond the scope of this review, examples of popular splitting criteria are Gini, entropy, and minimum error.⁵

Stopping Rules

To manage the size of a tree, CART analysis allows for predefined stopping rules to minimize the extent of growth while also establishing a minimal degree of statistical difference between nodes that is considered meaningful. To accomplish this task, two stopping rules are often used. The first defines the minimum number of observations in child, or “terminal,” nodes. The second defines the maximum number of levels a tree may grow, thus allowing the investigator to decide the total number of predictor variables that can define a terminal node. While several other stopping rules exist, these are the most commonly utilized.

Pruning

To avoid missing important associations due to premature stoppage, investigators may use another mechanism to limit tree growth called “pruning.” For pruning, the first step is to grow a considerably large tree that includes many levels or nodes, possibly to the point where there are just a few observations per terminal node. Then, similar to the residual sum of squares in a regression, the investigator can calculate a misclassification cost (ie, goodness of fit) and select the tree with the smallest cost.² Of note, stopping rules and pruning can be used simultaneously.

Classification Error

Similar to other forms of statistical inference it remains important to understand the uncertainty within the inference. In regression modeling, for example, classification errors can be calculated using standard errors of the parameter estimates. In CART analysis, because random samples from a population may produce different trees, measures of variability can be more complicated. One strategy is to generate a tree from a test sample and then use the remaining data to calculate a measure of the misclassification cost (a measure of how much additional accuracy a split must add to the entire tree to warrant the additional complexity). Alternatively, a “k-fold cross-validation” can be performed in which the data is broken down into k subsets from which a tree is created using all data except for one of the subsets. The computed tree is then applied to the remaining subset to determine a misclassification cost. These classification costs are important as they also impact the stopping and pruning processes. Ultimately, a final tree, which best limits classification errors, is selected.

This method can be useful in multiple settings in which an investigator wants to characterize a subpopulation from a larger cohort. Adaptation of this could include, but is not limited to, risk stratification,⁶ diagnostics,⁷ and patient identification for medical interventions.⁸ Moreover, CART analysis has the added benefit of creating visually interpretable predictive models that can be utilized for front-line clinical decision making.^9,10

CART analysis has been shown to have several advantages over other commonly used modeling methods. First, it is a nonparametric model that can handle highly skewed data and does not require that the predictor, or predictors, takes on a predetermined form (allowing them to be constructed from the data). This is helpful as many clinical variables can have wide degrees of variance.

Unlike other modeling techniques, CART can identify higher-order interactions between multiple variables, meaning it can handle interactions that occur whenever one variable affects the nature of an interaction between two other variables. Further, CART can handle multiple correlated independent variables, something logistic regression models classically cannot do.

From a clinical standpoint, the “logic” of the visual-based CART output can be easier to interpret than the probabilistic output (eg, odds ratio) associated with logistic regression modeling, making it more practical, applicable, and easier for clinicians to adopt.^10,12 Finally, CART software is easy to use for those who do not have strong statistical backgrounds, and it is less resource intensive than other statistical methods.²

Despite these features, CART does have several disadvantages. First, due to the ease with which CART analysis can be performed, “data dredging” can be a significant concern. Its ideal use is with a priori consideration of independent variables.² Second, while CART is most beneficial in describing links and cutoffs between variables, it may not be useful for hypothesis testing.² Third, large data sets are needed to perform CART, especially if the investigator is using the split sample approach mentioned above.¹¹ Finally, while CART is the most utilized decision tree methodology, several other types of decision tree methods exist: C4.5, CRUISE, Quick, Unbiased, Efficient Statistical Trees, Chi-square-Automatic-Interaction-Detection, and others. Many of these allow for splitting into more than two groups and have other features that may be more advantageous to one’s analysis.¹³

Decision trees offer simple, interpretable results of multiple factors that can be easily applied to clinical scenarios. In this case, the authors specifically used classification tree analysis to take advantage of CART’s machine-learning ability to consider higher-order interactions to build their model—as they lacked a priori evidence to help guide them in traditional (ie, logistic regression) model construction. Furthermore, CART analysis created an output that logically and visually illustrates which combination of characteristics is most associated with discharge placement and can potentially be utilized to help facilitate discharge planning in future hospitalized patients. To sum up, this machine-learning methodology allowed the investigators to determine which variables taken together were the most suitable in predicting their outcome of interest and present these findings in a manner that busy clinicians can interpret and apply.

Machine-learning is a type of artificial intelligence in which systems automatically learn and improve from experience without being explicitly programmed. Classification and Regression Tree (CART) analysis is a machine-learning algorithm that was developed to visually classify or segment populations into subgroups with similar characteristics and outcomes. CART analysis is a decision tree methodology that was initially developed in the 1960s for use in product marketing.¹ Since then, a number of health disciplines have used it to isolate patient subgroups from larger populations to guide clinical decision-making by better identifying those most likely to benefit.² The clinical utility of CART mirrors how most clinicians think, which is not in terms of coefficients (ie, regression output) but rather in terms of categories or classifications (eg, low vs high risk).

In this issue of the Journal of Hospital Medicine, Young and colleagues use classification trees to predict discharge placement (postacute care facility vs home) based on a patient’s hospital admission characteristics and mobility score. The resulting decision tree indicates that patients with the lowest mobility scores, as well as those 65 years and older, were most likely to be discharged to postacute care facilities.³ In this review, we orient the reader to the basics of CART analysis, discuss important intricacies, and weigh its pros, cons, and application as a statistical tool.

CART is a nonparametric (ie, makes no assumptions about data distribution) statistical tool that identifies subgroups within a population whose members share common characteristics as defined by the independent variables included in the model. CART analysis is unique in that it yields a visual output of the data in the form of a multisegmented structure that resembles the branches of a tree (Figure). CART analysis consists of four basic steps: (1) tree-building (including splitting criteria and estimation of classification error), (2) stopping the tree-building process, (3) tree “pruning,” and (4) tree selection.

wray06270318e_f1.jpg

In general, CART analysis begins with a single “node” or group, which contains the entire sample population. This is referred to as the “parent node.” The CART procedure simultaneously examines all available independent variables and selects one that results in two groups that are the most distinct with respect to the outcome variable of interest. In Young et al’s example, posthospital discharge placement is the outcome.³ This parent node then branches into two “child nodes” according to the independent variable that was selected. Within each of these “child nodes,” the tree-growing methodology recursively assesses each of the remaining independent variables to determine which will result in the best split according to the chosen splitting criterion.² Each subsequent “child node” will become a “parent node” to the two groups in which it splits. This process is repeated on the data in each subsequent “child node” and is stopped once a predefined stopping point is reached. Notably, while division into two groups is the most common application of CART modeling, there are models that can split data into more than two child nodes.

Since CART outcomes can be heavily dependent on the data being used (eg, electronic health records or administrative data), it is important to attempt to confirm results in a similar, but different, study cohort. Because obtaining separate data sources with similar cohorts can be difficult, many investigators using CART will utilize a “split sample approach” in which study data are split into separate training and validation sets.⁴ In the training set, which frequently comprises two-thirds of the available data, the algorithm is tested in exploratory analysis. Once the algorithm is defined and agreed upon, it is retested within a validation set, constructed from the remaining one-third of data. This approach, which Young et al utilize,³ allows for improved confidence and reduced risk of bias in the findings and allows for some degree of external validation. Further, the split sample approach supports more reliable measures of predictive accuracy: in Young et al’s case, the proportion of correctly classified patients discharged to a postacute care facility (sensitivity: 58%, 95% CI 49-68%) and the proportion of correctly classified patients discharged home (specificity: 84%, 95% CI 78-90%). Despite these advantages, the split sample approach is not universally used.

Classification Versus Regression Trees

While commonly grouped together, CARTs can be distinguished from one another based on the dependent, or outcome, variable. Categorical outcome variables require the use of a classification tree, while continuous outcomes utilize regression trees. Of note, the independent, or predictor, variables can be any combination of categorical or continuous variables. However, splitting at each node creates categorical output when using CART algorithms.

Splitting Criteria

The splitting of each node is based on reducing the degree of “impurity” (heterogeneity with respect to the outcome variable) within each node. For example, a node that has no impurity will have a zero error rate labeling its binary outcomes. While CART works well with categorical variables, continuous variables (eg, age) can also be assessed, though only with certain algorithms. Several different splitting criteria exist, each of which attempt to maximize the differences within each child node. While beyond the scope of this review, examples of popular splitting criteria are Gini, entropy, and minimum error.⁵

Stopping Rules

To manage the size of a tree, CART analysis allows for predefined stopping rules to minimize the extent of growth while also establishing a minimal degree of statistical difference between nodes that is considered meaningful. To accomplish this task, two stopping rules are often used. The first defines the minimum number of observations in child, or “terminal,” nodes. The second defines the maximum number of levels a tree may grow, thus allowing the investigator to decide the total number of predictor variables that can define a terminal node. While several other stopping rules exist, these are the most commonly utilized.

Pruning

To avoid missing important associations due to premature stoppage, investigators may use another mechanism to limit tree growth called “pruning.” For pruning, the first step is to grow a considerably large tree that includes many levels or nodes, possibly to the point where there are just a few observations per terminal node. Then, similar to the residual sum of squares in a regression, the investigator can calculate a misclassification cost (ie, goodness of fit) and select the tree with the smallest cost.² Of note, stopping rules and pruning can be used simultaneously.

Classification Error

Similar to other forms of statistical inference it remains important to understand the uncertainty within the inference. In regression modeling, for example, classification errors can be calculated using standard errors of the parameter estimates. In CART analysis, because random samples from a population may produce different trees, measures of variability can be more complicated. One strategy is to generate a tree from a test sample and then use the remaining data to calculate a measure of the misclassification cost (a measure of how much additional accuracy a split must add to the entire tree to warrant the additional complexity). Alternatively, a “k-fold cross-validation” can be performed in which the data is broken down into k subsets from which a tree is created using all data except for one of the subsets. The computed tree is then applied to the remaining subset to determine a misclassification cost. These classification costs are important as they also impact the stopping and pruning processes. Ultimately, a final tree, which best limits classification errors, is selected.

This method can be useful in multiple settings in which an investigator wants to characterize a subpopulation from a larger cohort. Adaptation of this could include, but is not limited to, risk stratification,⁶ diagnostics,⁷ and patient identification for medical interventions.⁸ Moreover, CART analysis has the added benefit of creating visually interpretable predictive models that can be utilized for front-line clinical decision making.^9,10

CART analysis has been shown to have several advantages over other commonly used modeling methods. First, it is a nonparametric model that can handle highly skewed data and does not require that the predictor, or predictors, takes on a predetermined form (allowing them to be constructed from the data). This is helpful as many clinical variables can have wide degrees of variance.

Unlike other modeling techniques, CART can identify higher-order interactions between multiple variables, meaning it can handle interactions that occur whenever one variable affects the nature of an interaction between two other variables. Further, CART can handle multiple correlated independent variables, something logistic regression models classically cannot do.

From a clinical standpoint, the “logic” of the visual-based CART output can be easier to interpret than the probabilistic output (eg, odds ratio) associated with logistic regression modeling, making it more practical, applicable, and easier for clinicians to adopt.^10,12 Finally, CART software is easy to use for those who do not have strong statistical backgrounds, and it is less resource intensive than other statistical methods.²

Despite these features, CART does have several disadvantages. First, due to the ease with which CART analysis can be performed, “data dredging” can be a significant concern. Its ideal use is with a priori consideration of independent variables.² Second, while CART is most beneficial in describing links and cutoffs between variables, it may not be useful for hypothesis testing.² Third, large data sets are needed to perform CART, especially if the investigator is using the split sample approach mentioned above.¹¹ Finally, while CART is the most utilized decision tree methodology, several other types of decision tree methods exist: C4.5, CRUISE, Quick, Unbiased, Efficient Statistical Trees, Chi-square-Automatic-Interaction-Detection, and others. Many of these allow for splitting into more than two groups and have other features that may be more advantageous to one’s analysis.¹³

Decision trees offer simple, interpretable results of multiple factors that can be easily applied to clinical scenarios. In this case, the authors specifically used classification tree analysis to take advantage of CART’s machine-learning ability to consider higher-order interactions to build their model—as they lacked a priori evidence to help guide them in traditional (ie, logistic regression) model construction. Furthermore, CART analysis created an output that logically and visually illustrates which combination of characteristics is most associated with discharge placement and can potentially be utilized to help facilitate discharge planning in future hospitalized patients. To sum up, this machine-learning methodology allowed the investigators to determine which variables taken together were the most suitable in predicting their outcome of interest and present these findings in a manner that busy clinicians can interpret and apply.

Using methodology created by the Centers for Medicare & Medicaid Services (CMS), the Department of Veterans Affairs (VA) calculates and reports hospital performance measures for several key conditions, including acute myocardial infarction (AMI), heart failure (HF), and pneumonia.¹ These measures are designed to benchmark individual hospitals against how average hospitals perform when caring for a similar case-mix index. Because readmissions to the hospital within 30-days of discharge are common and costly, this metric has garnered extensive attention in recent years.

To summarize the 30-day readmission metric, the VA utilizes the Strategic Analytics for Improvement and Learning (SAIL) system to present internally its findings to VA practitioners and leadership.² The VA provides these data as a means to drive quality improvement and allow for comparison of individual hospitals’ performance across measures throughout the VA healthcare system. Since 2010, the VA began using and publicly reporting the CMS-derived 30-day Risk-Stratified Readmission Rate (RSRR) on the Hospital Compare website.³ Similar to CMS, the VA uses three years of combined data so that patients, providers, and other stakeholders can compare individual hospitals’ performance across these measures.¹ In response to this, hospitals and healthcare organizations have implemented quality improvement and large-scale programmatic interventions in an attempt to improve quality around readmissions.^4-6 A recent assessment on how hospitals within the Medicare fee-for-service program have responded to such reporting found large degrees of variability, with more than half of the participating institutions facing penalties due to greater-than-expected readmission rates.⁵ Although the VA utilizes the same CMS-derived model in its assessments and reporting, the variability and distribution around this metric are not publicly reported—thus making it difficult to ascertain how individual VA hospitals compare with one another. Without such information, individual facilities may not know how to benchmark the quality of their care to others, nor would the VA recognize which interventions addressing readmissions are working, and which are not. Although previous assessments of interinstitutional variance have been performed in Medicare populations,⁷ a focused analysis of such variance within the VA has yet to be performed.

In this study, we performed a multiyear assessment of the CMS-derived 30-day RSRR metric for AMI, HF, and pneumonia as a useful measure to drive VA quality improvement or distinguish VA facility performance based on its ability to detect interfacility variability.

Data Source

We used VA administrative and Medicare claims data from 2010 to 2012. After identifying index hospitalizations to VA hospitals, we obtained patients’ respective inpatient Medicare claims data from the Medicare Provider Analysis and Review (MedPAR) and Outpatient files. All Medicare records were linked to VA records via scrambled Social Security numbers and were provided by the VA Information Resource Center. This study was approved by the San Francisco VA Medical Center Institutional Review Board.

Study Sample

Our cohort consisted of hospitalized VA beneficiary and Medicare fee-for-service patients who were aged ≥65 years and admitted to and discharged from a VA acute care center with a primary discharge diagnosis of AMI, HF, or pneumonia. These comorbidities were chosen as they are publicly reported and frequently used for interfacility comparisons. Because studies have found that inclusion of secondary payer data (ie, CMS data) may affect hospital-profiling outcomes, we included Medicare data on all available patients.⁸ We excluded hospitalizations that resulted in a transfer to another acute care facility and those admitted to observation status at their index admission. To ensure a full year of data for risk adjustment, beneficiaries were included only if they were enrolled in Medicare for 12 months prior to and including the date of the index admission.

Index hospitalizations were first identified using VA-only inpatient data similar to methods outlined by the CMS and endorsed by the National Quality Forum for Hospital Profiling.⁹ An index hospitalization was defined as an acute inpatient discharge between 2010 and 2012 in which the principal diagnosis was AMI, HF, or pneumonia. We excluded in-hospital deaths, discharges against medical advice, and--for the AMI cohort only--discharges on the same day as admission. Patients may have multiple admissions per year, but only admissions after 30 days of discharge from an index admission were eligible to be included as an additional index admission.

Outcomes

A readmission was defined as any unplanned rehospitalization to either non-VA or VA acute care facilities for any cause within 30 days of discharge from the index hospitalization. Readmissions to observation status or nonacute or rehabilitation units, such as skilled nursing facilities, were not included. Planned readmissions for elective procedures, such as elective chemotherapy and revascularization following an AMI index admission, were not considered as an outcome event.

Risk Standardization for 30-day Readmission

Using approaches developed by CMS,^10-12 we calculated hospital-specific 30-day RSRRs for each VA. Briefly, the RSRR is a ratio of the number of predicted readmissions within 30 days of discharge to the expected number of readmissions within 30 days of hospital discharge, multiplied by the national unadjusted 30-day readmission rate. This measure calculates hospital-specific RSRRs using hierarchical logistic regression models, which account for clustering of patients within hospitals and risk-adjusting for differences in case-mix, during the assessed time periods.¹³ This approach simultaneously models two levels (patient and hospital) to account for the variance in patient outcomes within and between hospitals.¹⁴ At the patient level, the model uses the log odds of readmissions as the dependent variable and age and selected comorbidities as the independent variables. The second level models the hospital-specific intercepts. According to CMS guidelines, the analysis was limited to facilities with at least 25 patient admissions annually for each condition. All readmissions were attributed to the hospital that initially discharged the patient to a nonacute setting.

Analysis

We examined and reported the distribution of patient and clinical characteristics at the hospital level. For each condition, we determined the number of hospitals that had a sufficient number of admissions (n ≥ 25) to be included in the analyses. We calculated the mean, median, and interquartile range for the observed unadjusted readmission rates across all included hospitals.

Similar to methods used by CMS, we used one year of data in the VA to assess hospital quality and variation in facility performance. First, we calculated the 30-day RSRRs using one year (2012) of data. To assess how variability changed with higher facility volume (ie, more years included in the analysis), we also calculated the 30-day RSRRs using two and three years of data. For this, we identified and quantified the number of hospitals whose RSRRs were calculated as being above or below the national VA average (mean ± 95% CI). Specifically, we calculated the number and percentage of hospitals that were classified as either above (+95% CI) or below the national average (−95% CI) using data from all three time periods. All analyses were conducted using SAS Enterprise Guide, Version 7.1. The SAS statistical packages made available by the CMS Measure Team were used to calculate RSRRs.

Patient Characteristics

Patients were predominantly older males (98.3%). Among those hospitalized for AMI, most of them had a history of previous coronary artery bypass graft (CABG) (69.1%), acute coronary syndrome (ACS; 66.2%), or documented coronary atherosclerosis (89.8%). Similarly, patients admitted for HF had high rates of CABG (71.3%) and HF (94.6%), in addition to cardiac arrhythmias (69.3%) and diabetes (60.8%). Patients admitted with a diagnosis of pneumonia had high rates of CABG (61.9%), chronic obstructive pulmonary disease (COPD; 58.1%), and previous diagnosis of pneumonia (78.8%; Table 1). Patient characteristics for two and three years of data are presented in Supplementary Table 1.

VA Hospitals with Sufficient Volume to Be Included in Profiling Assessments

There were 146 acute-care hospitals in the VA. In 2012, 56 (38%) VA hospitals had at least 25 admissions for AMI, 102 (70%) hospitals had at least 25 admissions for CHF, and 106 (73%) hospitals had at least 25 admissions for pneumonia (Table 1) and therefore qualified for analysis based on CMS criteria for 30-day RSRR calculation. The study sample included 3,571 patients with AMI, 10,609 patients with CHF, and 10,191 patients with pneumonia.

jhm014050266_t1.jpg

30-Day Readmission Rates

The mean observed readmission rates in 2012 were 20% (95% CI 19%-21%) among patients admitted for AMI, 20% (95% CI 19%-20%) for patients admitted with CHF, and 15% (95% CI 15%-16%) for patients admitted with pneumonia. No significant variation from these rates was noted following risk standardization across hospitals (Table 2). Observed and risk-standardized rates were also calculated for two and three years of data (Supplementary Table 2) but were not found to be grossly different when utilizing a single year of data.

wray06580220e_t2.jpg

In 2012, two hospitals (2%) exhibited HF RSRRs worse than the national average (+95% CI), whereas no hospital demonstrated worse-than-average rates (+95% CI) for AMI or pneumonia (Table 3, Figure 1). Similarly, in 2012, only three hospitals had RSRRs better than the national average (−95% CI) for HF and pneumonia.

jhm014050266_t3.jpg

wray06580220e_f1.jpg

We found that the CMS-derived 30-day risk-stratified readmission metric for AMI, HF, and pneumonia showed little variation among VA hospitals. The lack of institutional 30-day readmission volume appears to be a fundamental limitation that subsequently requires multiple years of data to make this metric clinically meaningful. As the largest integrated healthcare system in the United States, the VA relies upon and makes large-scale programmatic decisions based on such performance data. The inability to detect meaningful interhospital variation in a timely manner suggests that the CMS-derived 30-day RSRR may not be a sensitive metric to distinguish facility performance or drive quality improvement initiatives within the VA.

First, we found it notable that among the 146 VA medical centers available for analysis,¹⁵ between 38% and 77% of hospitals qualified for evaluation when using CMS-based participation criteria—which excludes institutions with fewer than 25 episodes per year. Although this low degree of qualification for profiling was most dramatic when using one year of data (range: 38%-72%), we noted that it did not dramatically improve when we combined three years of data (range: 52%-77%). These findings act to highlight the population and systems differences between CMS and VA populations¹⁶ and further support the idea that CMS-derived models may not be optimized for use in the VA healthcare system.

Our findings are particularly relevant within the VA given the quarterly rate with which these data are reported within the VA SAIL scorecard.² The VA designed SAIL for internal benchmarking to spotlight successful strategies of top performing institutions and promote high-quality, value-based care. Using one year of data, the minimum required to utilize CMS models, showed that quarterly feedback (ie, three months of data) may not be informative or useful given that few hospitals are able to differentiate themselves from the mean (±95% CI). Although the capacity to distinguish between high and low performers does improve by combining hospital admissions over three years, this is not a reasonable timeline for institutions to wait for quality comparisons. Furthermore, although the VA does present its data on CMS’s Hospital Compare website using three years of combined data, the variability and distribution of such results are not supplied.³

This lack of discriminability raises concerns about the ability to compare hospital performance between low- and high-volume institutions. Although these models function well in CMS settings with large patient volumes in which greater variability exists,⁵ they lose their capacity to discriminate when applied to low-volume settings such as the VA. Given that several hospitals in the US are small community hospitals with low patient volumes,¹⁷ this issue probably occurs in other non-VA settings. Although our study focuses on the VA, others have been able to compare VA and non-VA settings’ variation and distribution. For example, Nuti et al. explored the differences in 30-day RSRRs among hospitalized patients with AMI, HF, and pneumonia and similarly showed little variation, narrow distributions, and few outliers in the VA setting compared to those in the non-VA setting. For small patient volume institutions, including the VA, a focus on high-volume services, outcomes, and measures (ie, blood pressure control, medication reconciliation, etc.) may offer more discriminability between high- and low-performing facilities. For example, Patel et al. found that VA process measures in patients with HF (ie, beta-blocker and ACE-inhibitor use) can be used as valid quality measures as they exhibited consistent reliability over time and validity with adjusted mortality rates, whereas the 30-day RSRR did not.¹⁸

Our findings may have substantial financial, resource, and policy implications. Automatically developing and reporting measures created for the Medicare program in the VA may not be a good use of VA resources. In addition, facilities may react to these reported outcomes and expend local resources and finances to implement interventions to improve on a performance outcome whose measure is statistically no different than the vast majority of its comparators. Such events have been highlighted in the public media and have pointed to the fact that small changes in quality, or statistical errors themselves, can have large ramifications within the VA’s hospital rating system.¹⁹

These findings may also add to the discussion on whether public reporting of health and quality outcomes improves patient care. Since the CMS began public reporting on RSRRs in 2009, these rates have fallen for all three examined conditions (AMI, HF, and pneumonia),^7,20,21 in addition to several other health outcomes.¹⁷ Although recent studies have suggested that these decreased rates have been driven by the CMS-sponsored Hospital Readmissions Reduction Program (HRRP),²² others have suggested that these findings are consistent with ongoing secular trends toward decreased readmissions and may not be completely explained by public reporting alone.²³ Moreover, prior work has also found that readmissions may be strongly impacted by factors external to the hospital setting, such as patients’ social demographics (ie, household income, social isolation), that are not currently captured in risk-prediction models.²⁴ Given the small variability we see in our data, public reporting within the VA is probably not beneficial, as only a small number of facilities are outliers based on RSRR.

Our study has several limitations. First, although we adapted the CMS model to the VA, we did not include gender in the model because >99% of all patient admissions were male. Second, we assessed only three medical conditions that were being tracked by both CMS and VA during this time period, and these outcomes may not be representative of other aspects of care and cannot be generalized to other medical conditions. Finally, more contemporary data could lead to differing results – though we note that no large-scale structural or policy changes addressing readmission rates have been implemented within the VA since our study period.

The results of this study suggest that the CMS-derived 30-day risk-stratified readmission metric for AMI, HF, and pneumonia may not have the capacity to properly detect interfacility variance and thus may not be an optimal quality indicator within the VA. As the VA and other healthcare systems continually strive to improve the quality of care they provide, they will require more accurate and timely metrics for which to index their performance.

Disclosures

The authors have nothing to disclose

Using methodology created by the Centers for Medicare & Medicaid Services (CMS), the Department of Veterans Affairs (VA) calculates and reports hospital performance measures for several key conditions, including acute myocardial infarction (AMI), heart failure (HF), and pneumonia.¹ These measures are designed to benchmark individual hospitals against how average hospitals perform when caring for a similar case-mix index. Because readmissions to the hospital within 30-days of discharge are common and costly, this metric has garnered extensive attention in recent years.

To summarize the 30-day readmission metric, the VA utilizes the Strategic Analytics for Improvement and Learning (SAIL) system to present internally its findings to VA practitioners and leadership.² The VA provides these data as a means to drive quality improvement and allow for comparison of individual hospitals’ performance across measures throughout the VA healthcare system. Since 2010, the VA began using and publicly reporting the CMS-derived 30-day Risk-Stratified Readmission Rate (RSRR) on the Hospital Compare website.³ Similar to CMS, the VA uses three years of combined data so that patients, providers, and other stakeholders can compare individual hospitals’ performance across these measures.¹ In response to this, hospitals and healthcare organizations have implemented quality improvement and large-scale programmatic interventions in an attempt to improve quality around readmissions.^4-6 A recent assessment on how hospitals within the Medicare fee-for-service program have responded to such reporting found large degrees of variability, with more than half of the participating institutions facing penalties due to greater-than-expected readmission rates.⁵ Although the VA utilizes the same CMS-derived model in its assessments and reporting, the variability and distribution around this metric are not publicly reported—thus making it difficult to ascertain how individual VA hospitals compare with one another. Without such information, individual facilities may not know how to benchmark the quality of their care to others, nor would the VA recognize which interventions addressing readmissions are working, and which are not. Although previous assessments of interinstitutional variance have been performed in Medicare populations,⁷ a focused analysis of such variance within the VA has yet to be performed.

In this study, we performed a multiyear assessment of the CMS-derived 30-day RSRR metric for AMI, HF, and pneumonia as a useful measure to drive VA quality improvement or distinguish VA facility performance based on its ability to detect interfacility variability.

Data Source

We used VA administrative and Medicare claims data from 2010 to 2012. After identifying index hospitalizations to VA hospitals, we obtained patients’ respective inpatient Medicare claims data from the Medicare Provider Analysis and Review (MedPAR) and Outpatient files. All Medicare records were linked to VA records via scrambled Social Security numbers and were provided by the VA Information Resource Center. This study was approved by the San Francisco VA Medical Center Institutional Review Board.

Study Sample

Our cohort consisted of hospitalized VA beneficiary and Medicare fee-for-service patients who were aged ≥65 years and admitted to and discharged from a VA acute care center with a primary discharge diagnosis of AMI, HF, or pneumonia. These comorbidities were chosen as they are publicly reported and frequently used for interfacility comparisons. Because studies have found that inclusion of secondary payer data (ie, CMS data) may affect hospital-profiling outcomes, we included Medicare data on all available patients.⁸ We excluded hospitalizations that resulted in a transfer to another acute care facility and those admitted to observation status at their index admission. To ensure a full year of data for risk adjustment, beneficiaries were included only if they were enrolled in Medicare for 12 months prior to and including the date of the index admission.

Index hospitalizations were first identified using VA-only inpatient data similar to methods outlined by the CMS and endorsed by the National Quality Forum for Hospital Profiling.⁹ An index hospitalization was defined as an acute inpatient discharge between 2010 and 2012 in which the principal diagnosis was AMI, HF, or pneumonia. We excluded in-hospital deaths, discharges against medical advice, and--for the AMI cohort only--discharges on the same day as admission. Patients may have multiple admissions per year, but only admissions after 30 days of discharge from an index admission were eligible to be included as an additional index admission.

Outcomes

A readmission was defined as any unplanned rehospitalization to either non-VA or VA acute care facilities for any cause within 30 days of discharge from the index hospitalization. Readmissions to observation status or nonacute or rehabilitation units, such as skilled nursing facilities, were not included. Planned readmissions for elective procedures, such as elective chemotherapy and revascularization following an AMI index admission, were not considered as an outcome event.

Risk Standardization for 30-day Readmission

Using approaches developed by CMS,^10-12 we calculated hospital-specific 30-day RSRRs for each VA. Briefly, the RSRR is a ratio of the number of predicted readmissions within 30 days of discharge to the expected number of readmissions within 30 days of hospital discharge, multiplied by the national unadjusted 30-day readmission rate. This measure calculates hospital-specific RSRRs using hierarchical logistic regression models, which account for clustering of patients within hospitals and risk-adjusting for differences in case-mix, during the assessed time periods.¹³ This approach simultaneously models two levels (patient and hospital) to account for the variance in patient outcomes within and between hospitals.¹⁴ At the patient level, the model uses the log odds of readmissions as the dependent variable and age and selected comorbidities as the independent variables. The second level models the hospital-specific intercepts. According to CMS guidelines, the analysis was limited to facilities with at least 25 patient admissions annually for each condition. All readmissions were attributed to the hospital that initially discharged the patient to a nonacute setting.

Analysis

We examined and reported the distribution of patient and clinical characteristics at the hospital level. For each condition, we determined the number of hospitals that had a sufficient number of admissions (n ≥ 25) to be included in the analyses. We calculated the mean, median, and interquartile range for the observed unadjusted readmission rates across all included hospitals.

Similar to methods used by CMS, we used one year of data in the VA to assess hospital quality and variation in facility performance. First, we calculated the 30-day RSRRs using one year (2012) of data. To assess how variability changed with higher facility volume (ie, more years included in the analysis), we also calculated the 30-day RSRRs using two and three years of data. For this, we identified and quantified the number of hospitals whose RSRRs were calculated as being above or below the national VA average (mean ± 95% CI). Specifically, we calculated the number and percentage of hospitals that were classified as either above (+95% CI) or below the national average (−95% CI) using data from all three time periods. All analyses were conducted using SAS Enterprise Guide, Version 7.1. The SAS statistical packages made available by the CMS Measure Team were used to calculate RSRRs.

Patient Characteristics

Patients were predominantly older males (98.3%). Among those hospitalized for AMI, most of them had a history of previous coronary artery bypass graft (CABG) (69.1%), acute coronary syndrome (ACS; 66.2%), or documented coronary atherosclerosis (89.8%). Similarly, patients admitted for HF had high rates of CABG (71.3%) and HF (94.6%), in addition to cardiac arrhythmias (69.3%) and diabetes (60.8%). Patients admitted with a diagnosis of pneumonia had high rates of CABG (61.9%), chronic obstructive pulmonary disease (COPD; 58.1%), and previous diagnosis of pneumonia (78.8%; Table 1). Patient characteristics for two and three years of data are presented in Supplementary Table 1.

VA Hospitals with Sufficient Volume to Be Included in Profiling Assessments

There were 146 acute-care hospitals in the VA. In 2012, 56 (38%) VA hospitals had at least 25 admissions for AMI, 102 (70%) hospitals had at least 25 admissions for CHF, and 106 (73%) hospitals had at least 25 admissions for pneumonia (Table 1) and therefore qualified for analysis based on CMS criteria for 30-day RSRR calculation. The study sample included 3,571 patients with AMI, 10,609 patients with CHF, and 10,191 patients with pneumonia.

jhm014050266_t1.jpg

30-Day Readmission Rates

The mean observed readmission rates in 2012 were 20% (95% CI 19%-21%) among patients admitted for AMI, 20% (95% CI 19%-20%) for patients admitted with CHF, and 15% (95% CI 15%-16%) for patients admitted with pneumonia. No significant variation from these rates was noted following risk standardization across hospitals (Table 2). Observed and risk-standardized rates were also calculated for two and three years of data (Supplementary Table 2) but were not found to be grossly different when utilizing a single year of data.

wray06580220e_t2.jpg

In 2012, two hospitals (2%) exhibited HF RSRRs worse than the national average (+95% CI), whereas no hospital demonstrated worse-than-average rates (+95% CI) for AMI or pneumonia (Table 3, Figure 1). Similarly, in 2012, only three hospitals had RSRRs better than the national average (−95% CI) for HF and pneumonia.

jhm014050266_t3.jpg

wray06580220e_f1.jpg

We found that the CMS-derived 30-day risk-stratified readmission metric for AMI, HF, and pneumonia showed little variation among VA hospitals. The lack of institutional 30-day readmission volume appears to be a fundamental limitation that subsequently requires multiple years of data to make this metric clinically meaningful. As the largest integrated healthcare system in the United States, the VA relies upon and makes large-scale programmatic decisions based on such performance data. The inability to detect meaningful interhospital variation in a timely manner suggests that the CMS-derived 30-day RSRR may not be a sensitive metric to distinguish facility performance or drive quality improvement initiatives within the VA.

First, we found it notable that among the 146 VA medical centers available for analysis,¹⁵ between 38% and 77% of hospitals qualified for evaluation when using CMS-based participation criteria—which excludes institutions with fewer than 25 episodes per year. Although this low degree of qualification for profiling was most dramatic when using one year of data (range: 38%-72%), we noted that it did not dramatically improve when we combined three years of data (range: 52%-77%). These findings act to highlight the population and systems differences between CMS and VA populations¹⁶ and further support the idea that CMS-derived models may not be optimized for use in the VA healthcare system.

Our findings are particularly relevant within the VA given the quarterly rate with which these data are reported within the VA SAIL scorecard.² The VA designed SAIL for internal benchmarking to spotlight successful strategies of top performing institutions and promote high-quality, value-based care. Using one year of data, the minimum required to utilize CMS models, showed that quarterly feedback (ie, three months of data) may not be informative or useful given that few hospitals are able to differentiate themselves from the mean (±95% CI). Although the capacity to distinguish between high and low performers does improve by combining hospital admissions over three years, this is not a reasonable timeline for institutions to wait for quality comparisons. Furthermore, although the VA does present its data on CMS’s Hospital Compare website using three years of combined data, the variability and distribution of such results are not supplied.³

This lack of discriminability raises concerns about the ability to compare hospital performance between low- and high-volume institutions. Although these models function well in CMS settings with large patient volumes in which greater variability exists,⁵ they lose their capacity to discriminate when applied to low-volume settings such as the VA. Given that several hospitals in the US are small community hospitals with low patient volumes,¹⁷ this issue probably occurs in other non-VA settings. Although our study focuses on the VA, others have been able to compare VA and non-VA settings’ variation and distribution. For example, Nuti et al. explored the differences in 30-day RSRRs among hospitalized patients with AMI, HF, and pneumonia and similarly showed little variation, narrow distributions, and few outliers in the VA setting compared to those in the non-VA setting. For small patient volume institutions, including the VA, a focus on high-volume services, outcomes, and measures (ie, blood pressure control, medication reconciliation, etc.) may offer more discriminability between high- and low-performing facilities. For example, Patel et al. found that VA process measures in patients with HF (ie, beta-blocker and ACE-inhibitor use) can be used as valid quality measures as they exhibited consistent reliability over time and validity with adjusted mortality rates, whereas the 30-day RSRR did not.¹⁸

Our findings may have substantial financial, resource, and policy implications. Automatically developing and reporting measures created for the Medicare program in the VA may not be a good use of VA resources. In addition, facilities may react to these reported outcomes and expend local resources and finances to implement interventions to improve on a performance outcome whose measure is statistically no different than the vast majority of its comparators. Such events have been highlighted in the public media and have pointed to the fact that small changes in quality, or statistical errors themselves, can have large ramifications within the VA’s hospital rating system.¹⁹

These findings may also add to the discussion on whether public reporting of health and quality outcomes improves patient care. Since the CMS began public reporting on RSRRs in 2009, these rates have fallen for all three examined conditions (AMI, HF, and pneumonia),^7,20,21 in addition to several other health outcomes.¹⁷ Although recent studies have suggested that these decreased rates have been driven by the CMS-sponsored Hospital Readmissions Reduction Program (HRRP),²² others have suggested that these findings are consistent with ongoing secular trends toward decreased readmissions and may not be completely explained by public reporting alone.²³ Moreover, prior work has also found that readmissions may be strongly impacted by factors external to the hospital setting, such as patients’ social demographics (ie, household income, social isolation), that are not currently captured in risk-prediction models.²⁴ Given the small variability we see in our data, public reporting within the VA is probably not beneficial, as only a small number of facilities are outliers based on RSRR.

Our study has several limitations. First, although we adapted the CMS model to the VA, we did not include gender in the model because >99% of all patient admissions were male. Second, we assessed only three medical conditions that were being tracked by both CMS and VA during this time period, and these outcomes may not be representative of other aspects of care and cannot be generalized to other medical conditions. Finally, more contemporary data could lead to differing results – though we note that no large-scale structural or policy changes addressing readmission rates have been implemented within the VA since our study period.

The results of this study suggest that the CMS-derived 30-day risk-stratified readmission metric for AMI, HF, and pneumonia may not have the capacity to properly detect interfacility variance and thus may not be an optimal quality indicator within the VA. As the VA and other healthcare systems continually strive to improve the quality of care they provide, they will require more accurate and timely metrics for which to index their performance.

Disclosures

The authors have nothing to disclose

Using methodology created by the Centers for Medicare & Medicaid Services (CMS), the Department of Veterans Affairs (VA) calculates and reports hospital performance measures for several key conditions, including acute myocardial infarction (AMI), heart failure (HF), and pneumonia.¹ These measures are designed to benchmark individual hospitals against how average hospitals perform when caring for a similar case-mix index. Because readmissions to the hospital within 30-days of discharge are common and costly, this metric has garnered extensive attention in recent years.

To summarize the 30-day readmission metric, the VA utilizes the Strategic Analytics for Improvement and Learning (SAIL) system to present internally its findings to VA practitioners and leadership.² The VA provides these data as a means to drive quality improvement and allow for comparison of individual hospitals’ performance across measures throughout the VA healthcare system. Since 2010, the VA began using and publicly reporting the CMS-derived 30-day Risk-Stratified Readmission Rate (RSRR) on the Hospital Compare website.³ Similar to CMS, the VA uses three years of combined data so that patients, providers, and other stakeholders can compare individual hospitals’ performance across these measures.¹ In response to this, hospitals and healthcare organizations have implemented quality improvement and large-scale programmatic interventions in an attempt to improve quality around readmissions.^4-6 A recent assessment on how hospitals within the Medicare fee-for-service program have responded to such reporting found large degrees of variability, with more than half of the participating institutions facing penalties due to greater-than-expected readmission rates.⁵ Although the VA utilizes the same CMS-derived model in its assessments and reporting, the variability and distribution around this metric are not publicly reported—thus making it difficult to ascertain how individual VA hospitals compare with one another. Without such information, individual facilities may not know how to benchmark the quality of their care to others, nor would the VA recognize which interventions addressing readmissions are working, and which are not. Although previous assessments of interinstitutional variance have been performed in Medicare populations,⁷ a focused analysis of such variance within the VA has yet to be performed.

In this study, we performed a multiyear assessment of the CMS-derived 30-day RSRR metric for AMI, HF, and pneumonia as a useful measure to drive VA quality improvement or distinguish VA facility performance based on its ability to detect interfacility variability.

Data Source

We used VA administrative and Medicare claims data from 2010 to 2012. After identifying index hospitalizations to VA hospitals, we obtained patients’ respective inpatient Medicare claims data from the Medicare Provider Analysis and Review (MedPAR) and Outpatient files. All Medicare records were linked to VA records via scrambled Social Security numbers and were provided by the VA Information Resource Center. This study was approved by the San Francisco VA Medical Center Institutional Review Board.

Study Sample

Our cohort consisted of hospitalized VA beneficiary and Medicare fee-for-service patients who were aged ≥65 years and admitted to and discharged from a VA acute care center with a primary discharge diagnosis of AMI, HF, or pneumonia. These comorbidities were chosen as they are publicly reported and frequently used for interfacility comparisons. Because studies have found that inclusion of secondary payer data (ie, CMS data) may affect hospital-profiling outcomes, we included Medicare data on all available patients.⁸ We excluded hospitalizations that resulted in a transfer to another acute care facility and those admitted to observation status at their index admission. To ensure a full year of data for risk adjustment, beneficiaries were included only if they were enrolled in Medicare for 12 months prior to and including the date of the index admission.

Index hospitalizations were first identified using VA-only inpatient data similar to methods outlined by the CMS and endorsed by the National Quality Forum for Hospital Profiling.⁹ An index hospitalization was defined as an acute inpatient discharge between 2010 and 2012 in which the principal diagnosis was AMI, HF, or pneumonia. We excluded in-hospital deaths, discharges against medical advice, and--for the AMI cohort only--discharges on the same day as admission. Patients may have multiple admissions per year, but only admissions after 30 days of discharge from an index admission were eligible to be included as an additional index admission.

Outcomes

A readmission was defined as any unplanned rehospitalization to either non-VA or VA acute care facilities for any cause within 30 days of discharge from the index hospitalization. Readmissions to observation status or nonacute or rehabilitation units, such as skilled nursing facilities, were not included. Planned readmissions for elective procedures, such as elective chemotherapy and revascularization following an AMI index admission, were not considered as an outcome event.

Risk Standardization for 30-day Readmission

Using approaches developed by CMS,^10-12 we calculated hospital-specific 30-day RSRRs for each VA. Briefly, the RSRR is a ratio of the number of predicted readmissions within 30 days of discharge to the expected number of readmissions within 30 days of hospital discharge, multiplied by the national unadjusted 30-day readmission rate. This measure calculates hospital-specific RSRRs using hierarchical logistic regression models, which account for clustering of patients within hospitals and risk-adjusting for differences in case-mix, during the assessed time periods.¹³ This approach simultaneously models two levels (patient and hospital) to account for the variance in patient outcomes within and between hospitals.¹⁴ At the patient level, the model uses the log odds of readmissions as the dependent variable and age and selected comorbidities as the independent variables. The second level models the hospital-specific intercepts. According to CMS guidelines, the analysis was limited to facilities with at least 25 patient admissions annually for each condition. All readmissions were attributed to the hospital that initially discharged the patient to a nonacute setting.

Analysis

We examined and reported the distribution of patient and clinical characteristics at the hospital level. For each condition, we determined the number of hospitals that had a sufficient number of admissions (n ≥ 25) to be included in the analyses. We calculated the mean, median, and interquartile range for the observed unadjusted readmission rates across all included hospitals.

Similar to methods used by CMS, we used one year of data in the VA to assess hospital quality and variation in facility performance. First, we calculated the 30-day RSRRs using one year (2012) of data. To assess how variability changed with higher facility volume (ie, more years included in the analysis), we also calculated the 30-day RSRRs using two and three years of data. For this, we identified and quantified the number of hospitals whose RSRRs were calculated as being above or below the national VA average (mean ± 95% CI). Specifically, we calculated the number and percentage of hospitals that were classified as either above (+95% CI) or below the national average (−95% CI) using data from all three time periods. All analyses were conducted using SAS Enterprise Guide, Version 7.1. The SAS statistical packages made available by the CMS Measure Team were used to calculate RSRRs.

Patient Characteristics

Patients were predominantly older males (98.3%). Among those hospitalized for AMI, most of them had a history of previous coronary artery bypass graft (CABG) (69.1%), acute coronary syndrome (ACS; 66.2%), or documented coronary atherosclerosis (89.8%). Similarly, patients admitted for HF had high rates of CABG (71.3%) and HF (94.6%), in addition to cardiac arrhythmias (69.3%) and diabetes (60.8%). Patients admitted with a diagnosis of pneumonia had high rates of CABG (61.9%), chronic obstructive pulmonary disease (COPD; 58.1%), and previous diagnosis of pneumonia (78.8%; Table 1). Patient characteristics for two and three years of data are presented in Supplementary Table 1.

VA Hospitals with Sufficient Volume to Be Included in Profiling Assessments

There were 146 acute-care hospitals in the VA. In 2012, 56 (38%) VA hospitals had at least 25 admissions for AMI, 102 (70%) hospitals had at least 25 admissions for CHF, and 106 (73%) hospitals had at least 25 admissions for pneumonia (Table 1) and therefore qualified for analysis based on CMS criteria for 30-day RSRR calculation. The study sample included 3,571 patients with AMI, 10,609 patients with CHF, and 10,191 patients with pneumonia.

jhm014050266_t1.jpg

30-Day Readmission Rates

The mean observed readmission rates in 2012 were 20% (95% CI 19%-21%) among patients admitted for AMI, 20% (95% CI 19%-20%) for patients admitted with CHF, and 15% (95% CI 15%-16%) for patients admitted with pneumonia. No significant variation from these rates was noted following risk standardization across hospitals (Table 2). Observed and risk-standardized rates were also calculated for two and three years of data (Supplementary Table 2) but were not found to be grossly different when utilizing a single year of data.

wray06580220e_t2.jpg

In 2012, two hospitals (2%) exhibited HF RSRRs worse than the national average (+95% CI), whereas no hospital demonstrated worse-than-average rates (+95% CI) for AMI or pneumonia (Table 3, Figure 1). Similarly, in 2012, only three hospitals had RSRRs better than the national average (−95% CI) for HF and pneumonia.

jhm014050266_t3.jpg

wray06580220e_f1.jpg

We found that the CMS-derived 30-day risk-stratified readmission metric for AMI, HF, and pneumonia showed little variation among VA hospitals. The lack of institutional 30-day readmission volume appears to be a fundamental limitation that subsequently requires multiple years of data to make this metric clinically meaningful. As the largest integrated healthcare system in the United States, the VA relies upon and makes large-scale programmatic decisions based on such performance data. The inability to detect meaningful interhospital variation in a timely manner suggests that the CMS-derived 30-day RSRR may not be a sensitive metric to distinguish facility performance or drive quality improvement initiatives within the VA.

First, we found it notable that among the 146 VA medical centers available for analysis,¹⁵ between 38% and 77% of hospitals qualified for evaluation when using CMS-based participation criteria—which excludes institutions with fewer than 25 episodes per year. Although this low degree of qualification for profiling was most dramatic when using one year of data (range: 38%-72%), we noted that it did not dramatically improve when we combined three years of data (range: 52%-77%). These findings act to highlight the population and systems differences between CMS and VA populations¹⁶ and further support the idea that CMS-derived models may not be optimized for use in the VA healthcare system.

Our findings are particularly relevant within the VA given the quarterly rate with which these data are reported within the VA SAIL scorecard.² The VA designed SAIL for internal benchmarking to spotlight successful strategies of top performing institutions and promote high-quality, value-based care. Using one year of data, the minimum required to utilize CMS models, showed that quarterly feedback (ie, three months of data) may not be informative or useful given that few hospitals are able to differentiate themselves from the mean (±95% CI). Although the capacity to distinguish between high and low performers does improve by combining hospital admissions over three years, this is not a reasonable timeline for institutions to wait for quality comparisons. Furthermore, although the VA does present its data on CMS’s Hospital Compare website using three years of combined data, the variability and distribution of such results are not supplied.³

This lack of discriminability raises concerns about the ability to compare hospital performance between low- and high-volume institutions. Although these models function well in CMS settings with large patient volumes in which greater variability exists,⁵ they lose their capacity to discriminate when applied to low-volume settings such as the VA. Given that several hospitals in the US are small community hospitals with low patient volumes,¹⁷ this issue probably occurs in other non-VA settings. Although our study focuses on the VA, others have been able to compare VA and non-VA settings’ variation and distribution. For example, Nuti et al. explored the differences in 30-day RSRRs among hospitalized patients with AMI, HF, and pneumonia and similarly showed little variation, narrow distributions, and few outliers in the VA setting compared to those in the non-VA setting. For small patient volume institutions, including the VA, a focus on high-volume services, outcomes, and measures (ie, blood pressure control, medication reconciliation, etc.) may offer more discriminability between high- and low-performing facilities. For example, Patel et al. found that VA process measures in patients with HF (ie, beta-blocker and ACE-inhibitor use) can be used as valid quality measures as they exhibited consistent reliability over time and validity with adjusted mortality rates, whereas the 30-day RSRR did not.¹⁸

Our findings may have substantial financial, resource, and policy implications. Automatically developing and reporting measures created for the Medicare program in the VA may not be a good use of VA resources. In addition, facilities may react to these reported outcomes and expend local resources and finances to implement interventions to improve on a performance outcome whose measure is statistically no different than the vast majority of its comparators. Such events have been highlighted in the public media and have pointed to the fact that small changes in quality, or statistical errors themselves, can have large ramifications within the VA’s hospital rating system.¹⁹

These findings may also add to the discussion on whether public reporting of health and quality outcomes improves patient care. Since the CMS began public reporting on RSRRs in 2009, these rates have fallen for all three examined conditions (AMI, HF, and pneumonia),^7,20,21 in addition to several other health outcomes.¹⁷ Although recent studies have suggested that these decreased rates have been driven by the CMS-sponsored Hospital Readmissions Reduction Program (HRRP),²² others have suggested that these findings are consistent with ongoing secular trends toward decreased readmissions and may not be completely explained by public reporting alone.²³ Moreover, prior work has also found that readmissions may be strongly impacted by factors external to the hospital setting, such as patients’ social demographics (ie, household income, social isolation), that are not currently captured in risk-prediction models.²⁴ Given the small variability we see in our data, public reporting within the VA is probably not beneficial, as only a small number of facilities are outliers based on RSRR.

Our study has several limitations. First, although we adapted the CMS model to the VA, we did not include gender in the model because >99% of all patient admissions were male. Second, we assessed only three medical conditions that were being tracked by both CMS and VA during this time period, and these outcomes may not be representative of other aspects of care and cannot be generalized to other medical conditions. Finally, more contemporary data could lead to differing results – though we note that no large-scale structural or policy changes addressing readmission rates have been implemented within the VA since our study period.

The results of this study suggest that the CMS-derived 30-day risk-stratified readmission metric for AMI, HF, and pneumonia may not have the capacity to properly detect interfacility variance and thus may not be an optimal quality indicator within the VA. As the VA and other healthcare systems continually strive to improve the quality of care they provide, they will require more accurate and timely metrics for which to index their performance.

Disclosures

The authors have nothing to disclose