Martha Quinn, MPH

The demographic composition of physicians has shifted dramatically in the last five decades. The number of women matriculating into medical school rose from 6% in the 1960s¹ to 52% in 2019²; women accounted for 39% of full-time faculty in 2015.³ Despite this evolution of the physician gender array, many challenges remain.⁴ Women represented only 35% of all associate professors and 22% of full professors in 2015.³ Women experience gender-based discrimination, hostility, and unconscious bias as medical trainees^5-9 and as attending physicians^10-13 with significant deleterious effects including burnout and suicidal thoughts.¹⁴ While types of gender-based challenges are well described in the literature, strategies to navigate and respond to these challenges are less understood.

The approaches and techniques of exemplary teaching attending physicians (hereafter referred to as “attendings”) have previously been reported from groups of predominantly male attendings.^15-18 Because of gender-based challenges female physicians face that lead them to reduce their effort or leave the medical field,¹⁹ there is concern that prior scholarship in effective teaching may not adequately capture the approaches and techniques of female attendings. To our knowledge, no studies have specifically examined female attendings. Therefore, we sought to explore the lived experiences of six female attendings with particular emphasis on how they navigate and respond to gender-based challenges in clinical environments.

Study Design and Sampling

This was a multisite study using an exploratory qualitative approach to inquiry. We aimed to examine techniques, approaches, and attitudes of outstanding general medicine teaching attendings among groups previously not well represented (ie, women and self-identified underrepresented minorities [URMs] in medicine). URM was defined by the Association of American Medical Colleges as “those racial and ethnic populations that are underrepresented in the medical profession relative to their numbers in the general population.”²⁰ A modified snowball sampling approach²¹ was employed to identify attendings as delineated below.

To maintain quality while guaranteeing diversity in geography and population, potential institutions in which to observe attendings were determined by first creating the following lists: The top 20 hospitals in the U.S. News & World Report’s 2017-2018 Best Hospitals Honor Roll,²² top-rated institutions by Doximity in each geographic region and among rural training sites,²³ and four historically Black colleges and universities (HBCUs) with medical schools. Institutions visited during a previous similar study¹⁶ were excluded. Next, the list was narrowed to 25 by randomly selecting five in each main geographic region and five rural institutions. These were combined with all four HBCUs to create a final list of 29 institutions.

Next, division of hospital medicine chiefs (and/or general medicine chiefs) and internal medicine residency directors at each of these 29 institutions were asked to nominate exemplary attendings, particularly those who identified as women and URMs. Twelve attendings who were themselves observed in a previous study¹⁶ were also asked for nominations. Finally, recommendations were sought from leaders of relevant American Medical Association member groups.²⁴

Using this sampling method, 43 physicians were identified. An internet search was conducted to identify individual characteristics including medical education, training, clinical and research interests, and educational awards. These characteristics were considered and discussed by the research team. Preference was given to those attendings nominated by more than one individual (n = 3), those who had received teaching awards, and those with interests involving women in medicine. Research team members narrowed the list to seven attendings who were contacted via email and invited to participate. One did not respond, while six agreed to participate. The six attendings identified current team members who would be rounding on the visit date. Attendings were asked to recommend 6-10 former learners; we contacted these former learners and invited them to participate. Former learners were included to understand lasting effects from their attendings.

Data Collection

Observations

All 1-day site visits were conducted by two research team members, a physician (NH) and a qualitative research specialist (MQ). In four visits, an additional author accompanied the research team. In order to ensure consistency and diversity in perspectives, all authors attended at least one visit. These occurred between April 16 and August 28, 2018. Each visit began with direct observation of attendings (n = 6) and current learners (n = 24) during inpatient general medicine teaching rounds. Each researcher unobtrusively recorded their observations via handwritten, open field notes, paying particular attention to group interactions, teaching approach, conversations within and peripheral to the team, and patient–team interactions. After each visit, researchers met to compare and combine field notes.

Interviews and Focus Groups

Researchers then conducted individual, semistructured interviews with attendings and focus groups with current (n = 21) and former (n = 17) learners. Focus groups with learners varied in size from two to five participants. Former learners were occasionally not available for on-site focus groups and were interviewed separately by telephone after the visit. The interview guide for attendings (Appendix 1) was adapted from the prior study¹⁶ but expanded with questions related to experiences, challenges, and approaches of female and URM physicians. A separate guide was used to facilitate focus groups with learners (Appendix 1). Three current learners were unable to participate in focus groups due to clinical duties. All interviews and focus groups were audio recorded and transcribed.

This study was determined to be exempt by the University of Michigan Institutional Review Board. All participants were informed that their participation was completely voluntary and that they could terminate their involvement at any time.

Data Analysis

Data were analyzed using a content analysis approach.²⁵ Inductive coding was used to identify codes derived from the data. Two team members (MQ and MH) independently coded the first transcript to develop a codebook, then met to compare and discuss codes. Codes and definitions were entered into the codebook. These team members continued coding five additional transcripts, meeting to compare codes, discussing any discrepancies until agreement was reached, adding new codes identified, and ensuring consistent code application. They reviewed prior transcripts and recoded if necessary. Once no new codes were identified, one team member coded the remaining transcripts. The same codebook was used to code field note documents using the same iterative process. After all qualitative data were coded and verified, they were entered into NVivo 10. Code reports were generated and reviewed by three team members to identify themes and check for coding consistency.

Role of the Funding Source

This study received no external funding.

We examined six exemplary attendings through direct observation of rounds and individual interviews. We also discussed these attendings with 21 current learners and 17 former learners (Appendix 2). All attendings self-identified as female. The group was diverse in terms of race/ethnicity, with three identifying as Black or African American, two as Asian, and one as White or Caucasian. Levels of experience as an attending ranged from 8 to 20 years (mean, 15.3 years). At the time of observation, two were professors and four were associate professors. The group included all three attendings who had been nominated by more than one individual, and all six had won multiple teaching awards. The observation sites represented several areas of the United States (Table 1).

The coded interview data and field notes were categorized into three broad overlapping themes based on strategies our attendings used to respond to gender-based challenges. The following sections describe types of challenges faced by female attendings along with specific strategies they employed to actively position themselves as physician team leaders, manage gender-based stereotypes and perceptions, and identify and embrace their unique qualities. Illustrative quotations or observations that further elucidate meaning are provided.

Female Attendings Actively Position Themselves as Physician Team Leaders

Our attendings frequently stated that they were assumed to be other healthcare provider types, such as nurses or physical therapists, and that these assumptions originated from patients, faculty, and staff (Table 2). Attending 3 commented, “I think every woman in this role has been mistaken for a different caretaker role, so lots of requests for nursing help. I’m sure I have taken more patients off of bed pans and brought more cups of water than maybe some of my male counterparts.” Some attendings responded to this challenge with the strategy of routinely wearing a white coat during rounds and patient encounters. This external visual cue was seen as a necessary reminder of the female attending role.

We found that patients and healthcare providers often believe teams are led by men, leading to a feeling of invisibility for female attendings. One current learner remarked, “If it was a new patient, more than likely, if we had a female attending, the patient’s eyes would always divert to the male physician.” This was not limited to patients. Attending 6 remembered comments from her consultants including, “‘Who is your attending? Let me talk with them,’ kind of assuming that I’m not the person making the decisions.” Female attendings would respond to this challenge by clearly introducing team members, including themselves, with roles and responsibilities. At times, this would require reintroductions and redirection if individuals still misidentified female team members.

Female attendings’ decision-making and thought processes were frequently second-guessed. This would often lead to power struggles with consultants, nurses, and learners. Attending 5 commented, “Even in residency, I felt this sometimes adversarial relationship with...female nurses where they would treat [female attendings] differently...questioning our decisions.” Female attendings would respond to this challenge by asserting themselves and demonstrating confidence with colleagues and at the bedside. This was an active process for women, as one former learner described: “[Female] attendings have to be a little bit more ‘on’—whatever ‘on’ is—more forceful, more direct....There is more slack given to a male attending.”

Female Attendings Consciously Work to Manage Gender-Based Stereotypes and Perceptions

Our attendings navigated gender-based stereotypes and perceptions, ranging from subtle microaggressions to overt sexual harassment (Table 3). This required balance between extremes of being perceived as “too nice” and “too aggressive,” each of which was associated with negativity. Attending 1 remarked, “I know that other [female] faculty struggle with that a bit, with being...assertive. They are assertive, and it’s interpreted [negatively].” Attending 6 described insidiously sexist comments from patients: “‘You are too young to be a physician, you are too pretty to be a physician.’ ‘Oh, the woman doctor...rather than just ‘doctor.’” During one observation of rounds, a patient remarked to the attending, “You have cold hands. You know, I’m going to have to warm those up.” Our attendings responded to these challenges by proactively avoiding characteristics and behaviors considered to be stereotypically feminine in order to draw attention to their qualities as physicians rather than as women. During interviews, some attendings directed conversation away from themselves and instead placed emphasis on coaching female learners to navigate their own demeanors, behaviors, and responses to gender bias and harassment. This would include intentional planning of how to carry oneself, as well as feedback and debrief sessions after instances of harassment.

Our attendings grappled with how to physically portray themselves to avoid gender-based stereotypes. Attending 6 said, “Sometimes you might be taken less seriously if you pay more attention to your makeup or jewelry.” The same attending recalled “times where people would say inappropriate things based on what I was wearing—and I know that doesn’t happen with my male colleagues.” Our attendings responded to this challenge through purposeful choices of attire, personal appearance, and even external facial expressions that would avoid drawing unwanted or negative personal attention outside of the attending role.

Female Attendings Intentionally Identify and Embrace Their Unique Qualities

Our attendings identified societal gender norms and “traditional” masculine expectations in medicine (Table 4). Attending 4 drew attention to her institution’s healthcare leaders by remarking, “I think that women in medicine have similar challenges as women in other professional fields....Well, I guess it is different in that the pictures on the wall behind me are all White men.” Female attendings responded to this challenge by eschewing stereotypical qualities and intentionally finding and exhibiting their own unique strengths (eg, teaching approaches, areas of expertise, communication styles). By embracing their unique strengths, attendings gained confidence and felt more comfortable as physicians and educators. Advice from Attending 3 for other female physicians encapsulated this strategy: “But if [medicine] is what you love doing, then find a style that works for you, even if it’s different....Embrace being different.”

Several attendings identified patterns of thought in themselves that caused them to doubt their accomplishments and have a persistent fear of being exposed as a fraud, commonly known as impostor syndrome. Attending 2 summarized this with, “I know it’s irrational a little bit, but part of me [asks], ‘Am I getting all these opportunities because I’m female, because I’m a minority?’” Our attendings responded by recognizing impostor syndrome and addressing it through repeated positive self-reinforcing thoughts and language and by “letting go” of the doubt. Attending 4 recalled her feelings after being announced as a teaching award recipient for the fourth year in a row: “It was just like something changed in me....Maybe you are a good attending. Maybe you are doing something that is resonating with a unique class of medical students year after year.”

Our interviews also revealed strategies used by female attendings to support and advance their own careers, as well as those of other female faculty, to address the effects of impostor syndrome. Our participants noted the important role of female mentors and sponsors. One former learner mentioned, “I think some of the administration, there are definitely females that are helping promote [the attending].” During an observation, Attending 1 indicated that she was part of a network of women and junior faculty forged to promote each other’s work since “some people are good at self-promotion and some are not.” This group shares accomplishments by distributing and publicizing their accolades.

This multisite, qualitative study informs the complex ways in which exemplary female teaching attendings must navigate being women in medicine. We identified myriad challenges female attendings face originating from patients, from healthcare workers, and within themselves. Our attendings relied upon the following key strategies to mitigate such challenges: (1) they actively position themselves as physician team leaders, (2) they consciously work to manage gender-based stereotypes and perceptions, and (3) they intentionally identify and embrace their unique qualities.

Prior scholarship surrounding gender-based challenges has focused primarily on strategies to improve healthcare systems for women. Much scrutiny has been placed on elevating institutional culture,^26-29 enacting clear policy surrounding sexual harassment,³⁰ ensuring women are actively recruited and retained,³¹ providing resources to assist in work-life balance,^26,32 and cultivating effective mentorship and social networks.^11,33,34

While our findings support the importance of improving healthcare systems, they are more congruent with recent scholarship on explicit personal tactics to mitigate gender-based challenges. Researchers have suggested physicians use algorithmic responses to patient-initiated sexual harassment,³⁵ advocate for those who experience harassment in real time,³⁶ and engage in dedicated practice responding to harassment.^37,38 Our results build on these studies by outlining strategies intended to navigate complex gender dynamics and role model approaches for learners. Interestingly, it was more common for attendings to discuss how they guide their learners and debrief after difficult situations than to discuss how they personally respond to gender-based harassment. While we are not certain why this occurred, three factors may have contributed. First, attendings mentioned that these conversations are often uncomfortable. Second, attendings appeared to accept a higher level of gender-based challenges than they would have tolerated for their learners. Lastly, although we did not gather demographic data from learners, several attendings voiced a strong desire to advocate for and equip female learners with strategies to address and navigate these challenges for themselves.

Gender stereotypes are ubiquitous and firmly rooted in long-standing belief patterns. Certain characteristics are considered masculine (eg, aggressiveness, confidence) and others feminine (eg, kindness, cooperation).¹⁰ Role congruity theory purports that stereotypes lead women to demonstrate behaviors that reflect socially accepted gender norms³⁹ and that social approval is at risk if they behave in ways discordant with these norms.^10,40 Our study provides perspectives from female physicians who walk the tightrope of forcefully asserting themselves more than their male counterparts while not being overly aggressive, since both approaches may have negative connotations.

This study has several limitations. First, it was conducted with a limited number of site visits, attendings, and learners. Likewise, attendings were internists with relatively advanced academic rank. This may reduce the study’s generalizability since attendings in other fields and at earlier career stages may utilize different strategies. However, we believe that if more senior-level female attendings experienced difficulties being recognized and legitimized in their roles, then one can assume that junior-level female faculty would experience these challenges even more so. Likewise, data saturation was not the goal of this exploratory study. Through intensive qualitative data collection, we sought to obtain an in-depth understanding of challenges and strategies. Second, many exemplary female attendings were overlooked by our selection methodology, particularly since women are often underrepresented in the factors we chose. The multisite design, modified snowball sampling, and purposeful randomized selection methodology were used to ensure quality and diversity. Third, attendings provided lists of their former learners, and thus, selection and recall biases may have been introduced since attendings may have more readily identified learners with whom they formed positive relationships. Finally, we cannot eliminate a potential Hawthorne effect on data collection. Researchers attempted to lessen this by standing apart from teams and remaining unobtrusive.

We identified strategies employed by exemplary female attendings to navigate gender-based challenges in their workplaces. We found that female attendings face unconscious bias, labels, power struggles, and harassment, simply because of their gender. They consciously and constantly navigate these challenges by positioning themselves to be seen and heard as team leaders, balancing aspects of their outward appearance and demeanor, embracing their differences and avoiding assimilation to masculine stereotypes of physician leaders, working to manage self-doubt, and coaching their female learners in these areas.

The authors are indebted to Suzanne Winter, MS, for assisting with coordination of study participants and site visits.

The demographic composition of physicians has shifted dramatically in the last five decades. The number of women matriculating into medical school rose from 6% in the 1960s¹ to 52% in 2019²; women accounted for 39% of full-time faculty in 2015.³ Despite this evolution of the physician gender array, many challenges remain.⁴ Women represented only 35% of all associate professors and 22% of full professors in 2015.³ Women experience gender-based discrimination, hostility, and unconscious bias as medical trainees^5-9 and as attending physicians^10-13 with significant deleterious effects including burnout and suicidal thoughts.¹⁴ While types of gender-based challenges are well described in the literature, strategies to navigate and respond to these challenges are less understood.

The approaches and techniques of exemplary teaching attending physicians (hereafter referred to as “attendings”) have previously been reported from groups of predominantly male attendings.^15-18 Because of gender-based challenges female physicians face that lead them to reduce their effort or leave the medical field,¹⁹ there is concern that prior scholarship in effective teaching may not adequately capture the approaches and techniques of female attendings. To our knowledge, no studies have specifically examined female attendings. Therefore, we sought to explore the lived experiences of six female attendings with particular emphasis on how they navigate and respond to gender-based challenges in clinical environments.

Study Design and Sampling

This was a multisite study using an exploratory qualitative approach to inquiry. We aimed to examine techniques, approaches, and attitudes of outstanding general medicine teaching attendings among groups previously not well represented (ie, women and self-identified underrepresented minorities [URMs] in medicine). URM was defined by the Association of American Medical Colleges as “those racial and ethnic populations that are underrepresented in the medical profession relative to their numbers in the general population.”²⁰ A modified snowball sampling approach²¹ was employed to identify attendings as delineated below.

To maintain quality while guaranteeing diversity in geography and population, potential institutions in which to observe attendings were determined by first creating the following lists: The top 20 hospitals in the U.S. News & World Report’s 2017-2018 Best Hospitals Honor Roll,²² top-rated institutions by Doximity in each geographic region and among rural training sites,²³ and four historically Black colleges and universities (HBCUs) with medical schools. Institutions visited during a previous similar study¹⁶ were excluded. Next, the list was narrowed to 25 by randomly selecting five in each main geographic region and five rural institutions. These were combined with all four HBCUs to create a final list of 29 institutions.

Next, division of hospital medicine chiefs (and/or general medicine chiefs) and internal medicine residency directors at each of these 29 institutions were asked to nominate exemplary attendings, particularly those who identified as women and URMs. Twelve attendings who were themselves observed in a previous study¹⁶ were also asked for nominations. Finally, recommendations were sought from leaders of relevant American Medical Association member groups.²⁴

Using this sampling method, 43 physicians were identified. An internet search was conducted to identify individual characteristics including medical education, training, clinical and research interests, and educational awards. These characteristics were considered and discussed by the research team. Preference was given to those attendings nominated by more than one individual (n = 3), those who had received teaching awards, and those with interests involving women in medicine. Research team members narrowed the list to seven attendings who were contacted via email and invited to participate. One did not respond, while six agreed to participate. The six attendings identified current team members who would be rounding on the visit date. Attendings were asked to recommend 6-10 former learners; we contacted these former learners and invited them to participate. Former learners were included to understand lasting effects from their attendings.

Data Collection

Observations

All 1-day site visits were conducted by two research team members, a physician (NH) and a qualitative research specialist (MQ). In four visits, an additional author accompanied the research team. In order to ensure consistency and diversity in perspectives, all authors attended at least one visit. These occurred between April 16 and August 28, 2018. Each visit began with direct observation of attendings (n = 6) and current learners (n = 24) during inpatient general medicine teaching rounds. Each researcher unobtrusively recorded their observations via handwritten, open field notes, paying particular attention to group interactions, teaching approach, conversations within and peripheral to the team, and patient–team interactions. After each visit, researchers met to compare and combine field notes.

Interviews and Focus Groups

Researchers then conducted individual, semistructured interviews with attendings and focus groups with current (n = 21) and former (n = 17) learners. Focus groups with learners varied in size from two to five participants. Former learners were occasionally not available for on-site focus groups and were interviewed separately by telephone after the visit. The interview guide for attendings (Appendix 1) was adapted from the prior study¹⁶ but expanded with questions related to experiences, challenges, and approaches of female and URM physicians. A separate guide was used to facilitate focus groups with learners (Appendix 1). Three current learners were unable to participate in focus groups due to clinical duties. All interviews and focus groups were audio recorded and transcribed.

This study was determined to be exempt by the University of Michigan Institutional Review Board. All participants were informed that their participation was completely voluntary and that they could terminate their involvement at any time.

Data Analysis

Data were analyzed using a content analysis approach.²⁵ Inductive coding was used to identify codes derived from the data. Two team members (MQ and MH) independently coded the first transcript to develop a codebook, then met to compare and discuss codes. Codes and definitions were entered into the codebook. These team members continued coding five additional transcripts, meeting to compare codes, discussing any discrepancies until agreement was reached, adding new codes identified, and ensuring consistent code application. They reviewed prior transcripts and recoded if necessary. Once no new codes were identified, one team member coded the remaining transcripts. The same codebook was used to code field note documents using the same iterative process. After all qualitative data were coded and verified, they were entered into NVivo 10. Code reports were generated and reviewed by three team members to identify themes and check for coding consistency.

Role of the Funding Source

This study received no external funding.

We examined six exemplary attendings through direct observation of rounds and individual interviews. We also discussed these attendings with 21 current learners and 17 former learners (Appendix 2). All attendings self-identified as female. The group was diverse in terms of race/ethnicity, with three identifying as Black or African American, two as Asian, and one as White or Caucasian. Levels of experience as an attending ranged from 8 to 20 years (mean, 15.3 years). At the time of observation, two were professors and four were associate professors. The group included all three attendings who had been nominated by more than one individual, and all six had won multiple teaching awards. The observation sites represented several areas of the United States (Table 1).

The coded interview data and field notes were categorized into three broad overlapping themes based on strategies our attendings used to respond to gender-based challenges. The following sections describe types of challenges faced by female attendings along with specific strategies they employed to actively position themselves as physician team leaders, manage gender-based stereotypes and perceptions, and identify and embrace their unique qualities. Illustrative quotations or observations that further elucidate meaning are provided.

Female Attendings Actively Position Themselves as Physician Team Leaders

Our attendings frequently stated that they were assumed to be other healthcare provider types, such as nurses or physical therapists, and that these assumptions originated from patients, faculty, and staff (Table 2). Attending 3 commented, “I think every woman in this role has been mistaken for a different caretaker role, so lots of requests for nursing help. I’m sure I have taken more patients off of bed pans and brought more cups of water than maybe some of my male counterparts.” Some attendings responded to this challenge with the strategy of routinely wearing a white coat during rounds and patient encounters. This external visual cue was seen as a necessary reminder of the female attending role.

We found that patients and healthcare providers often believe teams are led by men, leading to a feeling of invisibility for female attendings. One current learner remarked, “If it was a new patient, more than likely, if we had a female attending, the patient’s eyes would always divert to the male physician.” This was not limited to patients. Attending 6 remembered comments from her consultants including, “‘Who is your attending? Let me talk with them,’ kind of assuming that I’m not the person making the decisions.” Female attendings would respond to this challenge by clearly introducing team members, including themselves, with roles and responsibilities. At times, this would require reintroductions and redirection if individuals still misidentified female team members.

Female attendings’ decision-making and thought processes were frequently second-guessed. This would often lead to power struggles with consultants, nurses, and learners. Attending 5 commented, “Even in residency, I felt this sometimes adversarial relationship with...female nurses where they would treat [female attendings] differently...questioning our decisions.” Female attendings would respond to this challenge by asserting themselves and demonstrating confidence with colleagues and at the bedside. This was an active process for women, as one former learner described: “[Female] attendings have to be a little bit more ‘on’—whatever ‘on’ is—more forceful, more direct....There is more slack given to a male attending.”

Female Attendings Consciously Work to Manage Gender-Based Stereotypes and Perceptions

Our attendings navigated gender-based stereotypes and perceptions, ranging from subtle microaggressions to overt sexual harassment (Table 3). This required balance between extremes of being perceived as “too nice” and “too aggressive,” each of which was associated with negativity. Attending 1 remarked, “I know that other [female] faculty struggle with that a bit, with being...assertive. They are assertive, and it’s interpreted [negatively].” Attending 6 described insidiously sexist comments from patients: “‘You are too young to be a physician, you are too pretty to be a physician.’ ‘Oh, the woman doctor...rather than just ‘doctor.’” During one observation of rounds, a patient remarked to the attending, “You have cold hands. You know, I’m going to have to warm those up.” Our attendings responded to these challenges by proactively avoiding characteristics and behaviors considered to be stereotypically feminine in order to draw attention to their qualities as physicians rather than as women. During interviews, some attendings directed conversation away from themselves and instead placed emphasis on coaching female learners to navigate their own demeanors, behaviors, and responses to gender bias and harassment. This would include intentional planning of how to carry oneself, as well as feedback and debrief sessions after instances of harassment.

Our attendings grappled with how to physically portray themselves to avoid gender-based stereotypes. Attending 6 said, “Sometimes you might be taken less seriously if you pay more attention to your makeup or jewelry.” The same attending recalled “times where people would say inappropriate things based on what I was wearing—and I know that doesn’t happen with my male colleagues.” Our attendings responded to this challenge through purposeful choices of attire, personal appearance, and even external facial expressions that would avoid drawing unwanted or negative personal attention outside of the attending role.

Female Attendings Intentionally Identify and Embrace Their Unique Qualities

Our attendings identified societal gender norms and “traditional” masculine expectations in medicine (Table 4). Attending 4 drew attention to her institution’s healthcare leaders by remarking, “I think that women in medicine have similar challenges as women in other professional fields....Well, I guess it is different in that the pictures on the wall behind me are all White men.” Female attendings responded to this challenge by eschewing stereotypical qualities and intentionally finding and exhibiting their own unique strengths (eg, teaching approaches, areas of expertise, communication styles). By embracing their unique strengths, attendings gained confidence and felt more comfortable as physicians and educators. Advice from Attending 3 for other female physicians encapsulated this strategy: “But if [medicine] is what you love doing, then find a style that works for you, even if it’s different....Embrace being different.”

Several attendings identified patterns of thought in themselves that caused them to doubt their accomplishments and have a persistent fear of being exposed as a fraud, commonly known as impostor syndrome. Attending 2 summarized this with, “I know it’s irrational a little bit, but part of me [asks], ‘Am I getting all these opportunities because I’m female, because I’m a minority?’” Our attendings responded by recognizing impostor syndrome and addressing it through repeated positive self-reinforcing thoughts and language and by “letting go” of the doubt. Attending 4 recalled her feelings after being announced as a teaching award recipient for the fourth year in a row: “It was just like something changed in me....Maybe you are a good attending. Maybe you are doing something that is resonating with a unique class of medical students year after year.”

Our interviews also revealed strategies used by female attendings to support and advance their own careers, as well as those of other female faculty, to address the effects of impostor syndrome. Our participants noted the important role of female mentors and sponsors. One former learner mentioned, “I think some of the administration, there are definitely females that are helping promote [the attending].” During an observation, Attending 1 indicated that she was part of a network of women and junior faculty forged to promote each other’s work since “some people are good at self-promotion and some are not.” This group shares accomplishments by distributing and publicizing their accolades.

This multisite, qualitative study informs the complex ways in which exemplary female teaching attendings must navigate being women in medicine. We identified myriad challenges female attendings face originating from patients, from healthcare workers, and within themselves. Our attendings relied upon the following key strategies to mitigate such challenges: (1) they actively position themselves as physician team leaders, (2) they consciously work to manage gender-based stereotypes and perceptions, and (3) they intentionally identify and embrace their unique qualities.

Prior scholarship surrounding gender-based challenges has focused primarily on strategies to improve healthcare systems for women. Much scrutiny has been placed on elevating institutional culture,^26-29 enacting clear policy surrounding sexual harassment,³⁰ ensuring women are actively recruited and retained,³¹ providing resources to assist in work-life balance,^26,32 and cultivating effective mentorship and social networks.^11,33,34

While our findings support the importance of improving healthcare systems, they are more congruent with recent scholarship on explicit personal tactics to mitigate gender-based challenges. Researchers have suggested physicians use algorithmic responses to patient-initiated sexual harassment,³⁵ advocate for those who experience harassment in real time,³⁶ and engage in dedicated practice responding to harassment.^37,38 Our results build on these studies by outlining strategies intended to navigate complex gender dynamics and role model approaches for learners. Interestingly, it was more common for attendings to discuss how they guide their learners and debrief after difficult situations than to discuss how they personally respond to gender-based harassment. While we are not certain why this occurred, three factors may have contributed. First, attendings mentioned that these conversations are often uncomfortable. Second, attendings appeared to accept a higher level of gender-based challenges than they would have tolerated for their learners. Lastly, although we did not gather demographic data from learners, several attendings voiced a strong desire to advocate for and equip female learners with strategies to address and navigate these challenges for themselves.

Gender stereotypes are ubiquitous and firmly rooted in long-standing belief patterns. Certain characteristics are considered masculine (eg, aggressiveness, confidence) and others feminine (eg, kindness, cooperation).¹⁰ Role congruity theory purports that stereotypes lead women to demonstrate behaviors that reflect socially accepted gender norms³⁹ and that social approval is at risk if they behave in ways discordant with these norms.^10,40 Our study provides perspectives from female physicians who walk the tightrope of forcefully asserting themselves more than their male counterparts while not being overly aggressive, since both approaches may have negative connotations.

This study has several limitations. First, it was conducted with a limited number of site visits, attendings, and learners. Likewise, attendings were internists with relatively advanced academic rank. This may reduce the study’s generalizability since attendings in other fields and at earlier career stages may utilize different strategies. However, we believe that if more senior-level female attendings experienced difficulties being recognized and legitimized in their roles, then one can assume that junior-level female faculty would experience these challenges even more so. Likewise, data saturation was not the goal of this exploratory study. Through intensive qualitative data collection, we sought to obtain an in-depth understanding of challenges and strategies. Second, many exemplary female attendings were overlooked by our selection methodology, particularly since women are often underrepresented in the factors we chose. The multisite design, modified snowball sampling, and purposeful randomized selection methodology were used to ensure quality and diversity. Third, attendings provided lists of their former learners, and thus, selection and recall biases may have been introduced since attendings may have more readily identified learners with whom they formed positive relationships. Finally, we cannot eliminate a potential Hawthorne effect on data collection. Researchers attempted to lessen this by standing apart from teams and remaining unobtrusive.

We identified strategies employed by exemplary female attendings to navigate gender-based challenges in their workplaces. We found that female attendings face unconscious bias, labels, power struggles, and harassment, simply because of their gender. They consciously and constantly navigate these challenges by positioning themselves to be seen and heard as team leaders, balancing aspects of their outward appearance and demeanor, embracing their differences and avoiding assimilation to masculine stereotypes of physician leaders, working to manage self-doubt, and coaching their female learners in these areas.

The authors are indebted to Suzanne Winter, MS, for assisting with coordination of study participants and site visits.

The demographic composition of physicians has shifted dramatically in the last five decades. The number of women matriculating into medical school rose from 6% in the 1960s¹ to 52% in 2019²; women accounted for 39% of full-time faculty in 2015.³ Despite this evolution of the physician gender array, many challenges remain.⁴ Women represented only 35% of all associate professors and 22% of full professors in 2015.³ Women experience gender-based discrimination, hostility, and unconscious bias as medical trainees^5-9 and as attending physicians^10-13 with significant deleterious effects including burnout and suicidal thoughts.¹⁴ While types of gender-based challenges are well described in the literature, strategies to navigate and respond to these challenges are less understood.

The approaches and techniques of exemplary teaching attending physicians (hereafter referred to as “attendings”) have previously been reported from groups of predominantly male attendings.^15-18 Because of gender-based challenges female physicians face that lead them to reduce their effort or leave the medical field,¹⁹ there is concern that prior scholarship in effective teaching may not adequately capture the approaches and techniques of female attendings. To our knowledge, no studies have specifically examined female attendings. Therefore, we sought to explore the lived experiences of six female attendings with particular emphasis on how they navigate and respond to gender-based challenges in clinical environments.

Study Design and Sampling

This was a multisite study using an exploratory qualitative approach to inquiry. We aimed to examine techniques, approaches, and attitudes of outstanding general medicine teaching attendings among groups previously not well represented (ie, women and self-identified underrepresented minorities [URMs] in medicine). URM was defined by the Association of American Medical Colleges as “those racial and ethnic populations that are underrepresented in the medical profession relative to their numbers in the general population.”²⁰ A modified snowball sampling approach²¹ was employed to identify attendings as delineated below.

To maintain quality while guaranteeing diversity in geography and population, potential institutions in which to observe attendings were determined by first creating the following lists: The top 20 hospitals in the U.S. News & World Report’s 2017-2018 Best Hospitals Honor Roll,²² top-rated institutions by Doximity in each geographic region and among rural training sites,²³ and four historically Black colleges and universities (HBCUs) with medical schools. Institutions visited during a previous similar study¹⁶ were excluded. Next, the list was narrowed to 25 by randomly selecting five in each main geographic region and five rural institutions. These were combined with all four HBCUs to create a final list of 29 institutions.

Next, division of hospital medicine chiefs (and/or general medicine chiefs) and internal medicine residency directors at each of these 29 institutions were asked to nominate exemplary attendings, particularly those who identified as women and URMs. Twelve attendings who were themselves observed in a previous study¹⁶ were also asked for nominations. Finally, recommendations were sought from leaders of relevant American Medical Association member groups.²⁴

Using this sampling method, 43 physicians were identified. An internet search was conducted to identify individual characteristics including medical education, training, clinical and research interests, and educational awards. These characteristics were considered and discussed by the research team. Preference was given to those attendings nominated by more than one individual (n = 3), those who had received teaching awards, and those with interests involving women in medicine. Research team members narrowed the list to seven attendings who were contacted via email and invited to participate. One did not respond, while six agreed to participate. The six attendings identified current team members who would be rounding on the visit date. Attendings were asked to recommend 6-10 former learners; we contacted these former learners and invited them to participate. Former learners were included to understand lasting effects from their attendings.

Data Collection

Observations

All 1-day site visits were conducted by two research team members, a physician (NH) and a qualitative research specialist (MQ). In four visits, an additional author accompanied the research team. In order to ensure consistency and diversity in perspectives, all authors attended at least one visit. These occurred between April 16 and August 28, 2018. Each visit began with direct observation of attendings (n = 6) and current learners (n = 24) during inpatient general medicine teaching rounds. Each researcher unobtrusively recorded their observations via handwritten, open field notes, paying particular attention to group interactions, teaching approach, conversations within and peripheral to the team, and patient–team interactions. After each visit, researchers met to compare and combine field notes.

Interviews and Focus Groups

Researchers then conducted individual, semistructured interviews with attendings and focus groups with current (n = 21) and former (n = 17) learners. Focus groups with learners varied in size from two to five participants. Former learners were occasionally not available for on-site focus groups and were interviewed separately by telephone after the visit. The interview guide for attendings (Appendix 1) was adapted from the prior study¹⁶ but expanded with questions related to experiences, challenges, and approaches of female and URM physicians. A separate guide was used to facilitate focus groups with learners (Appendix 1). Three current learners were unable to participate in focus groups due to clinical duties. All interviews and focus groups were audio recorded and transcribed.

This study was determined to be exempt by the University of Michigan Institutional Review Board. All participants were informed that their participation was completely voluntary and that they could terminate their involvement at any time.

Data Analysis

Data were analyzed using a content analysis approach.²⁵ Inductive coding was used to identify codes derived from the data. Two team members (MQ and MH) independently coded the first transcript to develop a codebook, then met to compare and discuss codes. Codes and definitions were entered into the codebook. These team members continued coding five additional transcripts, meeting to compare codes, discussing any discrepancies until agreement was reached, adding new codes identified, and ensuring consistent code application. They reviewed prior transcripts and recoded if necessary. Once no new codes were identified, one team member coded the remaining transcripts. The same codebook was used to code field note documents using the same iterative process. After all qualitative data were coded and verified, they were entered into NVivo 10. Code reports were generated and reviewed by three team members to identify themes and check for coding consistency.

Role of the Funding Source

This study received no external funding.

We examined six exemplary attendings through direct observation of rounds and individual interviews. We also discussed these attendings with 21 current learners and 17 former learners (Appendix 2). All attendings self-identified as female. The group was diverse in terms of race/ethnicity, with three identifying as Black or African American, two as Asian, and one as White or Caucasian. Levels of experience as an attending ranged from 8 to 20 years (mean, 15.3 years). At the time of observation, two were professors and four were associate professors. The group included all three attendings who had been nominated by more than one individual, and all six had won multiple teaching awards. The observation sites represented several areas of the United States (Table 1).

The coded interview data and field notes were categorized into three broad overlapping themes based on strategies our attendings used to respond to gender-based challenges. The following sections describe types of challenges faced by female attendings along with specific strategies they employed to actively position themselves as physician team leaders, manage gender-based stereotypes and perceptions, and identify and embrace their unique qualities. Illustrative quotations or observations that further elucidate meaning are provided.

Female Attendings Actively Position Themselves as Physician Team Leaders

Our attendings frequently stated that they were assumed to be other healthcare provider types, such as nurses or physical therapists, and that these assumptions originated from patients, faculty, and staff (Table 2). Attending 3 commented, “I think every woman in this role has been mistaken for a different caretaker role, so lots of requests for nursing help. I’m sure I have taken more patients off of bed pans and brought more cups of water than maybe some of my male counterparts.” Some attendings responded to this challenge with the strategy of routinely wearing a white coat during rounds and patient encounters. This external visual cue was seen as a necessary reminder of the female attending role.

We found that patients and healthcare providers often believe teams are led by men, leading to a feeling of invisibility for female attendings. One current learner remarked, “If it was a new patient, more than likely, if we had a female attending, the patient’s eyes would always divert to the male physician.” This was not limited to patients. Attending 6 remembered comments from her consultants including, “‘Who is your attending? Let me talk with them,’ kind of assuming that I’m not the person making the decisions.” Female attendings would respond to this challenge by clearly introducing team members, including themselves, with roles and responsibilities. At times, this would require reintroductions and redirection if individuals still misidentified female team members.

Female attendings’ decision-making and thought processes were frequently second-guessed. This would often lead to power struggles with consultants, nurses, and learners. Attending 5 commented, “Even in residency, I felt this sometimes adversarial relationship with...female nurses where they would treat [female attendings] differently...questioning our decisions.” Female attendings would respond to this challenge by asserting themselves and demonstrating confidence with colleagues and at the bedside. This was an active process for women, as one former learner described: “[Female] attendings have to be a little bit more ‘on’—whatever ‘on’ is—more forceful, more direct....There is more slack given to a male attending.”

Female Attendings Consciously Work to Manage Gender-Based Stereotypes and Perceptions

Our attendings navigated gender-based stereotypes and perceptions, ranging from subtle microaggressions to overt sexual harassment (Table 3). This required balance between extremes of being perceived as “too nice” and “too aggressive,” each of which was associated with negativity. Attending 1 remarked, “I know that other [female] faculty struggle with that a bit, with being...assertive. They are assertive, and it’s interpreted [negatively].” Attending 6 described insidiously sexist comments from patients: “‘You are too young to be a physician, you are too pretty to be a physician.’ ‘Oh, the woman doctor...rather than just ‘doctor.’” During one observation of rounds, a patient remarked to the attending, “You have cold hands. You know, I’m going to have to warm those up.” Our attendings responded to these challenges by proactively avoiding characteristics and behaviors considered to be stereotypically feminine in order to draw attention to their qualities as physicians rather than as women. During interviews, some attendings directed conversation away from themselves and instead placed emphasis on coaching female learners to navigate their own demeanors, behaviors, and responses to gender bias and harassment. This would include intentional planning of how to carry oneself, as well as feedback and debrief sessions after instances of harassment.

Our attendings grappled with how to physically portray themselves to avoid gender-based stereotypes. Attending 6 said, “Sometimes you might be taken less seriously if you pay more attention to your makeup or jewelry.” The same attending recalled “times where people would say inappropriate things based on what I was wearing—and I know that doesn’t happen with my male colleagues.” Our attendings responded to this challenge through purposeful choices of attire, personal appearance, and even external facial expressions that would avoid drawing unwanted or negative personal attention outside of the attending role.

Female Attendings Intentionally Identify and Embrace Their Unique Qualities

Our attendings identified societal gender norms and “traditional” masculine expectations in medicine (Table 4). Attending 4 drew attention to her institution’s healthcare leaders by remarking, “I think that women in medicine have similar challenges as women in other professional fields....Well, I guess it is different in that the pictures on the wall behind me are all White men.” Female attendings responded to this challenge by eschewing stereotypical qualities and intentionally finding and exhibiting their own unique strengths (eg, teaching approaches, areas of expertise, communication styles). By embracing their unique strengths, attendings gained confidence and felt more comfortable as physicians and educators. Advice from Attending 3 for other female physicians encapsulated this strategy: “But if [medicine] is what you love doing, then find a style that works for you, even if it’s different....Embrace being different.”

Several attendings identified patterns of thought in themselves that caused them to doubt their accomplishments and have a persistent fear of being exposed as a fraud, commonly known as impostor syndrome. Attending 2 summarized this with, “I know it’s irrational a little bit, but part of me [asks], ‘Am I getting all these opportunities because I’m female, because I’m a minority?’” Our attendings responded by recognizing impostor syndrome and addressing it through repeated positive self-reinforcing thoughts and language and by “letting go” of the doubt. Attending 4 recalled her feelings after being announced as a teaching award recipient for the fourth year in a row: “It was just like something changed in me....Maybe you are a good attending. Maybe you are doing something that is resonating with a unique class of medical students year after year.”

Our interviews also revealed strategies used by female attendings to support and advance their own careers, as well as those of other female faculty, to address the effects of impostor syndrome. Our participants noted the important role of female mentors and sponsors. One former learner mentioned, “I think some of the administration, there are definitely females that are helping promote [the attending].” During an observation, Attending 1 indicated that she was part of a network of women and junior faculty forged to promote each other’s work since “some people are good at self-promotion and some are not.” This group shares accomplishments by distributing and publicizing their accolades.

This multisite, qualitative study informs the complex ways in which exemplary female teaching attendings must navigate being women in medicine. We identified myriad challenges female attendings face originating from patients, from healthcare workers, and within themselves. Our attendings relied upon the following key strategies to mitigate such challenges: (1) they actively position themselves as physician team leaders, (2) they consciously work to manage gender-based stereotypes and perceptions, and (3) they intentionally identify and embrace their unique qualities.

Prior scholarship surrounding gender-based challenges has focused primarily on strategies to improve healthcare systems for women. Much scrutiny has been placed on elevating institutional culture,^26-29 enacting clear policy surrounding sexual harassment,³⁰ ensuring women are actively recruited and retained,³¹ providing resources to assist in work-life balance,^26,32 and cultivating effective mentorship and social networks.^11,33,34

While our findings support the importance of improving healthcare systems, they are more congruent with recent scholarship on explicit personal tactics to mitigate gender-based challenges. Researchers have suggested physicians use algorithmic responses to patient-initiated sexual harassment,³⁵ advocate for those who experience harassment in real time,³⁶ and engage in dedicated practice responding to harassment.^37,38 Our results build on these studies by outlining strategies intended to navigate complex gender dynamics and role model approaches for learners. Interestingly, it was more common for attendings to discuss how they guide their learners and debrief after difficult situations than to discuss how they personally respond to gender-based harassment. While we are not certain why this occurred, three factors may have contributed. First, attendings mentioned that these conversations are often uncomfortable. Second, attendings appeared to accept a higher level of gender-based challenges than they would have tolerated for their learners. Lastly, although we did not gather demographic data from learners, several attendings voiced a strong desire to advocate for and equip female learners with strategies to address and navigate these challenges for themselves.

Gender stereotypes are ubiquitous and firmly rooted in long-standing belief patterns. Certain characteristics are considered masculine (eg, aggressiveness, confidence) and others feminine (eg, kindness, cooperation).¹⁰ Role congruity theory purports that stereotypes lead women to demonstrate behaviors that reflect socially accepted gender norms³⁹ and that social approval is at risk if they behave in ways discordant with these norms.^10,40 Our study provides perspectives from female physicians who walk the tightrope of forcefully asserting themselves more than their male counterparts while not being overly aggressive, since both approaches may have negative connotations.

This study has several limitations. First, it was conducted with a limited number of site visits, attendings, and learners. Likewise, attendings were internists with relatively advanced academic rank. This may reduce the study’s generalizability since attendings in other fields and at earlier career stages may utilize different strategies. However, we believe that if more senior-level female attendings experienced difficulties being recognized and legitimized in their roles, then one can assume that junior-level female faculty would experience these challenges even more so. Likewise, data saturation was not the goal of this exploratory study. Through intensive qualitative data collection, we sought to obtain an in-depth understanding of challenges and strategies. Second, many exemplary female attendings were overlooked by our selection methodology, particularly since women are often underrepresented in the factors we chose. The multisite design, modified snowball sampling, and purposeful randomized selection methodology were used to ensure quality and diversity. Third, attendings provided lists of their former learners, and thus, selection and recall biases may have been introduced since attendings may have more readily identified learners with whom they formed positive relationships. Finally, we cannot eliminate a potential Hawthorne effect on data collection. Researchers attempted to lessen this by standing apart from teams and remaining unobtrusive.

We identified strategies employed by exemplary female attendings to navigate gender-based challenges in their workplaces. We found that female attendings face unconscious bias, labels, power struggles, and harassment, simply because of their gender. They consciously and constantly navigate these challenges by positioning themselves to be seen and heard as team leaders, balancing aspects of their outward appearance and demeanor, embracing their differences and avoiding assimilation to masculine stereotypes of physician leaders, working to manage self-doubt, and coaching their female learners in these areas.

The authors are indebted to Suzanne Winter, MS, for assisting with coordination of study participants and site visits.

Providing high-quality, patient-centered health care is a top priority for the US Department of Veterans Affairs (VA) Veteran Health Administration (VHA), whose core mission is to improve the health and well-being of US veterans. Thus, news of long wait times for medical appointments in the VHA sparked intense national attention and debate and led to changes in senior management and legislative action. ¹ On August 8, 2014, President Bara c k Obama signed the Veterans Access, Choice, and Accountability Act of 2014, also known as the Choice Act, which provided an additional $16 billion in emergency spending over 3 years to improve veterans’ access to timely health care. ² The Choice Act sought to develop an integrated health care network that allowed qualified VHA patients to receive specific health care services in their communities delivered by non-VHA health care providers (HCPs) but paid for by the VHA. The Choice Act also laid out explicit criteria for how to prioritize who would be eligible for VHA-purchased civilian care: (1) veterans who could not get timely appointments at a VHA medical facility within 30 days of referral; or (2) veterans who lived > 40 miles from the closest VHA medical facility.

VHA decision makers seeking to improve care delivery also need to weigh trade-offs between alternative approaches to providing rapid access. For instance, increasing access to non-VHA HCPs may not always decrease wait times and could result in loss of continuity, limited care coordination, limited ability to ensure and enforce high-quality standards at the VHA, and other challenges.^3-6 Although the concerns and views of elected representatives, advocacy groups, and health system leaders are important, it is unknown whether these views and preferences align with those of veterans. Arguably, the range of views and concerns of informed veterans whose health is at stake should be particularly prominent in such policy decision making.

To identify the considerations that were most important to veterans regarding VHA policy around decreasing wait times, a study was designed to engage a group of veterans who were eligible for civilian care under the Choice Act. The study took place 1 year after the Choice Act was passed. Veterans were asked to focus on 2 related questions: First, how should funding be used for building VHA capacity (build) vs purchasing civilian care (buy)? Second, under what circumstances should civilian care be prioritized?

The aim of this paper is to describe democratic deliberation (DD), a specific method that engaged veteran patients in complex policy decisions around access to care. DD methods have been used increasingly in health care for developing policy guidance, setting priorities, providing advice on ethical dilemmas, weighing risk-benefit trade-offs, and determining decision-making authority.^7-12 For example, DD helped guide national policy for mammography screening for breast cancer in New Zealand.¹³ The Agency for Healthcare Research and Quality has completed a systematic review and a large, randomized experiment on best practices for carrying out public deliberation.^8,13,14 However, despite the potential value of this approach, there has been little use of deliberative methods within the VHA for the explicit purpose of informing veteran health care delivery.

This paper describes the experience engaging veterans by using DD methodology and informing VHA leadership about the results of those deliberations. The specific aims were to understand whether DD is an acceptable approach to use to engage patients in the medical services policy-making process within VHA and whether veterans are able to come to an informed consensus.

Methods

Engaging patients and incorporating their needs and concerns within the policy-making process may improve health system policies and make those policies more patient centered. Such engagement also could be a way to generate creative solutions. However, because health-system decisions often involve making difficult trade-offs, effectively obtaining patient population input on complex care delivery issues can be challenging.

Although surveys can provide intuitive, top-of-mind input from respondents, these opinions are generally not sufficient for resolving complex problems.¹⁵ Focus groups and interviews may produce results that are more in-depth than surveys, but these methods tend to elicit settled private preferences rather than opinions about what the community should do.¹⁶ DD, on the other hand, is designed to elicit deeply informed public opinions on complex, value-laden topics to develop recommendations and policies for a larger community.¹⁷ The goal is to find collective solutions to challenging social problems. DD achieves this by giving participants an opportunity to explore a topic in-depth, question experts, and engage peers in reason-based discussions.^18,19 This method has its roots in political science and has been used over several decades to successfully inform policy making on a broad array of topics nationally and internationally—from health research ethics in the US to nuclear and energy policy in Japan.^7,16,20,21 DD has been found to promote ownership of public programs and lend legitimacy to policy decisions, political institutions, and democracy itself.¹⁸

A single day (8 hours) DD session was convened, following a Citizens Jury model of deliberation, which brings veteran patients together to learn about a topic, ask questions of experts, deliberate with peers, and generate a “citizen’s report” that contains a set of recommendations (Table 1). An overview of the different models of DD and rationale for each can be found elsewhere.^8,15

Recruitment Considerations

A purposively selected sample of civilian care-eligible veterans from a midwestern VHA health care system (1 medical center and 3 community-based outpatient clinics [CBOCs]) were invited to the DD session. The targeted number of participants was 30. Female veterans, who comprise only 7% of the local veteran population, were oversampled to account for their potentially different health care needs and to create balance between males and females in the session. Oversampling for other characteristics was not possible due to the relatively small sample size. Based on prior experience,⁷ it was assumed that 70% of willing participants would attend the session; therefore 34 veterans were invited and 24 attended. Each participant received a $200 incentive in appreciation for their substantial time commitment and to offset transportation costs.

Background Materials

A packet with educational materials (Flesch-Kincaid Grade Level of 10.5) was mailed to participants about 2 weeks before the DD session. Participants were asked to review prior to attending the session. These materials described the session (eg, purpose, organizers, importance) and provided factual information about the Choice Act (eg, eligibility, out-of-pocket costs, travel pay, prescription drug policies).

Session Overview

The session was structured to accomplish the following goals: (1) Elicit participants’ opinions about access to health care and reasons for those opinions; (2) Provide in-depth education about the Choice Act through presentations and discussions with topical experts; and (3) Elicit reasoning and recommendations on both the criteria by which participants prioritize candidates for civilian care and how participants would allocate additional funding to improve access (ie, by building VHA capacity to deliver more timely health care vs purchasing health care from civilian HCPs).

Participants were asked to fill out a survey on arrival in the morning and were assigned to 1 of 3 tables or small groups. Each table had a facilitator who had extensive experience in qualitative data collection methods and guided the dialogue using a scripted protocol that they helped develop and refine. The facilitation materials drew from and used previously published studies.^22,23 Each facilitator audio recorded the sessions and took notes. Three experts presented during plenary education sessions. Presentations were designed to provide balanced factual information and included a veteran’s perspective. One presenter was a clinician on the project team, another was a local clinical leader responsible for making decisions about what services to provide via civilian care (buy) vs enhancing the local VHA health system’s ability to provide those services (build), and the third presenter was a veteran who was on the project team.

Education Session 1

The first plenary education session with expert presentations was conducted after each table completed an icebreaker exercise. The project team physician provided a brief history and description of the Choice Act to reinforce educational materials sent to participants prior to the session. The health system clinical leader described his decision process and principles and highlighted constraints placed on him by the Choice Act that were in place at the time of the DD session. He also described existing local and national programs to provide civilian care (eg, local fee-basis non-VHA care programs) and how these programs sought to achieve goals similar to the Choice Act. The veteran presenter focused on the importance of session participants providing candid insight and observations and emphasized that this session was a significant opportunity to “have their voices heard.”

Deliberation 1: What criteria should be used to prioritize patients for receiving civilian care paid for by the VHA? To elicit preferences on the central question of this deliberation, participants were presented with 8 real-world cases that were based on interviews conducted with Choice Act-eligible veterans (Table 2 and eAppendices A

Education Session 2

In the second plenary session, the project team physician provided information about health care access issues, both inside and outside of the VHA, particularly between urban and rural areas. He also discussed factors related to the insufficient capacity to meet growing demand that contributed to the VHA wait-time crisis. The veteran presenter shared reflections on health care access from a veteran’s perspective.

Deliberation 2: How should additional funding be divided between increasing the ability of the VHA to (1) provide care by VHA HCPs; and (2) pay for care from non-VHA civilian HCPs? Participants were presented the patient examples and Choice Act funding scenarios (the buy policy option) and contrasted that with a build policy option. Participants were explicitly encouraged to shift their perspectives from thinking about individual cases to considering policy-level decisions and the broader social good (Table 2).

Ensuring Robust Deliberations

If participants do not adequately grasp the complexities of the topic, a deliberation can fail. To facilitate nuanced reasoning, real-world concrete examples were developed as the starting point of each deliberation based on interviews with actual patients (deliberation 1) and actual policy proposals relevant to the funding allocation decisions within the Choice Act (deliberation 2).

A deliberation also can fail with self-silencing, where participants withhold opinions that differ from those articulated first or by more vocal members of the group.²⁴ To combat self-silencing, highly experienced facilitators were used to ensure sharing from all participants and to support an open-minded, courteous, and reason-based environment for discourse. It was specified that the best solutions are achieved through reason-based and cordial disagreement and that success can be undermined when participants simply agree because it is easier or more comfortable.

A third way a deliberation can fail is if individuals do not adopt a group or system-level perspective. To counter this, facilitators reinforced at multiple points the importance of taking a broader social perspective rather than sharing only one’s personal preferences.

Finally, it is important to assess the quality of the deliberative process itself, to ensure that results are trustworthy.²⁵ To assess the quality of the deliberative process, participants knowledge about key issues pre- and postdeliberation were assessed. Participants also were asked to rate the quality of the facilitators and how well they felt their voice was heard and respected, and facilitators made qualitative assessments about the extent to which participants were engaged in reason-based and collaborative discussion.

Data

Quantitative data were collected via pre- and postsession surveys. The surveys contained items related to knowledge about the Choice Act, expectations for the DD session, beliefs and opinions about the provision of health care for veterans, recommended funding allocations between build vs buy policy options, and general demographics. Qualitative data were collected through detailed notes taken by the 3 facilitators. Each table’s deliberations were audio recorded so that gaps in the notes could be filled.

The 3 facilitators, who were all experienced qualitative researchers, typed their written notes into a template immediately after the session. Two of the 3 facilitators led the analysis of the session notes. Findings within and across the 3 deliberation tables were developed using content and matrix analysis methods.²⁶ Descriptive statistics were generated from survey responses and compared survey items pre- and postsession using paired t tests or χ² tests for categorical responses.

Results

Thirty-three percent of individuals invited (n = 127) agreed to participate. Those who declined cited conflicts related to distance, transportation, work/school, medical appointments, family commitments, or were not interested. In all, 24 (69%) of the 35 veterans who accepted the invitation attended the deliberation session. Of the 11 who accepted but did not attend, 5 cancelled ahead of time because of conflicts (Figure). Most participants were male (70%), 48% were aged 61 to 75 years, 65% were white, 43% had some college education, 43% reported an annual income of between $25,000 and $40,000, and only 35% reported very good health (eAppendix D).

Deliberation 1

During the deliberation on the prioritization criteria, the concept of “condition severity” emerged as an important criterion for veterans. This criterion captured simultaneous consideration of both clinical necessity and burden on the veteran to obtain care. For example, participants felt that patients with a life-threatening illness should be prioritized for civilian care over patients who need preventative or primary care (clinical necessity) and that elderly patients with substantial difficulty traveling to VHA appointments should be prioritized over patients who can travel more easily (burden). The Choice Act regulations at the time of the DD session did not reflect this nuanced perspective, stipulating only that veterans must live > 40 miles from the nearest VHA medical facility.

One of the 3 groups did not prioritize the patient cases because some members felt that no veteran should be constrained from receiving civilian care if they desired it. Nonetheless, this group did agree with prioritizing the first 2 cases in Table 3. The other groups prioritized all 8 cases in generally similar ways.

Deliberation 2

No clear consensus emerged on the buy vs build question. A representative from each table presented their group’s positions, rationale, and recommendations after deliberations were completed. After hearing the range of positions, the groups then had another opportunity to deliberate based on what they heard from the other tables; no new recommendations or consensus emerged.

Participants who were in favor of allocating more funds toward the build policy offered a range of rationales, saying that it would (1) increase access for rural veterans by building CBOCs and deploying more mobile units that could bring outlets for health care closer to their home communities; (2) provide critical and unique medical expertise to address veteran-specific issues such as prosthetics, traumatic brain injury, posttraumatic stress disorder, spinal cord injury, and shrapnel wounds that are typically not available through civilian providers; (3) give VHA more oversight over the quality and cost of care, which is more challenging to do with civilian providers; and (4) Improve VHA infrastructure by, for example, upgrading technology and attracting the best clinicians and staff to support “our VHA.”

Participants who were in favor of allocating more funds toward the buy policy also offered a range of rationales, saying that it would (1) decrease patient burden by increasing access through community providers, decreasing wait time, and lessening personal cost and travel time; (2) allow more patients to receive civilian care, which was generally seen as beneficial by a few participants because of perceptions that the VHA provides lower quality care due to a shortage of VHA providers, run-down/older facilities, lack of technology, and poorer-quality VHA providers; and (3) provide an opportunity to divest of costly facilities and invest in other innovative approaches. Regarding this last reason, a few participants felt that the VHA is “gouged” when building medical centers that overrun budgets. They also were concerned that investing in facilities tied VHA to specific locations when current locations of veterans may change “25 years from now.”

Survey Results

Twenty-three of the 24 participants completed both pre- and postsession surveys. The majority of participants in the session felt people in the group respected their opinion (96%); felt that the facilitator did not try to influence the group with her own opinions (96%); indicated they understood the information enough to participate as much as they wanted (100%); and were hopeful that their reasoning and recommendations would help inform VHA policy makers (82%).

The surveys also provided an opportunity to examine the extent to which knowledge, attitudes, and opinions changed from before to after the deliberation. Even with the small sample, responses revealed a trend toward improved knowledge about key elements of the Choice Act and its goals. Further, there was a shift in some participants’ opinions about how patients should be prioritized to receive civilian care. For example, before the deliberation participants generally felt that all veterans should be able to receive civilian care, whereas postdeliberation this was not the case. Postdeliberation, most participants felt that primary care should not be a high priority for civilian care but continued to endorse prioritizing civilian care for specialty services like orthopedic or cardiology-related care. Finally, participants moved from more diverse recommendations regarding additional funds allocations, toward consensus after the deliberation around allocating funds to the build policy. Eight participants supported a build policy beforehand, whereas 16 supported this policy afterward.

Discussion

This study explored DD as a method for deeply engaging veterans in complex policy making to guide funding allocation and prioritization decisions related to the Choice Act, decisions that are still very relevant today within the context of the Mission Act and have substantial implications for how health care is delivered in the VHA. The Mission Act passed on June 6, 2018, with the goal of improving access to and the reliability of civilian or community care for eligible veterans.²⁷ Decisions related to appropriating scarce funding to improve access to care is an emotional and value-laden topic that elicited strong and divergent opinions among the participants. Veterans were eager to have their voices heard and had strong expectations that VHA leadership would be briefed about their recommendations. The majority of participants were satisfied with the deliberation process, felt they understood the issues, and felt their opinions were respected. They expressed feelings of comradery and community throughout the process.

In this single deliberation session, the groups did not achieve a single, final consensus regarding how VHA funding should ultimately be allocated between buy and build policy options. Nonetheless, participants provided a rich array of recommendations and rationale for them. Session moderators observed rich, sophisticated, fair, and reason-based discussions on this complex topic. Participants left with a deeper knowledge and appreciation for the complex trade-offs and expressed strong rationales for both sides of the policy debate on build vs buy. In addition, the project yielded results of high interest to VHA policy makers.

This work was presented in multiple venues between 2015 to 2016, and to both local and national VHA leadership, including the local Executive Quality Leadership Boards, the VHA Central Office Committee on the Future State of VA Community Care, the VA Office of Patient Centered Care, and the National Veteran Experience Committee. Through these discussions and others, we saw great interest within the VHA system and high-level leaders to explore ways to include veterans’ voices in the policy-making process. This work was invaluable to our research team (eAppendix E 

Many health system decisions regarding what care should be delivered (and how) involve making difficult, value-laden choices in the context of limited resources. DD methods can be used to target and obtain specific viewpoints from diverse populations, such as the informed perspectives of minority and underrepresented populations within the VHA.¹⁹ For example, female veterans were oversampled to ensure that the informed preferences of this population was obtained. Thus, DD methods could provide a valuable tool for health systems to elicit in-depth diverse patient input on high-profile policies that will have a substantial impact on the system’s patient population.

Limitations

One potential downside of DD is that, because of the resource-intensive nature of deliberation sessions, they are often conducted with relatively small groups.⁹ Viewpoints of those within these small samples who are willing to spend an entire day discussing a complex topic may not be representative of the larger patient community. However, the core goal of DD is diversity of opinions rather than representativeness.

A stratified random sampling strategy that oversampled for underrepresented and minority populations was used to help select a diverse group that represents the population on key characteristics and partially addresses concern about representativeness. Efforts to optimize participation rates, including providing monetary incentives, also are helpful and have led to high participation rates in past deliberations.⁷

Health system communication strategies that promote the importance of becoming involved in DD sessions also may be helpful in improving rates of recruitment. On particularly important topics where health system leaders feel a larger resource investment is justified, conducting larger scale deliberations with many small groups may obtain more generalizable evidence about what individual patients and groups of patients recommend.⁷ However, due to the inherent limitations of surveys and focus group approaches for obtaining informed views on complex topics, there are no clear systematic alternatives to the DD approach.

Conclusion

DD is an effective method to meaningfully engage patients in deep deliberations to guide complex policy making. Although design of deliberative sessions is resource-intensive, patient engagement efforts, such as those described in this paper, could be an important aspect of a well-functioning learning health system. Further research into alternative, streamlined methods that can also engage veterans more deeply is needed. DD also can be combined with other approaches to broaden and confirm findings, including focus groups, town hall meetings, or surveys.

Although this study did not provide consensus on how the VHA should allocate funds with respect to the Choice Act, it did provide insight into the importance and feasibility of engaging veterans in the policy-making process. As more policies aimed at improving veterans’ access to civilian care are created, such as the most recent Mission Act, policy makers should strongly consider using the DD method of obtaining informed veteran input into future policy decisions.

Acknowledgments
Funding was provided by the US Department of Veterans Affairs Office of Analytics and Business Intelligence (OABI) and the VA Quality Enhancement Research Initiative (QUERI). Dr. Caverly was supported in part by a VA Career Development Award (CDA 16-151). Dr. Krein is supported by a VA Health Services Research and Development Research Career Scientist Award (RCS 11-222). The authors thank the veterans who participated in this work. They also thank Caitlin Reardon and Natalya Wawrin for their assistance in organizing the deliberation session.

Providing high-quality, patient-centered health care is a top priority for the US Department of Veterans Affairs (VA) Veteran Health Administration (VHA), whose core mission is to improve the health and well-being of US veterans. Thus, news of long wait times for medical appointments in the VHA sparked intense national attention and debate and led to changes in senior management and legislative action. ¹ On August 8, 2014, President Bara c k Obama signed the Veterans Access, Choice, and Accountability Act of 2014, also known as the Choice Act, which provided an additional $16 billion in emergency spending over 3 years to improve veterans’ access to timely health care. ² The Choice Act sought to develop an integrated health care network that allowed qualified VHA patients to receive specific health care services in their communities delivered by non-VHA health care providers (HCPs) but paid for by the VHA. The Choice Act also laid out explicit criteria for how to prioritize who would be eligible for VHA-purchased civilian care: (1) veterans who could not get timely appointments at a VHA medical facility within 30 days of referral; or (2) veterans who lived > 40 miles from the closest VHA medical facility.

VHA decision makers seeking to improve care delivery also need to weigh trade-offs between alternative approaches to providing rapid access. For instance, increasing access to non-VHA HCPs may not always decrease wait times and could result in loss of continuity, limited care coordination, limited ability to ensure and enforce high-quality standards at the VHA, and other challenges.^3-6 Although the concerns and views of elected representatives, advocacy groups, and health system leaders are important, it is unknown whether these views and preferences align with those of veterans. Arguably, the range of views and concerns of informed veterans whose health is at stake should be particularly prominent in such policy decision making.

To identify the considerations that were most important to veterans regarding VHA policy around decreasing wait times, a study was designed to engage a group of veterans who were eligible for civilian care under the Choice Act. The study took place 1 year after the Choice Act was passed. Veterans were asked to focus on 2 related questions: First, how should funding be used for building VHA capacity (build) vs purchasing civilian care (buy)? Second, under what circumstances should civilian care be prioritized?

The aim of this paper is to describe democratic deliberation (DD), a specific method that engaged veteran patients in complex policy decisions around access to care. DD methods have been used increasingly in health care for developing policy guidance, setting priorities, providing advice on ethical dilemmas, weighing risk-benefit trade-offs, and determining decision-making authority.^7-12 For example, DD helped guide national policy for mammography screening for breast cancer in New Zealand.¹³ The Agency for Healthcare Research and Quality has completed a systematic review and a large, randomized experiment on best practices for carrying out public deliberation.^8,13,14 However, despite the potential value of this approach, there has been little use of deliberative methods within the VHA for the explicit purpose of informing veteran health care delivery.

This paper describes the experience engaging veterans by using DD methodology and informing VHA leadership about the results of those deliberations. The specific aims were to understand whether DD is an acceptable approach to use to engage patients in the medical services policy-making process within VHA and whether veterans are able to come to an informed consensus.

Methods

Engaging patients and incorporating their needs and concerns within the policy-making process may improve health system policies and make those policies more patient centered. Such engagement also could be a way to generate creative solutions. However, because health-system decisions often involve making difficult trade-offs, effectively obtaining patient population input on complex care delivery issues can be challenging.

Although surveys can provide intuitive, top-of-mind input from respondents, these opinions are generally not sufficient for resolving complex problems.¹⁵ Focus groups and interviews may produce results that are more in-depth than surveys, but these methods tend to elicit settled private preferences rather than opinions about what the community should do.¹⁶ DD, on the other hand, is designed to elicit deeply informed public opinions on complex, value-laden topics to develop recommendations and policies for a larger community.¹⁷ The goal is to find collective solutions to challenging social problems. DD achieves this by giving participants an opportunity to explore a topic in-depth, question experts, and engage peers in reason-based discussions.^18,19 This method has its roots in political science and has been used over several decades to successfully inform policy making on a broad array of topics nationally and internationally—from health research ethics in the US to nuclear and energy policy in Japan.^7,16,20,21 DD has been found to promote ownership of public programs and lend legitimacy to policy decisions, political institutions, and democracy itself.¹⁸

A single day (8 hours) DD session was convened, following a Citizens Jury model of deliberation, which brings veteran patients together to learn about a topic, ask questions of experts, deliberate with peers, and generate a “citizen’s report” that contains a set of recommendations (Table 1). An overview of the different models of DD and rationale for each can be found elsewhere.^8,15

Recruitment Considerations

A purposively selected sample of civilian care-eligible veterans from a midwestern VHA health care system (1 medical center and 3 community-based outpatient clinics [CBOCs]) were invited to the DD session. The targeted number of participants was 30. Female veterans, who comprise only 7% of the local veteran population, were oversampled to account for their potentially different health care needs and to create balance between males and females in the session. Oversampling for other characteristics was not possible due to the relatively small sample size. Based on prior experience,⁷ it was assumed that 70% of willing participants would attend the session; therefore 34 veterans were invited and 24 attended. Each participant received a $200 incentive in appreciation for their substantial time commitment and to offset transportation costs.

Background Materials

A packet with educational materials (Flesch-Kincaid Grade Level of 10.5) was mailed to participants about 2 weeks before the DD session. Participants were asked to review prior to attending the session. These materials described the session (eg, purpose, organizers, importance) and provided factual information about the Choice Act (eg, eligibility, out-of-pocket costs, travel pay, prescription drug policies).

Session Overview

The session was structured to accomplish the following goals: (1) Elicit participants’ opinions about access to health care and reasons for those opinions; (2) Provide in-depth education about the Choice Act through presentations and discussions with topical experts; and (3) Elicit reasoning and recommendations on both the criteria by which participants prioritize candidates for civilian care and how participants would allocate additional funding to improve access (ie, by building VHA capacity to deliver more timely health care vs purchasing health care from civilian HCPs).

Participants were asked to fill out a survey on arrival in the morning and were assigned to 1 of 3 tables or small groups. Each table had a facilitator who had extensive experience in qualitative data collection methods and guided the dialogue using a scripted protocol that they helped develop and refine. The facilitation materials drew from and used previously published studies.^22,23 Each facilitator audio recorded the sessions and took notes. Three experts presented during plenary education sessions. Presentations were designed to provide balanced factual information and included a veteran’s perspective. One presenter was a clinician on the project team, another was a local clinical leader responsible for making decisions about what services to provide via civilian care (buy) vs enhancing the local VHA health system’s ability to provide those services (build), and the third presenter was a veteran who was on the project team.

Education Session 1

The first plenary education session with expert presentations was conducted after each table completed an icebreaker exercise. The project team physician provided a brief history and description of the Choice Act to reinforce educational materials sent to participants prior to the session. The health system clinical leader described his decision process and principles and highlighted constraints placed on him by the Choice Act that were in place at the time of the DD session. He also described existing local and national programs to provide civilian care (eg, local fee-basis non-VHA care programs) and how these programs sought to achieve goals similar to the Choice Act. The veteran presenter focused on the importance of session participants providing candid insight and observations and emphasized that this session was a significant opportunity to “have their voices heard.”

Deliberation 1: What criteria should be used to prioritize patients for receiving civilian care paid for by the VHA? To elicit preferences on the central question of this deliberation, participants were presented with 8 real-world cases that were based on interviews conducted with Choice Act-eligible veterans (Table 2 and eAppendices A

Education Session 2

In the second plenary session, the project team physician provided information about health care access issues, both inside and outside of the VHA, particularly between urban and rural areas. He also discussed factors related to the insufficient capacity to meet growing demand that contributed to the VHA wait-time crisis. The veteran presenter shared reflections on health care access from a veteran’s perspective.

Deliberation 2: How should additional funding be divided between increasing the ability of the VHA to (1) provide care by VHA HCPs; and (2) pay for care from non-VHA civilian HCPs? Participants were presented the patient examples and Choice Act funding scenarios (the buy policy option) and contrasted that with a build policy option. Participants were explicitly encouraged to shift their perspectives from thinking about individual cases to considering policy-level decisions and the broader social good (Table 2).

Ensuring Robust Deliberations

If participants do not adequately grasp the complexities of the topic, a deliberation can fail. To facilitate nuanced reasoning, real-world concrete examples were developed as the starting point of each deliberation based on interviews with actual patients (deliberation 1) and actual policy proposals relevant to the funding allocation decisions within the Choice Act (deliberation 2).

A deliberation also can fail with self-silencing, where participants withhold opinions that differ from those articulated first or by more vocal members of the group.²⁴ To combat self-silencing, highly experienced facilitators were used to ensure sharing from all participants and to support an open-minded, courteous, and reason-based environment for discourse. It was specified that the best solutions are achieved through reason-based and cordial disagreement and that success can be undermined when participants simply agree because it is easier or more comfortable.

A third way a deliberation can fail is if individuals do not adopt a group or system-level perspective. To counter this, facilitators reinforced at multiple points the importance of taking a broader social perspective rather than sharing only one’s personal preferences.

Finally, it is important to assess the quality of the deliberative process itself, to ensure that results are trustworthy.²⁵ To assess the quality of the deliberative process, participants knowledge about key issues pre- and postdeliberation were assessed. Participants also were asked to rate the quality of the facilitators and how well they felt their voice was heard and respected, and facilitators made qualitative assessments about the extent to which participants were engaged in reason-based and collaborative discussion.

Data

Quantitative data were collected via pre- and postsession surveys. The surveys contained items related to knowledge about the Choice Act, expectations for the DD session, beliefs and opinions about the provision of health care for veterans, recommended funding allocations between build vs buy policy options, and general demographics. Qualitative data were collected through detailed notes taken by the 3 facilitators. Each table’s deliberations were audio recorded so that gaps in the notes could be filled.

The 3 facilitators, who were all experienced qualitative researchers, typed their written notes into a template immediately after the session. Two of the 3 facilitators led the analysis of the session notes. Findings within and across the 3 deliberation tables were developed using content and matrix analysis methods.²⁶ Descriptive statistics were generated from survey responses and compared survey items pre- and postsession using paired t tests or χ² tests for categorical responses.

Results

Thirty-three percent of individuals invited (n = 127) agreed to participate. Those who declined cited conflicts related to distance, transportation, work/school, medical appointments, family commitments, or were not interested. In all, 24 (69%) of the 35 veterans who accepted the invitation attended the deliberation session. Of the 11 who accepted but did not attend, 5 cancelled ahead of time because of conflicts (Figure). Most participants were male (70%), 48% were aged 61 to 75 years, 65% were white, 43% had some college education, 43% reported an annual income of between $25,000 and $40,000, and only 35% reported very good health (eAppendix D).

Deliberation 1

During the deliberation on the prioritization criteria, the concept of “condition severity” emerged as an important criterion for veterans. This criterion captured simultaneous consideration of both clinical necessity and burden on the veteran to obtain care. For example, participants felt that patients with a life-threatening illness should be prioritized for civilian care over patients who need preventative or primary care (clinical necessity) and that elderly patients with substantial difficulty traveling to VHA appointments should be prioritized over patients who can travel more easily (burden). The Choice Act regulations at the time of the DD session did not reflect this nuanced perspective, stipulating only that veterans must live > 40 miles from the nearest VHA medical facility.

One of the 3 groups did not prioritize the patient cases because some members felt that no veteran should be constrained from receiving civilian care if they desired it. Nonetheless, this group did agree with prioritizing the first 2 cases in Table 3. The other groups prioritized all 8 cases in generally similar ways.

Deliberation 2

No clear consensus emerged on the buy vs build question. A representative from each table presented their group’s positions, rationale, and recommendations after deliberations were completed. After hearing the range of positions, the groups then had another opportunity to deliberate based on what they heard from the other tables; no new recommendations or consensus emerged.

Participants who were in favor of allocating more funds toward the build policy offered a range of rationales, saying that it would (1) increase access for rural veterans by building CBOCs and deploying more mobile units that could bring outlets for health care closer to their home communities; (2) provide critical and unique medical expertise to address veteran-specific issues such as prosthetics, traumatic brain injury, posttraumatic stress disorder, spinal cord injury, and shrapnel wounds that are typically not available through civilian providers; (3) give VHA more oversight over the quality and cost of care, which is more challenging to do with civilian providers; and (4) Improve VHA infrastructure by, for example, upgrading technology and attracting the best clinicians and staff to support “our VHA.”

Participants who were in favor of allocating more funds toward the buy policy also offered a range of rationales, saying that it would (1) decrease patient burden by increasing access through community providers, decreasing wait time, and lessening personal cost and travel time; (2) allow more patients to receive civilian care, which was generally seen as beneficial by a few participants because of perceptions that the VHA provides lower quality care due to a shortage of VHA providers, run-down/older facilities, lack of technology, and poorer-quality VHA providers; and (3) provide an opportunity to divest of costly facilities and invest in other innovative approaches. Regarding this last reason, a few participants felt that the VHA is “gouged” when building medical centers that overrun budgets. They also were concerned that investing in facilities tied VHA to specific locations when current locations of veterans may change “25 years from now.”

Survey Results

Twenty-three of the 24 participants completed both pre- and postsession surveys. The majority of participants in the session felt people in the group respected their opinion (96%); felt that the facilitator did not try to influence the group with her own opinions (96%); indicated they understood the information enough to participate as much as they wanted (100%); and were hopeful that their reasoning and recommendations would help inform VHA policy makers (82%).

The surveys also provided an opportunity to examine the extent to which knowledge, attitudes, and opinions changed from before to after the deliberation. Even with the small sample, responses revealed a trend toward improved knowledge about key elements of the Choice Act and its goals. Further, there was a shift in some participants’ opinions about how patients should be prioritized to receive civilian care. For example, before the deliberation participants generally felt that all veterans should be able to receive civilian care, whereas postdeliberation this was not the case. Postdeliberation, most participants felt that primary care should not be a high priority for civilian care but continued to endorse prioritizing civilian care for specialty services like orthopedic or cardiology-related care. Finally, participants moved from more diverse recommendations regarding additional funds allocations, toward consensus after the deliberation around allocating funds to the build policy. Eight participants supported a build policy beforehand, whereas 16 supported this policy afterward.

Discussion

This study explored DD as a method for deeply engaging veterans in complex policy making to guide funding allocation and prioritization decisions related to the Choice Act, decisions that are still very relevant today within the context of the Mission Act and have substantial implications for how health care is delivered in the VHA. The Mission Act passed on June 6, 2018, with the goal of improving access to and the reliability of civilian or community care for eligible veterans.²⁷ Decisions related to appropriating scarce funding to improve access to care is an emotional and value-laden topic that elicited strong and divergent opinions among the participants. Veterans were eager to have their voices heard and had strong expectations that VHA leadership would be briefed about their recommendations. The majority of participants were satisfied with the deliberation process, felt they understood the issues, and felt their opinions were respected. They expressed feelings of comradery and community throughout the process.

In this single deliberation session, the groups did not achieve a single, final consensus regarding how VHA funding should ultimately be allocated between buy and build policy options. Nonetheless, participants provided a rich array of recommendations and rationale for them. Session moderators observed rich, sophisticated, fair, and reason-based discussions on this complex topic. Participants left with a deeper knowledge and appreciation for the complex trade-offs and expressed strong rationales for both sides of the policy debate on build vs buy. In addition, the project yielded results of high interest to VHA policy makers.

This work was presented in multiple venues between 2015 to 2016, and to both local and national VHA leadership, including the local Executive Quality Leadership Boards, the VHA Central Office Committee on the Future State of VA Community Care, the VA Office of Patient Centered Care, and the National Veteran Experience Committee. Through these discussions and others, we saw great interest within the VHA system and high-level leaders to explore ways to include veterans’ voices in the policy-making process. This work was invaluable to our research team (eAppendix E 

Many health system decisions regarding what care should be delivered (and how) involve making difficult, value-laden choices in the context of limited resources. DD methods can be used to target and obtain specific viewpoints from diverse populations, such as the informed perspectives of minority and underrepresented populations within the VHA.¹⁹ For example, female veterans were oversampled to ensure that the informed preferences of this population was obtained. Thus, DD methods could provide a valuable tool for health systems to elicit in-depth diverse patient input on high-profile policies that will have a substantial impact on the system’s patient population.

Limitations

One potential downside of DD is that, because of the resource-intensive nature of deliberation sessions, they are often conducted with relatively small groups.⁹ Viewpoints of those within these small samples who are willing to spend an entire day discussing a complex topic may not be representative of the larger patient community. However, the core goal of DD is diversity of opinions rather than representativeness.

A stratified random sampling strategy that oversampled for underrepresented and minority populations was used to help select a diverse group that represents the population on key characteristics and partially addresses concern about representativeness. Efforts to optimize participation rates, including providing monetary incentives, also are helpful and have led to high participation rates in past deliberations.⁷

Health system communication strategies that promote the importance of becoming involved in DD sessions also may be helpful in improving rates of recruitment. On particularly important topics where health system leaders feel a larger resource investment is justified, conducting larger scale deliberations with many small groups may obtain more generalizable evidence about what individual patients and groups of patients recommend.⁷ However, due to the inherent limitations of surveys and focus group approaches for obtaining informed views on complex topics, there are no clear systematic alternatives to the DD approach.

Conclusion

DD is an effective method to meaningfully engage patients in deep deliberations to guide complex policy making. Although design of deliberative sessions is resource-intensive, patient engagement efforts, such as those described in this paper, could be an important aspect of a well-functioning learning health system. Further research into alternative, streamlined methods that can also engage veterans more deeply is needed. DD also can be combined with other approaches to broaden and confirm findings, including focus groups, town hall meetings, or surveys.

Although this study did not provide consensus on how the VHA should allocate funds with respect to the Choice Act, it did provide insight into the importance and feasibility of engaging veterans in the policy-making process. As more policies aimed at improving veterans’ access to civilian care are created, such as the most recent Mission Act, policy makers should strongly consider using the DD method of obtaining informed veteran input into future policy decisions.

Acknowledgments
Funding was provided by the US Department of Veterans Affairs Office of Analytics and Business Intelligence (OABI) and the VA Quality Enhancement Research Initiative (QUERI). Dr. Caverly was supported in part by a VA Career Development Award (CDA 16-151). Dr. Krein is supported by a VA Health Services Research and Development Research Career Scientist Award (RCS 11-222). The authors thank the veterans who participated in this work. They also thank Caitlin Reardon and Natalya Wawrin for their assistance in organizing the deliberation session.

Providing high-quality, patient-centered health care is a top priority for the US Department of Veterans Affairs (VA) Veteran Health Administration (VHA), whose core mission is to improve the health and well-being of US veterans. Thus, news of long wait times for medical appointments in the VHA sparked intense national attention and debate and led to changes in senior management and legislative action. ¹ On August 8, 2014, President Bara c k Obama signed the Veterans Access, Choice, and Accountability Act of 2014, also known as the Choice Act, which provided an additional $16 billion in emergency spending over 3 years to improve veterans’ access to timely health care. ² The Choice Act sought to develop an integrated health care network that allowed qualified VHA patients to receive specific health care services in their communities delivered by non-VHA health care providers (HCPs) but paid for by the VHA. The Choice Act also laid out explicit criteria for how to prioritize who would be eligible for VHA-purchased civilian care: (1) veterans who could not get timely appointments at a VHA medical facility within 30 days of referral; or (2) veterans who lived > 40 miles from the closest VHA medical facility.

VHA decision makers seeking to improve care delivery also need to weigh trade-offs between alternative approaches to providing rapid access. For instance, increasing access to non-VHA HCPs may not always decrease wait times and could result in loss of continuity, limited care coordination, limited ability to ensure and enforce high-quality standards at the VHA, and other challenges.^3-6 Although the concerns and views of elected representatives, advocacy groups, and health system leaders are important, it is unknown whether these views and preferences align with those of veterans. Arguably, the range of views and concerns of informed veterans whose health is at stake should be particularly prominent in such policy decision making.

To identify the considerations that were most important to veterans regarding VHA policy around decreasing wait times, a study was designed to engage a group of veterans who were eligible for civilian care under the Choice Act. The study took place 1 year after the Choice Act was passed. Veterans were asked to focus on 2 related questions: First, how should funding be used for building VHA capacity (build) vs purchasing civilian care (buy)? Second, under what circumstances should civilian care be prioritized?

The aim of this paper is to describe democratic deliberation (DD), a specific method that engaged veteran patients in complex policy decisions around access to care. DD methods have been used increasingly in health care for developing policy guidance, setting priorities, providing advice on ethical dilemmas, weighing risk-benefit trade-offs, and determining decision-making authority.^7-12 For example, DD helped guide national policy for mammography screening for breast cancer in New Zealand.¹³ The Agency for Healthcare Research and Quality has completed a systematic review and a large, randomized experiment on best practices for carrying out public deliberation.^8,13,14 However, despite the potential value of this approach, there has been little use of deliberative methods within the VHA for the explicit purpose of informing veteran health care delivery.

This paper describes the experience engaging veterans by using DD methodology and informing VHA leadership about the results of those deliberations. The specific aims were to understand whether DD is an acceptable approach to use to engage patients in the medical services policy-making process within VHA and whether veterans are able to come to an informed consensus.

Methods

Engaging patients and incorporating their needs and concerns within the policy-making process may improve health system policies and make those policies more patient centered. Such engagement also could be a way to generate creative solutions. However, because health-system decisions often involve making difficult trade-offs, effectively obtaining patient population input on complex care delivery issues can be challenging.

Although surveys can provide intuitive, top-of-mind input from respondents, these opinions are generally not sufficient for resolving complex problems.¹⁵ Focus groups and interviews may produce results that are more in-depth than surveys, but these methods tend to elicit settled private preferences rather than opinions about what the community should do.¹⁶ DD, on the other hand, is designed to elicit deeply informed public opinions on complex, value-laden topics to develop recommendations and policies for a larger community.¹⁷ The goal is to find collective solutions to challenging social problems. DD achieves this by giving participants an opportunity to explore a topic in-depth, question experts, and engage peers in reason-based discussions.^18,19 This method has its roots in political science and has been used over several decades to successfully inform policy making on a broad array of topics nationally and internationally—from health research ethics in the US to nuclear and energy policy in Japan.^7,16,20,21 DD has been found to promote ownership of public programs and lend legitimacy to policy decisions, political institutions, and democracy itself.¹⁸

A single day (8 hours) DD session was convened, following a Citizens Jury model of deliberation, which brings veteran patients together to learn about a topic, ask questions of experts, deliberate with peers, and generate a “citizen’s report” that contains a set of recommendations (Table 1). An overview of the different models of DD and rationale for each can be found elsewhere.^8,15

Recruitment Considerations

A purposively selected sample of civilian care-eligible veterans from a midwestern VHA health care system (1 medical center and 3 community-based outpatient clinics [CBOCs]) were invited to the DD session. The targeted number of participants was 30. Female veterans, who comprise only 7% of the local veteran population, were oversampled to account for their potentially different health care needs and to create balance between males and females in the session. Oversampling for other characteristics was not possible due to the relatively small sample size. Based on prior experience,⁷ it was assumed that 70% of willing participants would attend the session; therefore 34 veterans were invited and 24 attended. Each participant received a $200 incentive in appreciation for their substantial time commitment and to offset transportation costs.

Background Materials

A packet with educational materials (Flesch-Kincaid Grade Level of 10.5) was mailed to participants about 2 weeks before the DD session. Participants were asked to review prior to attending the session. These materials described the session (eg, purpose, organizers, importance) and provided factual information about the Choice Act (eg, eligibility, out-of-pocket costs, travel pay, prescription drug policies).

Session Overview

The session was structured to accomplish the following goals: (1) Elicit participants’ opinions about access to health care and reasons for those opinions; (2) Provide in-depth education about the Choice Act through presentations and discussions with topical experts; and (3) Elicit reasoning and recommendations on both the criteria by which participants prioritize candidates for civilian care and how participants would allocate additional funding to improve access (ie, by building VHA capacity to deliver more timely health care vs purchasing health care from civilian HCPs).

Participants were asked to fill out a survey on arrival in the morning and were assigned to 1 of 3 tables or small groups. Each table had a facilitator who had extensive experience in qualitative data collection methods and guided the dialogue using a scripted protocol that they helped develop and refine. The facilitation materials drew from and used previously published studies.^22,23 Each facilitator audio recorded the sessions and took notes. Three experts presented during plenary education sessions. Presentations were designed to provide balanced factual information and included a veteran’s perspective. One presenter was a clinician on the project team, another was a local clinical leader responsible for making decisions about what services to provide via civilian care (buy) vs enhancing the local VHA health system’s ability to provide those services (build), and the third presenter was a veteran who was on the project team.

Education Session 1

The first plenary education session with expert presentations was conducted after each table completed an icebreaker exercise. The project team physician provided a brief history and description of the Choice Act to reinforce educational materials sent to participants prior to the session. The health system clinical leader described his decision process and principles and highlighted constraints placed on him by the Choice Act that were in place at the time of the DD session. He also described existing local and national programs to provide civilian care (eg, local fee-basis non-VHA care programs) and how these programs sought to achieve goals similar to the Choice Act. The veteran presenter focused on the importance of session participants providing candid insight and observations and emphasized that this session was a significant opportunity to “have their voices heard.”

Deliberation 1: What criteria should be used to prioritize patients for receiving civilian care paid for by the VHA? To elicit preferences on the central question of this deliberation, participants were presented with 8 real-world cases that were based on interviews conducted with Choice Act-eligible veterans (Table 2 and eAppendices A

Education Session 2

In the second plenary session, the project team physician provided information about health care access issues, both inside and outside of the VHA, particularly between urban and rural areas. He also discussed factors related to the insufficient capacity to meet growing demand that contributed to the VHA wait-time crisis. The veteran presenter shared reflections on health care access from a veteran’s perspective.

Deliberation 2: How should additional funding be divided between increasing the ability of the VHA to (1) provide care by VHA HCPs; and (2) pay for care from non-VHA civilian HCPs? Participants were presented the patient examples and Choice Act funding scenarios (the buy policy option) and contrasted that with a build policy option. Participants were explicitly encouraged to shift their perspectives from thinking about individual cases to considering policy-level decisions and the broader social good (Table 2).

Ensuring Robust Deliberations

If participants do not adequately grasp the complexities of the topic, a deliberation can fail. To facilitate nuanced reasoning, real-world concrete examples were developed as the starting point of each deliberation based on interviews with actual patients (deliberation 1) and actual policy proposals relevant to the funding allocation decisions within the Choice Act (deliberation 2).

A deliberation also can fail with self-silencing, where participants withhold opinions that differ from those articulated first or by more vocal members of the group.²⁴ To combat self-silencing, highly experienced facilitators were used to ensure sharing from all participants and to support an open-minded, courteous, and reason-based environment for discourse. It was specified that the best solutions are achieved through reason-based and cordial disagreement and that success can be undermined when participants simply agree because it is easier or more comfortable.

A third way a deliberation can fail is if individuals do not adopt a group or system-level perspective. To counter this, facilitators reinforced at multiple points the importance of taking a broader social perspective rather than sharing only one’s personal preferences.

Finally, it is important to assess the quality of the deliberative process itself, to ensure that results are trustworthy.²⁵ To assess the quality of the deliberative process, participants knowledge about key issues pre- and postdeliberation were assessed. Participants also were asked to rate the quality of the facilitators and how well they felt their voice was heard and respected, and facilitators made qualitative assessments about the extent to which participants were engaged in reason-based and collaborative discussion.

Data

Quantitative data were collected via pre- and postsession surveys. The surveys contained items related to knowledge about the Choice Act, expectations for the DD session, beliefs and opinions about the provision of health care for veterans, recommended funding allocations between build vs buy policy options, and general demographics. Qualitative data were collected through detailed notes taken by the 3 facilitators. Each table’s deliberations were audio recorded so that gaps in the notes could be filled.

The 3 facilitators, who were all experienced qualitative researchers, typed their written notes into a template immediately after the session. Two of the 3 facilitators led the analysis of the session notes. Findings within and across the 3 deliberation tables were developed using content and matrix analysis methods.²⁶ Descriptive statistics were generated from survey responses and compared survey items pre- and postsession using paired t tests or χ² tests for categorical responses.

Results

Thirty-three percent of individuals invited (n = 127) agreed to participate. Those who declined cited conflicts related to distance, transportation, work/school, medical appointments, family commitments, or were not interested. In all, 24 (69%) of the 35 veterans who accepted the invitation attended the deliberation session. Of the 11 who accepted but did not attend, 5 cancelled ahead of time because of conflicts (Figure). Most participants were male (70%), 48% were aged 61 to 75 years, 65% were white, 43% had some college education, 43% reported an annual income of between $25,000 and $40,000, and only 35% reported very good health (eAppendix D).

Deliberation 1

During the deliberation on the prioritization criteria, the concept of “condition severity” emerged as an important criterion for veterans. This criterion captured simultaneous consideration of both clinical necessity and burden on the veteran to obtain care. For example, participants felt that patients with a life-threatening illness should be prioritized for civilian care over patients who need preventative or primary care (clinical necessity) and that elderly patients with substantial difficulty traveling to VHA appointments should be prioritized over patients who can travel more easily (burden). The Choice Act regulations at the time of the DD session did not reflect this nuanced perspective, stipulating only that veterans must live > 40 miles from the nearest VHA medical facility.

One of the 3 groups did not prioritize the patient cases because some members felt that no veteran should be constrained from receiving civilian care if they desired it. Nonetheless, this group did agree with prioritizing the first 2 cases in Table 3. The other groups prioritized all 8 cases in generally similar ways.

Deliberation 2

No clear consensus emerged on the buy vs build question. A representative from each table presented their group’s positions, rationale, and recommendations after deliberations were completed. After hearing the range of positions, the groups then had another opportunity to deliberate based on what they heard from the other tables; no new recommendations or consensus emerged.

Participants who were in favor of allocating more funds toward the build policy offered a range of rationales, saying that it would (1) increase access for rural veterans by building CBOCs and deploying more mobile units that could bring outlets for health care closer to their home communities; (2) provide critical and unique medical expertise to address veteran-specific issues such as prosthetics, traumatic brain injury, posttraumatic stress disorder, spinal cord injury, and shrapnel wounds that are typically not available through civilian providers; (3) give VHA more oversight over the quality and cost of care, which is more challenging to do with civilian providers; and (4) Improve VHA infrastructure by, for example, upgrading technology and attracting the best clinicians and staff to support “our VHA.”

Participants who were in favor of allocating more funds toward the buy policy also offered a range of rationales, saying that it would (1) decrease patient burden by increasing access through community providers, decreasing wait time, and lessening personal cost and travel time; (2) allow more patients to receive civilian care, which was generally seen as beneficial by a few participants because of perceptions that the VHA provides lower quality care due to a shortage of VHA providers, run-down/older facilities, lack of technology, and poorer-quality VHA providers; and (3) provide an opportunity to divest of costly facilities and invest in other innovative approaches. Regarding this last reason, a few participants felt that the VHA is “gouged” when building medical centers that overrun budgets. They also were concerned that investing in facilities tied VHA to specific locations when current locations of veterans may change “25 years from now.”

Survey Results

Twenty-three of the 24 participants completed both pre- and postsession surveys. The majority of participants in the session felt people in the group respected their opinion (96%); felt that the facilitator did not try to influence the group with her own opinions (96%); indicated they understood the information enough to participate as much as they wanted (100%); and were hopeful that their reasoning and recommendations would help inform VHA policy makers (82%).

The surveys also provided an opportunity to examine the extent to which knowledge, attitudes, and opinions changed from before to after the deliberation. Even with the small sample, responses revealed a trend toward improved knowledge about key elements of the Choice Act and its goals. Further, there was a shift in some participants’ opinions about how patients should be prioritized to receive civilian care. For example, before the deliberation participants generally felt that all veterans should be able to receive civilian care, whereas postdeliberation this was not the case. Postdeliberation, most participants felt that primary care should not be a high priority for civilian care but continued to endorse prioritizing civilian care for specialty services like orthopedic or cardiology-related care. Finally, participants moved from more diverse recommendations regarding additional funds allocations, toward consensus after the deliberation around allocating funds to the build policy. Eight participants supported a build policy beforehand, whereas 16 supported this policy afterward.

Discussion

This study explored DD as a method for deeply engaging veterans in complex policy making to guide funding allocation and prioritization decisions related to the Choice Act, decisions that are still very relevant today within the context of the Mission Act and have substantial implications for how health care is delivered in the VHA. The Mission Act passed on June 6, 2018, with the goal of improving access to and the reliability of civilian or community care for eligible veterans.²⁷ Decisions related to appropriating scarce funding to improve access to care is an emotional and value-laden topic that elicited strong and divergent opinions among the participants. Veterans were eager to have their voices heard and had strong expectations that VHA leadership would be briefed about their recommendations. The majority of participants were satisfied with the deliberation process, felt they understood the issues, and felt their opinions were respected. They expressed feelings of comradery and community throughout the process.

In this single deliberation session, the groups did not achieve a single, final consensus regarding how VHA funding should ultimately be allocated between buy and build policy options. Nonetheless, participants provided a rich array of recommendations and rationale for them. Session moderators observed rich, sophisticated, fair, and reason-based discussions on this complex topic. Participants left with a deeper knowledge and appreciation for the complex trade-offs and expressed strong rationales for both sides of the policy debate on build vs buy. In addition, the project yielded results of high interest to VHA policy makers.

This work was presented in multiple venues between 2015 to 2016, and to both local and national VHA leadership, including the local Executive Quality Leadership Boards, the VHA Central Office Committee on the Future State of VA Community Care, the VA Office of Patient Centered Care, and the National Veteran Experience Committee. Through these discussions and others, we saw great interest within the VHA system and high-level leaders to explore ways to include veterans’ voices in the policy-making process. This work was invaluable to our research team (eAppendix E 

Many health system decisions regarding what care should be delivered (and how) involve making difficult, value-laden choices in the context of limited resources. DD methods can be used to target and obtain specific viewpoints from diverse populations, such as the informed perspectives of minority and underrepresented populations within the VHA.¹⁹ For example, female veterans were oversampled to ensure that the informed preferences of this population was obtained. Thus, DD methods could provide a valuable tool for health systems to elicit in-depth diverse patient input on high-profile policies that will have a substantial impact on the system’s patient population.

Limitations

One potential downside of DD is that, because of the resource-intensive nature of deliberation sessions, they are often conducted with relatively small groups.⁹ Viewpoints of those within these small samples who are willing to spend an entire day discussing a complex topic may not be representative of the larger patient community. However, the core goal of DD is diversity of opinions rather than representativeness.

A stratified random sampling strategy that oversampled for underrepresented and minority populations was used to help select a diverse group that represents the population on key characteristics and partially addresses concern about representativeness. Efforts to optimize participation rates, including providing monetary incentives, also are helpful and have led to high participation rates in past deliberations.⁷

Health system communication strategies that promote the importance of becoming involved in DD sessions also may be helpful in improving rates of recruitment. On particularly important topics where health system leaders feel a larger resource investment is justified, conducting larger scale deliberations with many small groups may obtain more generalizable evidence about what individual patients and groups of patients recommend.⁷ However, due to the inherent limitations of surveys and focus group approaches for obtaining informed views on complex topics, there are no clear systematic alternatives to the DD approach.

Conclusion

DD is an effective method to meaningfully engage patients in deep deliberations to guide complex policy making. Although design of deliberative sessions is resource-intensive, patient engagement efforts, such as those described in this paper, could be an important aspect of a well-functioning learning health system. Further research into alternative, streamlined methods that can also engage veterans more deeply is needed. DD also can be combined with other approaches to broaden and confirm findings, including focus groups, town hall meetings, or surveys.

Although this study did not provide consensus on how the VHA should allocate funds with respect to the Choice Act, it did provide insight into the importance and feasibility of engaging veterans in the policy-making process. As more policies aimed at improving veterans’ access to civilian care are created, such as the most recent Mission Act, policy makers should strongly consider using the DD method of obtaining informed veteran input into future policy decisions.

Acknowledgments
Funding was provided by the US Department of Veterans Affairs Office of Analytics and Business Intelligence (OABI) and the VA Quality Enhancement Research Initiative (QUERI). Dr. Caverly was supported in part by a VA Career Development Award (CDA 16-151). Dr. Krein is supported by a VA Health Services Research and Development Research Career Scientist Award (RCS 11-222). The authors thank the veterans who participated in this work. They also thank Caitlin Reardon and Natalya Wawrin for their assistance in organizing the deliberation session.

Diagnostic error—defined as a failure to establish an accurate and timely explanation of the patient’s health problem—is an important source of patient harm.¹ Data suggest that all patients will experience at least 1 diagnostic error in their lifetime.^2-4 Not surprisingly, diagnostic errors are among the leading categories of paid malpractice claims in the United States.⁵

Despite diagnostic errors being morbid and sometimes deadly in the hospital,^6,7 little is known about how residents and learners approach diagnostic decision making. Errors in diagnosis are believed to stem from cognitive or system failures,⁸ with errors in cognition believed to occur due to rapid, reflexive thinking operating in the absence of a more analytical, deliberate process. System-based problems (eg, lack of expert availability, technology barriers, and access to data) have also been cited as contributors.⁹ However, whether and how these apply to trainees is not known.

Therefore, we conducted a focused ethnography of inpatient medicine teams (ie, attendings, residents, interns, and medical students) in 2 affiliated teaching hospitals, aiming to (a) observe the process of diagnosis by trainees and (b) identify methods to improve the diagnostic process and prevent errors.

We designed a multimethod, focused ethnographic study to examine diagnostic decision making in hospital settings.^10,11 In contrast to anthropologic ethnographies that study entire fields using open-ended questions, our study was designed to examine the process of diagnosis from the perspective of clinicians engaged in this activity.¹¹ This approach allowed us to capture diagnostic decisions and cognitive and system-based factors in a manner currently lacking in the literature.¹²

Setting and Participants

Between January 2016 and May 2016, we observed the members of four inpatient internal medicine teaching teams at 2 affiliated teaching hospitals. We purposefully selected teaching teams for observation because they are the primary model of care in academic settings and we have expertise in carrying out similar studies.^13,14 Teaching teams typically consisted of a medical attending (senior-level physician), 1 senior resident (a second- or third-year postgraduate trainee), two interns (a trainee in their first postgraduate year), and two to four  medical students. Teams were selected at random using existing schedules and followed Monday to Friday so as to permit observation of work on call and noncall days. Owing to manpower limitations, weekend and night shifts were not observed. However, overnight events were captured during morning rounds.

Most of the teams began rounds at 8:30 AM. Typically, rounds lasted for 90–120 min and concluded with a recap (ie, “running the list”) with a review of explicit plans for patients after they had been evaluated by the attending. This discussion often occurred in the team rooms, with the attending leading the discussion with the trainees.

Data Collection

A multidisciplinary team, including clinicians (eg, physicians, nurses), nonclinicians (eg, qualitative researchers, social scientists), and healthcare engineers, conducted the observations. We observed preround activities of interns and residents before arrival of the attending (7:00 AM - 8:30 AM), followed by morning rounds with the entire team, and afternoon work that included senior residents, interns, and students.

To capture multiple aspects of the diagnostic process, we collected data using field notes modeled on components of the National Academy of Science model for diagnosis (Appendix).^1,15 This model encompasses phases of the diagnostic process (eg, data gathering, integration, formulation of a working diagnosis, treatment delivery, and outcomes) and the work system (team members, organization, technology and tools, physical environment, tasks).

Focus Groups and Interviews

At the end of weekly observations, we conducted focus groups with the residents and one-on- one interviews with the attendings. Focus groups with the residents were conducted to encourage a group discussion about the diagnostic process. Separate interviews with the attendings were performed to ensure that power differentials did not influence discussions. During focus groups, we specifically asked about challenges and possible solutions to improve diagnosis. Experienced qualitative methodologists (J.F., M.H., M.Q.) used semistructured interview guides for discussions (Appendix).

Data Analysis

After aggregating and reading the data, three reviewers (V.C., S.K., S.S.) began inductive analysis by handwriting notes and initial reflective thoughts to create preliminary codes. Multiple team members then reread the original field notes and the focus group/interview data to refine the preliminary codes and develop additional codes. Next, relationships between codes were identified and used to develop key themes. Triangulation of data collected from observations and interview/focus group sessions was carried out to compare data that we surmised with data that were verbalized by the team. The developed themes were discussed as a group to ensure consistency of major findings.

Ethical and Regulatory Oversight

This study was reviewed and approved by the Institutional Review Boards at the University of Michigan Health System (HUM-00106657) and the VA Ann Arbor Healthcare System (1-2016-010040).

Four teaching teams (4 attendings, 4 senior residents, 9 interns, and 14 medical students) were observed over 33 distinct shifts and 168 hours. Observations included morning rounds (96 h), postround call days (52 h), and postround non-call days (20 h). Morning rounds lasted an average of 127 min (range: 48-232 min) and included an average of 9 patients (range: 4-16 patients).

Themes Regarding the Diagnostic Process

We identified the following 4 primary themes related to the diagnostic process in teaching hospitals: (1) diagnosis is a social phenomenon; (2) data necessary to make diagnoses are fragmented; (3) distractions undermine the diagnostic process; and (4) time pressures interfere with diagnostic decision making (Appendix Table 1).

(1) Diagnosis is a Social Phenomenon.

Team members viewed the process of diagnosis as a social exchange of facts, findings, and strategies within a defined structure. The opportunity to discuss impressions with others was valued as a means to share, test, and process assumptions.

“Rounds are the most important part of the process. That is where we make most decisions in a collective, collaborative way with the attending present. We bounce ideas off each other.” (Intern)

Typical of social processes, variations based on time of day and schedule were observed. For instance, during call days, learners gathered data and formed working diagnosis and treatment plans with minimal attending interaction. This separation of roles and responsibilities introduced a hierarchy within diagnosis as follows:

“The interns would not call me first; they would talk to the senior resident and then if the senior thought he should chat with me, then they would call. But for the most part, they gather information and come up with the plan.” (Attending).

The work system was suited to facilitate social interactions. For instance, designated rooms (with team members informally assigned to a computer) provided physical proximity of the resident to interns and medical students. In this space, numerous informal discussions between team members (eg, “What do you think about this test?” “I’m not sure what to do about this finding.” “Should I call a [consult] on this patient?”) were observed. Although proximity to each other was viewed as beneficial, dangers to the social nature of diagnosis in the form of anchoring (ie, a cognitive bias where emphasis is placed on the first piece of data)¹⁶ were also mentioned. Similarly, the paradox associated with social proof (ie, the pressure to assume conformity within a group) was also observed as disagreement between team members and attendings rarely occurred during observations.

“I mean, they’re the attending, right? It’s hard to argue with them when they want a test or something done. When I do push back, it’s rare that others will support me–so it’s usually me and the attending.” (Resident)

“I would push back if I think it’s really bad for the patient or could cause harm–but the truth is, it doesn’t happen much.” (Intern)

(2) Data Necessary to Make Diagnoses are Fragmented

Team members universally cited fragmentation in data delivery, retrieval, and processing as a barrier to diagnosis. Team members indicated that test results might not be looked at or acted upon in a timely manner, and participants pointed to the electronic medical record as a source of this challenge.

“Before I knew about [the app for Epic], I would literally sit on the computer to get all the information we would need on rounds. Its key to making decisions. We often say we will do something, only to find the test result doesn’t support it–and then we’re back to square 1.” (Intern)

Information used by teams came from myriad sources (eg, patients, family members, electronic records) and from various settings (eg, emergency department, patient rooms, discussions with consultants). Additionally, test results often appeared without warning. Thus, availability of information was poorly aligned with clinical duties.

“They (the lab) will call us when a blood culture is positive or something is off. That is very helpful but it often comes later in the day, when we’re done with rounds.” (Resident)

The work system was highlighted as a key contributor to data fragmentation. Peculiarities of our electronic medical record (EMR) and how data were collected, stored, or presented were described as “frustrating,” and “unsafe,” by team members. Correspondingly, we frequently observed interns asking for assistance for tasks such as ordering tests or finding information despite being “trained” to use the EMR.

“People have to learn how to filter, how to recognize the most important points and link data streams together in terms of causality. But we assume they know where to find that information. It’s actually a very hard thing to do, for both the house staff and me.” (Attending)

(3) Distractions Undermine the Diagnostic Process

Distractions often created cognitive difficulties. For example, ambient noise and interruptions from neighbors working on other teams were cited as barriers to diagnosis. In addition, we observed several team members using headphones to drown out ambient noise while working on the computer.

“I know I shouldn’t do it (wear headphones), but I have no other way of turning down the noise so I can concentrate.” (Intern)

Similarly, the unpredictable nature and the volume of pages often interrupted thinking about diagnosis.

“Sometimes the pager just goes off all the time and (after making sure its not an urgent issue), I will just ignore it for a bit, especially if I am in the middle of something. It would be great if I could finish my thought process knowing I would not be interrupted.” (Resident)

To mitigate this problem, 1 attending described how he would proactively seek out nurses caring for his patients to “head off” questions (eg, “I will renew the restraints and medications this morning,” and “Is there anything you need in terms of orders for this patient that I can take care of now?”) that might lead to pages. Another resident described his approach as follows:

“I make it a point to tell the nurses where I will be hanging out and where they can find me if they have any questions. I tell them to come talk to me rather than page me since that will be less distracting.” (Resident).

Most of the interns described documentation work such as writing admission and progress notes in negative terms (“an academic exercise,” “part of the billing activity”). However, in the context of interruptions, some described this as helpful.

“The most valuable part of the thinking process was writing the assessment and plan because that’s actually my schema for all problems. It literally is the only time where I can sit and collect my thoughts to formulate a diagnosis and plan.” (Intern)

(4) Time Pressures Interfere With Diagnostic Decision Making

All team members spoke about the challenge of finding time for diagnosis during the workday. Often, they had to skip learning sessions for this purpose.

“They tell us we should go to morning report or noon conference but when I’m running around trying to get things done. I hate having to choose between my education and doing what’s best for the patient–but that’s often what it comes down to.” (Intern)

When specifically asked whether setting aside dedicated time to specifically review and formulate diagnoses would be valuable, respondents were uniformly enthusiastic. Team members described attentional conflicts as being the worst when “cross covering” other teams on call days, as their patient load effectively doubled during this time. Of note, cross-covering occurred when teams were also on call—and thus took them away from important diagnostic activities such as data gathering or synthesis for patients they were admitting.

“If you were to ever design a system where errors were likely–this is how you would design it: take a team with little supervision, double their patient load, keep them busy with new challenging cases and then ask questions about patients they know little about.” (Resident)

Although diagnostic errors have been called “the next frontier for patient safety,”¹⁷ little is known about the process, barriers, and facilitators to diagnosis in teaching hospitals. In this focused ethnography conducted at 2 academic medical centers, we identified multiple cognitive and system-level challenges and potential strategies to improve diagnosis from trainees engaged in this activity. Key themes identified by those we observed included the social nature of diagnosis, fragmented information delivery, constant distractions and interruptions, and time pressures. In turn, these insights allow us to generate strategies that can be applied to improve the diagnostic process in teaching hospitals.

Our study underscores the importance of social interactions in diagnosis. In contrast, most of the interventions to prevent diagnostic errors target individual providers through practices such as metacognition and “thinking about thinking.”^18-20 These interventions are based on Daniel Kahnemann’s work on dual thought process. Type 1 thought processes are fast, subconscious, reflexive, largely intuitive, and more vulnerable to error. In contrast, Type 2 processes are slower, deliberate, analytic, and less prone to error.²¹ Although an individual’s Type 2 thought capacity is limited, a major goal of cognitive interventions is to encourage Type 2 over Type 1 thinking, an approach termed “de-biasing.”^22-24 Unfortunately, cognitive interventions testing such approaches have suffered mixed results–perhaps because of lack of focus on collective wisdom or group thinking, which may be key to diagnosis from our findings.^9,25 In this sense, morning rounds were a social gathering used to strategize and develop care plans, but with limited time to think about diagnosis.²⁶ Introduction of defined periods for individuals to engage in diagnostic activities such as de-biasing (ie, asking “what else could this be)²⁷ before or after rounds may provide an opportunity for reflection and improving diagnosis. In addition, embedding tools such as diagnosis expanders and checklists within these defined time slots^28,29 may prove to be useful in reflecting on diagnosis and preventing diagnostic errors.

An unexpected yet important finding from this study were the challenges posed by distractions and the physical environment. Potentially maladaptive workarounds to these interruptions included use of headphones; more productive strategies included updating nurses with plans to avert pages and creating a list of activities to ensure that key tasks were not forgotten.^30,31 Applying lessons from aviation, a focused effort to limit distractions during key portions of the day, might be worth considering for diagnostic safety.³² Similarly, improving the environment in which diagnosis occurs—including creating spaces that are quiet, orderly, and optimized for thinking—may be valuable.³³Our study has limitations. First, our findings are limited to direct observations; we are thus unable to comment on how unobserved aspects of care (eg, cognitive processes) might have influenced our findings. Our observations of clinical care might also have introduced a Hawthorne effect. However, because we were closely integrated with teams and conducted focus groups to corroborate our assessments, we believe that this was not the case. Second, we did not identify diagnostic errors or link processes we observed to errors. Third, our approach is limited to 2 teaching centers, thereby limiting the generalizability of findings. Relatedly, we were only able to conduct observations during weekdays; differences in weekend and night resources might affect our insights.

The cognitive and system-based barriers faced by clinicians in teaching hospitals suggest that new methods to improve diagnosis are needed. Future interventions such as defined “time-outs” for diagnosis, strategies focused on limiting distractions, and methods to improve communication between team members are novel and have parallels in other industries. As challenges to quantify diagnostic errors abound,³⁴ improving cognitive- and system-based factors via reflection through communication, concentration, and organization is necessary to improve medical decision making in academic medical centers.

Disclosures

None declared for all coauthors.

Funding

This project was supported by grant number P30HS024385 from the Agency for Healthcare Research and Quality. The funding source played no role in study design, data acquisition, analysis or decision to report these data. Dr. Chopra is supported by a career development award from the Agency of Healthcare Research and Quality (1-K08-HS022835-01). Dr. Krein is supported by a VA Health Services Research and Development Research Career Scientist Award (RCS 11-222). Dr. Singh is partially supported by Houston VA HSR&D Center for Innovations in Quality, Effectiveness and Safety (CIN 13-413). The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality or the Department of Veterans Affairs.

Diagnostic error—defined as a failure to establish an accurate and timely explanation of the patient’s health problem—is an important source of patient harm.¹ Data suggest that all patients will experience at least 1 diagnostic error in their lifetime.^2-4 Not surprisingly, diagnostic errors are among the leading categories of paid malpractice claims in the United States.⁵

Despite diagnostic errors being morbid and sometimes deadly in the hospital,^6,7 little is known about how residents and learners approach diagnostic decision making. Errors in diagnosis are believed to stem from cognitive or system failures,⁸ with errors in cognition believed to occur due to rapid, reflexive thinking operating in the absence of a more analytical, deliberate process. System-based problems (eg, lack of expert availability, technology barriers, and access to data) have also been cited as contributors.⁹ However, whether and how these apply to trainees is not known.

Therefore, we conducted a focused ethnography of inpatient medicine teams (ie, attendings, residents, interns, and medical students) in 2 affiliated teaching hospitals, aiming to (a) observe the process of diagnosis by trainees and (b) identify methods to improve the diagnostic process and prevent errors.

We designed a multimethod, focused ethnographic study to examine diagnostic decision making in hospital settings.^10,11 In contrast to anthropologic ethnographies that study entire fields using open-ended questions, our study was designed to examine the process of diagnosis from the perspective of clinicians engaged in this activity.¹¹ This approach allowed us to capture diagnostic decisions and cognitive and system-based factors in a manner currently lacking in the literature.¹²

Setting and Participants

Between January 2016 and May 2016, we observed the members of four inpatient internal medicine teaching teams at 2 affiliated teaching hospitals. We purposefully selected teaching teams for observation because they are the primary model of care in academic settings and we have expertise in carrying out similar studies.^13,14 Teaching teams typically consisted of a medical attending (senior-level physician), 1 senior resident (a second- or third-year postgraduate trainee), two interns (a trainee in their first postgraduate year), and two to four  medical students. Teams were selected at random using existing schedules and followed Monday to Friday so as to permit observation of work on call and noncall days. Owing to manpower limitations, weekend and night shifts were not observed. However, overnight events were captured during morning rounds.

Most of the teams began rounds at 8:30 AM. Typically, rounds lasted for 90–120 min and concluded with a recap (ie, “running the list”) with a review of explicit plans for patients after they had been evaluated by the attending. This discussion often occurred in the team rooms, with the attending leading the discussion with the trainees.

Data Collection

A multidisciplinary team, including clinicians (eg, physicians, nurses), nonclinicians (eg, qualitative researchers, social scientists), and healthcare engineers, conducted the observations. We observed preround activities of interns and residents before arrival of the attending (7:00 AM - 8:30 AM), followed by morning rounds with the entire team, and afternoon work that included senior residents, interns, and students.

To capture multiple aspects of the diagnostic process, we collected data using field notes modeled on components of the National Academy of Science model for diagnosis (Appendix).^1,15 This model encompasses phases of the diagnostic process (eg, data gathering, integration, formulation of a working diagnosis, treatment delivery, and outcomes) and the work system (team members, organization, technology and tools, physical environment, tasks).

Focus Groups and Interviews

At the end of weekly observations, we conducted focus groups with the residents and one-on- one interviews with the attendings. Focus groups with the residents were conducted to encourage a group discussion about the diagnostic process. Separate interviews with the attendings were performed to ensure that power differentials did not influence discussions. During focus groups, we specifically asked about challenges and possible solutions to improve diagnosis. Experienced qualitative methodologists (J.F., M.H., M.Q.) used semistructured interview guides for discussions (Appendix).

Data Analysis

After aggregating and reading the data, three reviewers (V.C., S.K., S.S.) began inductive analysis by handwriting notes and initial reflective thoughts to create preliminary codes. Multiple team members then reread the original field notes and the focus group/interview data to refine the preliminary codes and develop additional codes. Next, relationships between codes were identified and used to develop key themes. Triangulation of data collected from observations and interview/focus group sessions was carried out to compare data that we surmised with data that were verbalized by the team. The developed themes were discussed as a group to ensure consistency of major findings.

Ethical and Regulatory Oversight

This study was reviewed and approved by the Institutional Review Boards at the University of Michigan Health System (HUM-00106657) and the VA Ann Arbor Healthcare System (1-2016-010040).

Four teaching teams (4 attendings, 4 senior residents, 9 interns, and 14 medical students) were observed over 33 distinct shifts and 168 hours. Observations included morning rounds (96 h), postround call days (52 h), and postround non-call days (20 h). Morning rounds lasted an average of 127 min (range: 48-232 min) and included an average of 9 patients (range: 4-16 patients).

Themes Regarding the Diagnostic Process

We identified the following 4 primary themes related to the diagnostic process in teaching hospitals: (1) diagnosis is a social phenomenon; (2) data necessary to make diagnoses are fragmented; (3) distractions undermine the diagnostic process; and (4) time pressures interfere with diagnostic decision making (Appendix Table 1).

(1) Diagnosis is a Social Phenomenon.

Team members viewed the process of diagnosis as a social exchange of facts, findings, and strategies within a defined structure. The opportunity to discuss impressions with others was valued as a means to share, test, and process assumptions.

“Rounds are the most important part of the process. That is where we make most decisions in a collective, collaborative way with the attending present. We bounce ideas off each other.” (Intern)

Typical of social processes, variations based on time of day and schedule were observed. For instance, during call days, learners gathered data and formed working diagnosis and treatment plans with minimal attending interaction. This separation of roles and responsibilities introduced a hierarchy within diagnosis as follows:

“The interns would not call me first; they would talk to the senior resident and then if the senior thought he should chat with me, then they would call. But for the most part, they gather information and come up with the plan.” (Attending).

The work system was suited to facilitate social interactions. For instance, designated rooms (with team members informally assigned to a computer) provided physical proximity of the resident to interns and medical students. In this space, numerous informal discussions between team members (eg, “What do you think about this test?” “I’m not sure what to do about this finding.” “Should I call a [consult] on this patient?”) were observed. Although proximity to each other was viewed as beneficial, dangers to the social nature of diagnosis in the form of anchoring (ie, a cognitive bias where emphasis is placed on the first piece of data)¹⁶ were also mentioned. Similarly, the paradox associated with social proof (ie, the pressure to assume conformity within a group) was also observed as disagreement between team members and attendings rarely occurred during observations.

“I mean, they’re the attending, right? It’s hard to argue with them when they want a test or something done. When I do push back, it’s rare that others will support me–so it’s usually me and the attending.” (Resident)

“I would push back if I think it’s really bad for the patient or could cause harm–but the truth is, it doesn’t happen much.” (Intern)

(2) Data Necessary to Make Diagnoses are Fragmented

Team members universally cited fragmentation in data delivery, retrieval, and processing as a barrier to diagnosis. Team members indicated that test results might not be looked at or acted upon in a timely manner, and participants pointed to the electronic medical record as a source of this challenge.

“Before I knew about [the app for Epic], I would literally sit on the computer to get all the information we would need on rounds. Its key to making decisions. We often say we will do something, only to find the test result doesn’t support it–and then we’re back to square 1.” (Intern)

Information used by teams came from myriad sources (eg, patients, family members, electronic records) and from various settings (eg, emergency department, patient rooms, discussions with consultants). Additionally, test results often appeared without warning. Thus, availability of information was poorly aligned with clinical duties.

“They (the lab) will call us when a blood culture is positive or something is off. That is very helpful but it often comes later in the day, when we’re done with rounds.” (Resident)

The work system was highlighted as a key contributor to data fragmentation. Peculiarities of our electronic medical record (EMR) and how data were collected, stored, or presented were described as “frustrating,” and “unsafe,” by team members. Correspondingly, we frequently observed interns asking for assistance for tasks such as ordering tests or finding information despite being “trained” to use the EMR.

“People have to learn how to filter, how to recognize the most important points and link data streams together in terms of causality. But we assume they know where to find that information. It’s actually a very hard thing to do, for both the house staff and me.” (Attending)

(3) Distractions Undermine the Diagnostic Process

Distractions often created cognitive difficulties. For example, ambient noise and interruptions from neighbors working on other teams were cited as barriers to diagnosis. In addition, we observed several team members using headphones to drown out ambient noise while working on the computer.

“I know I shouldn’t do it (wear headphones), but I have no other way of turning down the noise so I can concentrate.” (Intern)

Similarly, the unpredictable nature and the volume of pages often interrupted thinking about diagnosis.

“Sometimes the pager just goes off all the time and (after making sure its not an urgent issue), I will just ignore it for a bit, especially if I am in the middle of something. It would be great if I could finish my thought process knowing I would not be interrupted.” (Resident)

To mitigate this problem, 1 attending described how he would proactively seek out nurses caring for his patients to “head off” questions (eg, “I will renew the restraints and medications this morning,” and “Is there anything you need in terms of orders for this patient that I can take care of now?”) that might lead to pages. Another resident described his approach as follows:

“I make it a point to tell the nurses where I will be hanging out and where they can find me if they have any questions. I tell them to come talk to me rather than page me since that will be less distracting.” (Resident).

Most of the interns described documentation work such as writing admission and progress notes in negative terms (“an academic exercise,” “part of the billing activity”). However, in the context of interruptions, some described this as helpful.

“The most valuable part of the thinking process was writing the assessment and plan because that’s actually my schema for all problems. It literally is the only time where I can sit and collect my thoughts to formulate a diagnosis and plan.” (Intern)

(4) Time Pressures Interfere With Diagnostic Decision Making

All team members spoke about the challenge of finding time for diagnosis during the workday. Often, they had to skip learning sessions for this purpose.

“They tell us we should go to morning report or noon conference but when I’m running around trying to get things done. I hate having to choose between my education and doing what’s best for the patient–but that’s often what it comes down to.” (Intern)

When specifically asked whether setting aside dedicated time to specifically review and formulate diagnoses would be valuable, respondents were uniformly enthusiastic. Team members described attentional conflicts as being the worst when “cross covering” other teams on call days, as their patient load effectively doubled during this time. Of note, cross-covering occurred when teams were also on call—and thus took them away from important diagnostic activities such as data gathering or synthesis for patients they were admitting.

“If you were to ever design a system where errors were likely–this is how you would design it: take a team with little supervision, double their patient load, keep them busy with new challenging cases and then ask questions about patients they know little about.” (Resident)

Although diagnostic errors have been called “the next frontier for patient safety,”¹⁷ little is known about the process, barriers, and facilitators to diagnosis in teaching hospitals. In this focused ethnography conducted at 2 academic medical centers, we identified multiple cognitive and system-level challenges and potential strategies to improve diagnosis from trainees engaged in this activity. Key themes identified by those we observed included the social nature of diagnosis, fragmented information delivery, constant distractions and interruptions, and time pressures. In turn, these insights allow us to generate strategies that can be applied to improve the diagnostic process in teaching hospitals.

Our study underscores the importance of social interactions in diagnosis. In contrast, most of the interventions to prevent diagnostic errors target individual providers through practices such as metacognition and “thinking about thinking.”^18-20 These interventions are based on Daniel Kahnemann’s work on dual thought process. Type 1 thought processes are fast, subconscious, reflexive, largely intuitive, and more vulnerable to error. In contrast, Type 2 processes are slower, deliberate, analytic, and less prone to error.²¹ Although an individual’s Type 2 thought capacity is limited, a major goal of cognitive interventions is to encourage Type 2 over Type 1 thinking, an approach termed “de-biasing.”^22-24 Unfortunately, cognitive interventions testing such approaches have suffered mixed results–perhaps because of lack of focus on collective wisdom or group thinking, which may be key to diagnosis from our findings.^9,25 In this sense, morning rounds were a social gathering used to strategize and develop care plans, but with limited time to think about diagnosis.²⁶ Introduction of defined periods for individuals to engage in diagnostic activities such as de-biasing (ie, asking “what else could this be)²⁷ before or after rounds may provide an opportunity for reflection and improving diagnosis. In addition, embedding tools such as diagnosis expanders and checklists within these defined time slots^28,29 may prove to be useful in reflecting on diagnosis and preventing diagnostic errors.

An unexpected yet important finding from this study were the challenges posed by distractions and the physical environment. Potentially maladaptive workarounds to these interruptions included use of headphones; more productive strategies included updating nurses with plans to avert pages and creating a list of activities to ensure that key tasks were not forgotten.^30,31 Applying lessons from aviation, a focused effort to limit distractions during key portions of the day, might be worth considering for diagnostic safety.³² Similarly, improving the environment in which diagnosis occurs—including creating spaces that are quiet, orderly, and optimized for thinking—may be valuable.³³Our study has limitations. First, our findings are limited to direct observations; we are thus unable to comment on how unobserved aspects of care (eg, cognitive processes) might have influenced our findings. Our observations of clinical care might also have introduced a Hawthorne effect. However, because we were closely integrated with teams and conducted focus groups to corroborate our assessments, we believe that this was not the case. Second, we did not identify diagnostic errors or link processes we observed to errors. Third, our approach is limited to 2 teaching centers, thereby limiting the generalizability of findings. Relatedly, we were only able to conduct observations during weekdays; differences in weekend and night resources might affect our insights.

The cognitive and system-based barriers faced by clinicians in teaching hospitals suggest that new methods to improve diagnosis are needed. Future interventions such as defined “time-outs” for diagnosis, strategies focused on limiting distractions, and methods to improve communication between team members are novel and have parallels in other industries. As challenges to quantify diagnostic errors abound,³⁴ improving cognitive- and system-based factors via reflection through communication, concentration, and organization is necessary to improve medical decision making in academic medical centers.

Disclosures

None declared for all coauthors.

Funding

This project was supported by grant number P30HS024385 from the Agency for Healthcare Research and Quality. The funding source played no role in study design, data acquisition, analysis or decision to report these data. Dr. Chopra is supported by a career development award from the Agency of Healthcare Research and Quality (1-K08-HS022835-01). Dr. Krein is supported by a VA Health Services Research and Development Research Career Scientist Award (RCS 11-222). Dr. Singh is partially supported by Houston VA HSR&D Center for Innovations in Quality, Effectiveness and Safety (CIN 13-413). The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality or the Department of Veterans Affairs.

Diagnostic error—defined as a failure to establish an accurate and timely explanation of the patient’s health problem—is an important source of patient harm.¹ Data suggest that all patients will experience at least 1 diagnostic error in their lifetime.^2-4 Not surprisingly, diagnostic errors are among the leading categories of paid malpractice claims in the United States.⁵

Despite diagnostic errors being morbid and sometimes deadly in the hospital,^6,7 little is known about how residents and learners approach diagnostic decision making. Errors in diagnosis are believed to stem from cognitive or system failures,⁸ with errors in cognition believed to occur due to rapid, reflexive thinking operating in the absence of a more analytical, deliberate process. System-based problems (eg, lack of expert availability, technology barriers, and access to data) have also been cited as contributors.⁹ However, whether and how these apply to trainees is not known.

Therefore, we conducted a focused ethnography of inpatient medicine teams (ie, attendings, residents, interns, and medical students) in 2 affiliated teaching hospitals, aiming to (a) observe the process of diagnosis by trainees and (b) identify methods to improve the diagnostic process and prevent errors.

We designed a multimethod, focused ethnographic study to examine diagnostic decision making in hospital settings.^10,11 In contrast to anthropologic ethnographies that study entire fields using open-ended questions, our study was designed to examine the process of diagnosis from the perspective of clinicians engaged in this activity.¹¹ This approach allowed us to capture diagnostic decisions and cognitive and system-based factors in a manner currently lacking in the literature.¹²

Setting and Participants

Between January 2016 and May 2016, we observed the members of four inpatient internal medicine teaching teams at 2 affiliated teaching hospitals. We purposefully selected teaching teams for observation because they are the primary model of care in academic settings and we have expertise in carrying out similar studies.^13,14 Teaching teams typically consisted of a medical attending (senior-level physician), 1 senior resident (a second- or third-year postgraduate trainee), two interns (a trainee in their first postgraduate year), and two to four  medical students. Teams were selected at random using existing schedules and followed Monday to Friday so as to permit observation of work on call and noncall days. Owing to manpower limitations, weekend and night shifts were not observed. However, overnight events were captured during morning rounds.

Most of the teams began rounds at 8:30 AM. Typically, rounds lasted for 90–120 min and concluded with a recap (ie, “running the list”) with a review of explicit plans for patients after they had been evaluated by the attending. This discussion often occurred in the team rooms, with the attending leading the discussion with the trainees.

Data Collection

A multidisciplinary team, including clinicians (eg, physicians, nurses), nonclinicians (eg, qualitative researchers, social scientists), and healthcare engineers, conducted the observations. We observed preround activities of interns and residents before arrival of the attending (7:00 AM - 8:30 AM), followed by morning rounds with the entire team, and afternoon work that included senior residents, interns, and students.

To capture multiple aspects of the diagnostic process, we collected data using field notes modeled on components of the National Academy of Science model for diagnosis (Appendix).^1,15 This model encompasses phases of the diagnostic process (eg, data gathering, integration, formulation of a working diagnosis, treatment delivery, and outcomes) and the work system (team members, organization, technology and tools, physical environment, tasks).

Focus Groups and Interviews

At the end of weekly observations, we conducted focus groups with the residents and one-on- one interviews with the attendings. Focus groups with the residents were conducted to encourage a group discussion about the diagnostic process. Separate interviews with the attendings were performed to ensure that power differentials did not influence discussions. During focus groups, we specifically asked about challenges and possible solutions to improve diagnosis. Experienced qualitative methodologists (J.F., M.H., M.Q.) used semistructured interview guides for discussions (Appendix).

Data Analysis

After aggregating and reading the data, three reviewers (V.C., S.K., S.S.) began inductive analysis by handwriting notes and initial reflective thoughts to create preliminary codes. Multiple team members then reread the original field notes and the focus group/interview data to refine the preliminary codes and develop additional codes. Next, relationships between codes were identified and used to develop key themes. Triangulation of data collected from observations and interview/focus group sessions was carried out to compare data that we surmised with data that were verbalized by the team. The developed themes were discussed as a group to ensure consistency of major findings.

Ethical and Regulatory Oversight

This study was reviewed and approved by the Institutional Review Boards at the University of Michigan Health System (HUM-00106657) and the VA Ann Arbor Healthcare System (1-2016-010040).

Four teaching teams (4 attendings, 4 senior residents, 9 interns, and 14 medical students) were observed over 33 distinct shifts and 168 hours. Observations included morning rounds (96 h), postround call days (52 h), and postround non-call days (20 h). Morning rounds lasted an average of 127 min (range: 48-232 min) and included an average of 9 patients (range: 4-16 patients).

Themes Regarding the Diagnostic Process

We identified the following 4 primary themes related to the diagnostic process in teaching hospitals: (1) diagnosis is a social phenomenon; (2) data necessary to make diagnoses are fragmented; (3) distractions undermine the diagnostic process; and (4) time pressures interfere with diagnostic decision making (Appendix Table 1).

(1) Diagnosis is a Social Phenomenon.

Team members viewed the process of diagnosis as a social exchange of facts, findings, and strategies within a defined structure. The opportunity to discuss impressions with others was valued as a means to share, test, and process assumptions.

“Rounds are the most important part of the process. That is where we make most decisions in a collective, collaborative way with the attending present. We bounce ideas off each other.” (Intern)

Typical of social processes, variations based on time of day and schedule were observed. For instance, during call days, learners gathered data and formed working diagnosis and treatment plans with minimal attending interaction. This separation of roles and responsibilities introduced a hierarchy within diagnosis as follows:

“The interns would not call me first; they would talk to the senior resident and then if the senior thought he should chat with me, then they would call. But for the most part, they gather information and come up with the plan.” (Attending).

The work system was suited to facilitate social interactions. For instance, designated rooms (with team members informally assigned to a computer) provided physical proximity of the resident to interns and medical students. In this space, numerous informal discussions between team members (eg, “What do you think about this test?” “I’m not sure what to do about this finding.” “Should I call a [consult] on this patient?”) were observed. Although proximity to each other was viewed as beneficial, dangers to the social nature of diagnosis in the form of anchoring (ie, a cognitive bias where emphasis is placed on the first piece of data)¹⁶ were also mentioned. Similarly, the paradox associated with social proof (ie, the pressure to assume conformity within a group) was also observed as disagreement between team members and attendings rarely occurred during observations.

“I mean, they’re the attending, right? It’s hard to argue with them when they want a test or something done. When I do push back, it’s rare that others will support me–so it’s usually me and the attending.” (Resident)

“I would push back if I think it’s really bad for the patient or could cause harm–but the truth is, it doesn’t happen much.” (Intern)

(2) Data Necessary to Make Diagnoses are Fragmented

Team members universally cited fragmentation in data delivery, retrieval, and processing as a barrier to diagnosis. Team members indicated that test results might not be looked at or acted upon in a timely manner, and participants pointed to the electronic medical record as a source of this challenge.

“Before I knew about [the app for Epic], I would literally sit on the computer to get all the information we would need on rounds. Its key to making decisions. We often say we will do something, only to find the test result doesn’t support it–and then we’re back to square 1.” (Intern)

Information used by teams came from myriad sources (eg, patients, family members, electronic records) and from various settings (eg, emergency department, patient rooms, discussions with consultants). Additionally, test results often appeared without warning. Thus, availability of information was poorly aligned with clinical duties.

“They (the lab) will call us when a blood culture is positive or something is off. That is very helpful but it often comes later in the day, when we’re done with rounds.” (Resident)

The work system was highlighted as a key contributor to data fragmentation. Peculiarities of our electronic medical record (EMR) and how data were collected, stored, or presented were described as “frustrating,” and “unsafe,” by team members. Correspondingly, we frequently observed interns asking for assistance for tasks such as ordering tests or finding information despite being “trained” to use the EMR.

“People have to learn how to filter, how to recognize the most important points and link data streams together in terms of causality. But we assume they know where to find that information. It’s actually a very hard thing to do, for both the house staff and me.” (Attending)

(3) Distractions Undermine the Diagnostic Process

Distractions often created cognitive difficulties. For example, ambient noise and interruptions from neighbors working on other teams were cited as barriers to diagnosis. In addition, we observed several team members using headphones to drown out ambient noise while working on the computer.

“I know I shouldn’t do it (wear headphones), but I have no other way of turning down the noise so I can concentrate.” (Intern)

Similarly, the unpredictable nature and the volume of pages often interrupted thinking about diagnosis.

“Sometimes the pager just goes off all the time and (after making sure its not an urgent issue), I will just ignore it for a bit, especially if I am in the middle of something. It would be great if I could finish my thought process knowing I would not be interrupted.” (Resident)

To mitigate this problem, 1 attending described how he would proactively seek out nurses caring for his patients to “head off” questions (eg, “I will renew the restraints and medications this morning,” and “Is there anything you need in terms of orders for this patient that I can take care of now?”) that might lead to pages. Another resident described his approach as follows:

“I make it a point to tell the nurses where I will be hanging out and where they can find me if they have any questions. I tell them to come talk to me rather than page me since that will be less distracting.” (Resident).

Most of the interns described documentation work such as writing admission and progress notes in negative terms (“an academic exercise,” “part of the billing activity”). However, in the context of interruptions, some described this as helpful.

“The most valuable part of the thinking process was writing the assessment and plan because that’s actually my schema for all problems. It literally is the only time where I can sit and collect my thoughts to formulate a diagnosis and plan.” (Intern)

(4) Time Pressures Interfere With Diagnostic Decision Making

All team members spoke about the challenge of finding time for diagnosis during the workday. Often, they had to skip learning sessions for this purpose.

“They tell us we should go to morning report or noon conference but when I’m running around trying to get things done. I hate having to choose between my education and doing what’s best for the patient–but that’s often what it comes down to.” (Intern)

When specifically asked whether setting aside dedicated time to specifically review and formulate diagnoses would be valuable, respondents were uniformly enthusiastic. Team members described attentional conflicts as being the worst when “cross covering” other teams on call days, as their patient load effectively doubled during this time. Of note, cross-covering occurred when teams were also on call—and thus took them away from important diagnostic activities such as data gathering or synthesis for patients they were admitting.

“If you were to ever design a system where errors were likely–this is how you would design it: take a team with little supervision, double their patient load, keep them busy with new challenging cases and then ask questions about patients they know little about.” (Resident)

Although diagnostic errors have been called “the next frontier for patient safety,”¹⁷ little is known about the process, barriers, and facilitators to diagnosis in teaching hospitals. In this focused ethnography conducted at 2 academic medical centers, we identified multiple cognitive and system-level challenges and potential strategies to improve diagnosis from trainees engaged in this activity. Key themes identified by those we observed included the social nature of diagnosis, fragmented information delivery, constant distractions and interruptions, and time pressures. In turn, these insights allow us to generate strategies that can be applied to improve the diagnostic process in teaching hospitals.

Our study underscores the importance of social interactions in diagnosis. In contrast, most of the interventions to prevent diagnostic errors target individual providers through practices such as metacognition and “thinking about thinking.”^18-20 These interventions are based on Daniel Kahnemann’s work on dual thought process. Type 1 thought processes are fast, subconscious, reflexive, largely intuitive, and more vulnerable to error. In contrast, Type 2 processes are slower, deliberate, analytic, and less prone to error.²¹ Although an individual’s Type 2 thought capacity is limited, a major goal of cognitive interventions is to encourage Type 2 over Type 1 thinking, an approach termed “de-biasing.”^22-24 Unfortunately, cognitive interventions testing such approaches have suffered mixed results–perhaps because of lack of focus on collective wisdom or group thinking, which may be key to diagnosis from our findings.^9,25 In this sense, morning rounds were a social gathering used to strategize and develop care plans, but with limited time to think about diagnosis.²⁶ Introduction of defined periods for individuals to engage in diagnostic activities such as de-biasing (ie, asking “what else could this be)²⁷ before or after rounds may provide an opportunity for reflection and improving diagnosis. In addition, embedding tools such as diagnosis expanders and checklists within these defined time slots^28,29 may prove to be useful in reflecting on diagnosis and preventing diagnostic errors.

An unexpected yet important finding from this study were the challenges posed by distractions and the physical environment. Potentially maladaptive workarounds to these interruptions included use of headphones; more productive strategies included updating nurses with plans to avert pages and creating a list of activities to ensure that key tasks were not forgotten.^30,31 Applying lessons from aviation, a focused effort to limit distractions during key portions of the day, might be worth considering for diagnostic safety.³² Similarly, improving the environment in which diagnosis occurs—including creating spaces that are quiet, orderly, and optimized for thinking—may be valuable.³³Our study has limitations. First, our findings are limited to direct observations; we are thus unable to comment on how unobserved aspects of care (eg, cognitive processes) might have influenced our findings. Our observations of clinical care might also have introduced a Hawthorne effect. However, because we were closely integrated with teams and conducted focus groups to corroborate our assessments, we believe that this was not the case. Second, we did not identify diagnostic errors or link processes we observed to errors. Third, our approach is limited to 2 teaching centers, thereby limiting the generalizability of findings. Relatedly, we were only able to conduct observations during weekdays; differences in weekend and night resources might affect our insights.

The cognitive and system-based barriers faced by clinicians in teaching hospitals suggest that new methods to improve diagnosis are needed. Future interventions such as defined “time-outs” for diagnosis, strategies focused on limiting distractions, and methods to improve communication between team members are novel and have parallels in other industries. As challenges to quantify diagnostic errors abound,³⁴ improving cognitive- and system-based factors via reflection through communication, concentration, and organization is necessary to improve medical decision making in academic medical centers.

Disclosures

None declared for all coauthors.

Funding

This project was supported by grant number P30HS024385 from the Agency for Healthcare Research and Quality. The funding source played no role in study design, data acquisition, analysis or decision to report these data. Dr. Chopra is supported by a career development award from the Agency of Healthcare Research and Quality (1-K08-HS022835-01). Dr. Krein is supported by a VA Health Services Research and Development Research Career Scientist Award (RCS 11-222). Dr. Singh is partially supported by Houston VA HSR&D Center for Innovations in Quality, Effectiveness and Safety (CIN 13-413). The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality or the Department of Veterans Affairs.