User login
For over 3 decades, AcademyHealth and its predecessor organizations and members have been studying how the healthcare system works and ways to improve health outcomes. The value of health services research (HSR) training programs that emphasize learning opportunities in delivery system settings was initially articulated at the 2009 AcademyHealth Summit on the Future of HSR Data and Methods.1 Two years later, the need for such programs was reiterated as a priority for AcademyHealth’s HSR Learning Consortium in their strategic plan.2 While HSR methods have become increasingly sophisticated, historical approaches largely relied on extant (usually academic-based) researchers.
To realize the goal of building learning health systems (considered here to be entities where applied, operationally relevant research is systematically designed, generated, and translated into high-quality care delivery), many healthcare organizations have begun to use researchers as an internal resource to inform and support higher quality and more efficient care delivery operations. However, there is a current dearth of scientists trained in research disciplines (eg, comparative effectiveness research, patient-centered outcomes research, implementation science) more directly applicable to operational settings.3,4 Conducting research within these “real-world” environments is challenging for a variety of well-documented reasons,5,6 and many important questions cannot be answered using traditional study designs and/or methodologies. Until more researchers are trained in research approaches that align better with care delivery needs, the field will continue to fall short of addressing topics identified by the National Academy of Medicine (NAM; formerly the Institute of Medicine) as priorities for real system improvement.
While researchers in academic and consulting settings play critical roles in knowledge generation, a substantial area for expansion is support for “embedded researchers” who work more directly with operational leaders and understand local context, data, and organizational-level goals of delivery systems. AcademyHealth’s Delivery System Science Fellowship (DSSF) was developed to forge a stronger link between rigorous research practice and pragmatic aspects of care delivery.
Since inception of the DSSF, national attention has further emphasized the need for specialized, experiential learning with specific competencies that extend formal HSR training. The Agency for Healthcare Research and Quality (AHRQ) is currently convening a Technical Expert Panel to guide Training the Next Generation of Learning Health System Researchers as part of its larger effort to provide support for evidence generation and uptake in these applied settings (see http://www.ahrq.gov/news/blog/ahrqviews/supporting-learning-health-systems.html for more information). This work is reinforced by an ongoing project at AcademyHealth focused on Understanding the Current Health Services Research Workforce and Maximizing Its Future,7 partially sponsored by AHRQ. These efforts aim to inform AHRQ-funded career and training program requirements in order to build a workforce in partnership with academia and delivery organizations in support of developing high-functioning researchers directly positioned to drive progress toward learning healthcare systems.
ACADEMYHEALTH’S DSSF
The AcademyHealth DSSF Program is a pioneering effort designed to meet the needs of learning healthcare systems for a human capital resource capable of generating insights from operational data and deploying this knowledge effectively. Established in 2012 in partnership with 3 initial host sites, the DSSF provides a paid, postdoctoral training opportunity to help highly qualified, early-career researchers gain applied experience in delivery system settings.8 The goal is to provide hands-on experience and professional leadership opportunities to enhance the array of skills needed to generate and apply evidence in delivery systems. The long-term program objective is for graduates to employ the methods and training garnered during the DSSF to produce new, practical insights required to transform healthcare delivery and achieve the “Triple Aim.”9
Sixteen delivery systems have participated in the DSSF program over the course of 5 years. These organizations represent a diverse group of innovative, high-performing systems that serve populations across the rural-urban continuum. Host site preceptors are nationally recognized experts in areas such as health economics, comparative effectiveness, pragmatic trials, clinical decision support, and implementation science. The fellowship is guided by an advisory committee that provides strategic direction and plays a key role in selecting fellows. AcademyHealth partners with 8 to 10 health systems annually, accepting new partnerships as interest and availability allows. The Figure summarizes program inputs, elements, antecedents, outputs, and intended outcomes. As a relatively new and evolving program, the DSSF is using a disciplined approach to assess both mid- and long-term outcomes. The Table is a comparative matrix that depicts the range in areas of investment offered by a sample of host sites during the fellowship program. Complementing development of the common core competencies and exposures presented in the Table, specific project choice at a host site is mutually determined according to system needs and the fellow’s interests.
To apply for DSSF, researchers must hold a doctoral degree in any relevant aspect of clinical medicine, HSR, or a related field. The review committee evaluates applicants based on their qualifications, a clear statement of professional goals, appropriateness for placement, the quality of a writing sample, and letters of recommendation. Host sites select a fellow based on their individual preferences and recommendations from the review committee. The minimum duration of the fellowship is 1 year, during which the fellow works full time at the host site. Host sites provide the fellows’ salary and benefits, financial support to attend AcademyHealth’s Annual Research Meeting, and mentorship and/or training. To ensure general continuity across sites, each fellow’s work focuses on “delivery system science (DSS),” with a significant part of the work intended for public dissemination (eg, a conference presentation) or publication in a peer-reviewed journal.
Host sites additionally provide AcademyHealth financial support to manage the process of recruiting promising candidates. AcademyHealth manages and convenes the advisory committee proceedings, facilitates the application cycle (including peer review of applicants), markets the fellowship, guides the interview and match process, and promotes placement of fellows upon program completion. AcademyHealth also convenes active and graduate fellows to foster engagement and professional development.
To date, 118 individuals have applied to the program and 25 fellows have been accepted. Nineteen have completed the fellowship, 2 are continuing as second-year fellows, and 4 started their fellowship in the fall of 2016. Fellows have a wide range of expertise in areas such as epidemiology, exercise physiology, health psychology, anthropology, clinical medicine, qualitative methods, organizational behavior, and systems engineering. Once individuals complete their fellowship, they become DSSF “alumni” and remain involved in program activities and as peer mentors.
EARLY EXPERIENCE WITH OUTCOMES
Program Level
AcademyHealth administers an annual evaluation to assess the program, understand impact on the fellow’s professional development and growth, track publications, and inform programmatic goals. To date, we have identified over 50 peer-reviewed publications resulting from work conducted through the fellowship (visit http://www.academyhealth.org/dssfpublications for a full listing of publications stemming from the DSSF program). As an example of continuous program improvement, this year staff implemented a fellow-led monthly call in response to requests to connect fellows. This has proven to be a useful response for fellows to understand how to enhance their own experience by learning from fellows in other systems to incorporate focused areas of development via cross-system sharing. As an indicator of continued value, of the 16 sites that have participated to date, 10 have participated in the DSSF for more than 1 cycle; 3 are new host sites currently participating in their first year of the program; and 3 host sites participated for 1 year. The 3 inaugural host sites that helped launch the DSSF continue to serve as host sites to date.
As a marker of longitudinal success, staff will continue to follow up with preceptors and fellows to understand fellows’ contributions to the host site and the field, as well as impact on the fellow career trajectory. Of the 19 fellow alumni to date, 8 have moved on to academic or research positions, and 11 have remained in care delivery systems to provide local expertise in study design, execution, and evidence uptake.
The program has also made some general contributions to advance the discipline of delivery system science, including:
1. Defining DSS and clarifying training needs for “embedded researchers” and health system analysts. To characterize the fellowship, AcademyHealth and the program advisory committee jointly developed the following definition for delivery system science (DSS):
“DSS includes research that seeks to understand how delivery systems operate, influence, change, and respond to external stimulus, among other topics. DSS may include efforts to examine how and under what circumstances interventions work and how delivery systems effectively implement evidence-based innovations. For the DSSF, DSS is conducted by researchers who are ‘embedded’ in delivery systems and respond to the decision-making needs of those systems.”
Additionally, closer connections with DSS leaders have led to a better understanding of challenges, opportunities, and needs of delivery systems.10,11
2. Cultivating a network of delivery systems and system leaders interested in expanding the cadre of embedded researchers, and trainees who intend to build careers in DSS. The DSSF also aims to enhance fellows’ skills and knowledge base, career opportunities, and professional network. To extend these relationships and support delivery system analytics, AcademyHealth worked with preceptors and fellows to inform creation of a new Community of Practice supported by AcademyHealth’s EDM Forum and guide planning for AcademyHealth’s Concordium conference to provide a national meeting to showcase DSS.
3. Strategic planning to ensure sustainable support for embedded research within delivery systems. Substantial interest in the program developed quickly, with rapid learning over the first few cycles to refine the program to meet host sites’ and fellows’ needs. Both efforts were critical to demonstrate that the DSSF fulfills an important need for our health system partners and members. As indicated previously, strong, sustained interest from prospective host sites and applicants demonstrates the program has created a win-win to jointly assess fit while building skills and supporting continuous learning.
Likewise, Lisa Simpson, President and CEO of AcademyHealth, and Lucy Savitz, DSSF host site preceptor at Intermountain Healthcare, participated in the Canadian Institutes of Health Research (CIHR) Invitational Workshop, “Modernizing Health Services and Policy Research Training in Canada” in March 2016. Shared learning largely informed by the DSSF led to CIHR creating a similar fellowship program with initial awards to be made in 2017 (see https://www.researchnet-recherchenet.ca/rnr16/vwOpprtntyDtls.do?prog=2540&view=browseActive&sponsor=CIHR-8&type=EXACT&resultCount=25 for more information). We are working to thread these efforts together in a way to leverage our learning community of government agencies, academia, and employers as a long-term funding stream for training in delivery science.
Participant Host Site Level
Two selected examples of how DSSF researchers have engaged high-priority topics that contributed to health system operations are provided here.
Kaiser Permanente Southern California: Assessing adherence with “Choosing Wisely” recommendations in oncology. In partnership with preceptor Dr. Michael Gould, 2013-2014 DSSF fellow Dr. Erin Hahn worked with the Kaiser Permanente Southern California (KPSC) Care Improvement Research Team to lead a project addressing several KPSC priority areas. Focusing on “Choosing Wisely” recommendations from the American Society of Clinical Oncology,12 the project evaluated appropriateness of imaging and laboratory services for early-stage cancer patients and survivors between and within 2 integrated health systems, Kaiser Permanente (KP) and Intermountain Healthcare.13,14 Results were presented to KP national leaders, including an external health policy advisory board. In close collaboration with clinical and operational leaders in medical oncology, this multiregional, multisystem project is contributing to targeted quality improvement efforts and improved healthcare value, including audit and feedback of nonrecommended labs.
Dr. Hahn subsequently received a KPSC Incubator Award, a competitive internal grant, to further study factors associated with use of nonrecommended surveillance lab tests for early-stage breast cancer patients. The study focused on medical oncologists within KPSC, categorizing them as high or low utilizers of the tests.15 Results indicate that high utilizers perceive that the tests help manage patient anxiety about recurrence, while acknowledging that the tests do not provide clinical utility. These findings are contributing to the development of targeted survivorship services across the organization.
Intermountain Healthcare: Formative evaluation of large-scale implementation of shared decision-making. Preceptor Dr. Lucy Savitz assembled a team to conduct a formative evaluation of Intermountain Healthcare’s efforts to implement shared decision-making (SDM) as part of its Center for Medicare and Medicaid Innovation Challenge Award. The 2015-2017 DSSF fellow at Intermountain Healthcare, Dr. Kim Brunisholz, served as a core member of the project team, focusing primarily on a mixed-methods evaluation of the SDM program.
Dr. Brunisholz engaged operational leads, clinical teams, patient and family advisory councils, and senior executives to conduct the program assessment. Results demonstrated significant variation in invited participation in SDM among eligible patient populations: preference sensitive conditions (1 in 30 patients), oncology-related diagnosis (1 in 3 patients), and chronic conditions (1 in 74 patients). Provisional analysis of patient-level clinical outcomes demonstrated that among those invited to the SDM program compared to those that were not, total joint replacement was decreased (10.1% vs 17.3%; P < 0.001) and a trend towards breast conservation emerged (61.8% vs 56.4%; P = 0.10). No difference in treatment choice for lower back pain was observed. Qualitative program analysis suggested need for improvements in the areas of (1) routine and continuous staff training, (2) workflow standardization, and (3) active data monitoring with meaningful, actionable feedback to caregivers. In response to these results, a chartered SDM Steering Committee was created (Dr. Brunisholz is a member of that group) to develop a strategic plan for SDM, with an accompanying organizational response to reimplement SDM in a targeted manner. Learning from this program is being leveraged to support a subproject analysis on a large scale using data from the High Value Healthcare Collaborative as part of an AHRQ-funded Center of Excellence award. (See https://www.ahrq.gov/news/newsroom/press-releases/2015/pcorawards.html. For more information on the High Value Healthcare Collaborative, please visit: https://www.highvaluehealthcare.org.)
CONCLUSION
Moving forward, the DSSF will continue working with progressive delivery systems. Partnerships between organizations that are interested in integrating rigorous research practice to drive continuous system improvement and maximize the value of care will have substantial need for technical skills and analytic capacity. They will also need to ensure that researchers working in their systems have sufficient understanding of cultural and political context within the organization to be effective leaders who can manage change.
AcademyHealth created the DSSF in response to the field’s request to build a research workforce that reflects the vision for a 21st Century Health System, as laid out by the NAM.16 We anticipate that as the US Department of Health & Human Services’ goals for payment reform and new measures to promote quality and high-value care are implemented, the DSSF trainees’ skill set will be increasingly valuable and will provide needed thought leadership on strategies to generate and apply evidence in practice.
Disclosure
Ms. Kanani received funding from Intermountain Healthcare, Kaiser Permanente Southern California for support for the Delivery System Science Fellowship. Drs. Hahn, Gould, and Brunisholz have no conflicts to disclose. Dr. Savitz has received funding from HRQ COE, PCORI LHSNet; received funding for lectures from the Institute from Healthcare Improvement, Department of Epidemiology, University of Utah; received funding for travel, accommodations, and meeting expenses from AHRQ NAC, EDM Forum, AH CAPP, AARP NPC, and PROM TEP; and received additional funding from Dartmouth University. Dr. Holve received funding from Intermountain Healthcare, Kaiser Permanente Southern California for support for the Delivery System Science Fellowship, provided by our delivery system partners, several of whom are coauthors on this manuscript.
1. AcademyHealth. Health Services Research in 2020: Summit on the Future of HSR Data and Methods. http://www.academyhealth.org/About/content.cfm?ItemNumber=2529. Accessed March 21, 2016.
2. AcademyHealth. Health Services Research (HSR) Learning Consortium Strategic Plan. http://www.academyhealth.org/files/ProfDev/Files/HSRstrategicplan2011FINAL.pdf. Accessed March 21, 2016.
3. Institute of Medicine. Initial National Priorities for Comparative Effectiveness Research. http://www.nationalacademies.org/hmd/~/media/Files/Report%20Files/2009/ComparativeEffectivenessResearchPriorities/CER%20report%20brief%2008-13-09.pdf. Accessed March 21, 2016.
4. Bonham A, Rich E, Davis D, Longnecker D, Heinig S. Putting evidence to work: an expanded research agenda for academic medicine in the era of health care reform. Acad Med. 2010;85(10):1551-1553. PubMed
5. Zerhouni E. Translational and clinical science—time for a new vision. N Engl J Med. 2005;353(15):1621-1623. PubMed
6. AcademyHealth. “Getting Answers We Can Believe In: Methodological Considerations When Using Electronic Clinical Data for Research,” EDM Forum, December 2012.
7. Rich G, Collins A. Current and Future Demand for Health Services Researchers. Funded by the Agency for Healthcare Research and Quality (AHRQ). Presented at the AcademyHealth HSR Workforce Conference, Understanding the Current Health Services Research Workforce and Maximizing Its Future, funded by AHRQ, Patient-Centered Outcomes Research Institute, and the Robert Wood Johnson Foundation. October 2016.
8. AcademyHealth. Delivery System Science Fellowship. http://www.academyhealth.org/dssf. Accessed March 21, 2016.
9. Institute for Healthcare Improvement. IHI Triple Aim Initiative. http://www.ihi.org/engage/initiatives/tripleaim/Pages/default.aspx. Accessed March 21, 2016.
10. Psek W, Stametz R, Bailey-Davis L, et al. Operationalizing the learning health care system in an integrated delivery system. eGEMs. 2015;3(1):1122. PubMed
11. Thompson C, Kurian A, Luft H. Linking electronic health records to better understand breast cancer patient pathways within and between two health systems. eGEMs. 2015;3(1):1127. PubMed
12. Schnipper L, Smith TJ, Raghavan D, et al. American Society of Clinical Oncology identifies five key opportunities to improve care and reduce costs: The top five list for oncology. J Clin Oncol. 2012;30(14):1715-1724. PubMed
13. Hahn E, Tang T, Lee JS, et al. Use of posttreatment imaging and biomarkers in survivors of early‐stage breast cancer: Inappropriate surveillance or necessary care? Cancer. 2015;122(6):908-916. PubMed
14. Hahn E, Tang T, Lee JS, et al. Use of imaging for staging of early-stage breast cancer in two integrated health care systems: Adherence with a choosing wisely recommendation. J Oncol Pract. 2015;11(3):e320-e328. PubMed
15. Hahn EE, Munoz-Plaza C, Wang J, et al. Anxiety, culture, expectations: Oncologist-perceived factors associated with use of non-recommended serum tumor marker tests for surveillance of early stage breast cancer. J Oncol Pract. 2016;13(1):e77-e290. PubMed
16. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy of Sciences; 2000.
For over 3 decades, AcademyHealth and its predecessor organizations and members have been studying how the healthcare system works and ways to improve health outcomes. The value of health services research (HSR) training programs that emphasize learning opportunities in delivery system settings was initially articulated at the 2009 AcademyHealth Summit on the Future of HSR Data and Methods.1 Two years later, the need for such programs was reiterated as a priority for AcademyHealth’s HSR Learning Consortium in their strategic plan.2 While HSR methods have become increasingly sophisticated, historical approaches largely relied on extant (usually academic-based) researchers.
To realize the goal of building learning health systems (considered here to be entities where applied, operationally relevant research is systematically designed, generated, and translated into high-quality care delivery), many healthcare organizations have begun to use researchers as an internal resource to inform and support higher quality and more efficient care delivery operations. However, there is a current dearth of scientists trained in research disciplines (eg, comparative effectiveness research, patient-centered outcomes research, implementation science) more directly applicable to operational settings.3,4 Conducting research within these “real-world” environments is challenging for a variety of well-documented reasons,5,6 and many important questions cannot be answered using traditional study designs and/or methodologies. Until more researchers are trained in research approaches that align better with care delivery needs, the field will continue to fall short of addressing topics identified by the National Academy of Medicine (NAM; formerly the Institute of Medicine) as priorities for real system improvement.
While researchers in academic and consulting settings play critical roles in knowledge generation, a substantial area for expansion is support for “embedded researchers” who work more directly with operational leaders and understand local context, data, and organizational-level goals of delivery systems. AcademyHealth’s Delivery System Science Fellowship (DSSF) was developed to forge a stronger link between rigorous research practice and pragmatic aspects of care delivery.
Since inception of the DSSF, national attention has further emphasized the need for specialized, experiential learning with specific competencies that extend formal HSR training. The Agency for Healthcare Research and Quality (AHRQ) is currently convening a Technical Expert Panel to guide Training the Next Generation of Learning Health System Researchers as part of its larger effort to provide support for evidence generation and uptake in these applied settings (see http://www.ahrq.gov/news/blog/ahrqviews/supporting-learning-health-systems.html for more information). This work is reinforced by an ongoing project at AcademyHealth focused on Understanding the Current Health Services Research Workforce and Maximizing Its Future,7 partially sponsored by AHRQ. These efforts aim to inform AHRQ-funded career and training program requirements in order to build a workforce in partnership with academia and delivery organizations in support of developing high-functioning researchers directly positioned to drive progress toward learning healthcare systems.
ACADEMYHEALTH’S DSSF
The AcademyHealth DSSF Program is a pioneering effort designed to meet the needs of learning healthcare systems for a human capital resource capable of generating insights from operational data and deploying this knowledge effectively. Established in 2012 in partnership with 3 initial host sites, the DSSF provides a paid, postdoctoral training opportunity to help highly qualified, early-career researchers gain applied experience in delivery system settings.8 The goal is to provide hands-on experience and professional leadership opportunities to enhance the array of skills needed to generate and apply evidence in delivery systems. The long-term program objective is for graduates to employ the methods and training garnered during the DSSF to produce new, practical insights required to transform healthcare delivery and achieve the “Triple Aim.”9
Sixteen delivery systems have participated in the DSSF program over the course of 5 years. These organizations represent a diverse group of innovative, high-performing systems that serve populations across the rural-urban continuum. Host site preceptors are nationally recognized experts in areas such as health economics, comparative effectiveness, pragmatic trials, clinical decision support, and implementation science. The fellowship is guided by an advisory committee that provides strategic direction and plays a key role in selecting fellows. AcademyHealth partners with 8 to 10 health systems annually, accepting new partnerships as interest and availability allows. The Figure summarizes program inputs, elements, antecedents, outputs, and intended outcomes. As a relatively new and evolving program, the DSSF is using a disciplined approach to assess both mid- and long-term outcomes. The Table is a comparative matrix that depicts the range in areas of investment offered by a sample of host sites during the fellowship program. Complementing development of the common core competencies and exposures presented in the Table, specific project choice at a host site is mutually determined according to system needs and the fellow’s interests.
To apply for DSSF, researchers must hold a doctoral degree in any relevant aspect of clinical medicine, HSR, or a related field. The review committee evaluates applicants based on their qualifications, a clear statement of professional goals, appropriateness for placement, the quality of a writing sample, and letters of recommendation. Host sites select a fellow based on their individual preferences and recommendations from the review committee. The minimum duration of the fellowship is 1 year, during which the fellow works full time at the host site. Host sites provide the fellows’ salary and benefits, financial support to attend AcademyHealth’s Annual Research Meeting, and mentorship and/or training. To ensure general continuity across sites, each fellow’s work focuses on “delivery system science (DSS),” with a significant part of the work intended for public dissemination (eg, a conference presentation) or publication in a peer-reviewed journal.
Host sites additionally provide AcademyHealth financial support to manage the process of recruiting promising candidates. AcademyHealth manages and convenes the advisory committee proceedings, facilitates the application cycle (including peer review of applicants), markets the fellowship, guides the interview and match process, and promotes placement of fellows upon program completion. AcademyHealth also convenes active and graduate fellows to foster engagement and professional development.
To date, 118 individuals have applied to the program and 25 fellows have been accepted. Nineteen have completed the fellowship, 2 are continuing as second-year fellows, and 4 started their fellowship in the fall of 2016. Fellows have a wide range of expertise in areas such as epidemiology, exercise physiology, health psychology, anthropology, clinical medicine, qualitative methods, organizational behavior, and systems engineering. Once individuals complete their fellowship, they become DSSF “alumni” and remain involved in program activities and as peer mentors.
EARLY EXPERIENCE WITH OUTCOMES
Program Level
AcademyHealth administers an annual evaluation to assess the program, understand impact on the fellow’s professional development and growth, track publications, and inform programmatic goals. To date, we have identified over 50 peer-reviewed publications resulting from work conducted through the fellowship (visit http://www.academyhealth.org/dssfpublications for a full listing of publications stemming from the DSSF program). As an example of continuous program improvement, this year staff implemented a fellow-led monthly call in response to requests to connect fellows. This has proven to be a useful response for fellows to understand how to enhance their own experience by learning from fellows in other systems to incorporate focused areas of development via cross-system sharing. As an indicator of continued value, of the 16 sites that have participated to date, 10 have participated in the DSSF for more than 1 cycle; 3 are new host sites currently participating in their first year of the program; and 3 host sites participated for 1 year. The 3 inaugural host sites that helped launch the DSSF continue to serve as host sites to date.
As a marker of longitudinal success, staff will continue to follow up with preceptors and fellows to understand fellows’ contributions to the host site and the field, as well as impact on the fellow career trajectory. Of the 19 fellow alumni to date, 8 have moved on to academic or research positions, and 11 have remained in care delivery systems to provide local expertise in study design, execution, and evidence uptake.
The program has also made some general contributions to advance the discipline of delivery system science, including:
1. Defining DSS and clarifying training needs for “embedded researchers” and health system analysts. To characterize the fellowship, AcademyHealth and the program advisory committee jointly developed the following definition for delivery system science (DSS):
“DSS includes research that seeks to understand how delivery systems operate, influence, change, and respond to external stimulus, among other topics. DSS may include efforts to examine how and under what circumstances interventions work and how delivery systems effectively implement evidence-based innovations. For the DSSF, DSS is conducted by researchers who are ‘embedded’ in delivery systems and respond to the decision-making needs of those systems.”
Additionally, closer connections with DSS leaders have led to a better understanding of challenges, opportunities, and needs of delivery systems.10,11
2. Cultivating a network of delivery systems and system leaders interested in expanding the cadre of embedded researchers, and trainees who intend to build careers in DSS. The DSSF also aims to enhance fellows’ skills and knowledge base, career opportunities, and professional network. To extend these relationships and support delivery system analytics, AcademyHealth worked with preceptors and fellows to inform creation of a new Community of Practice supported by AcademyHealth’s EDM Forum and guide planning for AcademyHealth’s Concordium conference to provide a national meeting to showcase DSS.
3. Strategic planning to ensure sustainable support for embedded research within delivery systems. Substantial interest in the program developed quickly, with rapid learning over the first few cycles to refine the program to meet host sites’ and fellows’ needs. Both efforts were critical to demonstrate that the DSSF fulfills an important need for our health system partners and members. As indicated previously, strong, sustained interest from prospective host sites and applicants demonstrates the program has created a win-win to jointly assess fit while building skills and supporting continuous learning.
Likewise, Lisa Simpson, President and CEO of AcademyHealth, and Lucy Savitz, DSSF host site preceptor at Intermountain Healthcare, participated in the Canadian Institutes of Health Research (CIHR) Invitational Workshop, “Modernizing Health Services and Policy Research Training in Canada” in March 2016. Shared learning largely informed by the DSSF led to CIHR creating a similar fellowship program with initial awards to be made in 2017 (see https://www.researchnet-recherchenet.ca/rnr16/vwOpprtntyDtls.do?prog=2540&view=browseActive&sponsor=CIHR-8&type=EXACT&resultCount=25 for more information). We are working to thread these efforts together in a way to leverage our learning community of government agencies, academia, and employers as a long-term funding stream for training in delivery science.
Participant Host Site Level
Two selected examples of how DSSF researchers have engaged high-priority topics that contributed to health system operations are provided here.
Kaiser Permanente Southern California: Assessing adherence with “Choosing Wisely” recommendations in oncology. In partnership with preceptor Dr. Michael Gould, 2013-2014 DSSF fellow Dr. Erin Hahn worked with the Kaiser Permanente Southern California (KPSC) Care Improvement Research Team to lead a project addressing several KPSC priority areas. Focusing on “Choosing Wisely” recommendations from the American Society of Clinical Oncology,12 the project evaluated appropriateness of imaging and laboratory services for early-stage cancer patients and survivors between and within 2 integrated health systems, Kaiser Permanente (KP) and Intermountain Healthcare.13,14 Results were presented to KP national leaders, including an external health policy advisory board. In close collaboration with clinical and operational leaders in medical oncology, this multiregional, multisystem project is contributing to targeted quality improvement efforts and improved healthcare value, including audit and feedback of nonrecommended labs.
Dr. Hahn subsequently received a KPSC Incubator Award, a competitive internal grant, to further study factors associated with use of nonrecommended surveillance lab tests for early-stage breast cancer patients. The study focused on medical oncologists within KPSC, categorizing them as high or low utilizers of the tests.15 Results indicate that high utilizers perceive that the tests help manage patient anxiety about recurrence, while acknowledging that the tests do not provide clinical utility. These findings are contributing to the development of targeted survivorship services across the organization.
Intermountain Healthcare: Formative evaluation of large-scale implementation of shared decision-making. Preceptor Dr. Lucy Savitz assembled a team to conduct a formative evaluation of Intermountain Healthcare’s efforts to implement shared decision-making (SDM) as part of its Center for Medicare and Medicaid Innovation Challenge Award. The 2015-2017 DSSF fellow at Intermountain Healthcare, Dr. Kim Brunisholz, served as a core member of the project team, focusing primarily on a mixed-methods evaluation of the SDM program.
Dr. Brunisholz engaged operational leads, clinical teams, patient and family advisory councils, and senior executives to conduct the program assessment. Results demonstrated significant variation in invited participation in SDM among eligible patient populations: preference sensitive conditions (1 in 30 patients), oncology-related diagnosis (1 in 3 patients), and chronic conditions (1 in 74 patients). Provisional analysis of patient-level clinical outcomes demonstrated that among those invited to the SDM program compared to those that were not, total joint replacement was decreased (10.1% vs 17.3%; P < 0.001) and a trend towards breast conservation emerged (61.8% vs 56.4%; P = 0.10). No difference in treatment choice for lower back pain was observed. Qualitative program analysis suggested need for improvements in the areas of (1) routine and continuous staff training, (2) workflow standardization, and (3) active data monitoring with meaningful, actionable feedback to caregivers. In response to these results, a chartered SDM Steering Committee was created (Dr. Brunisholz is a member of that group) to develop a strategic plan for SDM, with an accompanying organizational response to reimplement SDM in a targeted manner. Learning from this program is being leveraged to support a subproject analysis on a large scale using data from the High Value Healthcare Collaborative as part of an AHRQ-funded Center of Excellence award. (See https://www.ahrq.gov/news/newsroom/press-releases/2015/pcorawards.html. For more information on the High Value Healthcare Collaborative, please visit: https://www.highvaluehealthcare.org.)
CONCLUSION
Moving forward, the DSSF will continue working with progressive delivery systems. Partnerships between organizations that are interested in integrating rigorous research practice to drive continuous system improvement and maximize the value of care will have substantial need for technical skills and analytic capacity. They will also need to ensure that researchers working in their systems have sufficient understanding of cultural and political context within the organization to be effective leaders who can manage change.
AcademyHealth created the DSSF in response to the field’s request to build a research workforce that reflects the vision for a 21st Century Health System, as laid out by the NAM.16 We anticipate that as the US Department of Health & Human Services’ goals for payment reform and new measures to promote quality and high-value care are implemented, the DSSF trainees’ skill set will be increasingly valuable and will provide needed thought leadership on strategies to generate and apply evidence in practice.
Disclosure
Ms. Kanani received funding from Intermountain Healthcare, Kaiser Permanente Southern California for support for the Delivery System Science Fellowship. Drs. Hahn, Gould, and Brunisholz have no conflicts to disclose. Dr. Savitz has received funding from HRQ COE, PCORI LHSNet; received funding for lectures from the Institute from Healthcare Improvement, Department of Epidemiology, University of Utah; received funding for travel, accommodations, and meeting expenses from AHRQ NAC, EDM Forum, AH CAPP, AARP NPC, and PROM TEP; and received additional funding from Dartmouth University. Dr. Holve received funding from Intermountain Healthcare, Kaiser Permanente Southern California for support for the Delivery System Science Fellowship, provided by our delivery system partners, several of whom are coauthors on this manuscript.
For over 3 decades, AcademyHealth and its predecessor organizations and members have been studying how the healthcare system works and ways to improve health outcomes. The value of health services research (HSR) training programs that emphasize learning opportunities in delivery system settings was initially articulated at the 2009 AcademyHealth Summit on the Future of HSR Data and Methods.1 Two years later, the need for such programs was reiterated as a priority for AcademyHealth’s HSR Learning Consortium in their strategic plan.2 While HSR methods have become increasingly sophisticated, historical approaches largely relied on extant (usually academic-based) researchers.
To realize the goal of building learning health systems (considered here to be entities where applied, operationally relevant research is systematically designed, generated, and translated into high-quality care delivery), many healthcare organizations have begun to use researchers as an internal resource to inform and support higher quality and more efficient care delivery operations. However, there is a current dearth of scientists trained in research disciplines (eg, comparative effectiveness research, patient-centered outcomes research, implementation science) more directly applicable to operational settings.3,4 Conducting research within these “real-world” environments is challenging for a variety of well-documented reasons,5,6 and many important questions cannot be answered using traditional study designs and/or methodologies. Until more researchers are trained in research approaches that align better with care delivery needs, the field will continue to fall short of addressing topics identified by the National Academy of Medicine (NAM; formerly the Institute of Medicine) as priorities for real system improvement.
While researchers in academic and consulting settings play critical roles in knowledge generation, a substantial area for expansion is support for “embedded researchers” who work more directly with operational leaders and understand local context, data, and organizational-level goals of delivery systems. AcademyHealth’s Delivery System Science Fellowship (DSSF) was developed to forge a stronger link between rigorous research practice and pragmatic aspects of care delivery.
Since inception of the DSSF, national attention has further emphasized the need for specialized, experiential learning with specific competencies that extend formal HSR training. The Agency for Healthcare Research and Quality (AHRQ) is currently convening a Technical Expert Panel to guide Training the Next Generation of Learning Health System Researchers as part of its larger effort to provide support for evidence generation and uptake in these applied settings (see http://www.ahrq.gov/news/blog/ahrqviews/supporting-learning-health-systems.html for more information). This work is reinforced by an ongoing project at AcademyHealth focused on Understanding the Current Health Services Research Workforce and Maximizing Its Future,7 partially sponsored by AHRQ. These efforts aim to inform AHRQ-funded career and training program requirements in order to build a workforce in partnership with academia and delivery organizations in support of developing high-functioning researchers directly positioned to drive progress toward learning healthcare systems.
ACADEMYHEALTH’S DSSF
The AcademyHealth DSSF Program is a pioneering effort designed to meet the needs of learning healthcare systems for a human capital resource capable of generating insights from operational data and deploying this knowledge effectively. Established in 2012 in partnership with 3 initial host sites, the DSSF provides a paid, postdoctoral training opportunity to help highly qualified, early-career researchers gain applied experience in delivery system settings.8 The goal is to provide hands-on experience and professional leadership opportunities to enhance the array of skills needed to generate and apply evidence in delivery systems. The long-term program objective is for graduates to employ the methods and training garnered during the DSSF to produce new, practical insights required to transform healthcare delivery and achieve the “Triple Aim.”9
Sixteen delivery systems have participated in the DSSF program over the course of 5 years. These organizations represent a diverse group of innovative, high-performing systems that serve populations across the rural-urban continuum. Host site preceptors are nationally recognized experts in areas such as health economics, comparative effectiveness, pragmatic trials, clinical decision support, and implementation science. The fellowship is guided by an advisory committee that provides strategic direction and plays a key role in selecting fellows. AcademyHealth partners with 8 to 10 health systems annually, accepting new partnerships as interest and availability allows. The Figure summarizes program inputs, elements, antecedents, outputs, and intended outcomes. As a relatively new and evolving program, the DSSF is using a disciplined approach to assess both mid- and long-term outcomes. The Table is a comparative matrix that depicts the range in areas of investment offered by a sample of host sites during the fellowship program. Complementing development of the common core competencies and exposures presented in the Table, specific project choice at a host site is mutually determined according to system needs and the fellow’s interests.
To apply for DSSF, researchers must hold a doctoral degree in any relevant aspect of clinical medicine, HSR, or a related field. The review committee evaluates applicants based on their qualifications, a clear statement of professional goals, appropriateness for placement, the quality of a writing sample, and letters of recommendation. Host sites select a fellow based on their individual preferences and recommendations from the review committee. The minimum duration of the fellowship is 1 year, during which the fellow works full time at the host site. Host sites provide the fellows’ salary and benefits, financial support to attend AcademyHealth’s Annual Research Meeting, and mentorship and/or training. To ensure general continuity across sites, each fellow’s work focuses on “delivery system science (DSS),” with a significant part of the work intended for public dissemination (eg, a conference presentation) or publication in a peer-reviewed journal.
Host sites additionally provide AcademyHealth financial support to manage the process of recruiting promising candidates. AcademyHealth manages and convenes the advisory committee proceedings, facilitates the application cycle (including peer review of applicants), markets the fellowship, guides the interview and match process, and promotes placement of fellows upon program completion. AcademyHealth also convenes active and graduate fellows to foster engagement and professional development.
To date, 118 individuals have applied to the program and 25 fellows have been accepted. Nineteen have completed the fellowship, 2 are continuing as second-year fellows, and 4 started their fellowship in the fall of 2016. Fellows have a wide range of expertise in areas such as epidemiology, exercise physiology, health psychology, anthropology, clinical medicine, qualitative methods, organizational behavior, and systems engineering. Once individuals complete their fellowship, they become DSSF “alumni” and remain involved in program activities and as peer mentors.
EARLY EXPERIENCE WITH OUTCOMES
Program Level
AcademyHealth administers an annual evaluation to assess the program, understand impact on the fellow’s professional development and growth, track publications, and inform programmatic goals. To date, we have identified over 50 peer-reviewed publications resulting from work conducted through the fellowship (visit http://www.academyhealth.org/dssfpublications for a full listing of publications stemming from the DSSF program). As an example of continuous program improvement, this year staff implemented a fellow-led monthly call in response to requests to connect fellows. This has proven to be a useful response for fellows to understand how to enhance their own experience by learning from fellows in other systems to incorporate focused areas of development via cross-system sharing. As an indicator of continued value, of the 16 sites that have participated to date, 10 have participated in the DSSF for more than 1 cycle; 3 are new host sites currently participating in their first year of the program; and 3 host sites participated for 1 year. The 3 inaugural host sites that helped launch the DSSF continue to serve as host sites to date.
As a marker of longitudinal success, staff will continue to follow up with preceptors and fellows to understand fellows’ contributions to the host site and the field, as well as impact on the fellow career trajectory. Of the 19 fellow alumni to date, 8 have moved on to academic or research positions, and 11 have remained in care delivery systems to provide local expertise in study design, execution, and evidence uptake.
The program has also made some general contributions to advance the discipline of delivery system science, including:
1. Defining DSS and clarifying training needs for “embedded researchers” and health system analysts. To characterize the fellowship, AcademyHealth and the program advisory committee jointly developed the following definition for delivery system science (DSS):
“DSS includes research that seeks to understand how delivery systems operate, influence, change, and respond to external stimulus, among other topics. DSS may include efforts to examine how and under what circumstances interventions work and how delivery systems effectively implement evidence-based innovations. For the DSSF, DSS is conducted by researchers who are ‘embedded’ in delivery systems and respond to the decision-making needs of those systems.”
Additionally, closer connections with DSS leaders have led to a better understanding of challenges, opportunities, and needs of delivery systems.10,11
2. Cultivating a network of delivery systems and system leaders interested in expanding the cadre of embedded researchers, and trainees who intend to build careers in DSS. The DSSF also aims to enhance fellows’ skills and knowledge base, career opportunities, and professional network. To extend these relationships and support delivery system analytics, AcademyHealth worked with preceptors and fellows to inform creation of a new Community of Practice supported by AcademyHealth’s EDM Forum and guide planning for AcademyHealth’s Concordium conference to provide a national meeting to showcase DSS.
3. Strategic planning to ensure sustainable support for embedded research within delivery systems. Substantial interest in the program developed quickly, with rapid learning over the first few cycles to refine the program to meet host sites’ and fellows’ needs. Both efforts were critical to demonstrate that the DSSF fulfills an important need for our health system partners and members. As indicated previously, strong, sustained interest from prospective host sites and applicants demonstrates the program has created a win-win to jointly assess fit while building skills and supporting continuous learning.
Likewise, Lisa Simpson, President and CEO of AcademyHealth, and Lucy Savitz, DSSF host site preceptor at Intermountain Healthcare, participated in the Canadian Institutes of Health Research (CIHR) Invitational Workshop, “Modernizing Health Services and Policy Research Training in Canada” in March 2016. Shared learning largely informed by the DSSF led to CIHR creating a similar fellowship program with initial awards to be made in 2017 (see https://www.researchnet-recherchenet.ca/rnr16/vwOpprtntyDtls.do?prog=2540&view=browseActive&sponsor=CIHR-8&type=EXACT&resultCount=25 for more information). We are working to thread these efforts together in a way to leverage our learning community of government agencies, academia, and employers as a long-term funding stream for training in delivery science.
Participant Host Site Level
Two selected examples of how DSSF researchers have engaged high-priority topics that contributed to health system operations are provided here.
Kaiser Permanente Southern California: Assessing adherence with “Choosing Wisely” recommendations in oncology. In partnership with preceptor Dr. Michael Gould, 2013-2014 DSSF fellow Dr. Erin Hahn worked with the Kaiser Permanente Southern California (KPSC) Care Improvement Research Team to lead a project addressing several KPSC priority areas. Focusing on “Choosing Wisely” recommendations from the American Society of Clinical Oncology,12 the project evaluated appropriateness of imaging and laboratory services for early-stage cancer patients and survivors between and within 2 integrated health systems, Kaiser Permanente (KP) and Intermountain Healthcare.13,14 Results were presented to KP national leaders, including an external health policy advisory board. In close collaboration with clinical and operational leaders in medical oncology, this multiregional, multisystem project is contributing to targeted quality improvement efforts and improved healthcare value, including audit and feedback of nonrecommended labs.
Dr. Hahn subsequently received a KPSC Incubator Award, a competitive internal grant, to further study factors associated with use of nonrecommended surveillance lab tests for early-stage breast cancer patients. The study focused on medical oncologists within KPSC, categorizing them as high or low utilizers of the tests.15 Results indicate that high utilizers perceive that the tests help manage patient anxiety about recurrence, while acknowledging that the tests do not provide clinical utility. These findings are contributing to the development of targeted survivorship services across the organization.
Intermountain Healthcare: Formative evaluation of large-scale implementation of shared decision-making. Preceptor Dr. Lucy Savitz assembled a team to conduct a formative evaluation of Intermountain Healthcare’s efforts to implement shared decision-making (SDM) as part of its Center for Medicare and Medicaid Innovation Challenge Award. The 2015-2017 DSSF fellow at Intermountain Healthcare, Dr. Kim Brunisholz, served as a core member of the project team, focusing primarily on a mixed-methods evaluation of the SDM program.
Dr. Brunisholz engaged operational leads, clinical teams, patient and family advisory councils, and senior executives to conduct the program assessment. Results demonstrated significant variation in invited participation in SDM among eligible patient populations: preference sensitive conditions (1 in 30 patients), oncology-related diagnosis (1 in 3 patients), and chronic conditions (1 in 74 patients). Provisional analysis of patient-level clinical outcomes demonstrated that among those invited to the SDM program compared to those that were not, total joint replacement was decreased (10.1% vs 17.3%; P < 0.001) and a trend towards breast conservation emerged (61.8% vs 56.4%; P = 0.10). No difference in treatment choice for lower back pain was observed. Qualitative program analysis suggested need for improvements in the areas of (1) routine and continuous staff training, (2) workflow standardization, and (3) active data monitoring with meaningful, actionable feedback to caregivers. In response to these results, a chartered SDM Steering Committee was created (Dr. Brunisholz is a member of that group) to develop a strategic plan for SDM, with an accompanying organizational response to reimplement SDM in a targeted manner. Learning from this program is being leveraged to support a subproject analysis on a large scale using data from the High Value Healthcare Collaborative as part of an AHRQ-funded Center of Excellence award. (See https://www.ahrq.gov/news/newsroom/press-releases/2015/pcorawards.html. For more information on the High Value Healthcare Collaborative, please visit: https://www.highvaluehealthcare.org.)
CONCLUSION
Moving forward, the DSSF will continue working with progressive delivery systems. Partnerships between organizations that are interested in integrating rigorous research practice to drive continuous system improvement and maximize the value of care will have substantial need for technical skills and analytic capacity. They will also need to ensure that researchers working in their systems have sufficient understanding of cultural and political context within the organization to be effective leaders who can manage change.
AcademyHealth created the DSSF in response to the field’s request to build a research workforce that reflects the vision for a 21st Century Health System, as laid out by the NAM.16 We anticipate that as the US Department of Health & Human Services’ goals for payment reform and new measures to promote quality and high-value care are implemented, the DSSF trainees’ skill set will be increasingly valuable and will provide needed thought leadership on strategies to generate and apply evidence in practice.
Disclosure
Ms. Kanani received funding from Intermountain Healthcare, Kaiser Permanente Southern California for support for the Delivery System Science Fellowship. Drs. Hahn, Gould, and Brunisholz have no conflicts to disclose. Dr. Savitz has received funding from HRQ COE, PCORI LHSNet; received funding for lectures from the Institute from Healthcare Improvement, Department of Epidemiology, University of Utah; received funding for travel, accommodations, and meeting expenses from AHRQ NAC, EDM Forum, AH CAPP, AARP NPC, and PROM TEP; and received additional funding from Dartmouth University. Dr. Holve received funding from Intermountain Healthcare, Kaiser Permanente Southern California for support for the Delivery System Science Fellowship, provided by our delivery system partners, several of whom are coauthors on this manuscript.
1. AcademyHealth. Health Services Research in 2020: Summit on the Future of HSR Data and Methods. http://www.academyhealth.org/About/content.cfm?ItemNumber=2529. Accessed March 21, 2016.
2. AcademyHealth. Health Services Research (HSR) Learning Consortium Strategic Plan. http://www.academyhealth.org/files/ProfDev/Files/HSRstrategicplan2011FINAL.pdf. Accessed March 21, 2016.
3. Institute of Medicine. Initial National Priorities for Comparative Effectiveness Research. http://www.nationalacademies.org/hmd/~/media/Files/Report%20Files/2009/ComparativeEffectivenessResearchPriorities/CER%20report%20brief%2008-13-09.pdf. Accessed March 21, 2016.
4. Bonham A, Rich E, Davis D, Longnecker D, Heinig S. Putting evidence to work: an expanded research agenda for academic medicine in the era of health care reform. Acad Med. 2010;85(10):1551-1553. PubMed
5. Zerhouni E. Translational and clinical science—time for a new vision. N Engl J Med. 2005;353(15):1621-1623. PubMed
6. AcademyHealth. “Getting Answers We Can Believe In: Methodological Considerations When Using Electronic Clinical Data for Research,” EDM Forum, December 2012.
7. Rich G, Collins A. Current and Future Demand for Health Services Researchers. Funded by the Agency for Healthcare Research and Quality (AHRQ). Presented at the AcademyHealth HSR Workforce Conference, Understanding the Current Health Services Research Workforce and Maximizing Its Future, funded by AHRQ, Patient-Centered Outcomes Research Institute, and the Robert Wood Johnson Foundation. October 2016.
8. AcademyHealth. Delivery System Science Fellowship. http://www.academyhealth.org/dssf. Accessed March 21, 2016.
9. Institute for Healthcare Improvement. IHI Triple Aim Initiative. http://www.ihi.org/engage/initiatives/tripleaim/Pages/default.aspx. Accessed March 21, 2016.
10. Psek W, Stametz R, Bailey-Davis L, et al. Operationalizing the learning health care system in an integrated delivery system. eGEMs. 2015;3(1):1122. PubMed
11. Thompson C, Kurian A, Luft H. Linking electronic health records to better understand breast cancer patient pathways within and between two health systems. eGEMs. 2015;3(1):1127. PubMed
12. Schnipper L, Smith TJ, Raghavan D, et al. American Society of Clinical Oncology identifies five key opportunities to improve care and reduce costs: The top five list for oncology. J Clin Oncol. 2012;30(14):1715-1724. PubMed
13. Hahn E, Tang T, Lee JS, et al. Use of posttreatment imaging and biomarkers in survivors of early‐stage breast cancer: Inappropriate surveillance or necessary care? Cancer. 2015;122(6):908-916. PubMed
14. Hahn E, Tang T, Lee JS, et al. Use of imaging for staging of early-stage breast cancer in two integrated health care systems: Adherence with a choosing wisely recommendation. J Oncol Pract. 2015;11(3):e320-e328. PubMed
15. Hahn EE, Munoz-Plaza C, Wang J, et al. Anxiety, culture, expectations: Oncologist-perceived factors associated with use of non-recommended serum tumor marker tests for surveillance of early stage breast cancer. J Oncol Pract. 2016;13(1):e77-e290. PubMed
16. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy of Sciences; 2000.
1. AcademyHealth. Health Services Research in 2020: Summit on the Future of HSR Data and Methods. http://www.academyhealth.org/About/content.cfm?ItemNumber=2529. Accessed March 21, 2016.
2. AcademyHealth. Health Services Research (HSR) Learning Consortium Strategic Plan. http://www.academyhealth.org/files/ProfDev/Files/HSRstrategicplan2011FINAL.pdf. Accessed March 21, 2016.
3. Institute of Medicine. Initial National Priorities for Comparative Effectiveness Research. http://www.nationalacademies.org/hmd/~/media/Files/Report%20Files/2009/ComparativeEffectivenessResearchPriorities/CER%20report%20brief%2008-13-09.pdf. Accessed March 21, 2016.
4. Bonham A, Rich E, Davis D, Longnecker D, Heinig S. Putting evidence to work: an expanded research agenda for academic medicine in the era of health care reform. Acad Med. 2010;85(10):1551-1553. PubMed
5. Zerhouni E. Translational and clinical science—time for a new vision. N Engl J Med. 2005;353(15):1621-1623. PubMed
6. AcademyHealth. “Getting Answers We Can Believe In: Methodological Considerations When Using Electronic Clinical Data for Research,” EDM Forum, December 2012.
7. Rich G, Collins A. Current and Future Demand for Health Services Researchers. Funded by the Agency for Healthcare Research and Quality (AHRQ). Presented at the AcademyHealth HSR Workforce Conference, Understanding the Current Health Services Research Workforce and Maximizing Its Future, funded by AHRQ, Patient-Centered Outcomes Research Institute, and the Robert Wood Johnson Foundation. October 2016.
8. AcademyHealth. Delivery System Science Fellowship. http://www.academyhealth.org/dssf. Accessed March 21, 2016.
9. Institute for Healthcare Improvement. IHI Triple Aim Initiative. http://www.ihi.org/engage/initiatives/tripleaim/Pages/default.aspx. Accessed March 21, 2016.
10. Psek W, Stametz R, Bailey-Davis L, et al. Operationalizing the learning health care system in an integrated delivery system. eGEMs. 2015;3(1):1122. PubMed
11. Thompson C, Kurian A, Luft H. Linking electronic health records to better understand breast cancer patient pathways within and between two health systems. eGEMs. 2015;3(1):1127. PubMed
12. Schnipper L, Smith TJ, Raghavan D, et al. American Society of Clinical Oncology identifies five key opportunities to improve care and reduce costs: The top five list for oncology. J Clin Oncol. 2012;30(14):1715-1724. PubMed
13. Hahn E, Tang T, Lee JS, et al. Use of posttreatment imaging and biomarkers in survivors of early‐stage breast cancer: Inappropriate surveillance or necessary care? Cancer. 2015;122(6):908-916. PubMed
14. Hahn E, Tang T, Lee JS, et al. Use of imaging for staging of early-stage breast cancer in two integrated health care systems: Adherence with a choosing wisely recommendation. J Oncol Pract. 2015;11(3):e320-e328. PubMed
15. Hahn EE, Munoz-Plaza C, Wang J, et al. Anxiety, culture, expectations: Oncologist-perceived factors associated with use of non-recommended serum tumor marker tests for surveillance of early stage breast cancer. J Oncol Pract. 2016;13(1):e77-e290. PubMed
16. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy of Sciences; 2000.
© 2017 Society of Hospital Medicine