User login
The VA is charged with providing high-quality, efficient care for those who have served in the armed forces. Starting in 2009, leadership within the VA articulated a set of “transformational” initiatives, which included greater implementation of telemedicine to improve the coordination of veterans’ care and access to specialty care.
The VA Boston Healthcare System (VABHS) consists of 3 major facilities and multiple community-based outpatient clinics (CBOCs) across eastern Massachusetts. Most medical specialty care is concentrated near the single acute-care site, resulting in a diffuse network of primary and specialty care providers. Consequently, many patients travel beyond their primary medical facility to access specialists. To address this fragmentation, the Medical Service at VABHS implemented a system of electronic chart review consultations (e-Consults) to provide high-quality, efficient care while maintaining the integrity of the patient’s medical home and improving access to specialty providers.
Designed to address clinical questions that may not require a face-to-face (FTF) encounter, e-Consults are a form of store-and-forward telemedicine (SAFT), with information-sharing between requesting providers (typically primary care) and answering specialists, using the VA’s existing electronic health record (EHR), the Computerized Patient Record System (CPRS). Store-and-forward telemedicine increases patient access to specialists and timeliness in initiating a consultation while decreasing patient travel and associated costs.1-15
Although previous studies have usually found high concordance between SAFT and FTF consultations in both diagnosis and disease management, the feasibility of SAFT has been historically limited by the lack of an integrated shared EHR.16-18 The VA implemented CPRS in 1996 and was soon recognized as a fully integrated EHR, allowing any VA clinician to view any veteran’s medical record nationwide. This technology, combined with the geographically dispersed population of veterans and the concentration of specialty services within mostly urban tertiary care centers, makes the VA an optimal environment to benefit from implementation of SAFT.
This study examines the implementation of e-Consults in a large VA medical service with the primary goal of facilitating timely specialty care access. This article details the initial experience of implementation and clinicians’ early perceptions of the system.
E-Consult Implementation
The VABHS provides tertiary specialty care for much of New England. It also serves a primary care population of more than 30,000 veterans in eastern Massachusetts through a network of 3 larger primary care practices based at 3 main campuses and 5 smaller CBOCs. Most specialty care is provided by the 2 campuses in Boston.
Discussions with primary care leadership and e-mail announcements to primary care providers (PCPs) throughout VABHS announced the availability and purpose of the new e-Consult. The initiative was initially piloted within 2 specialties, allergy and cardiology, beginning in January 2011. After establishing the logistics for routing, processing, and encounter data capture, e-Consults were expanded to the remaining subspecialties by December 2011.
Within 9 months, 12 different specialties had e-Consult options: endocrinology, gastroenterology, hematology, infectious disease, nephrology, oncology, palliative care, pulmonary disease, rheumatology, and sleep medicine. The e-Consult use was tracked for this study on a monthly basis.
Process and Work Flow
Any PCP with consult ordering privileges can request an e-Consult. Expectations were set that e-Consults should be used to request help with nonemergent clinical issues and be completed within 3 nonholiday weekdays.
The e-Consult uses CPRS modular programming to create an event-based electronic consultation request. Once initiated, the consult is routed to the consulting service. Each specialty defines how to triage the e-Consult request; computer alerts can be immediately generated to a single individual or a defined team within each specialty. Options to alert or print consult requests to administrative personnel also exist. As with requests for FTF consultation requests, e-Consults allow specialists the options to deny a request if they feel it is not appropriate for their specialty. In addition, specialists are able to convert the e-Consult request to a FTF encounter, to forward the e-Consult to another specialty, or to add a “comment” before proceeding. These comments generate an immediate electronic alert to the requestor, thus providing a forum for asynchronous preconsultation dialogue between providers. Once the consultation is complete, the requestor receives an alert that directs them to view the completed consultation note in the EHR.
The entirety of the EHR is accessible to the consultant, including all primary care and specialty progress notes, inpatient and outpatient laboratory testing, and all diagnostic testing performed nationally within the VA.
User Feedback
Based on the literature, separate online questionnaires were developed to ascertain primary care clinicians’ and specialists’ perceptions of the e-Consult system. This 26-item survey for PCPs and 13-item survey for specialists assessed use of e-Consults, satisfaction with the system, and for primary care clinicians, perceived benefits of the e-Consult process
and specific feedback by specialty.
Nine months after full implementation, perceptions of e-Consult processes were solicited. All PCPs, staff physicians, nurse practitioners, and internal medicine residents assigned to a VABHS primary care clinic were surveyed. Eligible specialist consultants included attending physicians and fellows who were identified by the specialty chief as potentially having responded to an e-Consult in the past 6 months. The survey was administered in June 2012 and queried the previous 6 months’ experience.
The monthly number of e-Consults ranged from 96 to 137 over the 6-month study period. Cardiology and hematology were the most commonly requested e-Consults, with > 20 per specialty per month. The 146 cardiology e-Consults during those 6 months represented 20% of all new cardiology consults, whereas the 159 e-Consults to hematology represented 53% of their new consults. The least used specialties for e-Consults were allergy, oncology, palliative care, and rheumatology, each averaging < 2 e-Consults per month.
Provider Perceptions
Overall, 85 of the 163 providers completed the survey (52%). Permanent staff had a better response rate than did trainees: 58% vs 33% among PCPs (P < .02), and 71% vs 44% among specialists (P = .03).
Of the 47 primary care respondents, 4 residents and 2 staff physicians had not previously used e-Consults; all 4 residents and 1 staff physician stated that this was due to lack of awareness of the system. The remaining 41 PCPs had requested ≥ 1 e-Consult in the past 6 months. All 38 specialist provider respondents had answered ≥ 1 e-Consult in study period, with 9 providers completing > 20.
e-Consult Satisfaction
Overall, 93% of PCPs were satisfied with e-Consults, and none were dissatisfied, compared with 53% of specialty providers who were satisfied and 26% who were dissatisfied (P < .001) (Figure 1). All specialists who were dissatisfied were in the sections of cardiology or gastroenterology. Further inspection identified several differences between those specialty providers who were generally satisfied with e-Consults (n = 20) compared with those providers who were dissatisfied (n = 10). Eight PCPs reported being neither satisfied nor dissatisfied.
In analyses comparing satisfied specialists (n = 20) with dissatisfied specialists (n = 10), excluding the 8 respondents who were neither satisfied nor dissatisfied, those who were satisfied reported a lower rate of conversion of e-Consults to FTF visits than did their dissatisfied peers, 26% vs 52%, respectively (P = .01). Satisfied specialists were more likely to view e-Consults as replacing work they otherwise would have done, such as a replacement for a FTF consult and prior informal “curbside” consults, 79% vs 50%, respectively (P < .02). Satisfied specialists were somewhat more likely to complete their e-Consults, on average, in ≤ 15 minutes, compared with dissatisfied specialists, although this trend did not reach statistical significance (59% vs 49%, respectively; (P = .07) (Figure 2).
Improving Quality of Care
PCPs were asked to note which benefits were realized among patients for whom e-Consults replaced FTF visits. Nearly all (98%) indicated that the e-Consults enabled patients to avoid unnecessary travel, and 95% of PCPs indicated that they requested e-consults to receive faster clinical input about a patient. A total of 59% believed e-Consults helped their patients avoid additional copays, and 56% obtained specialty input for a patient who would otherwise refuse to travel. When an e-Consult did not avoid a FTF visit (ie, despite the e-Consult, the patient ultimately proceeded to a FTF consultation with the specialist), the majority of PCPs believed that the e-Consult provided them with reassurance and initiated diagnostic testing that would be useful during the patient’s subsequent FTF consult (Figure 3).
Overall, 61% of specialists agreed with the statement, “e-Consults improve the quality of care VA Boston Healthcare System provides.” Even when specialists perceived that e-Consults did not avoid FTF visits, most agreed that e-Consults helped initiate diagnostic testing prior to a FTF visit. In addition, satisfied specialists saw the benefit in reassuring the PCP and initiating additional management prior to a FTF visit.
Dynamics of e-Consults
Primary care providers reported that 78% of their requested e-Consults were completed within 2 days and 95% of all e-Consults were completed by day 3. In aggregate, primary care clinicians estimated that about one-third of their e-Consults replaced FTF visits; one-third replaced prior mechanisms for informal consultation (eg, “curbside” or e-mail); and one-third of e-Consults represented new requests that would not have involved specialty consultation in the absence of the e-Consult mechanism.
Specialists estimated that 27% of e-Consults were new work (ie, consultations that would not have occurred formally or informally in the absence of the e-Consult mechanism); 32% replaced FTF visits; and 42% were substitutes for prior informal communications. Specialists reported a wide range of time required to answer e-Consults. For specialists, on average, 54% of e-Consults took < 15 minutes to complete, but 20% took > 25 minutes, with 6% requiring ≥ 45 minutes to complete (Figure 2).
Discussion
This article reports the initial experience with the implementation of an e-Consult system for PCPs and medical subspecialists in a large VA health care system. Primary care providers were generally satisfied with the e-Consult system and reported that the system yielded tangible benefits to patients, such as quicker specialty input and avoidance of FTF visits and travel. Specialists were somewhat less satisfied than were their primary care colleagues; nevertheless, specialists perceived similar benefits for patients. Despite the general satisfaction among PCPs, > 1 in 4 specialists expressed dissatisfaction with the system.
Satisfaction with e-Consults may be influenced by the typology of the specialty itself. Some specialties (eg, hematology) rely more heavily on laboratory tests, compared with specialties such as cardiology, gastroenterology, or pulmonary whose subtleties of history and physical examination (H&P), along with review of imaging data, are more commonly required for clinical decision making.2 Hence, the EHR may allow some specialties to provide e-Consults with greater facility. For example, because the EHR enables clinicians to trend the results of years of complete blood counts with a few simple keystrokes, a hematologist may have more confidence in making a clinical assessment without seeing the patient based on historical information in the EHR.
Anecdotal evidence from specialists suggests discomfort when clinical input is based on H&P findings of others. While the sample size was small, this reasoning could explain the differences in satisfaction among specialties: Satisfied specialists may be taking slightly less time to complete their e-Consults, but they were less likely to view the e-Consult as new work and less likely to convert to a FTF consultation.
Store-and-forward telemedicine cannot completely replace FTF visits, but rather supplement them. If e-Consults obviate patient travel and copays while stimulating more timely completion of the consultation, the benefits of cost savings and improved specialty access by veterans merit further attention. When an e-Consult did not avoid a FTF consultation, the majority of clinicians perceived that the e-Consult allowed the PCP to initiate diagnostic testing or alterations in treatment prior to the eventual FTF consultation with the specialist. This finding could be considered as a proxy for increased coordination of care between primary care and specialty providers in anticipation of a FTF visit.
During the study period, workload capture for e-Consults was administratively fixed at the level of a brief consultation, regardless of the effort expended. Validation of the current finding regarding time spent completing consults has resulted in a change. On January 10, 2014, VA Central Office updated its policy to allow 3 levels of workload credit for e-Consults based on time: up to 15 minutes, 16 to 30 minutes, and > 30 minutes.
E-consults are just one example of innovations that broadly fall under the rubric of telehealth initiatives being deployed across VA health care facilities. Other examples of SAFT include pictures collected by technicians and sent to specialists for review to screen for diabetic retinopathy, a practice done within VA for many years.19 Clinical video telehealth provides real-time videoconferencing between patient and specialist and can obviate the need for long-distance patient travel. Initially used only for interviews, newer equipment amplifies sound and enhances optical imaging to allow thorough physical examinations when trained personnel are handling the equipment.
Each technology has its challenges for implementation, which vary in degree of coordination and cost. Overall, the application of new technologies and repurposing existing ones may be limited by creativity alone in the efforts to improve access and quality of the care provided to veterans. The Table highlights telehealth uses being deployed by VA.
Conclusion
Electronic consults have been well received by PCPs and most specialists in the VABHS, seeming to meet the goal of using telehealth to improve veterans’ access to specialty care and coordination of care between PCPs and specialists. While not examined in this initial report, e-Consults may lead to reduced costs, and this possibility should be further explored.
Despite concerns expressed by some specialists, most believed that e-Consults improved the quality of care their patients received. Future work needs to validate these findings, examine patient perspectives, delve more deeply into ways to improve and bring more value to the process, and address the specialty effort in relation to workload awarded.
Acknowledgements
The authors would like to thank Yisraela Elstein and Pauline Benedetti for their assistance in tracking workload data and Steven Simon for his detailed review of the manuscript. Funding for this work was supported by the Department of Veterans Affairs Specialty Care Transformation Initiative for the 21st Century—T21 innovation grant for telehealth initiatives.
Author disclosures
The authors report no actual or potential conflicts of interest with regard to this article.
Disclaimer
The opinions expressed herein are those of the authors and do not necessarily reflect those of Federal Practitioner, Frontline Medical Communications Inc., the U.S. Government, or any of its agencies. This article may discuss unlabeled or investigational use of certain drugs. Please review complete prescribing information for specific drugs or drug combinations—including indications, contraindications, warnings, and adverse effects—before administering pharmacologic therapy to patients.
1. Cusack CM, Pan E, Hook JM, et al. The Value of Provider-to-Provider Telehealth Technologies. Charlestown, MA: Center for Information Technology Leadership; 2007.
2. Horner K, Wagner E, Tufano J. Electronic consultations between primary and specialty care clinicians: Early insights. Issue Brief (Commonw Fund). 2011;23:1-14.
3. Angstman KB, Rohrer JE, Adamson SC, Chaudhry R. Impact of e-consults on return visits of primary care patients. Health Care Manag (Frederick). 2009;28(3):253-257.
4. Kim Y, Chen AH, Keith E, Yee HF Jr, Kushel MB. Not perfect, but better: Primary care providers’ experiences with electronic referrals in a safety net health system. J Gen Intern Med. 2009;24(5):614-619.
5. Angstman KB, Adamson SC, Furst JW, Houston MS, Rohrer JE. Provider satisfaction with virtual specialist consultations in a family medicine department. Health Care Manag (Frederick). 2009;28(1):14-18.
6. Mahnke CB, Jordan CP, Bergvall E, Person DA, Pinsker JE. The Pacific Asynchronous TeleHealth (PATH) system: Review of 1,000 pediatric teleconsultations. Telemed J E Health. 2011;17(1):35-39.
7. Sood S, Mbarika V, Jugoo S, et al. What is telemedicine? A collection of 104 peer-reviewed perspectives and theoretical underpinnings. Telemed J E Health. 2007;13(5):573-590.
8. Maeder A. Telehealth and remote access. Stud Health Technol Inform. 2010;151:239-254.
9. Schmidt T, Lappan CM, Hospenthal DR, Murray CK. Deployed provider satisfaction with infectious disease teleconsulation. Mil Med. 2011;176(12):1417-1420.
10. Verhoeven F, Tanja-Dijkstra K, Nijland N, Eysenbach G, van Gemert-Pijnen L. Asynchronous and synchronous teleconsultation for diabetes care: A systematic literature review. J Diabetes Sci Technol. 2010;4(3):666-684.
11. Krupinski E, Nypaver M, Poropatich R, Ellis D, Safwat R, Sapci H. Telemedicine/telehealth: An international perspective. Clinical applications in telemedicine/telehealth. Telemed J E Health. 2002;8(1):13-34.
12. Moreno-Ramirez D, Ferrandiz L, Ruiz-de-Casas A, et al. Economic evaluation of a store-and-forward teledermatology system for skin cancer patients. J Telemed Telecare. 2009;15(1):40-45.
13. Callahan CW, Malone F, Estroff D, Person DA. Effectiveness of an Internet-based store-and-forward telemedicine system for pediatric subspecialty consultation. Arch Pediatr Adolesc Med. 2005;159(4):389-393.
14. Gagnon MP, Duplantie J, Fortin JP, Landry R. Implementing telehealth to support medical practice in rural/remote regions: What are the conditions for success? Implement Sci. 2006;1:18.
15. Hailey D, Ohinmaa A, Roine R. Study quality and evidence of benefit in recent assessments of telemedicine. J Telemed Telecare. 2004;10(6):318-324.
16. High WA, Houston MS, Calobrisi SD, Drage LA, McEvoy MT. Assessment of the accuracy of low-cost store-and-forward teledermatology consultation. J Am Acad Dermatol. 2000;42(5 pt 1):776-783.
17. Houston MS, Myers JD, Levens SP, et al. Clinical consultations using store-and-forward telemedicine technology. Mayo Clin Proc. 1999;74(8):764-769.
18. Hersh WR, Hickam DH, Severance SM, Dana TL, Pyle Krages K, Helfand M. Diagnosis, access and outcomes: Update of a systematic review of telemedicine services. J Telemed Telecare. 2006;12(suppl 2):S3-S31.
19. Cavallerano AA, Conlin PR. Teleretinal imaging to screen for diabetic retinopathy in the Veterans Health Administration. J Diabetes Sci Technol. 2008;2(1):33-39.
20. Arora S, Thornton K, Murata G, et al. Outcomes of treatment for hepatitis C virus infection by primary care providers. N Engl J Med. 2011;364(23):2199-2207.
The VA is charged with providing high-quality, efficient care for those who have served in the armed forces. Starting in 2009, leadership within the VA articulated a set of “transformational” initiatives, which included greater implementation of telemedicine to improve the coordination of veterans’ care and access to specialty care.
The VA Boston Healthcare System (VABHS) consists of 3 major facilities and multiple community-based outpatient clinics (CBOCs) across eastern Massachusetts. Most medical specialty care is concentrated near the single acute-care site, resulting in a diffuse network of primary and specialty care providers. Consequently, many patients travel beyond their primary medical facility to access specialists. To address this fragmentation, the Medical Service at VABHS implemented a system of electronic chart review consultations (e-Consults) to provide high-quality, efficient care while maintaining the integrity of the patient’s medical home and improving access to specialty providers.
Designed to address clinical questions that may not require a face-to-face (FTF) encounter, e-Consults are a form of store-and-forward telemedicine (SAFT), with information-sharing between requesting providers (typically primary care) and answering specialists, using the VA’s existing electronic health record (EHR), the Computerized Patient Record System (CPRS). Store-and-forward telemedicine increases patient access to specialists and timeliness in initiating a consultation while decreasing patient travel and associated costs.1-15
Although previous studies have usually found high concordance between SAFT and FTF consultations in both diagnosis and disease management, the feasibility of SAFT has been historically limited by the lack of an integrated shared EHR.16-18 The VA implemented CPRS in 1996 and was soon recognized as a fully integrated EHR, allowing any VA clinician to view any veteran’s medical record nationwide. This technology, combined with the geographically dispersed population of veterans and the concentration of specialty services within mostly urban tertiary care centers, makes the VA an optimal environment to benefit from implementation of SAFT.
This study examines the implementation of e-Consults in a large VA medical service with the primary goal of facilitating timely specialty care access. This article details the initial experience of implementation and clinicians’ early perceptions of the system.
E-Consult Implementation
The VABHS provides tertiary specialty care for much of New England. It also serves a primary care population of more than 30,000 veterans in eastern Massachusetts through a network of 3 larger primary care practices based at 3 main campuses and 5 smaller CBOCs. Most specialty care is provided by the 2 campuses in Boston.
Discussions with primary care leadership and e-mail announcements to primary care providers (PCPs) throughout VABHS announced the availability and purpose of the new e-Consult. The initiative was initially piloted within 2 specialties, allergy and cardiology, beginning in January 2011. After establishing the logistics for routing, processing, and encounter data capture, e-Consults were expanded to the remaining subspecialties by December 2011.
Within 9 months, 12 different specialties had e-Consult options: endocrinology, gastroenterology, hematology, infectious disease, nephrology, oncology, palliative care, pulmonary disease, rheumatology, and sleep medicine. The e-Consult use was tracked for this study on a monthly basis.
Process and Work Flow
Any PCP with consult ordering privileges can request an e-Consult. Expectations were set that e-Consults should be used to request help with nonemergent clinical issues and be completed within 3 nonholiday weekdays.
The e-Consult uses CPRS modular programming to create an event-based electronic consultation request. Once initiated, the consult is routed to the consulting service. Each specialty defines how to triage the e-Consult request; computer alerts can be immediately generated to a single individual or a defined team within each specialty. Options to alert or print consult requests to administrative personnel also exist. As with requests for FTF consultation requests, e-Consults allow specialists the options to deny a request if they feel it is not appropriate for their specialty. In addition, specialists are able to convert the e-Consult request to a FTF encounter, to forward the e-Consult to another specialty, or to add a “comment” before proceeding. These comments generate an immediate electronic alert to the requestor, thus providing a forum for asynchronous preconsultation dialogue between providers. Once the consultation is complete, the requestor receives an alert that directs them to view the completed consultation note in the EHR.
The entirety of the EHR is accessible to the consultant, including all primary care and specialty progress notes, inpatient and outpatient laboratory testing, and all diagnostic testing performed nationally within the VA.
User Feedback
Based on the literature, separate online questionnaires were developed to ascertain primary care clinicians’ and specialists’ perceptions of the e-Consult system. This 26-item survey for PCPs and 13-item survey for specialists assessed use of e-Consults, satisfaction with the system, and for primary care clinicians, perceived benefits of the e-Consult process
and specific feedback by specialty.
Nine months after full implementation, perceptions of e-Consult processes were solicited. All PCPs, staff physicians, nurse practitioners, and internal medicine residents assigned to a VABHS primary care clinic were surveyed. Eligible specialist consultants included attending physicians and fellows who were identified by the specialty chief as potentially having responded to an e-Consult in the past 6 months. The survey was administered in June 2012 and queried the previous 6 months’ experience.
The monthly number of e-Consults ranged from 96 to 137 over the 6-month study period. Cardiology and hematology were the most commonly requested e-Consults, with > 20 per specialty per month. The 146 cardiology e-Consults during those 6 months represented 20% of all new cardiology consults, whereas the 159 e-Consults to hematology represented 53% of their new consults. The least used specialties for e-Consults were allergy, oncology, palliative care, and rheumatology, each averaging < 2 e-Consults per month.
Provider Perceptions
Overall, 85 of the 163 providers completed the survey (52%). Permanent staff had a better response rate than did trainees: 58% vs 33% among PCPs (P < .02), and 71% vs 44% among specialists (P = .03).
Of the 47 primary care respondents, 4 residents and 2 staff physicians had not previously used e-Consults; all 4 residents and 1 staff physician stated that this was due to lack of awareness of the system. The remaining 41 PCPs had requested ≥ 1 e-Consult in the past 6 months. All 38 specialist provider respondents had answered ≥ 1 e-Consult in study period, with 9 providers completing > 20.
e-Consult Satisfaction
Overall, 93% of PCPs were satisfied with e-Consults, and none were dissatisfied, compared with 53% of specialty providers who were satisfied and 26% who were dissatisfied (P < .001) (Figure 1). All specialists who were dissatisfied were in the sections of cardiology or gastroenterology. Further inspection identified several differences between those specialty providers who were generally satisfied with e-Consults (n = 20) compared with those providers who were dissatisfied (n = 10). Eight PCPs reported being neither satisfied nor dissatisfied.
In analyses comparing satisfied specialists (n = 20) with dissatisfied specialists (n = 10), excluding the 8 respondents who were neither satisfied nor dissatisfied, those who were satisfied reported a lower rate of conversion of e-Consults to FTF visits than did their dissatisfied peers, 26% vs 52%, respectively (P = .01). Satisfied specialists were more likely to view e-Consults as replacing work they otherwise would have done, such as a replacement for a FTF consult and prior informal “curbside” consults, 79% vs 50%, respectively (P < .02). Satisfied specialists were somewhat more likely to complete their e-Consults, on average, in ≤ 15 minutes, compared with dissatisfied specialists, although this trend did not reach statistical significance (59% vs 49%, respectively; (P = .07) (Figure 2).
Improving Quality of Care
PCPs were asked to note which benefits were realized among patients for whom e-Consults replaced FTF visits. Nearly all (98%) indicated that the e-Consults enabled patients to avoid unnecessary travel, and 95% of PCPs indicated that they requested e-consults to receive faster clinical input about a patient. A total of 59% believed e-Consults helped their patients avoid additional copays, and 56% obtained specialty input for a patient who would otherwise refuse to travel. When an e-Consult did not avoid a FTF visit (ie, despite the e-Consult, the patient ultimately proceeded to a FTF consultation with the specialist), the majority of PCPs believed that the e-Consult provided them with reassurance and initiated diagnostic testing that would be useful during the patient’s subsequent FTF consult (Figure 3).
Overall, 61% of specialists agreed with the statement, “e-Consults improve the quality of care VA Boston Healthcare System provides.” Even when specialists perceived that e-Consults did not avoid FTF visits, most agreed that e-Consults helped initiate diagnostic testing prior to a FTF visit. In addition, satisfied specialists saw the benefit in reassuring the PCP and initiating additional management prior to a FTF visit.
Dynamics of e-Consults
Primary care providers reported that 78% of their requested e-Consults were completed within 2 days and 95% of all e-Consults were completed by day 3. In aggregate, primary care clinicians estimated that about one-third of their e-Consults replaced FTF visits; one-third replaced prior mechanisms for informal consultation (eg, “curbside” or e-mail); and one-third of e-Consults represented new requests that would not have involved specialty consultation in the absence of the e-Consult mechanism.
Specialists estimated that 27% of e-Consults were new work (ie, consultations that would not have occurred formally or informally in the absence of the e-Consult mechanism); 32% replaced FTF visits; and 42% were substitutes for prior informal communications. Specialists reported a wide range of time required to answer e-Consults. For specialists, on average, 54% of e-Consults took < 15 minutes to complete, but 20% took > 25 minutes, with 6% requiring ≥ 45 minutes to complete (Figure 2).
Discussion
This article reports the initial experience with the implementation of an e-Consult system for PCPs and medical subspecialists in a large VA health care system. Primary care providers were generally satisfied with the e-Consult system and reported that the system yielded tangible benefits to patients, such as quicker specialty input and avoidance of FTF visits and travel. Specialists were somewhat less satisfied than were their primary care colleagues; nevertheless, specialists perceived similar benefits for patients. Despite the general satisfaction among PCPs, > 1 in 4 specialists expressed dissatisfaction with the system.
Satisfaction with e-Consults may be influenced by the typology of the specialty itself. Some specialties (eg, hematology) rely more heavily on laboratory tests, compared with specialties such as cardiology, gastroenterology, or pulmonary whose subtleties of history and physical examination (H&P), along with review of imaging data, are more commonly required for clinical decision making.2 Hence, the EHR may allow some specialties to provide e-Consults with greater facility. For example, because the EHR enables clinicians to trend the results of years of complete blood counts with a few simple keystrokes, a hematologist may have more confidence in making a clinical assessment without seeing the patient based on historical information in the EHR.
Anecdotal evidence from specialists suggests discomfort when clinical input is based on H&P findings of others. While the sample size was small, this reasoning could explain the differences in satisfaction among specialties: Satisfied specialists may be taking slightly less time to complete their e-Consults, but they were less likely to view the e-Consult as new work and less likely to convert to a FTF consultation.
Store-and-forward telemedicine cannot completely replace FTF visits, but rather supplement them. If e-Consults obviate patient travel and copays while stimulating more timely completion of the consultation, the benefits of cost savings and improved specialty access by veterans merit further attention. When an e-Consult did not avoid a FTF consultation, the majority of clinicians perceived that the e-Consult allowed the PCP to initiate diagnostic testing or alterations in treatment prior to the eventual FTF consultation with the specialist. This finding could be considered as a proxy for increased coordination of care between primary care and specialty providers in anticipation of a FTF visit.
During the study period, workload capture for e-Consults was administratively fixed at the level of a brief consultation, regardless of the effort expended. Validation of the current finding regarding time spent completing consults has resulted in a change. On January 10, 2014, VA Central Office updated its policy to allow 3 levels of workload credit for e-Consults based on time: up to 15 minutes, 16 to 30 minutes, and > 30 minutes.
E-consults are just one example of innovations that broadly fall under the rubric of telehealth initiatives being deployed across VA health care facilities. Other examples of SAFT include pictures collected by technicians and sent to specialists for review to screen for diabetic retinopathy, a practice done within VA for many years.19 Clinical video telehealth provides real-time videoconferencing between patient and specialist and can obviate the need for long-distance patient travel. Initially used only for interviews, newer equipment amplifies sound and enhances optical imaging to allow thorough physical examinations when trained personnel are handling the equipment.
Each technology has its challenges for implementation, which vary in degree of coordination and cost. Overall, the application of new technologies and repurposing existing ones may be limited by creativity alone in the efforts to improve access and quality of the care provided to veterans. The Table highlights telehealth uses being deployed by VA.
Conclusion
Electronic consults have been well received by PCPs and most specialists in the VABHS, seeming to meet the goal of using telehealth to improve veterans’ access to specialty care and coordination of care between PCPs and specialists. While not examined in this initial report, e-Consults may lead to reduced costs, and this possibility should be further explored.
Despite concerns expressed by some specialists, most believed that e-Consults improved the quality of care their patients received. Future work needs to validate these findings, examine patient perspectives, delve more deeply into ways to improve and bring more value to the process, and address the specialty effort in relation to workload awarded.
Acknowledgements
The authors would like to thank Yisraela Elstein and Pauline Benedetti for their assistance in tracking workload data and Steven Simon for his detailed review of the manuscript. Funding for this work was supported by the Department of Veterans Affairs Specialty Care Transformation Initiative for the 21st Century—T21 innovation grant for telehealth initiatives.
Author disclosures
The authors report no actual or potential conflicts of interest with regard to this article.
Disclaimer
The opinions expressed herein are those of the authors and do not necessarily reflect those of Federal Practitioner, Frontline Medical Communications Inc., the U.S. Government, or any of its agencies. This article may discuss unlabeled or investigational use of certain drugs. Please review complete prescribing information for specific drugs or drug combinations—including indications, contraindications, warnings, and adverse effects—before administering pharmacologic therapy to patients.
The VA is charged with providing high-quality, efficient care for those who have served in the armed forces. Starting in 2009, leadership within the VA articulated a set of “transformational” initiatives, which included greater implementation of telemedicine to improve the coordination of veterans’ care and access to specialty care.
The VA Boston Healthcare System (VABHS) consists of 3 major facilities and multiple community-based outpatient clinics (CBOCs) across eastern Massachusetts. Most medical specialty care is concentrated near the single acute-care site, resulting in a diffuse network of primary and specialty care providers. Consequently, many patients travel beyond their primary medical facility to access specialists. To address this fragmentation, the Medical Service at VABHS implemented a system of electronic chart review consultations (e-Consults) to provide high-quality, efficient care while maintaining the integrity of the patient’s medical home and improving access to specialty providers.
Designed to address clinical questions that may not require a face-to-face (FTF) encounter, e-Consults are a form of store-and-forward telemedicine (SAFT), with information-sharing between requesting providers (typically primary care) and answering specialists, using the VA’s existing electronic health record (EHR), the Computerized Patient Record System (CPRS). Store-and-forward telemedicine increases patient access to specialists and timeliness in initiating a consultation while decreasing patient travel and associated costs.1-15
Although previous studies have usually found high concordance between SAFT and FTF consultations in both diagnosis and disease management, the feasibility of SAFT has been historically limited by the lack of an integrated shared EHR.16-18 The VA implemented CPRS in 1996 and was soon recognized as a fully integrated EHR, allowing any VA clinician to view any veteran’s medical record nationwide. This technology, combined with the geographically dispersed population of veterans and the concentration of specialty services within mostly urban tertiary care centers, makes the VA an optimal environment to benefit from implementation of SAFT.
This study examines the implementation of e-Consults in a large VA medical service with the primary goal of facilitating timely specialty care access. This article details the initial experience of implementation and clinicians’ early perceptions of the system.
E-Consult Implementation
The VABHS provides tertiary specialty care for much of New England. It also serves a primary care population of more than 30,000 veterans in eastern Massachusetts through a network of 3 larger primary care practices based at 3 main campuses and 5 smaller CBOCs. Most specialty care is provided by the 2 campuses in Boston.
Discussions with primary care leadership and e-mail announcements to primary care providers (PCPs) throughout VABHS announced the availability and purpose of the new e-Consult. The initiative was initially piloted within 2 specialties, allergy and cardiology, beginning in January 2011. After establishing the logistics for routing, processing, and encounter data capture, e-Consults were expanded to the remaining subspecialties by December 2011.
Within 9 months, 12 different specialties had e-Consult options: endocrinology, gastroenterology, hematology, infectious disease, nephrology, oncology, palliative care, pulmonary disease, rheumatology, and sleep medicine. The e-Consult use was tracked for this study on a monthly basis.
Process and Work Flow
Any PCP with consult ordering privileges can request an e-Consult. Expectations were set that e-Consults should be used to request help with nonemergent clinical issues and be completed within 3 nonholiday weekdays.
The e-Consult uses CPRS modular programming to create an event-based electronic consultation request. Once initiated, the consult is routed to the consulting service. Each specialty defines how to triage the e-Consult request; computer alerts can be immediately generated to a single individual or a defined team within each specialty. Options to alert or print consult requests to administrative personnel also exist. As with requests for FTF consultation requests, e-Consults allow specialists the options to deny a request if they feel it is not appropriate for their specialty. In addition, specialists are able to convert the e-Consult request to a FTF encounter, to forward the e-Consult to another specialty, or to add a “comment” before proceeding. These comments generate an immediate electronic alert to the requestor, thus providing a forum for asynchronous preconsultation dialogue between providers. Once the consultation is complete, the requestor receives an alert that directs them to view the completed consultation note in the EHR.
The entirety of the EHR is accessible to the consultant, including all primary care and specialty progress notes, inpatient and outpatient laboratory testing, and all diagnostic testing performed nationally within the VA.
User Feedback
Based on the literature, separate online questionnaires were developed to ascertain primary care clinicians’ and specialists’ perceptions of the e-Consult system. This 26-item survey for PCPs and 13-item survey for specialists assessed use of e-Consults, satisfaction with the system, and for primary care clinicians, perceived benefits of the e-Consult process
and specific feedback by specialty.
Nine months after full implementation, perceptions of e-Consult processes were solicited. All PCPs, staff physicians, nurse practitioners, and internal medicine residents assigned to a VABHS primary care clinic were surveyed. Eligible specialist consultants included attending physicians and fellows who were identified by the specialty chief as potentially having responded to an e-Consult in the past 6 months. The survey was administered in June 2012 and queried the previous 6 months’ experience.
The monthly number of e-Consults ranged from 96 to 137 over the 6-month study period. Cardiology and hematology were the most commonly requested e-Consults, with > 20 per specialty per month. The 146 cardiology e-Consults during those 6 months represented 20% of all new cardiology consults, whereas the 159 e-Consults to hematology represented 53% of their new consults. The least used specialties for e-Consults were allergy, oncology, palliative care, and rheumatology, each averaging < 2 e-Consults per month.
Provider Perceptions
Overall, 85 of the 163 providers completed the survey (52%). Permanent staff had a better response rate than did trainees: 58% vs 33% among PCPs (P < .02), and 71% vs 44% among specialists (P = .03).
Of the 47 primary care respondents, 4 residents and 2 staff physicians had not previously used e-Consults; all 4 residents and 1 staff physician stated that this was due to lack of awareness of the system. The remaining 41 PCPs had requested ≥ 1 e-Consult in the past 6 months. All 38 specialist provider respondents had answered ≥ 1 e-Consult in study period, with 9 providers completing > 20.
e-Consult Satisfaction
Overall, 93% of PCPs were satisfied with e-Consults, and none were dissatisfied, compared with 53% of specialty providers who were satisfied and 26% who were dissatisfied (P < .001) (Figure 1). All specialists who were dissatisfied were in the sections of cardiology or gastroenterology. Further inspection identified several differences between those specialty providers who were generally satisfied with e-Consults (n = 20) compared with those providers who were dissatisfied (n = 10). Eight PCPs reported being neither satisfied nor dissatisfied.
In analyses comparing satisfied specialists (n = 20) with dissatisfied specialists (n = 10), excluding the 8 respondents who were neither satisfied nor dissatisfied, those who were satisfied reported a lower rate of conversion of e-Consults to FTF visits than did their dissatisfied peers, 26% vs 52%, respectively (P = .01). Satisfied specialists were more likely to view e-Consults as replacing work they otherwise would have done, such as a replacement for a FTF consult and prior informal “curbside” consults, 79% vs 50%, respectively (P < .02). Satisfied specialists were somewhat more likely to complete their e-Consults, on average, in ≤ 15 minutes, compared with dissatisfied specialists, although this trend did not reach statistical significance (59% vs 49%, respectively; (P = .07) (Figure 2).
Improving Quality of Care
PCPs were asked to note which benefits were realized among patients for whom e-Consults replaced FTF visits. Nearly all (98%) indicated that the e-Consults enabled patients to avoid unnecessary travel, and 95% of PCPs indicated that they requested e-consults to receive faster clinical input about a patient. A total of 59% believed e-Consults helped their patients avoid additional copays, and 56% obtained specialty input for a patient who would otherwise refuse to travel. When an e-Consult did not avoid a FTF visit (ie, despite the e-Consult, the patient ultimately proceeded to a FTF consultation with the specialist), the majority of PCPs believed that the e-Consult provided them with reassurance and initiated diagnostic testing that would be useful during the patient’s subsequent FTF consult (Figure 3).
Overall, 61% of specialists agreed with the statement, “e-Consults improve the quality of care VA Boston Healthcare System provides.” Even when specialists perceived that e-Consults did not avoid FTF visits, most agreed that e-Consults helped initiate diagnostic testing prior to a FTF visit. In addition, satisfied specialists saw the benefit in reassuring the PCP and initiating additional management prior to a FTF visit.
Dynamics of e-Consults
Primary care providers reported that 78% of their requested e-Consults were completed within 2 days and 95% of all e-Consults were completed by day 3. In aggregate, primary care clinicians estimated that about one-third of their e-Consults replaced FTF visits; one-third replaced prior mechanisms for informal consultation (eg, “curbside” or e-mail); and one-third of e-Consults represented new requests that would not have involved specialty consultation in the absence of the e-Consult mechanism.
Specialists estimated that 27% of e-Consults were new work (ie, consultations that would not have occurred formally or informally in the absence of the e-Consult mechanism); 32% replaced FTF visits; and 42% were substitutes for prior informal communications. Specialists reported a wide range of time required to answer e-Consults. For specialists, on average, 54% of e-Consults took < 15 minutes to complete, but 20% took > 25 minutes, with 6% requiring ≥ 45 minutes to complete (Figure 2).
Discussion
This article reports the initial experience with the implementation of an e-Consult system for PCPs and medical subspecialists in a large VA health care system. Primary care providers were generally satisfied with the e-Consult system and reported that the system yielded tangible benefits to patients, such as quicker specialty input and avoidance of FTF visits and travel. Specialists were somewhat less satisfied than were their primary care colleagues; nevertheless, specialists perceived similar benefits for patients. Despite the general satisfaction among PCPs, > 1 in 4 specialists expressed dissatisfaction with the system.
Satisfaction with e-Consults may be influenced by the typology of the specialty itself. Some specialties (eg, hematology) rely more heavily on laboratory tests, compared with specialties such as cardiology, gastroenterology, or pulmonary whose subtleties of history and physical examination (H&P), along with review of imaging data, are more commonly required for clinical decision making.2 Hence, the EHR may allow some specialties to provide e-Consults with greater facility. For example, because the EHR enables clinicians to trend the results of years of complete blood counts with a few simple keystrokes, a hematologist may have more confidence in making a clinical assessment without seeing the patient based on historical information in the EHR.
Anecdotal evidence from specialists suggests discomfort when clinical input is based on H&P findings of others. While the sample size was small, this reasoning could explain the differences in satisfaction among specialties: Satisfied specialists may be taking slightly less time to complete their e-Consults, but they were less likely to view the e-Consult as new work and less likely to convert to a FTF consultation.
Store-and-forward telemedicine cannot completely replace FTF visits, but rather supplement them. If e-Consults obviate patient travel and copays while stimulating more timely completion of the consultation, the benefits of cost savings and improved specialty access by veterans merit further attention. When an e-Consult did not avoid a FTF consultation, the majority of clinicians perceived that the e-Consult allowed the PCP to initiate diagnostic testing or alterations in treatment prior to the eventual FTF consultation with the specialist. This finding could be considered as a proxy for increased coordination of care between primary care and specialty providers in anticipation of a FTF visit.
During the study period, workload capture for e-Consults was administratively fixed at the level of a brief consultation, regardless of the effort expended. Validation of the current finding regarding time spent completing consults has resulted in a change. On January 10, 2014, VA Central Office updated its policy to allow 3 levels of workload credit for e-Consults based on time: up to 15 minutes, 16 to 30 minutes, and > 30 minutes.
E-consults are just one example of innovations that broadly fall under the rubric of telehealth initiatives being deployed across VA health care facilities. Other examples of SAFT include pictures collected by technicians and sent to specialists for review to screen for diabetic retinopathy, a practice done within VA for many years.19 Clinical video telehealth provides real-time videoconferencing between patient and specialist and can obviate the need for long-distance patient travel. Initially used only for interviews, newer equipment amplifies sound and enhances optical imaging to allow thorough physical examinations when trained personnel are handling the equipment.
Each technology has its challenges for implementation, which vary in degree of coordination and cost. Overall, the application of new technologies and repurposing existing ones may be limited by creativity alone in the efforts to improve access and quality of the care provided to veterans. The Table highlights telehealth uses being deployed by VA.
Conclusion
Electronic consults have been well received by PCPs and most specialists in the VABHS, seeming to meet the goal of using telehealth to improve veterans’ access to specialty care and coordination of care between PCPs and specialists. While not examined in this initial report, e-Consults may lead to reduced costs, and this possibility should be further explored.
Despite concerns expressed by some specialists, most believed that e-Consults improved the quality of care their patients received. Future work needs to validate these findings, examine patient perspectives, delve more deeply into ways to improve and bring more value to the process, and address the specialty effort in relation to workload awarded.
Acknowledgements
The authors would like to thank Yisraela Elstein and Pauline Benedetti for their assistance in tracking workload data and Steven Simon for his detailed review of the manuscript. Funding for this work was supported by the Department of Veterans Affairs Specialty Care Transformation Initiative for the 21st Century—T21 innovation grant for telehealth initiatives.
Author disclosures
The authors report no actual or potential conflicts of interest with regard to this article.
Disclaimer
The opinions expressed herein are those of the authors and do not necessarily reflect those of Federal Practitioner, Frontline Medical Communications Inc., the U.S. Government, or any of its agencies. This article may discuss unlabeled or investigational use of certain drugs. Please review complete prescribing information for specific drugs or drug combinations—including indications, contraindications, warnings, and adverse effects—before administering pharmacologic therapy to patients.
1. Cusack CM, Pan E, Hook JM, et al. The Value of Provider-to-Provider Telehealth Technologies. Charlestown, MA: Center for Information Technology Leadership; 2007.
2. Horner K, Wagner E, Tufano J. Electronic consultations between primary and specialty care clinicians: Early insights. Issue Brief (Commonw Fund). 2011;23:1-14.
3. Angstman KB, Rohrer JE, Adamson SC, Chaudhry R. Impact of e-consults on return visits of primary care patients. Health Care Manag (Frederick). 2009;28(3):253-257.
4. Kim Y, Chen AH, Keith E, Yee HF Jr, Kushel MB. Not perfect, but better: Primary care providers’ experiences with electronic referrals in a safety net health system. J Gen Intern Med. 2009;24(5):614-619.
5. Angstman KB, Adamson SC, Furst JW, Houston MS, Rohrer JE. Provider satisfaction with virtual specialist consultations in a family medicine department. Health Care Manag (Frederick). 2009;28(1):14-18.
6. Mahnke CB, Jordan CP, Bergvall E, Person DA, Pinsker JE. The Pacific Asynchronous TeleHealth (PATH) system: Review of 1,000 pediatric teleconsultations. Telemed J E Health. 2011;17(1):35-39.
7. Sood S, Mbarika V, Jugoo S, et al. What is telemedicine? A collection of 104 peer-reviewed perspectives and theoretical underpinnings. Telemed J E Health. 2007;13(5):573-590.
8. Maeder A. Telehealth and remote access. Stud Health Technol Inform. 2010;151:239-254.
9. Schmidt T, Lappan CM, Hospenthal DR, Murray CK. Deployed provider satisfaction with infectious disease teleconsulation. Mil Med. 2011;176(12):1417-1420.
10. Verhoeven F, Tanja-Dijkstra K, Nijland N, Eysenbach G, van Gemert-Pijnen L. Asynchronous and synchronous teleconsultation for diabetes care: A systematic literature review. J Diabetes Sci Technol. 2010;4(3):666-684.
11. Krupinski E, Nypaver M, Poropatich R, Ellis D, Safwat R, Sapci H. Telemedicine/telehealth: An international perspective. Clinical applications in telemedicine/telehealth. Telemed J E Health. 2002;8(1):13-34.
12. Moreno-Ramirez D, Ferrandiz L, Ruiz-de-Casas A, et al. Economic evaluation of a store-and-forward teledermatology system for skin cancer patients. J Telemed Telecare. 2009;15(1):40-45.
13. Callahan CW, Malone F, Estroff D, Person DA. Effectiveness of an Internet-based store-and-forward telemedicine system for pediatric subspecialty consultation. Arch Pediatr Adolesc Med. 2005;159(4):389-393.
14. Gagnon MP, Duplantie J, Fortin JP, Landry R. Implementing telehealth to support medical practice in rural/remote regions: What are the conditions for success? Implement Sci. 2006;1:18.
15. Hailey D, Ohinmaa A, Roine R. Study quality and evidence of benefit in recent assessments of telemedicine. J Telemed Telecare. 2004;10(6):318-324.
16. High WA, Houston MS, Calobrisi SD, Drage LA, McEvoy MT. Assessment of the accuracy of low-cost store-and-forward teledermatology consultation. J Am Acad Dermatol. 2000;42(5 pt 1):776-783.
17. Houston MS, Myers JD, Levens SP, et al. Clinical consultations using store-and-forward telemedicine technology. Mayo Clin Proc. 1999;74(8):764-769.
18. Hersh WR, Hickam DH, Severance SM, Dana TL, Pyle Krages K, Helfand M. Diagnosis, access and outcomes: Update of a systematic review of telemedicine services. J Telemed Telecare. 2006;12(suppl 2):S3-S31.
19. Cavallerano AA, Conlin PR. Teleretinal imaging to screen for diabetic retinopathy in the Veterans Health Administration. J Diabetes Sci Technol. 2008;2(1):33-39.
20. Arora S, Thornton K, Murata G, et al. Outcomes of treatment for hepatitis C virus infection by primary care providers. N Engl J Med. 2011;364(23):2199-2207.
1. Cusack CM, Pan E, Hook JM, et al. The Value of Provider-to-Provider Telehealth Technologies. Charlestown, MA: Center for Information Technology Leadership; 2007.
2. Horner K, Wagner E, Tufano J. Electronic consultations between primary and specialty care clinicians: Early insights. Issue Brief (Commonw Fund). 2011;23:1-14.
3. Angstman KB, Rohrer JE, Adamson SC, Chaudhry R. Impact of e-consults on return visits of primary care patients. Health Care Manag (Frederick). 2009;28(3):253-257.
4. Kim Y, Chen AH, Keith E, Yee HF Jr, Kushel MB. Not perfect, but better: Primary care providers’ experiences with electronic referrals in a safety net health system. J Gen Intern Med. 2009;24(5):614-619.
5. Angstman KB, Adamson SC, Furst JW, Houston MS, Rohrer JE. Provider satisfaction with virtual specialist consultations in a family medicine department. Health Care Manag (Frederick). 2009;28(1):14-18.
6. Mahnke CB, Jordan CP, Bergvall E, Person DA, Pinsker JE. The Pacific Asynchronous TeleHealth (PATH) system: Review of 1,000 pediatric teleconsultations. Telemed J E Health. 2011;17(1):35-39.
7. Sood S, Mbarika V, Jugoo S, et al. What is telemedicine? A collection of 104 peer-reviewed perspectives and theoretical underpinnings. Telemed J E Health. 2007;13(5):573-590.
8. Maeder A. Telehealth and remote access. Stud Health Technol Inform. 2010;151:239-254.
9. Schmidt T, Lappan CM, Hospenthal DR, Murray CK. Deployed provider satisfaction with infectious disease teleconsulation. Mil Med. 2011;176(12):1417-1420.
10. Verhoeven F, Tanja-Dijkstra K, Nijland N, Eysenbach G, van Gemert-Pijnen L. Asynchronous and synchronous teleconsultation for diabetes care: A systematic literature review. J Diabetes Sci Technol. 2010;4(3):666-684.
11. Krupinski E, Nypaver M, Poropatich R, Ellis D, Safwat R, Sapci H. Telemedicine/telehealth: An international perspective. Clinical applications in telemedicine/telehealth. Telemed J E Health. 2002;8(1):13-34.
12. Moreno-Ramirez D, Ferrandiz L, Ruiz-de-Casas A, et al. Economic evaluation of a store-and-forward teledermatology system for skin cancer patients. J Telemed Telecare. 2009;15(1):40-45.
13. Callahan CW, Malone F, Estroff D, Person DA. Effectiveness of an Internet-based store-and-forward telemedicine system for pediatric subspecialty consultation. Arch Pediatr Adolesc Med. 2005;159(4):389-393.
14. Gagnon MP, Duplantie J, Fortin JP, Landry R. Implementing telehealth to support medical practice in rural/remote regions: What are the conditions for success? Implement Sci. 2006;1:18.
15. Hailey D, Ohinmaa A, Roine R. Study quality and evidence of benefit in recent assessments of telemedicine. J Telemed Telecare. 2004;10(6):318-324.
16. High WA, Houston MS, Calobrisi SD, Drage LA, McEvoy MT. Assessment of the accuracy of low-cost store-and-forward teledermatology consultation. J Am Acad Dermatol. 2000;42(5 pt 1):776-783.
17. Houston MS, Myers JD, Levens SP, et al. Clinical consultations using store-and-forward telemedicine technology. Mayo Clin Proc. 1999;74(8):764-769.
18. Hersh WR, Hickam DH, Severance SM, Dana TL, Pyle Krages K, Helfand M. Diagnosis, access and outcomes: Update of a systematic review of telemedicine services. J Telemed Telecare. 2006;12(suppl 2):S3-S31.
19. Cavallerano AA, Conlin PR. Teleretinal imaging to screen for diabetic retinopathy in the Veterans Health Administration. J Diabetes Sci Technol. 2008;2(1):33-39.
20. Arora S, Thornton K, Murata G, et al. Outcomes of treatment for hepatitis C virus infection by primary care providers. N Engl J Med. 2011;364(23):2199-2207.