User login
Institutions conducting research involving animals have established operational frameworks, referred to as animal care and use programs (ACUPs), to ensure research animal welfare and high-quality research data and to meet ethical and regulatory requirements.1-4 The Institutional Animal Care and Use Committee (IACUC) is a critical component of the ACUP and is responsible for the oversight and evaluation of all aspects of the ACUP.5 However, investigators, IACUCs, institutions, the research sponsor, and the federal government share responsibilities for ensuring research animal welfare.
Effective policies, procedures, practices, and systems in the ACUP are critical to an institution’s ability to ensure that animal research is conducted humanely and complies with applicable regulations, policies, and guidelines. To this end, considerable effort and resources have been devoted to improve the effectiveness of ACUPs, including external accreditation of ACUPs by the Association for Assessment and Accreditation of Laboratory Animal Care International (AAALAC International) and implementation of science-based performance standards, postapproval monitoring, and risk assessments and mitigation of identified vulnerability.6-9 However, the impact of these quality improvement measures remains unclear. There have been no valid, reliable, and quantifiable measures to assess the effectiveness and quality of ACUPs.
Compliance with federal regulations is not only required, but also essential in protecting laboratory animals. However, the goal is not to ensure compliance but to prevent unnecessary harm, injury, and suffering to those research animals. Overemphasis on compliance and documentation may negatively impact the system by diverting resources away from ensuring research animal welfare. The authors propose that although research animal welfare cannot be directly measured, it is possible to assess the quality of ACUPs. High-quality ACUPs are expected to minimize risk to research animals to the extent possible while maintaining the integrity of the research.
The authors previously developed a set of quality indicators (QIs) for human research protection programs (HRPPs) at the VA, emphasizing performance outcomes built on a foundation of compliance.10 Implementation of these QIs allowed the research team to collect data to assess the quality of VA HRPPs.11 It also allowed the team to answer important questions, such as whether there were significant differences in the quality of HRPPs among facilities using their own institutional review boards (IRBs) and those using affiliated university IRBs as their IRBs of record.12
Background
The VA health care system (VAHCS) is the largest integrated health care system in the U.S. Currently, there are 77 VA facilities conducting research involving laboratory animals. In addition to federal regulations governing research with animals, researchers in the VAHCS must comply with requirements established by VA.1-4 For example, in the VAHCS, the IACUC is a subcommittee of the Research and Development Committee (R&DC). Research involving animals may not be initiated until it has been approved by both the IACUC and the R&DC.13,14 All investigators, including animal research investigators, are required to have approved scopes of practice.14 Furthermore, all VA facilities that conduct animal research are required to have their ACUPs accredited by the AAALAC International.13
Based on the experience gained from the VA HRPP QIs, the authors developed a set of QIs that emphasize assessing the outcome of ACUPs rather than solely on IACUC review or compliance with animal research regulations and policies. This report describes the proposed QIs for assessing the quality of VA ACUPs and presents preliminary data using some of these QIs.
Methods
The VA ACUP QIs were developed through a process that included (1) identification of a set of potential indicators by the authors; (2) review and comments on the potential indicators by individuals within and outside VA who have expertise in protecting research animal welfare, including veterinarians with board certification in laboratory animal medicine, IACUC chairs, and individuals involved in the accreditation and oversight of ACUPs; and (3) review and revision by the authors of the proposed QIs in light of the suggestions and comments received. After 6 months of deliberation, a set of 13 QIs was finalized for consideration for implementation.
Data Collection
As part of the VA ACUP quality assurance program, each VA research facility is required to conduct regulatory audits of all animal research protocols once every 3 years by qualified research compliance officers (RCOs).15 Audit tools were developed for the triennial animal protocol regulatory audits (available at http://www.va.gov/oro/rcep.asp).11,12 Facility RCOs were then trained to use these tools to conduct audits throughout the year.
Results of the protocol regulatory audits, conducted between June 1, 2011, and May 31, 2012, were collected through a Web-based system from all 74 VA facilities conducting animal research during that period. Information collected included IACUC and R&DC initial approval of human research protocols; for-cause suspension or termination of animal research protocols; compliance with continuing review requirements; research personnel scopes of practice; and investigator animal research protection training requirements.
Because this study did not involve the use of laboratory animals, no ACUC review and approval was required.
Data Analysis
All data collected were entered into a database for analysis. When necessary, facilities were contacted to verify the accuracy and uniformity of data reported. Only descriptive statistics were obtained and presented.
Quality Indicators
As shown in the Box, a total of 13 QIs covering a broad range of areas that may have significant impact on research animal welfare were selected.
QI 1. ACUP accreditation status was chosen, because accreditation of an institutional ACUP by AAALAC International, the sole widely accepted ACUP accrediting organization, suggests that the institution establish acceptable operational frameworks to ensure research animal welfare. Because VA policy requires that all facilities conducting animal research be accredited, failure to achieve full accreditation may indicate that research animals are at an elevated risk due to a less than optimal system to protect research animals.13
QI 2. IACUC and R&DC initial approval of animal research protocols was chosen because of the importance of IACUC and R&DC review and approval in ensuring the scientific merit of the research and the adequacy of research animal protection. The number and the percentage of protocols conducted without or initiated prior to IACUC and/or R&DC approval, which may put animals at risk, is a good measure of the adequacy of the institution’s ACUP.
QI 3. For-cause suspension or termination of animal research protocols was chosen, because this is a serious event. Protocols can be suspended or prematurely terminated by IACUCs due to investigators’ serious or continuing noncompliance or due to serious adverse events/injuries to the animals or research personnel. The number and percentage of protocols suspended reflect the adequacy of the IACUC oversight of the institution’s animal research program.
QI 4. Investigator sanction was chosen, because investigators and research personnel play an important role in protecting research animals. The number and percentage of investigators or technicians whose research privileges were suspended due to noncompliance reflect the adequacy of the institution’s education and training program as well as oversight of the ACUP.
QI 5. Annual review requirement was chosen because of the importance of ongoing oversight of approved animal research by the IACUC. The number and percentage of protocols lapsed in annual reviews, particularly when research activities continued during the lapse reflects the adequacy of IACUC oversight.
QI 6. Unanticipated loss of animal lives was chosen, because loss of animal lives is the most serious harm to animals that the ACUP is intended to prevent. The number and percentage of animals whose lives are unnecessarily lost due to heating, ventilation, or air-conditioning failure reflect the adequacy of the institution’s animal care infrastructure and effectiveness of the emergency response plan.
QI 7. Serious or continuing noncompliance resulting in actual harm to animals was chosen, because actual harm to animals is an important outcome measure of the adequacy of ACUP. The number and percentage of animals harmed due to investigator noncompliance or inadequate care reflect the adequacy of the institution’s veterinarian and IACUC oversight.
QI 8. Semi-annual program review and facility inspection was chosen because of the importance of semi-annual program review and facility inspection in IACUC’s oversight of the institution’s ACUP. This QI emphasizes the timely correction and remediation of both major and minor deficiencies identified during semi-annual program reviews and facility inspections. Failure to promptly address identified deficiencies in a timely manner may place research animals at significant risk.
QI 9. Scope of practice was chosen because of the importance of the investigator’s qualification in ensuring not only high-quality research data, but also adequate protection of research animals. Certain animal procedures can be safely performed only by investigators with adequate training and experience. Allowing investigators who are unqualified to perform these procedures places animals at significant risk of being harmed.
QI 10. Work- or research-related injuries was chosen because of the importance of the safety of investigators and animal caretakers in the institution’s ACUP. The importance of the institution’s occupational health and safety program in protecting investigators and animal care workers cannot be overemphasized. The number and percentage of investigators and animal care workers covered by the occupational health and safety program and work- or research-related injuries reflect the adequacy of the ACUP.
QI 11. Investigator animal care and use education/training requirements was chosen because of the important role of investigators in protecting animal welfare. The number and percentage of investigators who fail to maintain required animal care and use education/training reflect the adequacy of the institution’s IACUC oversight.
QI 12. IACUC chair and members’ animal care and use education and training requirements was chosen because of the important role of the IACUC chair and members in the institution’s ACUP. To appropriately evaluate and approve/disapprove animal research protocols, the chair and members of IACUC must maintain sufficient knowledge of federal regulations and VA policies regarding animal protections.
QI 13. Veterinarian and veterinary medical unit staff qualification was chosen because of the important role of veterinarian and veterinary medical unit staff in the day-to-day care of research animals and the specialized knowledge and qualification they need to maintain the animal research facilities. The number of veterinarians and nonveterinary animal care staff with appropriate board certifications reflects the strength of an institution’s ACUP.
Results
Recognizing the importance of assessing the quality of VA ACUPs, the authors started to collect some QI data of VA ACUPs parallel to those of VA HRPPs before the aforementioned proposed QIs for VA ACUPs were fully developed. These preliminary data are included here to demonstrate the feasibility of implementing these proposed VA ACUP QIs.
IACUC and R&DC Approvals (QI 2)
VA policies require that all animal research protocols be reviewed and approved first by the IACUC and then by the R&DC.13,14 The IACUC is a subcommittee of the R&DC. No animal research activities in VA may be initiated before receiving both IACUC and R&DC approval.13,14
Between June 1, 2011, and May 31, 2012, regulatory audits were conducted on 1,286 animal research protocols. Among them, 1 (0.08%) protocol was conducted and completed without the required IACUC approval, 1 (0.08%) was conducted and completed without the required R&DC approval, 1 (0.08%) was initiated prior to IACUC approval, and 2 (0.16%) were initiated prior to R&DC approval.
For-Cause Suspension or Termination (QI 3)
Among the 1,286 animal research protocols audited, 14 (1.09%) protocols were suspended or terminated for cause; 10 (0.78%) protocols were suspended or terminated due to animal safety concerns; and 4 (0.31%) protocols were suspended or terminated due to investigator-related concerns.
Lapse in Continuing Reviews (QI 5)
Federal regulations and VA policies require that IACUC conduct continuing review of all animal research protocols annually.2,13 Of the 1,286 animal research protocols audited, 1,159 protocols required IACUC continuing reviews during the auditing period. Fifty-three protocols (4.57%) lapsed in IACUC annual reviews, and in 25 of these 53 protocols, investigators continued research activities during the lapse.
Scope of Practice (QI 9)
VA policies require all research personnel to have an approved research scope of practice or functional statement that defines the duties that the individual is qualified and allowed to perform for research purposes.14
A total of 4,604 research personnel records were reviewed from the 1,286 animal research protocols audited. Of these, 276 (5.99%) did not have an approved research scope of practice; 1 (0.02%) had an approved research scope of practice but was working outside the approved research scope of practice.
Training Requirements (QI 11)
VA policies require that all research personnel who participate in animal research complete initial and annual training to ensure that they can competently and humanely perform their duties related to animal research.14
Among the 4,604 animal research personnel records reviewed, 186 (4.04%) did not maintain their training requirements, including 26 (0.56%) without required initial training and 160 (3.48%) with lapses in required continuing training.
Discussion
Collectively, these proposed QIs should provide useful information about the overall quality of an ACUP. This allows semiquantitative assessment of the quality and performance of VA facilities’ ACUPs over time and comparison of the performance of ACUPs across research facilities in the VAHCS. The information obtained may also help administrators identify program vulnerabilities and make management decisions regarding where improvements are most needed. Specifically, QI data will be collected from all VA research facilities’ ACUPs annually. National averages for all QIs will be calculated. Each facility will then be provided with the results of its own ACUP QI data as well as the national averages, allowing the facility to compare its QI data with the national averages and determine how its ACUP performs compared with the overall VA ACUP performance.
These QIs were designed for use in assessing the quality of ACUPs at VA research facilities annually or at least once every other year. With the recent requirement that a full-time RCO at each VA research facility conduct regulatory audits of all animal research protocols once every 3 years, it is feasible that an assessment of the VA ACUPs using these QIs could be conducted annually as demonstrated by the preliminary data for QIs 2, 3, 5, 9, and 11 reported here.15,16 These preliminary data also showed high rates of lapses in IACUC continuing review (4.57%), lack of research personnel scopes of practice (5.99%), and noncompliance with training requirements (4.04%). These are areas that need improvements.
The size and complexity of animal research programs are different among different facilities, which can make it difficult to compare different facilities’ ACUPs using the same quality measures. In addition, VA facilities may use their own IACUCs or the affiliate university IACUCs as the IACUCs of record. However, based on the authors’ experience using HRPP QIs to assess the quality of VA HRPPs, the collected data using ACUP QIs will help determine whether such variables as the size and complexity of a program or the kind of IACUCs used (either VA, own IACUC, or affiliate IACUC) affect the quality of VA ACUPs.10-12
Limitations
There is no evidence proving that these QIs are the most optimal measures for evaluating the quality of a VA facility’s ACUP. It is also unknown whether these QIs correlate directly with the protection of research animals. Furthermore, a quantitative, numerical value cannot be put on each indicator to allow evaluators to rank facilities’ ACUPs.
Some QIs, such as QIs 3, 4, 7, and 8, may depend on how stringent an IACUC is. For example, it is possible that a conscientious IACUC may report more noncompliance or suspend more protocols, giving the appearance of a poor quality ACUP, whereas in fact it might be an excellent program. However, the authors want to emphasize that no single QI by itself is sufficient to assess the quality of a program. It is the combination of various QIs that provides information about the overall quality of a program. It is also through the data collected that the usefulness of any particular indicators may be determined.
Conclusion
These proposed QIs provide a useful first step toward developing a robust and valid assessment of VA ACUPs. As these QIs are used at VA facilities, they will likely be redefined and modified. The authors hope that other institutions will find these indicators useful as they develop instruments to assess their own ACUPs.
Acknowledgement
The authors thank Dr. Kathryn Bayne, Global Director, Association for Assessment and Accreditation of Laboratory Animal Care International, for her suggestions and comments during the development of these quality indicators and critical review of the manuscript, and Dr. J. Thomas Puglisi, Chief Officer, VA Office of Research Oversight, for his support and critical review of the manuscript.
Author disclosures
The authors report no actual or potential conflicts of interest with regard to this article.
Disclaimer
The opinions expressed herein are those of the authors and do not necessarily reflect those of Federal Practitioner, Frontline Medical Communications Inc., the U.S. Government, or any of its agencies. This article may discuss unlabeled or investigational use of certain drugs. Please review complete prescribing information for specific drugs or drug combinations—including indications, contraindications, warnings, and adverse effects—before administering pharmacologic therapy to patients.
1. Animal Welfare Act, 7 USC §2131-2156 (2008).
2. Animal Welfare Regulations, 9 CFR §1-4 (2008).
3. National Research Council of the National Academies. Guide for the Care and Use of Laboratory Animals. 8th ed. Washington, DC: National Academies Press; 2011.
4. Office of Laboratory Animal Welfare. Public Health Service Policy On Humane Care And Use Of Laboratory Animals. Bethesda, MD: National Institutes of Health, U.S. Department of Health and Human Services; 2015. NIH publication 15-8013. http://grants.nih.gov/grants/olaw//PHSPolicyLabAnimals.pdf. Revised 2015. Accessed August 3, 2015.
5. Sandgren EP. Defining the animal care and use program. Lab Anim (NY). 2005;34(10):41-44.
6. Association for Assessment and Accreditation of Laboratory Animal Care International. The AAALAC International accreditation program. The Association for Assessment and Accreditation of Laboratory Animal Care International Website. http://www.aaalac.org/accreditation/index.cfm. Updated 2015. Accessed August 3, 2015.
7. Klein HJ, Bayne KA. Establishing a culture of care, conscience, and responsibility: addressing the improvement of scientific discovery and animal welfare through science-based performance standards. ILAR J. 2007;48(1):3-11.
8. Banks RE, Norton JN. A sample postapproval monitoring program in academia. ILAR J. 2008;49(4):402-418.
9. Van Sluyters RC. A guide to risk assessment in animal care and use programs: the metaphor of the 3-legged stool. ILAR J. 2008;49(4):372-378.
10. Tsan MF, Smith K, Gao B. Assessing the quality of human research protection programs: the experience at the Department of Veterans Affairs. IRB. 2010;32(4):16-19.
11. Tsan MF, Nguyen Y, Brooks R. Using quality indicators to assess human research protection programs at the Department of Veterans Affairs. IRB. 2013;35(1):10-14.
12. Tsan MF, Nguyen Y, Brooks B. Assessing the quality of VA Human Research Protection Programs: VA vs. affiliated University Institutional Review Board. J Emp Res Hum Res Ethics. 2013;8(2):153-160.
13. VA Research and Development Service. Use of Animals in Research. VHA Handbook 1200.07. Washington, DC: Department of Veterans Affairs, Veterans Health Administration; 2011.
14. VA Research and Development Service. Research and Development (R&D) Committee. VHA Handbook 1200.01. Washington, DC: Veterans Health Administration; 2009.
15. Research Compliance Officers and the Auditing of VHA Human Subjects Research to Determine Compliance with Applicable Laws, Regulations, and Policies. VHA Directive 2008-064. Washington, DC: Veterans Health Administration; 2008.
16. VA Office of Research Oversight. Research Compliance Reporting Requirements. VHA Handbook 1058.01. Washington, DC: Veterans Health Administration; 2015.
Institutions conducting research involving animals have established operational frameworks, referred to as animal care and use programs (ACUPs), to ensure research animal welfare and high-quality research data and to meet ethical and regulatory requirements.1-4 The Institutional Animal Care and Use Committee (IACUC) is a critical component of the ACUP and is responsible for the oversight and evaluation of all aspects of the ACUP.5 However, investigators, IACUCs, institutions, the research sponsor, and the federal government share responsibilities for ensuring research animal welfare.
Effective policies, procedures, practices, and systems in the ACUP are critical to an institution’s ability to ensure that animal research is conducted humanely and complies with applicable regulations, policies, and guidelines. To this end, considerable effort and resources have been devoted to improve the effectiveness of ACUPs, including external accreditation of ACUPs by the Association for Assessment and Accreditation of Laboratory Animal Care International (AAALAC International) and implementation of science-based performance standards, postapproval monitoring, and risk assessments and mitigation of identified vulnerability.6-9 However, the impact of these quality improvement measures remains unclear. There have been no valid, reliable, and quantifiable measures to assess the effectiveness and quality of ACUPs.
Compliance with federal regulations is not only required, but also essential in protecting laboratory animals. However, the goal is not to ensure compliance but to prevent unnecessary harm, injury, and suffering to those research animals. Overemphasis on compliance and documentation may negatively impact the system by diverting resources away from ensuring research animal welfare. The authors propose that although research animal welfare cannot be directly measured, it is possible to assess the quality of ACUPs. High-quality ACUPs are expected to minimize risk to research animals to the extent possible while maintaining the integrity of the research.
The authors previously developed a set of quality indicators (QIs) for human research protection programs (HRPPs) at the VA, emphasizing performance outcomes built on a foundation of compliance.10 Implementation of these QIs allowed the research team to collect data to assess the quality of VA HRPPs.11 It also allowed the team to answer important questions, such as whether there were significant differences in the quality of HRPPs among facilities using their own institutional review boards (IRBs) and those using affiliated university IRBs as their IRBs of record.12
Background
The VA health care system (VAHCS) is the largest integrated health care system in the U.S. Currently, there are 77 VA facilities conducting research involving laboratory animals. In addition to federal regulations governing research with animals, researchers in the VAHCS must comply with requirements established by VA.1-4 For example, in the VAHCS, the IACUC is a subcommittee of the Research and Development Committee (R&DC). Research involving animals may not be initiated until it has been approved by both the IACUC and the R&DC.13,14 All investigators, including animal research investigators, are required to have approved scopes of practice.14 Furthermore, all VA facilities that conduct animal research are required to have their ACUPs accredited by the AAALAC International.13
Based on the experience gained from the VA HRPP QIs, the authors developed a set of QIs that emphasize assessing the outcome of ACUPs rather than solely on IACUC review or compliance with animal research regulations and policies. This report describes the proposed QIs for assessing the quality of VA ACUPs and presents preliminary data using some of these QIs.
Methods
The VA ACUP QIs were developed through a process that included (1) identification of a set of potential indicators by the authors; (2) review and comments on the potential indicators by individuals within and outside VA who have expertise in protecting research animal welfare, including veterinarians with board certification in laboratory animal medicine, IACUC chairs, and individuals involved in the accreditation and oversight of ACUPs; and (3) review and revision by the authors of the proposed QIs in light of the suggestions and comments received. After 6 months of deliberation, a set of 13 QIs was finalized for consideration for implementation.
Data Collection
As part of the VA ACUP quality assurance program, each VA research facility is required to conduct regulatory audits of all animal research protocols once every 3 years by qualified research compliance officers (RCOs).15 Audit tools were developed for the triennial animal protocol regulatory audits (available at http://www.va.gov/oro/rcep.asp).11,12 Facility RCOs were then trained to use these tools to conduct audits throughout the year.
Results of the protocol regulatory audits, conducted between June 1, 2011, and May 31, 2012, were collected through a Web-based system from all 74 VA facilities conducting animal research during that period. Information collected included IACUC and R&DC initial approval of human research protocols; for-cause suspension or termination of animal research protocols; compliance with continuing review requirements; research personnel scopes of practice; and investigator animal research protection training requirements.
Because this study did not involve the use of laboratory animals, no ACUC review and approval was required.
Data Analysis
All data collected were entered into a database for analysis. When necessary, facilities were contacted to verify the accuracy and uniformity of data reported. Only descriptive statistics were obtained and presented.
Quality Indicators
As shown in the Box, a total of 13 QIs covering a broad range of areas that may have significant impact on research animal welfare were selected.
QI 1. ACUP accreditation status was chosen, because accreditation of an institutional ACUP by AAALAC International, the sole widely accepted ACUP accrediting organization, suggests that the institution establish acceptable operational frameworks to ensure research animal welfare. Because VA policy requires that all facilities conducting animal research be accredited, failure to achieve full accreditation may indicate that research animals are at an elevated risk due to a less than optimal system to protect research animals.13
QI 2. IACUC and R&DC initial approval of animal research protocols was chosen because of the importance of IACUC and R&DC review and approval in ensuring the scientific merit of the research and the adequacy of research animal protection. The number and the percentage of protocols conducted without or initiated prior to IACUC and/or R&DC approval, which may put animals at risk, is a good measure of the adequacy of the institution’s ACUP.
QI 3. For-cause suspension or termination of animal research protocols was chosen, because this is a serious event. Protocols can be suspended or prematurely terminated by IACUCs due to investigators’ serious or continuing noncompliance or due to serious adverse events/injuries to the animals or research personnel. The number and percentage of protocols suspended reflect the adequacy of the IACUC oversight of the institution’s animal research program.
QI 4. Investigator sanction was chosen, because investigators and research personnel play an important role in protecting research animals. The number and percentage of investigators or technicians whose research privileges were suspended due to noncompliance reflect the adequacy of the institution’s education and training program as well as oversight of the ACUP.
QI 5. Annual review requirement was chosen because of the importance of ongoing oversight of approved animal research by the IACUC. The number and percentage of protocols lapsed in annual reviews, particularly when research activities continued during the lapse reflects the adequacy of IACUC oversight.
QI 6. Unanticipated loss of animal lives was chosen, because loss of animal lives is the most serious harm to animals that the ACUP is intended to prevent. The number and percentage of animals whose lives are unnecessarily lost due to heating, ventilation, or air-conditioning failure reflect the adequacy of the institution’s animal care infrastructure and effectiveness of the emergency response plan.
QI 7. Serious or continuing noncompliance resulting in actual harm to animals was chosen, because actual harm to animals is an important outcome measure of the adequacy of ACUP. The number and percentage of animals harmed due to investigator noncompliance or inadequate care reflect the adequacy of the institution’s veterinarian and IACUC oversight.
QI 8. Semi-annual program review and facility inspection was chosen because of the importance of semi-annual program review and facility inspection in IACUC’s oversight of the institution’s ACUP. This QI emphasizes the timely correction and remediation of both major and minor deficiencies identified during semi-annual program reviews and facility inspections. Failure to promptly address identified deficiencies in a timely manner may place research animals at significant risk.
QI 9. Scope of practice was chosen because of the importance of the investigator’s qualification in ensuring not only high-quality research data, but also adequate protection of research animals. Certain animal procedures can be safely performed only by investigators with adequate training and experience. Allowing investigators who are unqualified to perform these procedures places animals at significant risk of being harmed.
QI 10. Work- or research-related injuries was chosen because of the importance of the safety of investigators and animal caretakers in the institution’s ACUP. The importance of the institution’s occupational health and safety program in protecting investigators and animal care workers cannot be overemphasized. The number and percentage of investigators and animal care workers covered by the occupational health and safety program and work- or research-related injuries reflect the adequacy of the ACUP.
QI 11. Investigator animal care and use education/training requirements was chosen because of the important role of investigators in protecting animal welfare. The number and percentage of investigators who fail to maintain required animal care and use education/training reflect the adequacy of the institution’s IACUC oversight.
QI 12. IACUC chair and members’ animal care and use education and training requirements was chosen because of the important role of the IACUC chair and members in the institution’s ACUP. To appropriately evaluate and approve/disapprove animal research protocols, the chair and members of IACUC must maintain sufficient knowledge of federal regulations and VA policies regarding animal protections.
QI 13. Veterinarian and veterinary medical unit staff qualification was chosen because of the important role of veterinarian and veterinary medical unit staff in the day-to-day care of research animals and the specialized knowledge and qualification they need to maintain the animal research facilities. The number of veterinarians and nonveterinary animal care staff with appropriate board certifications reflects the strength of an institution’s ACUP.
Results
Recognizing the importance of assessing the quality of VA ACUPs, the authors started to collect some QI data of VA ACUPs parallel to those of VA HRPPs before the aforementioned proposed QIs for VA ACUPs were fully developed. These preliminary data are included here to demonstrate the feasibility of implementing these proposed VA ACUP QIs.
IACUC and R&DC Approvals (QI 2)
VA policies require that all animal research protocols be reviewed and approved first by the IACUC and then by the R&DC.13,14 The IACUC is a subcommittee of the R&DC. No animal research activities in VA may be initiated before receiving both IACUC and R&DC approval.13,14
Between June 1, 2011, and May 31, 2012, regulatory audits were conducted on 1,286 animal research protocols. Among them, 1 (0.08%) protocol was conducted and completed without the required IACUC approval, 1 (0.08%) was conducted and completed without the required R&DC approval, 1 (0.08%) was initiated prior to IACUC approval, and 2 (0.16%) were initiated prior to R&DC approval.
For-Cause Suspension or Termination (QI 3)
Among the 1,286 animal research protocols audited, 14 (1.09%) protocols were suspended or terminated for cause; 10 (0.78%) protocols were suspended or terminated due to animal safety concerns; and 4 (0.31%) protocols were suspended or terminated due to investigator-related concerns.
Lapse in Continuing Reviews (QI 5)
Federal regulations and VA policies require that IACUC conduct continuing review of all animal research protocols annually.2,13 Of the 1,286 animal research protocols audited, 1,159 protocols required IACUC continuing reviews during the auditing period. Fifty-three protocols (4.57%) lapsed in IACUC annual reviews, and in 25 of these 53 protocols, investigators continued research activities during the lapse.
Scope of Practice (QI 9)
VA policies require all research personnel to have an approved research scope of practice or functional statement that defines the duties that the individual is qualified and allowed to perform for research purposes.14
A total of 4,604 research personnel records were reviewed from the 1,286 animal research protocols audited. Of these, 276 (5.99%) did not have an approved research scope of practice; 1 (0.02%) had an approved research scope of practice but was working outside the approved research scope of practice.
Training Requirements (QI 11)
VA policies require that all research personnel who participate in animal research complete initial and annual training to ensure that they can competently and humanely perform their duties related to animal research.14
Among the 4,604 animal research personnel records reviewed, 186 (4.04%) did not maintain their training requirements, including 26 (0.56%) without required initial training and 160 (3.48%) with lapses in required continuing training.
Discussion
Collectively, these proposed QIs should provide useful information about the overall quality of an ACUP. This allows semiquantitative assessment of the quality and performance of VA facilities’ ACUPs over time and comparison of the performance of ACUPs across research facilities in the VAHCS. The information obtained may also help administrators identify program vulnerabilities and make management decisions regarding where improvements are most needed. Specifically, QI data will be collected from all VA research facilities’ ACUPs annually. National averages for all QIs will be calculated. Each facility will then be provided with the results of its own ACUP QI data as well as the national averages, allowing the facility to compare its QI data with the national averages and determine how its ACUP performs compared with the overall VA ACUP performance.
These QIs were designed for use in assessing the quality of ACUPs at VA research facilities annually or at least once every other year. With the recent requirement that a full-time RCO at each VA research facility conduct regulatory audits of all animal research protocols once every 3 years, it is feasible that an assessment of the VA ACUPs using these QIs could be conducted annually as demonstrated by the preliminary data for QIs 2, 3, 5, 9, and 11 reported here.15,16 These preliminary data also showed high rates of lapses in IACUC continuing review (4.57%), lack of research personnel scopes of practice (5.99%), and noncompliance with training requirements (4.04%). These are areas that need improvements.
The size and complexity of animal research programs are different among different facilities, which can make it difficult to compare different facilities’ ACUPs using the same quality measures. In addition, VA facilities may use their own IACUCs or the affiliate university IACUCs as the IACUCs of record. However, based on the authors’ experience using HRPP QIs to assess the quality of VA HRPPs, the collected data using ACUP QIs will help determine whether such variables as the size and complexity of a program or the kind of IACUCs used (either VA, own IACUC, or affiliate IACUC) affect the quality of VA ACUPs.10-12
Limitations
There is no evidence proving that these QIs are the most optimal measures for evaluating the quality of a VA facility’s ACUP. It is also unknown whether these QIs correlate directly with the protection of research animals. Furthermore, a quantitative, numerical value cannot be put on each indicator to allow evaluators to rank facilities’ ACUPs.
Some QIs, such as QIs 3, 4, 7, and 8, may depend on how stringent an IACUC is. For example, it is possible that a conscientious IACUC may report more noncompliance or suspend more protocols, giving the appearance of a poor quality ACUP, whereas in fact it might be an excellent program. However, the authors want to emphasize that no single QI by itself is sufficient to assess the quality of a program. It is the combination of various QIs that provides information about the overall quality of a program. It is also through the data collected that the usefulness of any particular indicators may be determined.
Conclusion
These proposed QIs provide a useful first step toward developing a robust and valid assessment of VA ACUPs. As these QIs are used at VA facilities, they will likely be redefined and modified. The authors hope that other institutions will find these indicators useful as they develop instruments to assess their own ACUPs.
Acknowledgement
The authors thank Dr. Kathryn Bayne, Global Director, Association for Assessment and Accreditation of Laboratory Animal Care International, for her suggestions and comments during the development of these quality indicators and critical review of the manuscript, and Dr. J. Thomas Puglisi, Chief Officer, VA Office of Research Oversight, for his support and critical review of the manuscript.
Author disclosures
The authors report no actual or potential conflicts of interest with regard to this article.
Disclaimer
The opinions expressed herein are those of the authors and do not necessarily reflect those of Federal Practitioner, Frontline Medical Communications Inc., the U.S. Government, or any of its agencies. This article may discuss unlabeled or investigational use of certain drugs. Please review complete prescribing information for specific drugs or drug combinations—including indications, contraindications, warnings, and adverse effects—before administering pharmacologic therapy to patients.
Institutions conducting research involving animals have established operational frameworks, referred to as animal care and use programs (ACUPs), to ensure research animal welfare and high-quality research data and to meet ethical and regulatory requirements.1-4 The Institutional Animal Care and Use Committee (IACUC) is a critical component of the ACUP and is responsible for the oversight and evaluation of all aspects of the ACUP.5 However, investigators, IACUCs, institutions, the research sponsor, and the federal government share responsibilities for ensuring research animal welfare.
Effective policies, procedures, practices, and systems in the ACUP are critical to an institution’s ability to ensure that animal research is conducted humanely and complies with applicable regulations, policies, and guidelines. To this end, considerable effort and resources have been devoted to improve the effectiveness of ACUPs, including external accreditation of ACUPs by the Association for Assessment and Accreditation of Laboratory Animal Care International (AAALAC International) and implementation of science-based performance standards, postapproval monitoring, and risk assessments and mitigation of identified vulnerability.6-9 However, the impact of these quality improvement measures remains unclear. There have been no valid, reliable, and quantifiable measures to assess the effectiveness and quality of ACUPs.
Compliance with federal regulations is not only required, but also essential in protecting laboratory animals. However, the goal is not to ensure compliance but to prevent unnecessary harm, injury, and suffering to those research animals. Overemphasis on compliance and documentation may negatively impact the system by diverting resources away from ensuring research animal welfare. The authors propose that although research animal welfare cannot be directly measured, it is possible to assess the quality of ACUPs. High-quality ACUPs are expected to minimize risk to research animals to the extent possible while maintaining the integrity of the research.
The authors previously developed a set of quality indicators (QIs) for human research protection programs (HRPPs) at the VA, emphasizing performance outcomes built on a foundation of compliance.10 Implementation of these QIs allowed the research team to collect data to assess the quality of VA HRPPs.11 It also allowed the team to answer important questions, such as whether there were significant differences in the quality of HRPPs among facilities using their own institutional review boards (IRBs) and those using affiliated university IRBs as their IRBs of record.12
Background
The VA health care system (VAHCS) is the largest integrated health care system in the U.S. Currently, there are 77 VA facilities conducting research involving laboratory animals. In addition to federal regulations governing research with animals, researchers in the VAHCS must comply with requirements established by VA.1-4 For example, in the VAHCS, the IACUC is a subcommittee of the Research and Development Committee (R&DC). Research involving animals may not be initiated until it has been approved by both the IACUC and the R&DC.13,14 All investigators, including animal research investigators, are required to have approved scopes of practice.14 Furthermore, all VA facilities that conduct animal research are required to have their ACUPs accredited by the AAALAC International.13
Based on the experience gained from the VA HRPP QIs, the authors developed a set of QIs that emphasize assessing the outcome of ACUPs rather than solely on IACUC review or compliance with animal research regulations and policies. This report describes the proposed QIs for assessing the quality of VA ACUPs and presents preliminary data using some of these QIs.
Methods
The VA ACUP QIs were developed through a process that included (1) identification of a set of potential indicators by the authors; (2) review and comments on the potential indicators by individuals within and outside VA who have expertise in protecting research animal welfare, including veterinarians with board certification in laboratory animal medicine, IACUC chairs, and individuals involved in the accreditation and oversight of ACUPs; and (3) review and revision by the authors of the proposed QIs in light of the suggestions and comments received. After 6 months of deliberation, a set of 13 QIs was finalized for consideration for implementation.
Data Collection
As part of the VA ACUP quality assurance program, each VA research facility is required to conduct regulatory audits of all animal research protocols once every 3 years by qualified research compliance officers (RCOs).15 Audit tools were developed for the triennial animal protocol regulatory audits (available at http://www.va.gov/oro/rcep.asp).11,12 Facility RCOs were then trained to use these tools to conduct audits throughout the year.
Results of the protocol regulatory audits, conducted between June 1, 2011, and May 31, 2012, were collected through a Web-based system from all 74 VA facilities conducting animal research during that period. Information collected included IACUC and R&DC initial approval of human research protocols; for-cause suspension or termination of animal research protocols; compliance with continuing review requirements; research personnel scopes of practice; and investigator animal research protection training requirements.
Because this study did not involve the use of laboratory animals, no ACUC review and approval was required.
Data Analysis
All data collected were entered into a database for analysis. When necessary, facilities were contacted to verify the accuracy and uniformity of data reported. Only descriptive statistics were obtained and presented.
Quality Indicators
As shown in the Box, a total of 13 QIs covering a broad range of areas that may have significant impact on research animal welfare were selected.
QI 1. ACUP accreditation status was chosen, because accreditation of an institutional ACUP by AAALAC International, the sole widely accepted ACUP accrediting organization, suggests that the institution establish acceptable operational frameworks to ensure research animal welfare. Because VA policy requires that all facilities conducting animal research be accredited, failure to achieve full accreditation may indicate that research animals are at an elevated risk due to a less than optimal system to protect research animals.13
QI 2. IACUC and R&DC initial approval of animal research protocols was chosen because of the importance of IACUC and R&DC review and approval in ensuring the scientific merit of the research and the adequacy of research animal protection. The number and the percentage of protocols conducted without or initiated prior to IACUC and/or R&DC approval, which may put animals at risk, is a good measure of the adequacy of the institution’s ACUP.
QI 3. For-cause suspension or termination of animal research protocols was chosen, because this is a serious event. Protocols can be suspended or prematurely terminated by IACUCs due to investigators’ serious or continuing noncompliance or due to serious adverse events/injuries to the animals or research personnel. The number and percentage of protocols suspended reflect the adequacy of the IACUC oversight of the institution’s animal research program.
QI 4. Investigator sanction was chosen, because investigators and research personnel play an important role in protecting research animals. The number and percentage of investigators or technicians whose research privileges were suspended due to noncompliance reflect the adequacy of the institution’s education and training program as well as oversight of the ACUP.
QI 5. Annual review requirement was chosen because of the importance of ongoing oversight of approved animal research by the IACUC. The number and percentage of protocols lapsed in annual reviews, particularly when research activities continued during the lapse reflects the adequacy of IACUC oversight.
QI 6. Unanticipated loss of animal lives was chosen, because loss of animal lives is the most serious harm to animals that the ACUP is intended to prevent. The number and percentage of animals whose lives are unnecessarily lost due to heating, ventilation, or air-conditioning failure reflect the adequacy of the institution’s animal care infrastructure and effectiveness of the emergency response plan.
QI 7. Serious or continuing noncompliance resulting in actual harm to animals was chosen, because actual harm to animals is an important outcome measure of the adequacy of ACUP. The number and percentage of animals harmed due to investigator noncompliance or inadequate care reflect the adequacy of the institution’s veterinarian and IACUC oversight.
QI 8. Semi-annual program review and facility inspection was chosen because of the importance of semi-annual program review and facility inspection in IACUC’s oversight of the institution’s ACUP. This QI emphasizes the timely correction and remediation of both major and minor deficiencies identified during semi-annual program reviews and facility inspections. Failure to promptly address identified deficiencies in a timely manner may place research animals at significant risk.
QI 9. Scope of practice was chosen because of the importance of the investigator’s qualification in ensuring not only high-quality research data, but also adequate protection of research animals. Certain animal procedures can be safely performed only by investigators with adequate training and experience. Allowing investigators who are unqualified to perform these procedures places animals at significant risk of being harmed.
QI 10. Work- or research-related injuries was chosen because of the importance of the safety of investigators and animal caretakers in the institution’s ACUP. The importance of the institution’s occupational health and safety program in protecting investigators and animal care workers cannot be overemphasized. The number and percentage of investigators and animal care workers covered by the occupational health and safety program and work- or research-related injuries reflect the adequacy of the ACUP.
QI 11. Investigator animal care and use education/training requirements was chosen because of the important role of investigators in protecting animal welfare. The number and percentage of investigators who fail to maintain required animal care and use education/training reflect the adequacy of the institution’s IACUC oversight.
QI 12. IACUC chair and members’ animal care and use education and training requirements was chosen because of the important role of the IACUC chair and members in the institution’s ACUP. To appropriately evaluate and approve/disapprove animal research protocols, the chair and members of IACUC must maintain sufficient knowledge of federal regulations and VA policies regarding animal protections.
QI 13. Veterinarian and veterinary medical unit staff qualification was chosen because of the important role of veterinarian and veterinary medical unit staff in the day-to-day care of research animals and the specialized knowledge and qualification they need to maintain the animal research facilities. The number of veterinarians and nonveterinary animal care staff with appropriate board certifications reflects the strength of an institution’s ACUP.
Results
Recognizing the importance of assessing the quality of VA ACUPs, the authors started to collect some QI data of VA ACUPs parallel to those of VA HRPPs before the aforementioned proposed QIs for VA ACUPs were fully developed. These preliminary data are included here to demonstrate the feasibility of implementing these proposed VA ACUP QIs.
IACUC and R&DC Approvals (QI 2)
VA policies require that all animal research protocols be reviewed and approved first by the IACUC and then by the R&DC.13,14 The IACUC is a subcommittee of the R&DC. No animal research activities in VA may be initiated before receiving both IACUC and R&DC approval.13,14
Between June 1, 2011, and May 31, 2012, regulatory audits were conducted on 1,286 animal research protocols. Among them, 1 (0.08%) protocol was conducted and completed without the required IACUC approval, 1 (0.08%) was conducted and completed without the required R&DC approval, 1 (0.08%) was initiated prior to IACUC approval, and 2 (0.16%) were initiated prior to R&DC approval.
For-Cause Suspension or Termination (QI 3)
Among the 1,286 animal research protocols audited, 14 (1.09%) protocols were suspended or terminated for cause; 10 (0.78%) protocols were suspended or terminated due to animal safety concerns; and 4 (0.31%) protocols were suspended or terminated due to investigator-related concerns.
Lapse in Continuing Reviews (QI 5)
Federal regulations and VA policies require that IACUC conduct continuing review of all animal research protocols annually.2,13 Of the 1,286 animal research protocols audited, 1,159 protocols required IACUC continuing reviews during the auditing period. Fifty-three protocols (4.57%) lapsed in IACUC annual reviews, and in 25 of these 53 protocols, investigators continued research activities during the lapse.
Scope of Practice (QI 9)
VA policies require all research personnel to have an approved research scope of practice or functional statement that defines the duties that the individual is qualified and allowed to perform for research purposes.14
A total of 4,604 research personnel records were reviewed from the 1,286 animal research protocols audited. Of these, 276 (5.99%) did not have an approved research scope of practice; 1 (0.02%) had an approved research scope of practice but was working outside the approved research scope of practice.
Training Requirements (QI 11)
VA policies require that all research personnel who participate in animal research complete initial and annual training to ensure that they can competently and humanely perform their duties related to animal research.14
Among the 4,604 animal research personnel records reviewed, 186 (4.04%) did not maintain their training requirements, including 26 (0.56%) without required initial training and 160 (3.48%) with lapses in required continuing training.
Discussion
Collectively, these proposed QIs should provide useful information about the overall quality of an ACUP. This allows semiquantitative assessment of the quality and performance of VA facilities’ ACUPs over time and comparison of the performance of ACUPs across research facilities in the VAHCS. The information obtained may also help administrators identify program vulnerabilities and make management decisions regarding where improvements are most needed. Specifically, QI data will be collected from all VA research facilities’ ACUPs annually. National averages for all QIs will be calculated. Each facility will then be provided with the results of its own ACUP QI data as well as the national averages, allowing the facility to compare its QI data with the national averages and determine how its ACUP performs compared with the overall VA ACUP performance.
These QIs were designed for use in assessing the quality of ACUPs at VA research facilities annually or at least once every other year. With the recent requirement that a full-time RCO at each VA research facility conduct regulatory audits of all animal research protocols once every 3 years, it is feasible that an assessment of the VA ACUPs using these QIs could be conducted annually as demonstrated by the preliminary data for QIs 2, 3, 5, 9, and 11 reported here.15,16 These preliminary data also showed high rates of lapses in IACUC continuing review (4.57%), lack of research personnel scopes of practice (5.99%), and noncompliance with training requirements (4.04%). These are areas that need improvements.
The size and complexity of animal research programs are different among different facilities, which can make it difficult to compare different facilities’ ACUPs using the same quality measures. In addition, VA facilities may use their own IACUCs or the affiliate university IACUCs as the IACUCs of record. However, based on the authors’ experience using HRPP QIs to assess the quality of VA HRPPs, the collected data using ACUP QIs will help determine whether such variables as the size and complexity of a program or the kind of IACUCs used (either VA, own IACUC, or affiliate IACUC) affect the quality of VA ACUPs.10-12
Limitations
There is no evidence proving that these QIs are the most optimal measures for evaluating the quality of a VA facility’s ACUP. It is also unknown whether these QIs correlate directly with the protection of research animals. Furthermore, a quantitative, numerical value cannot be put on each indicator to allow evaluators to rank facilities’ ACUPs.
Some QIs, such as QIs 3, 4, 7, and 8, may depend on how stringent an IACUC is. For example, it is possible that a conscientious IACUC may report more noncompliance or suspend more protocols, giving the appearance of a poor quality ACUP, whereas in fact it might be an excellent program. However, the authors want to emphasize that no single QI by itself is sufficient to assess the quality of a program. It is the combination of various QIs that provides information about the overall quality of a program. It is also through the data collected that the usefulness of any particular indicators may be determined.
Conclusion
These proposed QIs provide a useful first step toward developing a robust and valid assessment of VA ACUPs. As these QIs are used at VA facilities, they will likely be redefined and modified. The authors hope that other institutions will find these indicators useful as they develop instruments to assess their own ACUPs.
Acknowledgement
The authors thank Dr. Kathryn Bayne, Global Director, Association for Assessment and Accreditation of Laboratory Animal Care International, for her suggestions and comments during the development of these quality indicators and critical review of the manuscript, and Dr. J. Thomas Puglisi, Chief Officer, VA Office of Research Oversight, for his support and critical review of the manuscript.
Author disclosures
The authors report no actual or potential conflicts of interest with regard to this article.
Disclaimer
The opinions expressed herein are those of the authors and do not necessarily reflect those of Federal Practitioner, Frontline Medical Communications Inc., the U.S. Government, or any of its agencies. This article may discuss unlabeled or investigational use of certain drugs. Please review complete prescribing information for specific drugs or drug combinations—including indications, contraindications, warnings, and adverse effects—before administering pharmacologic therapy to patients.
1. Animal Welfare Act, 7 USC §2131-2156 (2008).
2. Animal Welfare Regulations, 9 CFR §1-4 (2008).
3. National Research Council of the National Academies. Guide for the Care and Use of Laboratory Animals. 8th ed. Washington, DC: National Academies Press; 2011.
4. Office of Laboratory Animal Welfare. Public Health Service Policy On Humane Care And Use Of Laboratory Animals. Bethesda, MD: National Institutes of Health, U.S. Department of Health and Human Services; 2015. NIH publication 15-8013. http://grants.nih.gov/grants/olaw//PHSPolicyLabAnimals.pdf. Revised 2015. Accessed August 3, 2015.
5. Sandgren EP. Defining the animal care and use program. Lab Anim (NY). 2005;34(10):41-44.
6. Association for Assessment and Accreditation of Laboratory Animal Care International. The AAALAC International accreditation program. The Association for Assessment and Accreditation of Laboratory Animal Care International Website. http://www.aaalac.org/accreditation/index.cfm. Updated 2015. Accessed August 3, 2015.
7. Klein HJ, Bayne KA. Establishing a culture of care, conscience, and responsibility: addressing the improvement of scientific discovery and animal welfare through science-based performance standards. ILAR J. 2007;48(1):3-11.
8. Banks RE, Norton JN. A sample postapproval monitoring program in academia. ILAR J. 2008;49(4):402-418.
9. Van Sluyters RC. A guide to risk assessment in animal care and use programs: the metaphor of the 3-legged stool. ILAR J. 2008;49(4):372-378.
10. Tsan MF, Smith K, Gao B. Assessing the quality of human research protection programs: the experience at the Department of Veterans Affairs. IRB. 2010;32(4):16-19.
11. Tsan MF, Nguyen Y, Brooks R. Using quality indicators to assess human research protection programs at the Department of Veterans Affairs. IRB. 2013;35(1):10-14.
12. Tsan MF, Nguyen Y, Brooks B. Assessing the quality of VA Human Research Protection Programs: VA vs. affiliated University Institutional Review Board. J Emp Res Hum Res Ethics. 2013;8(2):153-160.
13. VA Research and Development Service. Use of Animals in Research. VHA Handbook 1200.07. Washington, DC: Department of Veterans Affairs, Veterans Health Administration; 2011.
14. VA Research and Development Service. Research and Development (R&D) Committee. VHA Handbook 1200.01. Washington, DC: Veterans Health Administration; 2009.
15. Research Compliance Officers and the Auditing of VHA Human Subjects Research to Determine Compliance with Applicable Laws, Regulations, and Policies. VHA Directive 2008-064. Washington, DC: Veterans Health Administration; 2008.
16. VA Office of Research Oversight. Research Compliance Reporting Requirements. VHA Handbook 1058.01. Washington, DC: Veterans Health Administration; 2015.
1. Animal Welfare Act, 7 USC §2131-2156 (2008).
2. Animal Welfare Regulations, 9 CFR §1-4 (2008).
3. National Research Council of the National Academies. Guide for the Care and Use of Laboratory Animals. 8th ed. Washington, DC: National Academies Press; 2011.
4. Office of Laboratory Animal Welfare. Public Health Service Policy On Humane Care And Use Of Laboratory Animals. Bethesda, MD: National Institutes of Health, U.S. Department of Health and Human Services; 2015. NIH publication 15-8013. http://grants.nih.gov/grants/olaw//PHSPolicyLabAnimals.pdf. Revised 2015. Accessed August 3, 2015.
5. Sandgren EP. Defining the animal care and use program. Lab Anim (NY). 2005;34(10):41-44.
6. Association for Assessment and Accreditation of Laboratory Animal Care International. The AAALAC International accreditation program. The Association for Assessment and Accreditation of Laboratory Animal Care International Website. http://www.aaalac.org/accreditation/index.cfm. Updated 2015. Accessed August 3, 2015.
7. Klein HJ, Bayne KA. Establishing a culture of care, conscience, and responsibility: addressing the improvement of scientific discovery and animal welfare through science-based performance standards. ILAR J. 2007;48(1):3-11.
8. Banks RE, Norton JN. A sample postapproval monitoring program in academia. ILAR J. 2008;49(4):402-418.
9. Van Sluyters RC. A guide to risk assessment in animal care and use programs: the metaphor of the 3-legged stool. ILAR J. 2008;49(4):372-378.
10. Tsan MF, Smith K, Gao B. Assessing the quality of human research protection programs: the experience at the Department of Veterans Affairs. IRB. 2010;32(4):16-19.
11. Tsan MF, Nguyen Y, Brooks R. Using quality indicators to assess human research protection programs at the Department of Veterans Affairs. IRB. 2013;35(1):10-14.
12. Tsan MF, Nguyen Y, Brooks B. Assessing the quality of VA Human Research Protection Programs: VA vs. affiliated University Institutional Review Board. J Emp Res Hum Res Ethics. 2013;8(2):153-160.
13. VA Research and Development Service. Use of Animals in Research. VHA Handbook 1200.07. Washington, DC: Department of Veterans Affairs, Veterans Health Administration; 2011.
14. VA Research and Development Service. Research and Development (R&D) Committee. VHA Handbook 1200.01. Washington, DC: Veterans Health Administration; 2009.
15. Research Compliance Officers and the Auditing of VHA Human Subjects Research to Determine Compliance with Applicable Laws, Regulations, and Policies. VHA Directive 2008-064. Washington, DC: Veterans Health Administration; 2008.
16. VA Office of Research Oversight. Research Compliance Reporting Requirements. VHA Handbook 1058.01. Washington, DC: Veterans Health Administration; 2015.