New priorities will shift the focus of the ACGME Accreditation site visit
Ingrid Philibert, PhD, MBA
Senior Vice President, Field Activities
Accreditation Council for Graduate Medical Education (ACGME)
The Accreditation Council for Graduate Medical Education’s (ACGME) 2006 strategic plan endorsed four strategic priorities: 1) foster innovation and improvement in the learning environment; 2) increase the accreditation emphasis on educational outcomes; 3) enhance efficiency and reduce burden in accreditation; and 4) improve communication and collaboration with key internal and external stakeholders. These priorities were selected to facilitate the emergence of a new model of accreditation. Goals of the new system include streamlining the accreditation process and bringing it in line with current directions in accreditation and professional education. Primary attributes of the new system include an emphasis on validity and relevance; basing accreditation on educational outcomes; and enabling longer review cycles by adapting the amount and type of data collected and the frequency of on-site visits to programs’ performance. The new approach also will continue early work to promote improvement and innovation in programs and sponsoring institutions.
Each year, 2,000 program directors are informed about an upcoming ACGME site visit. New and experienced program directors want to know what to expect and how to prepare for the process. This article offers an update on the accreditation site visit, summarizing the current process and offering practical advice for surgical program directors. It also foreshadows some of the changes in the process expected for the coming five years.
The present-day site visit
In the current accreditation model, the review of the program is based on the Program Information Form (PIF), a comprehensive self-study document that describes how the program meets the common and specialty-specific program requirements. The site visit seeks to verify and clarify the information in the PIF to confirm program strengths as well as areas in need of improvement and to add information the program may have left out that facilitates the assessment of compliance. The importance of the PIF means that, in addition to the program’s actual performance, an important component for a successful accreditation review is a well-prepared PIF that accurately and completely describes the program. It reduces the need to correct errors or fill in missing information during the site visit, and it makes the accreditation process easier and more effective.
A well-prepared PIF is comprehensive, specific, concise, and answers all questions. It helps to keep in mind that the Residency Review Committee (RRC) members reviewing the document are not familiar with the program’s specifics, and the program should not rely on the site visit report to convey information. If something is important, it should be included in the PIF. Finally, whether the program director completes the PIF or delegates parts to others, a third party should review the entire document for consistency, accuracy, and clarity.
The day of the site visit
The visit begins with the program director interview. The first order of business is a detailed exploration of citations from the last RRC review. This exploration provides an opportunity to showcase what the program has done to correct citations and how it now complies with the standards. This first step is followed by a review of the PIF with the program director, accompanied by the verification of on-site information, such as resident and faculty evaluations, goals and objectives, curriculum information, affiliation agreements, and the like.
Next, come interviews with faculty and the residents which focus on the information in the PIF. In addition, the site visitor wants to find out how the residents and faculty perceive the program. The resident interview often starts with a brief overview of the accreditation process. The site visitor tells the residents the interview is confidential but that the report may be shared with the program if a proposed adverse action results from the review. Two questions are asked each time: the opening question, “How were you selected for the interview?” And the closing one, “What are the strengths and weaknesses of this program?” The remainder of the interview is determined by the information in the PIF and needs verification or clarification. Site visitors also interview the department leadership and a representative for administration. The goal for all interviews is to obtain a deeper understanding of the program and verify and clarify the information in the PIF. For information to be reported to the RRC, a consensus must be reached or the report will identify the comments as coming from an individual or a small group.
A major focus of the accreditation process for surgical programs is the assessment of residents’ operative volume. The accuracy and completeness of the information the residents provide to the ACGME on the volume and variety of procedures is critical to this evaluation. The site visitor does not assess whether the operative volume is sufficient (that is the prerogative of the RRC); rather, the field representative collects information on the quality of the data collection process (whether resident data are verified and by whom) and if there may be marginal or variable data, the site visitor will verify the circumstances and reasons with the program director, residents, and the faculty, and any others involved in collecting and ensuring the accuracy of this information,.
Another area of focus is to assess compliance with the common and specialty-specific duty hour standards. The resident interview is important in this process, as is verification of the information from the ACGME resident survey. In addition, the verification of duty hours uses other program data, including rotation and call schedules and information from program or institutional systems for tracking resident hours. The goal is to distinguish between isolated instances of individual residents working above the limits and duty hours for a significant number of the residents that exceed the ACGME’s common requirements. At the same time, programs and sponsoring institutions should explore any data suggesting non-compliance to assess whether it is a scheduling problem, an isolated incident, or an individual resident with time management or work organization problems. All but the second constitute non-compliance that should be addressed by the program/institution, but it would be unduly intrusive for the ACGME to follow up on all these instances.
The site visit concludes with a meeting with the program director to complete the clarification process by resolving any ambiguities discovered throughout the day. If the program director expects to learn “how the program has done,” he or she may be disappointed. The site visit is just one step in the accreditation process, and the site surveyor is not the decision-maker. That prerogative is reserved for the RRC, in keeping with the ACGME’s peer review process. At the same time, site visitors are experts in their own right, who see programs in every setting, ranging from the excellent to the very worst. Their view of residency education is comprehensive and, while clearly not the RRC’s accreditation decision, their comments on the program may offer some helpful insights.
After the visit, the site visitor’s report is transmitted to ACGME. At the RRC meeting, the program is discussed by one or two reviewers and the RRC makes an accreditation decision that is a group consensus. RRC members with a real or perceived conflict of interest with the program being reviewed are excused from the discussions. Site visitors do not participate in this process; their job is completed when the finished report is transmitted. The ACGME strives to get each program reviewed in a timely fashion, but there is no guarantee that a program site visited three months before the RRC meets will make the agenda. After the RRC has met, the program is notified within 72 hours of the new accreditation status and the time interval to the next site visit. A detailed letter of notification is sent within four to six weeks.
The planned future accreditation approach
Two key changes in the planned future approach will be to base accreditation on a program’s achievements in producing competent graduates and to decrease the amount and importance of the information the program provides in the PIF in favor of ongoing collection of data. This new approach also will allow the ACGME to extend cycles by enabling it to select programs for added scrutiny when its regular data collection suggests negative trends. The ACGME envisions that much of the surveillance process will use data sources already generated by the program, reducing burden and the extent to which the quality of the PIF may determine accreditation outcomes.
This new model will produce changes in the site visit. Future visits may be regular accreditation visits with a status update, likely with extended time intervals between them. Seven to up to 10 years are envisioned to be possible with ongoing data surveillance around the extent to which programs’ residents meet specialty-determined educational milestones. A second form of the visit of the future may be a focused assessment of the program when routine data collection suggests a problem in an important area such as competency attainment, curricula, or duty hours. Changes in the site visit will be necessary to accommodate it to this new role. The PIF may be significantly shorter and much of the data for the accreditation process may come from data collected annually or from a program portfolio that shows residents’ attainment of educational milestones determined by the individual RRCs in concert with the specialty organizations. The combined data of this program “Dashboard” could also trigger an early site visit, if the information suggests non-compliance with the accreditation standards.
Aspects of the future model in today’s approach
A small change in keeping with this vision of the future are “focused” site visits. The ACGME has begun to schedule in 2008 to assess compliance with the duty-hour standards in programs where a significant percentage of the residents responded in a way that suggests non-compliance with the standards. “Significant” is defined as 20 percent or five or more residents reporting non-compliance in the survey for three separate numeric duty-hour standards. These visits do not use a PIF. Instead, the program will complete a short focused form that responds to the areas of non-compliance identified by the survey. To identify programs for the focused visit, the ACGME uses a cut off of several consecutive years of resident survey data suggesting significant non-compliance with three or more of the numeric duty-hour standards.
This type of focused on-site analysis could also be the forerunner of a branching logic that could customize a future PIF to focus on the areas where the RRC may have questions about compliance. From a philosophical perspective, the ACGME is exploring approaches that preserve the strengths of the peer review process while informing it with valid, reliable information to facilitate assessment of programs’ progress over time, allow national comparisons among programs, and promote a more data-driven and outcomes-based accreditation process. A third philosophical goal is meeting expectations for public accountability without these becoming the sole driver of the accreditation process.
In the envisioned new accreditation site visit, the site visitor would be primarily concerned with verifying the areas identified in a reduced PIF and the program’s accomplishments as demonstrated in the program portfolio. The use of a program portfolio, particularly if it allows national comparisons to other programs, would allow the future accreditation process to provide more advice to programs on how to address problems identified via the surveillance or site visit process, although the details of this, including how to provide guidance to programs without avoiding real or perceived conflicts of interest, have yet to be determined.
Ingrid Philibert is the Senior Vice President, Field Activities, ACGME.
Revised December 12, 2009