ACS Entering Resident Readiness Assessment (ACS ERRA)

The Entering Resident Readiness Assessment (ERRA), developed by the ACS Division of Education, is an innovative online, case-based instrument to measure the preparedness of entering surgical residents (PGY-1).

The ERRA focuses clinical decision-making for cases frequently encountered at the beginning of resident training. The assessment provides surgery program directors and residents with psychometrically rigorous measures of key skills needed to safely assume their new clinical responsibilities.

Short case scenarios are used to address essential clinical topics through special "key features" questions that specifically assess clinical decision-making skills instead of simple recall of factual knowledge. Results of the assessment can be used to identify strengths and to develop learning plans in areas that may need reinforcement for individual residents or the entering group as a whole.

Sample reports and an ACS ERRA demonstration for program staff are available upon request.

Assessment Description

The ACS ERRA is a psychometrically rigorous, formative assessment of PGY-1 preparedness, focusing on 20 clinical topics that incoming surgical residents are likely to encounter. Residents will respond to 40 short clinical cases, each followed by questions that specifically assess key clinical decisions. As a formative tool, the ERRA is intended to assess residents’ current decision-making at the time they enter surgical residency; as such, it is not intended to measure the memory and recall of surgical fact and procedure for use as a summative assessment. Over the past several years, the ACS Division of Education has methodically defined a validated domain of cases and recruited expert faculty to develop the assessment. Residents will respond to 40 short clinical cases, each followed by questions that specifically assess key clinical decisions. As a formative tool, the ACS ERRA is intended to assess residents' current decision-making at the time they enter surgical residency. The ACS ERRA is not intended to measure fact and procedure memory and recall for use as a summative assessment.

Response formats include short, write-in answers (for example, listing possible diagnoses) or the selection of options from menus containing possible next steps in the investigation or management of the problem. The specially designed questions use the internationally proven “key features” approach as established by Georges Bordage, MD, MSc, PhD, and Gordon Page, MA, EdD, in which the case scenarios require medical knowledge to be applied appropriately at critical decision points during management of the case. Key features may include essential steps required to resolve a problem, a particularly challenging aspect in the identification and management of the problem, or actions most likely associated with errors in practice. Because not all steps in the management of patient problems are equally important, assessment time is best spent by focusing exclusively on the most challenging aspects that will distinguish appropriate performance.

Development and Authors

The ACS Division of Education has methodically defined a validated domain of cases and recruited expert faculty to develop the assessment. Four pilot studies were conducted in a step-wise sequence with improvements made after each pilot. The first pilot was a small study that focused on the test takers’ understanding of the clinical cases. The second pilot study asked users to provide feedback on the assessment’s online presentation and functionality. The technical feasibility study tested the technical operations and capacity of the online server. Most importantly, the psychometric pilot study was conducted during the summer of 2017 across 18 surgical residency programs nationwide: the results demonstrated validity and psychometric rigor. (doi: 10.1097/SLA.0000000000003241).

Since its launch in 2018, more than 3,687 residents across 152 general surgery residency programs have participated nationwide. Development is ongoing to protect the integrity and quality of the assessment.

Leadership Team

• Kathy R. Liscum, MD, FACS, Co-Chair
  
• Ajit K. Sachdeva, MD, FACS, FRCSC, FSACME, MAMSE, Co-Chair
• Adnan A. Alseidi, MD, EdM, FACS
• Edgardo Salcedo, MD, FACS
• Yoon Soo Park, PhD
• Maura E. Sullivan, MSN, PhD
• Patrice Gabler Blair, DrPH, MPH
• Kevin M. Wasielewski, MPH, MBA

ACS ERRA Authors

• Georges Bordage, MD, MSc, PhD
• Rebecca C. Britt, MD, FACS
• Melissa E. Brunsvold, MD, FACS
• Benjamin W. Dart IV, MD, FACS
• Celia M. Divino, MD, FACS
• Robert M. Goldstein, MD, FACS
• Enjae Jung, MD, FACS
• Alan P. Ladd, MD, FACS
• Jennifer LaFemina, MD, FACS
• Eric Lazar, MD, FACS
• John D. Mellinger, MD, FACS
• Sarkis H. Meterissian, MD, FACS
• Rebecca M. Minter, MD, FACS
• John T. Mullen, MD, FACS
• Patricia J. Numann, MD, FACS
• Barbara J. Pettitt, MD, FACS, FAAP
• Douglas S. Smink, MD, FACS
• Ranjan Sudan, MD, FACS

The cost of the 2024 ERRA is $135 per resident.

Enrollment dates: March 18–May 31, 2024
Administration dates: June 10–July 3, 2024

Only surgery residency programs are eligible to enroll their entering PGY-1 residents in the assessment.

Following enrollment, programs will be invoiced for payment. ACS staff will then contact programs to provide details regarding the assessment, including login requirements, specific instructions for the proctored assessment environment, details regarding computer needs and accessing the assessment online, time frames and dates for the assessment, confidentiality agreements, and other information.

ACS staff provide training and support to all program administrators throughout the entire process.

Process

The ACS ERRA is proctored in-person at a secure location within your institution by a designated program staff member. Residents will access the online assessment via institutional computers (or laptops) on a secure browser. Programs that have enrolled their residents will be asked to identify a time between mid-June and early July (specific dates identified above), to administer the assessment; many programs use the ACS ERRA as part of their orientation activities. Residents will need to be excused from clinical duties for about three (3) hours and can complete the assessment at the same time as a group or individually at different times, as long as they have access to computers with an Internet connection in a proctored environment. Residents will be asked to sign confidentiality agreements. Results of the assessment will be made available as soon as possible and reports should be received by program directors by the first week of August. All data will be considered and handled as confidential.

Requirements

• Three (3) hours of protected time for entering residents
• Proctored environment with computers or tablets connected to the Internet. Personal equipment should not be used.

ACS staff will be happy to provide training and assistance with the technical and administrative details of the assessment.

Following the assessment, program directors will receive confidential reports with data on the preparedness of their residents in each clinical topic. Data are provided for each individual resident, at the program level, and in aggregate form for comparison with other participating programs nationwide. Results list the topic areas and resident levels of performance based on the percent answered correctly in each area, as well as the overall distribution of scores on the assessment.

Reports are also provided for each resident and include information regarding individual scores. These reports will allow program directors and each resident to recognize specific areas of strengths and to develop learning plans to address focused areas for increasing preparedness.

An Individual Learning Plan (ILP) template is included with each individual score report. The template is designed to facilitate conversations between the program director and resident to help define focused learning goals and to identify specific actions for residents to take.