Unique Aspects of the American Board of Surgery Certification Exam
The American Board of Surgery Certifying Exam (ABS-CE) may be the most unusual testing experience surgeons encounter. Previous exams such as the US Medical Licensing Exams, ABS Inservice Training Exam (ABSITE), and the ABS Qualifying Exam are all multiple choice and focus on applied recall. Examinees may return to a question, change a response, and not face limited time pressure; the correct response to one question is not dependent upon correct responses to previous questions.
On the contrary, the ABS-CE is a dynamic, interactive experience focusing on clinical decision-making. The intent is to simulate the clinical environment such that the examinee is tested on the sequential evaluation, planning, treatment, and management of general surgical problems. Unlike oral interactions residents typically experience—such as morbidity and mortality conferences or the use of the Socratic method—ABS examiners provide no affirmation or correction of the responses given.
The ABS-CE is also important for residency programs. To remain accredited, the program's aggregate first-time pass rate must be higher than the bottom five percentile of programs or an 80% pass rate over three years.1 In addition, it is possible for a single resident to hurt a program twice, first by failing the ABS Qualifying Exam then passing it, followed by failing the ABS-CE.
Given the ABS-CE's uniqueness and importance, many programs organize mock oral exams. The balance of the literature supports the use of mock oral exams.2-7
Benefits of Mock Oral Exams
A mock oral exam provides a simulated experience allowing residents to practice this unique format. While the ABSITE provides information about a resident’s fund of factual knowledge, the mock oral exam provides information about clinical decision-making. This exam also identifies areas of weakness and an opportunity to track one’s progress as they advance through their training. Finally, the mock oral exam provides a mechanism to identify areas of weaknesses within the residency.
Problems with Mock Oral Exams
The most significant problem with mock oral exams is that most faculty have never given the actual ABS-CE. Previously, the ABS prohibited examiners from participating in mock oral exams. The ABS has recognized this problem and now requires only a 3-year hiatus between giving the ABS-CE and participating in mock oral exams. While a good step forward, the majority of examiners’ only experience with the ABS-CE is their remembrance of their own ABS-CE. As a result, recall bias corrupts the fidelity of the mock oral exam.
There are other sources of potential bias which do not occur during ABS-CE. These include past experiences with a specific resident that may positively or negatively influence the mock oral exam score. A resident’s work ethic, attentiveness to detail, or previous complications are unknown by the ABS examiners. A resident’s knowledge of each faculty’s specific preferences may influence their responses even if the faculty member’s practice pattern is incorrect or outdated. Faculty may also be dogmatic in their approach to specific clinical scenarios, thereby downgrading a resident who has selected a safe, effective alternative.
While commonly overemphasized during mock oral exam,3,6 professional attire, personal appearance, linguistic skill, and eye contact are never evaluated during the ABS-CE. However, these factors may influence how the examiner views the examinee.
The behavior of faculty examiners may vary significantly from what the ABS expects of their examiners. Some faculty intentionally amplify the level of stress, whereas ABS examiners intentionally try to place the examinee at ease. Further, during the ABS-CE, only one examiner interacts with an examinee regarding a specific case, while the other ABS examiner sits quietly and evaluates the examinee. During the ABS-CE there is neither affirmation nor condemnation of an examinee’s response and, the examiner takes great care to present a clear, logical scenario, intentionally avoiding any additional confusion.
Finally, to be a fair exam, all examinees must complete four cases in each room and scoring of each case is tallied independently by the ABS examiner.
Checklist for a High-Fidelity Mock Oral Exam
Before the Exam
- To decrease bias, if possible, residents should be examined by faculty they do not frequently interact with. During the ABS-CE, examiners will test both within and outside their normal scope of clinical practice. However, for mock oral exam, it is recommended that faculty test outside their areas of practice. This will eliminate the temptation for the resident to answer a question based on past experiences with a specific faculty member. This concern also supports the benefits of doing regional or city-wide mock oral exams.
- The goal is to have a standard exam. Case scenarios should be developed in advance. Each scenario consists of a brief stem, additional information the examinee should request, and agreed upon pathways for case. Understanding that faculty may be examining outside their normal scope, proper responses should also be provided.
- All examiners should meet before the mock oral exam to review, discuss, and reach a consensus as to what constitutes a correct response to each question.
- To accurately simulate the ABS-CE experience, the faculty must be in appropriate professional attire, a 7-minute timer should be available, and televideo-conferencing utilized.
During the Exam
Resident Entry into the Room
- It is important that the faculty’s mindset is to figure out what the resident knows rather than what the resident does not know.
- The faculty should inform the resident that they will be taking notes and the case may end abruptly to stay on time. This should not be interpreted as the resident doing well or poorly.
- The resident should know that if he or she does not understand something or recognize they have made an error, it is permissible to ask for clarification or redirect themselves.
Case Presentation and Management
- Each case should be presented in a clear, concise, and straightforward manner. Information should be presented in small enough bites that it can be understood and acted upon.
- Only one faculty should manage each scenario, without giving any positive or negative feedback. If a resident has misunderstood the situation and is heading in the wrong direction it is permissible to redirect the resident in the proper direction.
- The rapid pace of the ABS-CE may be surprising. Therefore, the faculty needs to manage the resident through the entirety of each case in seven minutes. Each resident must complete four cases per room.
Scoring
- For each case, the ABS essentially uses a 3-tier scoring system (Pass, Equivocal, Fail). The same is recommended for mock oral exam.
- Each examiner scores the case independently. Scores are not discussed and should only be based on the resident’s ability to manage the case and not any of the attributes contained in the attributes list below.
- Each of the four cases should be evaluated separately and no room score is given.
After the Exam
Resident Performance
- The ABS only provides a passing or failing grade to the examinee. However, in the learning environment of the MOE, it may be more beneficial to calculate numeric scores for each case. This will provide more granular information to improve performance.
- Notes taken during each case should also be shared with the resident and program director.
- The MOE should be used as a formative, not summative feedback mechanism.
Faculty and Exam Performance
- To improve the quality of future mock oral exams, data comparing each examiner should be collected and shared.
- The ABS requires that each examiner evaluate their co-examiner’s performance during the exam. The same should hold true for mock oral exams.
- The exam questions should also be carefully reviewed to ensure each is logical, clear, up-to-date, and not duplicated.
In summary, the ABS-CE is unique and high stakes for both residents and the residency programs. Mock oral exams provide an effective, simulated experience to prepare for this exam. To be most beneficial, the fidelity of this exam should be reflective of the actual ABS examination process.
