“Holistic evaluation” is a term often repeated by program directors when quizzed about how residency applicants are assessed. The Merriam-Webster dictionary defines “holistic” as “relating to or concerned with wholes or with complete systems rather than with the analysis of, treatment of, or dissection into parts.”* But what holistic truly means in this context is a $1 million question for the stakeholders.
In 2020–2021, I had the opportunity to interact with program directors, faculty, residents, and medical students over the course of an arduous virtual interview season. My recent experience gave me profound insight into the complexities that prevail in this holistic evaluation process for the selection of candidates.
Making the Right Match
A holistic review of applications is a time-consuming and laborious task. It is impossible for even the most efficient program directors to diligently review more than 1,000 applications. Any discussion of how to improve this evaluation process must coexist with a simultaneous discussion attempting to solve the hapless prisoner’s dilemma that applicants face when applying to programs.
The lack of transparency in the selection process inevitably leads to applicants overapplying with the fear of not matching with a particular program for reasons entirely unknown to them. The obvious solution is that the programs should reveal their selection filters, including minimum US Medical Licensing Examination (USMLE) score requirements, clerkship grades, international medical graduate status, visa restrictions, and so on, before the Electronic Residency Application Service (ERAS) submission deadline.
A novel idea to address this issue was implemented by otorhinolaryngologists in the 2021 National Residency Match Program (NRMP). Each ear, nose, and throat (ENT) candidate was given five tokens to preferentially signal their interest to their top five programs. This schematic was intended to reassure the applicants by giving them an opportunity to objectively demonstrate their interest in their preferred program. This idea was carried forward by internal medicine, dermatology, and general surgery for the NRMP Match 2022. Unless the predicament of overapplying is addressed, any debate on the holistic evaluation of applicants will remain academic rather than practical.
USMLE: Pros and Cons of Pass/Fail
According to the results of the 2020 NRMP Program Director Survey, the USMLE Step 1 score was the highest cited factor in selecting applicants for interview. With the USMLE Step 1 changing to pass/fail, program directors will be forced to change their selection criteria, hopefully for the better. Deemphasizing the enormous weight given to the USMLE Step 1 will reduce the mental strain on medical students early in their medical education. The current system rewards super-efficient test-takers and rote learners rather than focusing on evaluating logical and conceptual learning. It also ignores other critical aspects of medicine, such as interpersonal skills, teamwork, patient communication, and so on.
It is anticipated that with the USMLE Step 1 now being pass/fail, students will have an opportunity to grasp important preclinical concepts and hone their soft skills more diligently. It will enable students to build a strong preclinical foundation, which will pave the way toward better medical practice.
With that thought in mind, it would be careless to completely do away with using USMLE scores as a barometer for selecting residents. The USMLE is the great equalizer. The exam does not have intrinsic or extrinsic bias. Furthermore, the exam does not discriminate based on race, gender, nationality, prestige of medical school, international medical graduate status, and so on. Whereas access to research opportunities, visiting rotations, membership in prestigious honorary societies, and the like are inequitably distributed among applicants, the USMLE score gives an opportunity to medical students from disadvantaged backgrounds to excel. I believe that the USMLE Step 2 CK (clinical knowledge) is a superior benchmark exam in the evaluation process, as it tests the fundamentals of medicine and surgery with a special focus on diagnosis and treatment. As compared with the basic science-oriented USMLE Step 1, the USMLE Step 2 CK, which is clinically oriented, is perhaps a better criterion to predict future performance in residency.
A Truly Holistic Experience
I had the opportunity to interview at a wide spectrum of programs—from large academic centers at R1 universities (doctoral universities with the highest research activity) to standalone community hospitals. By the end of the interview season, the entirety of my ERAS application was covered in one way or another. Different programs underscored different parts of my application according to their requirements. My clinical experiences, letters of recommendation, research publications, volunteerism activities, visiting rotations, leadership skills, and hobbies were discussed and dissected thoroughly. I believe that this methodology of multifaceted evaluation is fantastic and holistic. This same methodology should be adopted during the initial screening of applicants as well.
Although programs could tailor their selection criteria to find candidates who are the perfect fit, they should be wary of overemphasis on just one aspect of the application. Applicants who are well-rounded and demonstrate the ability to balance extracurricular activities with academic coursework are more likely to thrive in a balanced way and less likely to drop out of a rigorous surgical residency.
Lastly, a few programs used supplementary questionnaires to get additional information to gauge the interest and aptitude of the applicants. These questionnaires may help programs to narrow down the applicant pool. However, discussions with peers revealed mixed reactions to these supplementary questionnaires. In fact, they came as a surprise to most applicants, as traditionally submitting the ERAS application is considered the final hurdle. The ERAS curriculum vitae along with the personal statement, letters of recommendation, and medical student performance evaluation are sufficient to judge the caliber of an applicant, and the utility of these supplementary questionnaires was called into question. If a program does want to use a supplementary questionnaire, it should be mentioned in advance on their website so that applicants are not blindsided.
The change of USMLE Step 1 to a pass/fail exam signifies a paradigm shift. The former system of evaluation reduced medical students to just numbers, which has had detrimental effects on their mental health, as they have been led to believe that a three-digit score determines their entire future. A holistic review of applicants—composed of a transparent and multifaceted evaluation accounting for diversity and inclusion—is a necessary component of today’s match process. By adopting a new system of evaluation, we cannot say whether things will get better. What is certain is that things must change if they are to get better.
*Merriam-Webster Dictionary. Holistic. Available at: https://www.merriam-webster.com/dictionary/holistic. Accessed January 29, 2022.
