How to develop competency-based promotion criteria: Enhancing surgery resident assessment and evaluation

Laura Torbeck, PhD, and David Canal, MD
Department of Surgery, Indiana University, Indianapolis

Whether it is in the operating room, on the wards, in a clinic, or at a conference, surgery residents spend the majority of their training time performing.  The assessment of resident per-formance, therefore, needs to be structured and implemented in sound fashion.  The education literature indicate that two components are mandatory to a performance assessment: the task and the criteria.  The task for surgical residents is largely composed of various clinical experiences along with skills labs, conference presentations, and teaching.  However, the criteria are often

ill-defined, leaving residents to make assumptions as to what is expected of them.  Goals and objectives certainly give some direction for residents, but many times residents gloss over them and fail to use the objectives to guide their performances.

To enhance our resident assessment and evaluation process, our program recently developed performance-based criteria that are competency-based and will be used in considering residents for promotion.  This article will outline the recommended steps for developing competency-based promotion criteria.

How to develop competency-based performance criteria    

1.  Form a focus group of physicians by selecting five to six surgeons who are interested and well-suited to work collaboratively to define performance-based criteria for each resident level.

2.  Schedule four to five one-hour meetings with the focus group over the course of two months to brainstorm criteria. 
At our first meeting, the surgical educator introduced the group’s purpose and task.  Promotion criteria used for family medicine residents (see references) were shared with the group, providing a foundation for its work.  During the subsequent meetings, the group considered what was important for PGY 1 residents to achieve by the end of their intern year and devised cor-responding criteria.  The same was done for PGY 2/3 residents, research residents, and PGY 4/5 residents.

3.  Distribute the performance-based criteria to the other members of the faculty and select residents for their feedback and buy-in to the process. 
The surgical educator passed out paper copies of the criteria at faculty meetings and asked faculty to agree or disagree with the level-specific criteria as well as suggest additions or modifications.  Those faculty members who did not attend the meetings were E-mailed the criteria and encouraged to provide feedback.  The surgical educator also met with representa-tives from the resident education committee to get resident feedback and buy-in to the criteria.

4.  Take the faculty suggestions back to the focus group to finalize the criteria. 
The focus group gathered one last time to consider the faculty feedback and rework the criteria into its final format.

5.  Organize the criteria into a matrix detailing corresponding resident level, Accreditation Council for Graduate Medical Education (ACGME) subcompetency label, setting, and evaluation method. 
Once the performance-based criteria were devised for each resident level, the surgical educator identified which ACGME competency each criterion met by reworking the competency and subcompetency information provided by ACGME on the Outcome Project’s Web site to include a label for each competency and subcompetency.  For example, the label “PC1” stands for the first subcompetency of the patient care competency.  The surgical educator also identified which setting and what evaluation method(s) would be used to assess each criterion.  All of this was formatted into a matrix.

Example of Matrix

Year	Promotion Criteria	ACGME Label	Setting	Evaluation
PGY 1	Gathers complete and reliable patient history	PC2	Clinic, Ward, OR	Faculty reviews H&P’s; resident evaluation; resident self-evaluation
PGY 1	Demonstrates efficient and thorough patient sign-out	PC4, PC9, ICS2,3	Change-over	Peer evaluation; resident self-evaluation
PGY 2/3	Shows ability to organize and prioritize in emergent situations and on the wards	PC1-4, PC7, MK1	Trauma bay, ED, Wards	Resident evaluation; resident self-evaluation
PGY2/3	Presents working diagnosis to patient	ICS2, PC5	Clinic, ED	Resident evaluation; resident self-evaluation; patient evaluation
Research	Shows ability to analyze and interpret data	MK2, PBL4	Lab	Lab faculty and staff evaluation
Research	Conducts a critical interpretation of literature relating to area of research	PBL2, 4, 5	Lab	Lab faculty and staff evaluation
PGY 4/5	Shows appropriate assertiveness in the OR	ICS2-3, P1,3, SBP1	OR	Resident evaluation; resident self-evaluation
PGY 4/5	Demonstrates an understanding of practice management issues pertinent to resident’s future career plan	SBP1-3	Practice Management Courses	Portfolio assignments

6.  Identify a date on which to implement this new competency-based evaluation process.
We started this process a year before we wished to implement it.  It took approximately two months to meet with the focus group, two months to gather faculty and resident feedback, and a month to organize and format the criteria into a matrix.  The remaining steps were completed during the last six months prior to the implementation date of July 1.

7.  Familiarize the residents with the competency-based performance criteria by having the residents perform self-assessments midyear. 
In January, we used the latter part of a conference time to have the residents complete a self-assessment.  Their self-assessments of the performance criteria were placed in their files and used as a discussion topic during their biannual review with the program director.

8.  Continue to develop the faculty on the meaning of the competencies and how they relate to the newly developed performance criteria. 
The surgical educator took advantage of every opportunity when the faculty assembled to remind them of what the competencies are, what they mean, and how our residency plans to incorporate them into our evaluation process.

9.  Incorporate the performance-based criteria into the curricular goals and objectives. 
Since residents are evaluated based upon the goals and objectives for specific curricular assignments, it is essential that the performance-based criteria be included.

10.  Develop evaluation tools or enhance existing evaluation methods to include the competency-based performance criteria. 
Our program has decided to focus first on enhancing our global resident performance evaluation that the faculty complete after each rotation.  The new evaluation will include the level-specific performance criteria using a rating scale of needs improvement (completes < 80 percent of the time), competent (completes ~ 80 percent of the time), exceeds competent (completes > 80 percent of the time), and did not achieve enough direct observation to rate.  The other evaluation tools we plan to utilize are a portfolio and multisource feedback evaluations.

Using the performance-based criteria as promotion criteria

In the past, our residency’s methodology of promoting residents was not conducted in the most systematic way.  With the advent of the ACGME competencies, our program educators have come to realize and embrace the importance of using the competencies to develop an accountable method for ensuring our trainees develop into competent surgeons.  It’s no longer a matter of a “thumbs up, thumbs down” approach when it comes to resident promotion; rather, it’s now a matter of closely tracking how our residents develop each year and deciding collaboratively if each resident should be promoted.  Using the level-specific performance-based criteria that our program developed is one way to begin clarifying expectations.  Specific, well-defined criteria, we expect, will allow the residents to better understand what is expected of them during each year in residency and it also helps the faculty to better evaluate and coach our trainees.  Another benefit to defining promotion criteria is it allows the program to identify resident deficiencies earlier so that remediation can occur in a timely fashion.  By tracking the performance ratings of each resident throughout the year, our program will decide whether or not to promote residents based on achievement of the level-specific, competency-based “promotion” (performance-based) criteria and more critically defines the areas in need of remediation. 

Introducing the idea of “promotion” criteria has raised several concerns within our program.  For instance, both the program director and residents have questioned whether all the level-specific promotion criteria must be achieved by the end of the year in order to be promoted.  This is an issue that programs must decide for themselves.  Our belief is all criteria need to be met, and if they are not, then a resident could potentially not be promoted, be placed on probation with remediation, or terminated.  We suspect that program directors will likely have different comfort thresholds when making such decisions.  As long as a rationale is defined for the promotion process and that rationale is made transparent to the faculty and residents, the program should feel secure in their decisions.

Another concern raised within our program revolved around giving and receiving a “needs improvement” mark on the resident evaluations.  Our surgical educator has tried very hard to impress upon the faculty and residents the developmental nature of residents in training.  Through both faculty and resident development sessions, she has explained that residents need to be achieving competent marks by April for end of the year promotion.  Early in the year, residents should not be overly concerned about receiving “needs improvement” marks.  Residents, however, should be worried if they continue to receive “needs improvement” scores well into the latter half of the academic year.  These are two main concerns worth mentioning that our program experienced so far, and it is best to have answers for these issues before implementation.

Summary

Developing level-specific, competency-based promotion criteria is one way that our program chose to incorporate assessment of the ACGME competencies.  More importantly, it was also a way to devise a more accountable and systematic way of evaluating our residents.  The steps outlined above can be adapted to any program.  We see the value of promotion criteria as being threefold:

• Criteria can raise the awareness of what is expected of the residents for both the residents and faculty.
• Criteria can improve faculty development, especially in terms of how to assess competence.
• Criteria can target remediation in those residents failing to meet the criteria in a timely fashion.

References and Resources

1. Torbeck L, Wrightson S.  A Method for Defining Competency-Based Promotion Criteria for Family Medicine Residents.  Academic Medicine.  2005; 80:832-839.
2. Nitko AJ.  Educational Assessment of Students.  Upper Saddle River, NJ: Prentice Hall, 2001.
3. The ACGME Outcome Project http://www.acgme.org/outcome/.    

Revised December 8, 2006