Resident Assessment Tools and Measures

Numeric and global rating assessment tools have been used to assess resident performance for quite some time. Rating systems provide a “unit of measure” or “grade,” which most learners (and teachers) have likely grown accustomed to during their own formalized educational process. One argument for numeric scoring summaries is that they may indeed provide some useful data, assisting the program director to “objectively” assess residents and the program as a whole. On the other hand, there is debatably greater value to providing highly meaningful detailed feedback to individual surgeons in training.¹¹ Isolating a competency to a single numeric value provides feedback that often fulfills requirements but fails to provide the resident feedback concerning targeted areas of improvement. When measuring across multiple competencies, it is quite likely that there will be some residents who achieve an overall satisfactory score but who perform poorly in a single competency. Global ratings have been shown to be highly subjective when graders are not well trained.¹² At times, some raters may rate all competencies equally, regardless of performance. Scores may be biased when raters inappropriately make severe or lenient judgments or avoid using the extreme ends of a rating scale. Reproducibility of scores by raters appears easier to achieve for the domain of knowledge and more difficult to achieve for patient care and interpersonal and communication skills. Perhaps the greatest criticism against the use of global rating tools is that a numeric score can provide suboptimal feedback to the individual resident attempting to learn from the evaluation process.

More specific and useful information used to create a summary of an individual resident’s performance can come from the narrative components of assessment tools. In many programs, emphasis on resident assessment numeric scoring is reduced or eliminated, and specific competency-based narrative commentary is heavily relied on. Self, peer, faculty, as well as 360-degree assessments make up a collective formative database. Strengths and weaknesses can be specifically and immediately queried by the resident, mentor, and program director. The resident should be responsible for keeping track of the formative database. The downside of a system that heavily relies up narrative feedback is that greater time is required to compose a summary compared with scoring schemes. There are also varying degrees of “narrative skill sets” that faculty have in regard to providing rich and meaningful feedback, which limits an assessment system that purely depends on the written word.

Portfolio for Assessment of Professional Development

Several residency programs have begun to use educational portfolios to assess graduate medical education performance.¹³ Principal functions are to lay the groundwork for reflective learning, assess to promote learning, provide useful feedback for improving performance, inform rather than just measure, and document the longitudinal progress. Portfolios were initially described as a purposeful collection of student work that exhibits to the student (and/or to others) the student’s efforts, progress, or achievement in one or more given area(s). This collection must include student participation in the selection of portfolio content, the criteria for selection, the criteria for judging merit, and evidence of student reflection.¹⁴ Essential to the core of the revamped assessment plan is that the resident must take responsibility for creating his or her own portfolio.

The ACGME issued an Executive Summary Report in 2007, launching their own pilot portfolio project¹⁵:

The ACGME is currently in the beta phase of this pilot project, and expected global presentation in the next few years should be anticipated.