How to synthesize narratives in program, rotation, or faculty evaluations to use for program improvement

Ilene B. Harris, PhD
Professor and Director of Graduate Studies in Health Professions Education Leadership
Department of Medical Education, University of Illinois at Chicago College of Medicine

Introduction

Program directors endeavor to identify the strengths of their program and areas for improvement by using trainee written evaluations of programs overall, as well as of rotations in their programs, faculty, and residents.  Typically, two types of data comprise written evaluations:

(1) responses to structured questions, which are summarized in descriptive statistics (quantitative data), and (2) responses to open-ended questions referred to as comments, narratives, or qualitative data.  The responses to structured questions are synthesized in descriptive statistics and are immediately usable to identify strengths and areas for improvement.  However, the comments, which are usually presented in long lists of quotations, need to be synthesized to identify themes before they become truly useful for program and faculty improvement.  Similarly, other methods used to collect data for program improvement, such as focus group sessions with trainees, interviews with individual trainees, and observations summarized in notes, also yield narrative or qualitative data that must be synthesized to identify themes before they become optimally useful for program improvement.  The purpose of this article is to provide an overview of the nature and purposes of qualitative data; to demonstrate how qualitative data may be analyzed to provide a useful synthesis of themes; and to outline standards for rigor in analyzing qualitative data so that it can be trustworthy to use for program evaluation and improvement.

The nature and purposes of qualitative data

What is qualitative data?  Fundamentally, it is data in the form of words rather than numbers, such as the responses to questions in interviews or surveys, the comments on trainee performance evaluation forms, notes from examining documents such as course syllabi, and notes based on observations in natural settings such as the operating room.

Here is an example of a response to an open-ended question in a survey about areas for improvement in a surgery rotation.

"Strength of rotation is chance to work independently, but that can be frustrating. There were times I really had no idea if I was doing what was expected of me or if I was learning at an appropriate pace.  Would encourage senior staff to give us a bit more feedback on how we are doing.  I wrote many daily notes on patients and wrote management plans, but never received any feedback if my plan was appropriate or if I was totally off base.  Would be nice if someone could review daily notes and the patient care plan with me sometime in the morning."

Notice the complex nuances embedded in this comment-issues related to independent work, the role of expectations, the role of feedback, organization for instruction, trainee self-confidence, and so on.  Qualitative data has long been a staple of some social sciences, such as anthropology and history.  It is increasingly used in medical education for program development, program evaluation, performance evaluation, and various scholarship applications; in turn, methods have been developed to achieve rigor in the analysis of qualitative data.  (Harris, 2002)

What purposes do qualitative data serve when compared to quantitative data?  For example, the mean rating (quantitative data) for a surgery rotation for overall quality may be 2.8 (when n=25, 5=Excellent, 4=Very Good, 3=Good, 2=Fair, and 1=Poor).  What do we know about the rotation?  We know that trainees are not very positive about the rotation, but we don’t know why.  By contrast, students’ comments, such as the comment above, may provide perspectives to explain, illuminate, supplement, validate, or reinterpret the quantitative data.

How to analyze qualitative data

By what method do we get from a long list of responses to open-ended questions about program effectiveness to a usable summary of results?  By what method do we get from a lengthy focus group transcript to a usable summary of results?  The fundamental process for analyzing and synthesizing qualitative data is to identify themes.  This can be done inductively by reading each comment in a set, and from this reading, identifying themes.  It can be done deductively.  If you have a set of themes developed based on prior course evaluations, you can use this set to assign comments to themes/categories, and additional themes can be added as needed.  An example below demonstrates this process.  Here are some of the comments for the surgery rotation above (with the mean rating of 2.8), in no particular order, in response to the following request:

Describe the weaknesses of this surgery rotation.

"Poor feedback and evaluation of performance."

"Sometimes mocking students during intraoperative pimping was rather humiliating.  Scrub nurses even commented that they felt badly for me."

"Expectations not entirely clear as far as following patients, writing notes, etc."

"Students weren’t involved in many initial work-ups."

"A lot of autonomy in this rotation, which proved problematic for finding out changes  in schedule, knowing exactly what to focus on."

"Few responsibilities delegated to students in the face of an exclusively private patient population."

"Would have liked clearer expectations about my role as student."

"Chief resident was quite fond of her new status.  Seemed to feel it was her job to belittle myself, other students, and nurses at every available opportunity.  Made my life miserable."

"More feedback would have been helpful.  I would have liked to have known area where I could have improved while there was still time to improve."

"Student role on team should be better defined, regarding note writing and staying until all work is done by all members of the team."

This is a set of possible themes to provide a summary of these comments.

Comments related to clarity of expectations for trainees (4)

"Would have liked clearer expectations about my role as student."  "Expectations not entirely clear as far as following patients, writing notes, etc."  "A lot of autonomy in this rotation, which proved problematic for finding out changes in schedule, knowing exactly what to focus on."  "Student role on team should be better defined, regarding note writing and staying until all work is done by all members of the team."

Comments related to trainee role in patient care as members of team (3)

"Few responsibilities delegated to students in the face of an exclusively private patient population."  "Student role on team should be better defined, regarding note writing and staying until all work is done by all members of the team."  "Students weren’t involved in many initial work-ups."

Comments related to the "belittling" of trainees (2)

"Sometimes mocking students during intraoperative pimping was rather humiliating.  Scrub nurses even commented that they felt badly for me."  "Chief resident was quite fond of her new status.  Seemed to feel it was her job to belittle myself, other students, and nurses at every available opportunity.  Made my life miserable."

Comments related to evaluation and feedback (2)

"Poor feedback and evaluation of performance."  "More feedback would have been helpful.  I would have liked to have known area where I could have improved while there was still time to improve."

For the surgery rotation with the mean rating of 2.8 (when 5=Excellent, 4=Very Good, 3=Good, 2=Fair, and 1=Poor), we only know that trainees are not very positive about the rotation.  With an analysis of the qualitative data, we learn that some of the problems in the rotation relate to clarity of expectations for trainees, trainee role and responsibility in the care team, the "belittling" of trainees, and evaluation and feedback on performance.  Identification of these problems provides a starting point for efforts to improve the rotation.

Standards for rigor in analyzing qualitative data

How do we know that our qualitative data analysis is trustworthy (that we can have trust in our identification of themes), valid (that we can have confidence in our interpretations), and reliable (that it is accurate)?  The standards for rigor used for quantitative data.  A number of methods have been developed for establishing the trustworthiness of qualitative data analysis (Harris, 2002; Miles and Huberman, 1994; Strauss and Corbin, 1999), and the following are   among the methods used:

• Peer Review (Audit Trail): Review of themes by peer(s)
• Member Checking: Checking themes and interpretations with individuals who were data sources
• Triangulation: Analyzing themes in relation to data from other sources, eg, the quantitative data in a survey, the results of a focus group discussion
• Saturation: Refining themes by reviewing additional comments until no new themes are identified

Conclusion

Using the various methods to collect data for program evaluation and improvement, two types of data are collected: (1) responses to structured questions, which are summarized in descriptive statistics (quantitative data) and (2) responses to open-ended questions, referred to as comments, narratives, or qualitative data.   Program directors, and their colleagues, may have trust in the quantitative data and perhaps dismiss the qualitative data as mere anecdotes.  Envision the following scenario: You are about to present the results of a survey evaluating your program to colleagues in your department.  The results include both quantitative data (descriptive statistics) for responses to structured questions and an analysis of responses to the open-ended questions.  One of your colleagues says, "Let’s just look at the numbers, the hard data, not the anecdotes."  You could now say, "The numbers are useful, but the comments that have been synthesized into themes, following standards for rigor in analysis of qualitative data, may provide us with perspectives to explain, illuminate, supplement, validate, or reinterpret the quantitative data so that we can better use our data to work on improving our program."

In this article, we have provided an overview of the nature and purposes of qualitative data, demonstrated how qualitative data may be analyzed to provide a useful synthesis of themes in evaluations for program improvement, and outlined standards for rigor in analyzing qualitative data.  We have provided a clear example of the fundamentals of qualitative data analysis.  There are, of course, issues and complexities in the use and analysis of qualitative data.  The resources listed below provide helpful conceptual perspectives and practical strategies for analyzing and using qualitative data for program improvement.

Resources

Harris I.  Qualitative Methods.  In Geoff Norman et al (eds).  International Handbook of Research in Medical Education.  Dordrecht: Kluwer, 2002.  pp. 45S95.

Miles M, Huberman AM.  Qualitative Data Analysis: An Extended Sourcebook (2^nd edition).  Thousand Oakes: Sage, 1994.

Strauss A, Corbin J.  Basics of Qualitative Research: Grounded Theory Procedures and Techniques.  Newbury Park: Sage, 1999.

 

 

Online November 14, 2007