Graduate Follow-up Surveys Are a Good Idea

Brian E. Mavis, PhD
Office of Medical Education Research and Development
Michigan State University

Graduates represent a unique perspective as the individuals who have experienced the curriculum from beginning to end; they represent the product of our collective educational efforts.¹ In addition, they have been shaped by recruitment processes, support and advising, and other components that complement the curriculum. After graduation, former students can reflect on their experience from a new perspective with the added benefit of comparing their experiences to those of graduates from other programs. For these reasons, many medical schools and residency training programs have implemented graduate follow-up surveys. There are generally three types of questions that can be addressed through the use of graduate survey data:

1. Where do our graduates practice and who are their patients? Information derived from questions such as geographic location, practice types, employment setting, and patient characteristics can be used to document outcomes in support of public policy analyses to justify funding, as well as to determine the extent to which programs are achieving mission-related goals.²
2. How well did we prepare our graduates? This question addresses the training adequacy of the curriculum for meeting the needs of graduates: for medical school graduates, this is often in terms of preparation for residency, while for graduates of residency programs, the focus is on the usefulness of the residency curriculum for post-residency practice.² The adequacy of training can include the importance of the inclusion of various components of training for subsequent practice as well as the sufficiency of the preparation provided.³
3. To what extent are program changes and curricular innovations evident in graduate outcomes? Graduate surveys can be used to monitor outcomes associated with curricular changes adopted over time, or to compare programs incorporating curricular changes that distinguish them from more traditional programs.³

Information provided by program graduates is an integral part of program evaluation efforts to monitor program outcomes and enhance program quality. Alumni feedback is useful for the reassessment and reorientation of education and training programs.⁴ Many educational programs use graduate survey data to document outcomes related to accreditation standards.⁵ Additionally, graduate surveys communicate that program leaders are interested in current and former trainees' opinions, and can yield useful information for program recruitment. For new program directors, graduate surveys can be used as part of a baseline dataset for planning their own activities.⁶

While gathering information about performance from learners enrolled in a program is relatively simple, it is much more challenging to design and implement a system for collecting information once they have graduated.¹ The implementation of graduate follow-up surveys requires thoughtful planning as well as human and technical resources. A variety of issues related to sampling, content, informed consent, format, and reporting need to be addressed in order to create an effective system for follow-up of graduates.

Survey Purpose

As suggested by the questions above, a survey of program graduates can be used for multiple purposes. Educational goals and objectives, mission statements and other internal documents are a good resource for identifying performance outcomes, educational strategies, curricular content and other features around which to structure questions.⁵ Also, considering the audience for whom the data are being gathered can help set priorities among the issues and questions of importance. Clearly defining the purpose of the survey and the kinds of information needed to answer significant questions will help focus the structure of the questionnaire and the individual questions comprising the content. This is an important step in creating a questionnaire that engages the respondent insofar as it makes sense, is a reasonable length, and has relevance.

Structure and Format

In structuring the content of the questionnaire, the use of subtitles and directions are important as guides to the respondent, helping to communicate the logic underlying the document. Questions related by content should be grouped together. Generally questions fall into two types: factual and subjective. Questions designed to gather factual data focus on information that could be objectively verified, although objective verification is seldom practical, hence the use of a self-report survey. Subjective questions include those related to feelings, opinions, or judgments, and are frequently measured through the use of rating scales. There is both art and science involved in writing good survey questions; many resources (listed below) are available to guide the development of questions that are clear and minimize respondent bias.

Sampling

Some programs regularly survey cohorts of graduates and use the information as part of an ongoing system of program evaluation that allows monitoring changes over time. In some cases, graduate surveys are only used periodically, as a means of collecting information to address an issue of concern. In either case, the target audience is well defined; the challenge is finding contact information for graduates. The longer the time since graduation, the more likely graduates will have moved or otherwise become more difficult to locate. Many professional organizations maintain databases that can be used to help locate graduates. Once located, various strategies have been used to maximize response rates, including personalized cover letters, incentives, and multiple mailings. The review by Parrino and Kern reported response rates ranging from 50 percent to 90 percent for mail surveys to graduates of Internal Medicine programs.⁴ Advances in e-mail and Web-based surveys present new opportunities that can have an impact on cost and response time for data collection.^{7, 8}

Troubleshooting

The single easiest strategy for avoiding problems in survey implementation is through pilot testing. The first phase involves having several individuals proofread the questionnaire, with particular attention to content, item wording, and the response options used for the questions. Ideally, these should be people who have not had a significant role in the development of the questionnaire, but have some knowledge about the program. The second phase involves distributing the questionnaire to a small group of individuals similar to the respondent group, and having them complete the survey as well as critique the content in terms of relevance and meaning. Current students or recent graduates can provide insight into the survey and any misinterpretation of questions that often can be easily remedied prior to full implementation.

Informed Consent

Graduate surveys represent an important component of continuous quality improvement efforts by gathering information for judging the training process.⁴ Information gathered for program monitoring or improvement is not typically considered research, and in many cases does not require internal review board (IRB) approval. A noteworthy exception is when the information gathered will be used for scholarship, whether conference presentations, posters, manuscripts, or some other form.⁹ If there is any potential at all that the information gathered from a graduate survey will result in generalizable knowledge that can be used for scholarship, consult with your local IRB. In many cases, this type of data collection does not require a full committee review and would be eligible for expedited or exempt review.

Limitations

As with any information source, graduate surveys are subject to biases and the interpretation of the results subject to limitations. Foremost, survey research is based on respondents' self-reports. The accuracy of personal self-reports depend on many factors related to survey sponsorship, question wording, perceived confidentiality, etc. The conclusions that can be drawn from survey data also are limited by the response rate: the lower the response rate, the less likely that data are representative of the target group as a whole. Program evaluation strategies that rely on multiple data sources (faculty, current students, patients, etc) can provide more accurate information by bringing multiple perspectives to address questions related to training program outcomes.

Resources

To the extent that there is careful deliberation at each step of survey implementation, many problems can be avoided to result in high quality information. Many offices of medical education have individuals experienced with survey development and implementation, and who can provide consultation. In addition, there are many print and Web-based resources available:

• The Survey System: http://www.surveysystem.com/sdesign.htm
• Council of American Survey Research Organizations: http://www.casro.org/guidelines.cfm
• How to design a questionnaire: http://www.studentbmj.com/back_issues/0601/education/187.html
• American Statistical Association: http://www.amstat.org/sections/srms/brochures/designquest.pdf
• Alreck, P. & Settle, R. The survey research handbook. Irwin/McGraw-Hill, 1995.
• Dillman, D. Mail and Internet surveys: the tailored design method. Second Edition. Wiley, 2000.
• Fowler, F. Improving survey questions. Sage Publications, 1995.
• Stone, D. Designing a questionnaire. British J Medicine. 1994; 308 (6922): 202-3.

References

1. Distlehorst L. What graduate follow-up studies can tell us. Medical Education. 2000; 34: 976-7.
2. Davidson R, Nesbitt T & Fox-Garcia J. Trend analysis of changes in family practice residency graduates' practice characteristics and comparison with recent graduates. Family Medicine. 1995; 27: 112-6.
3. Kiel, D., O'Sullivan, P., Ellis, P. & Wartman, S. Alumni perspectives comparing a general internal medicine program and a traditional medicine program. Journal of General Internal Medicine. 1991; 6: 544-52.
4. Parrino, T. and Kern, D. The alumni survey as an instrument for program evaluation in internal medicine. Journal of General Internal Medicine. 1994; 9: 92-5.
5. Richter, R. & Ruebling, I. Model for development of outcome assessment surveys for allied health educational programs. Journal of Allied Health. 2003; 32: 179-84.
6. Dale, D. Alumni surveys for evaluation of innovations in medical education. Journal of General Internal Medicine. 1991; 6: 587.
7. Mavis, B. & Brocato, J. Postal surveys versus electronic mail surveys: The tortoise and the hare revisited. Evaluation and the Health Professions. 1998; 21 (3): 395-408.
8. Dillman, D. Mail and Internet Surveys: The Tailored Design Method. Second Edition. Wiley and Sons: New York, 2000.
9. Henry, R. & Wright, D. When do medical students become human subjects of research? The case of program evaluation. Academic Medicine. 2001; 76(9): 871-875.