Reviewing Qualitative Research in Surgical Education Literature

• Qualitative research enriches understanding of complex phenomena in education and requires a different analytic approach than quantitative studies for both readers and reviewers.
• Evaluation of qualitative education literature should include consideration of the research question, study design, methodology, data collection, analysis techniques, and presentation of data.
• Reference tools may aid in evaluating the rigor of qualitative education literature.

Medical education research has increased since its recognition as a distinct field of study.¹ The field is diverse and includes quantitative, qualitative, and mixed-methods methodologies. This practical guide is a resource for surgeons to strengthen their understanding and evaluation of qualitative research by focusing on study design and methodology, data collection and analysis, and limitations. It may be used as a resource for reading and interpreting qualitative research and providing peer review feedback.

Qualitative research differs from quantitative research in many important ways. The most profound difference is the epistemology—or view on reality—that each technique is founded upon. Quantitative research is based upon objectivism or positivism, the belief in the existence of an absolute truth that, through effective research, can be discovered or described.² Alternatively, qualitative research is rooted in the philosophy of constructivism, the view that reality is a construct of historical, social, and individual perspectives. Therefore, the focus is interpreting or understanding social settings and/or human experiences.³ Research questions that seek to understand, explore, describe, or explain experiences are suited for qualitative research. Examples include questions starting with "how," "what," or "why," which are not fully answered using numbers and statistical analyses.³ Commonly used qualitative methodological approaches include case study, ethnography, grounded theory, and phenomenology (Table 1).

Table 1: Commonly Used Qualitative Methodological Approaches

The goal of data collection in qualitative research is depth of understanding, which places particular importance on subject selection and engagement.⁷ Subjects can be selected through a variety of sampling methods (Table 2).

Table 2: Common Sampling Methods²

Though differences exist in each methodological approach, similar techniques are used for data collection. Common techniques include content analysis, interviews, and direct observation of social interactions. Unlike power analyses that dictate sample size in quantitative research, the concept of saturation drives the data acquisition process in qualitative research. Saturation is achieved when the data collection process no longer uncovers novel insights or themes.⁶

Qualitative manuscripts should clearly describe the data analysis process. There are many ways to investigate a question. The quality of data analysis does not depend on a single approach or interpretation. Each of the approaches can be equally rigorous. The process typically begins with a systematic review, or "coding," of data. Coding represents the link between data and the investigator's interpretation of those data. Coding may be done inductively or deductively.⁵ An explanation should be provided for why an approach was used, based on the research question and methodology. Components of the analysis process for interview data may be described as follows:

Initial codes emerged while reviewing a sampling of data focused on the research question. Using the initial codes, a codebook was developed, which was iteratively refined as further data was reviewed. Similar codes were grouped to form categories, which were used to structure the codebook.

Qualitative software can help support and organize the coding process but using software is not required. Once data are coded, investigators look for themes emerging from the categories. A simplified example of the analysis process is illustrated in Figure 1.

Figure 1. Example of thematic analysis process for qualitative interview data

An analysis may include a variable number of themes. Traditionally, two to five themes are included. The significance of a theme relates to the ability to address the research question, not how often it appears. For studies with multiple coders, the Kappa coefficient may be used to determine intercoder reliability or the consistency among coders to apply the same codes to the same text.⁸ Investigators may use additional techniques to promote quality in the analysis process (Table 3).

Table 3. Examples of Quality Techniques

Study results should be presented sequentially and include detailed explanations of the themes identified, paired with data, such as illustrative participant quotes. Data may be presented either within the text, as a table, or in a figure, such as a concept map. Data selected to highlight themes should be logical and transparent to both investigators and readers. Based on the presentation of themes, investigators may create a model or framework, which can illustrate the results and how results add and relate to current literature.

Specific considerations apply to the discussion of qualitative results. The human and social experiences studied in qualitative research are complex. The research product is not expected or required to describe every facet of the phenomenon under investigation. Results should reflect the concept of dependability in that they are consistent with the context in which they were created.¹⁰ Results should also reflect credibility in the sense that they are trustworthy and believable. Investigators should further discuss the transferability to other settings of the facets they do address. This concept aligns with the concept of generalizability in quantitative research.¹⁰ As qualitative data are obtained by the investigator, the investigator's background and experience can influence how they interpret, process, and code data, including coding differently than co-investigators, or from participants' intent. This influence should be clearly described, and the confirmability of the findings should be based on whether the results reflect participant responses as opposed to investigator biases.¹⁰ Critical readers will interpret the results based on their perception of the rigor of the research (Table 4).⁴

Table 4. Aspects of Rigor⁴

Formal and informal checklists exist to guide study design and writing and can assist in the peer review of qualitative research. These checklists are limited in application and do not guarantee quality research. However, they can be used as reference tools for constructing and/or evaluating the methodology and reporting of qualitative studies. The two most utilized and cited tools are the Standards for Reporting Qualitative Research (SRQR), and the Consolidated Criteria for Reporting Qualitative Research (COREQ).^11,12 Both require a fundamental understanding of the components of qualitative research and may be used when designing or reviewing qualitative studies.

Qualitative research is an important component of medical education that can expand knowledge and deepen understanding of complex phenomena. Surgical educators should develop skills to understand, interpret, and review qualitative education research.

The authors would like to thank Dr. Roy Phitayakorn for facilitating collaboration on this article.

1. Reeves S, Peller J, Goldman J, Kitto S. Ethnography in qualitative educational research: AMEE Guide No. 80. Medical Teacher. 2013;35(8). doi:10.3109/0142159X.2013.804977
2. Sawatsky AP, Ratelle JT, Beckman TJ. Qualitative Research Methods in Medical Education. Anesthesiology. 2019;131(1):14-22. doi:10.1097/ALN.0000000000002728
3. Cleland JA. The qualitative orientation in medical education research. Korean Journal of Medical Education. 2017;29(2):61-71. doi:10.3946/kjme.2017.53
4. Cristancho SM, Goldszmidt M, Lingard L, Watling C. Qualitative research essentials for medical education. Singapore Medical Journal. 2018;59(12):622-627. doi:10.11622/smedj.2018093
5. Kiger ME, Varpio L. Thematic analysis of qualitative data: AMEE Guide No. 131. Medical Teacher. 2020;42(8):846-854. doi:10.1080/0142159X.2020.1755030
6. Stalmeijer R, McNaughton N, van Mook W. Using focus groups in medical education research: AMEE Guide No. 91. Medical Teacher. 2014;(36):923-939.
7. Cleland JA, Durning SJ, Driessen E. Medical education research: Aligning design and research goals. Medical Journal of Australia. 2018;208(11). doi:10.5694/mja17.00941
8. O'Connor C, Joffe H. Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines. International Journal of Qualitative Methods. 2020;19:160940691989922. doi:10.1177/1609406919899220
9. Mann K, Macleod A. Constructivism: Learning Theories and Approaches to Research.; 2015.
10. Frambach JM, Cees PM van der V, Durning S. Quality Criteria in Qualitative and Quantitative Research. Academic Medicine. 2013;88(4):552.
11. O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for Reporting Qualitative Research. Academic Medicine. 2014;89(9):1245-1251. doi:10.1097/ACM.0000000000000388
12. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care. 2007;19(6):349-357. doi:10.1093/intqhc/mzm042

Robert D. Sinyard, MD, MBA, is a general surgery resident currently completing a surgery education research and simulation fellowship at Massachusetts General Hospital, Harvard Medical School in Boston, MA. 

Roi Anteby, MD, is a postdoctoral surgery education research and simulation fellow at Massachusetts General Hospital, Harvard Medical School in Boston, MA.

Carl Gustaf S. Axelsson, MD, MPhil, MMSc, is a postdoctoral medical education researcher at Massachusetts General Hospital, Harvard Medical School in Boston, MA.

Michael G. Healy, EdD, is a research fellow at Massachusetts General Hospital, Harvard Medical School in Boston, MA.

Halle B. Ellison, MD, FACS, is an assistant professor of surgery and palliative care at Geisinger Commonwealth School of Medicine, Danville, PA.