Unsupported Browser
The American College of Surgeons website is not compatible with Internet Explorer 11, IE 11. For the best experience please update your browser.
Become a member and receive career-enhancing benefits

Our top priority is providing value to members. Your Member Services team is here to ensure you maximize your ACS member benefits, participate in College activities, and engage with your ACS colleagues. It's all here.

Become a Member
Become a member and receive career-enhancing benefits

Our top priority is providing value to members. Your Member Services team is here to ensure you maximize your ACS member benefits, participate in College activities, and engage with your ACS colleagues. It's all here.

Membership Benefits
ACS
RISE

Reviewing Qualitative Research in Surgical Education Literature

Robert D. Sinyard, MD, MBA; Roi Anteby, MD; Carl Gustaf S. Axelsson, MD, MPhil, MMSc; Michael G. Healy, EdD; Halle B. Ellison, MD, FACS

September 1, 2021

Key Learning Points

  • Qualitative research enriches understanding of complex phenomena in education and requires a different analytic approach than quantitative studies for both readers and reviewers.
  • Evaluation of qualitative education literature should include consideration of the research question, study design, methodology, data collection, analysis techniques, and presentation of data.
  • Reference tools may aid in evaluating the rigor of qualitative education literature.

Introduction

Medical education research has increased since its recognition as a distinct field of study.1 The field is diverse and includes quantitative, qualitative, and mixed-methods methodologies. This practical guide is a resource for surgeons to strengthen their understanding and evaluation of qualitative research by focusing on study design and methodology, data collection and analysis, and limitations. It may be used as a resource for reading and interpreting qualitative research and providing peer review feedback.

Qualitative Methodologies

Qualitative research differs from quantitative research in many important ways. The most profound difference is the epistemology—or view on reality—that each technique is founded upon. Quantitative research is based upon objectivism or positivism, the belief in the existence of an absolute truth that, through effective research, can be discovered or described.2 Alternatively, qualitative research is rooted in the philosophy of constructivism, the view that reality is a construct of historical, social, and individual perspectives. Therefore, the focus is interpreting or understanding social settings and/or human experiences.3 Research questions that seek to understand, explore, describe, or explain experiences are suited for qualitative research. Examples include questions starting with "how," "what," or "why," which are not fully answered using numbers and statistical analyses.3 Commonly used qualitative methodological approaches include case study, ethnography, grounded theory, and phenomenology (Table 1).

Table 1: Commonly Used Qualitative Methodological Approaches

Case Study

Seeks to understand a case or case-series that reflects an underlying problem of interest4

Ethnography

Seeks to understand people in their cultural and social setting4

Grounded Theory

Seeks to understand social processes in order to inform development of new theory4,5

Phenomenology

Seeks to understand the meaning of individuals' lived experiences6

The goal of data collection in qualitative research is depth of understanding, which places particular importance on subject selection and engagement.7 Subjects can be selected through a variety of sampling methods (Table 2).

Table 2: Common Sampling Methods2

Though differences exist in each methodological approach, similar techniques are used for data collection. Common techniques include content analysis, interviews, and direct observation of social interactions. Unlike power analyses that dictate sample size in quantitative research, the concept of saturation drives the data acquisition process in qualitative research. Saturation is achieved when the data collection process no longer uncovers novel insights or themes.6

Convenience Sampling

Subjects are selected based on availability and accessibility to the research team

Purposive Sampling

Subjects are selected based on pertinent experiences or perspectives

Random Sampling

Subjects are selected completely by chance from the general population or a subset of interest

Snowball Sampling

Selection of future subjects is based upon the recommendations of current subjects

Qualitative Data Analysis

Qualitative manuscripts should clearly describe the data analysis process. There are many ways to investigate a question. The quality of data analysis does not depend on a single approach or interpretation. Each of the approaches can be equally rigorous. The process typically begins with a systematic review, or "coding," of data. Coding represents the link between data and the investigator's interpretation of those data. Coding may be done inductively or deductively.5 An explanation should be provided for why an approach was used, based on the research question and methodology. Components of the analysis process for interview data may be described as follows:

Initial codes emerged while reviewing a sampling of data focused on the research question. Using the initial codes, a codebook was developed, which was iteratively refined as further data was reviewed. Similar codes were grouped to form categories, which were used to structure the codebook.

Initial codes emerged while reviewing a sampling of data focused on the research question. Using the initial codes, a codebook was developed, which was iteratively refined as further data was reviewed. Similar codes were grouped to form categories, which were used to structure the codebook.

Qualitative software can help support and organize the coding process but using software is not required. Once data are coded, investigators look for themes emerging from the categories. A simplified example of the analysis process is illustrated in Figure 1.

Figure 1. Example of thematic analysis process for qualitative interview data

Figure 1. Example of thematic analysis process for qualitative interview data
Figure 1. Example of thematic analysis process for qualitative interview data

An analysis may include a variable number of themes. Traditionally, two to five themes are included. The significance of a theme relates to the ability to address the research question, not how often it appears. For studies with multiple coders, the Kappa coefficient may be used to determine intercoder reliability or the consistency among coders to apply the same codes to the same text.8 Investigators may use additional techniques to promote quality in the analysis process (Table 3).

Table 3. Examples of Quality Techniques

Audit Trail

Maintaining detailed research records, including changes occurring during the research process9

Member Checking

Inviting participant feedback on data and/or interpretation of data10

Triangulation

Combining different perspectives, methods, and data sources to verify convergence on specific themes4

Presentation of Data and Discussion

Study results should be presented sequentially and include detailed explanations of the themes identified, paired with data, such as illustrative participant quotes. Data may be presented either within the text, as a table, or in a figure, such as a concept map. Data selected to highlight themes should be logical and transparent to both investigators and readers. Based on the presentation of themes, investigators may create a model or framework, which can illustrate the results and how results add and relate to current literature.

Specific considerations apply to the discussion of qualitative results. The human and social experiences studied in qualitative research are complex. The research product is not expected or required to describe every facet of the phenomenon under investigation. Results should reflect the concept of dependability in that they are consistent with the context in which they were created.10 Results should also reflect credibility in the sense that they are trustworthy and believable. Investigators should further discuss the transferability to other settings of the facets they do address. This concept aligns with the concept of generalizability in quantitative research.10 As qualitative data are obtained by the investigator, the investigator's background and experience can influence how they interpret, process, and code data, including coding differently than co-investigators, or from participants' intent. This influence should be clearly described, and the confirmability of the findings should be based on whether the results reflect participant responses as opposed to investigator biases.10 Critical readers will interpret the results based on their perception of the rigor of the research (Table 4).4

Table 4. Aspects of Rigor4

Reflexivity

Has the investigator's perspective about and influence on the work been considered?

Adequacy

Was the data sample adequate to represent the breadth, depth, and nature of the phenomenon being studied?

Authenticity

Is the data an authentic reflection of the phenomenon being studied?

Trustworthiness

Are analysis techniques described clearly and carried out systematically?

Resonance

Are interpretations of data meaningful for those who live in the social experience studied?

Tools for Evaluating Qualitative Research

Formal and informal checklists exist to guide study design and writing and can assist in the peer review of qualitative research. These checklists are limited in application and do not guarantee quality research. However, they can be used as reference tools for constructing and/or evaluating the methodology and reporting of qualitative studies. The two most utilized and cited tools are the Standards for Reporting Qualitative Research (SRQR), and the Consolidated Criteria for Reporting Qualitative Research (COREQ).11,12 Both require a fundamental understanding of the components of qualitative research and may be used when designing or reviewing qualitative studies.

Conclusion

Qualitative research is an important component of medical education that can expand knowledge and deepen understanding of complex phenomena. Surgical educators should develop skills to understand, interpret, and review qualitative education research.

Acknowledgements

The authors would like to thank Dr. Roy Phitayakorn for facilitating collaboration on this article.

References

  1. Reeves S, Peller J, Goldman J, Kitto S. Ethnography in qualitative educational research: AMEE Guide No. 80. Medical Teacher. 2013;35(8). doi:10.3109/0142159X.2013.804977
  2. Sawatsky AP, Ratelle JT, Beckman TJ. Qualitative Research Methods in Medical Education. Anesthesiology. 2019;131(1):14-22. doi:10.1097/ALN.0000000000002728
  3. Cleland JA. The qualitative orientation in medical education research. Korean Journal of Medical Education. 2017;29(2):61-71. doi:10.3946/kjme.2017.53
  4. Cristancho SM, Goldszmidt M, Lingard L, Watling C. Qualitative research essentials for medical education. Singapore Medical Journal. 2018;59(12):622-627. doi:10.11622/smedj.2018093
  5. Kiger ME, Varpio L. Thematic analysis of qualitative data: AMEE Guide No. 131. Medical Teacher. 2020;42(8):846-854. doi:10.1080/0142159X.2020.1755030
  6. Stalmeijer R, McNaughton N, van Mook W. Using focus groups in medical education research: AMEE Guide No. 91. Medical Teacher. 2014;(36):923-939.
  7. Cleland JA, Durning SJ, Driessen E. Medical education research: Aligning design and research goals. Medical Journal of Australia. 2018;208(11). doi:10.5694/mja17.00941
  8. O'Connor C, Joffe H. Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines. International Journal of Qualitative Methods. 2020;19:160940691989922. doi:10.1177/1609406919899220
  9. Mann K, Macleod A. Constructivism: Learning Theories and Approaches to Research.; 2015.
  10. Frambach JM, Cees PM van der V, Durning S. Quality Criteria in Qualitative and Quantitative Research. Academic Medicine. 2013;88(4):552.
  11. O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for Reporting Qualitative Research. Academic Medicine. 2014;89(9):1245-1251. doi:10.1097/ACM.0000000000000388
  12. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care. 2007;19(6):349-357. doi:10.1093/intqhc/mzm042

About the Authors

Robert D. Sinyard, MD, MBA, is a general surgery resident currently completing a surgery education research and simulation fellowship at Massachusetts General Hospital, Harvard Medical School in Boston, MA. 

Roi Anteby, MD, is a postdoctoral surgery education research and simulation fellow at Massachusetts General Hospital, Harvard Medical School in Boston, MA.

Carl Gustaf S. Axelsson, MD, MPhil, MMSc, is a postdoctoral medical education researcher at Massachusetts General Hospital, Harvard Medical School in Boston, MA.

Michael G. Healy, EdD, is a research fellow at Massachusetts General Hospital, Harvard Medical School in Boston, MA.

Halle B. Ellison, MD, FACS, is an assistant professor of surgery and palliative care at Geisinger Commonwealth School of Medicine, Danville, PA.