REMINDER: The ACS would never email, call, or text asking for your personal details like IDs, passwords, or account numbers. Safeguard your cybersecurity by being vigilant against phishing messages, not connecting to public Wi-Fi, and verifying the identity of unexpected phone callers.
Unsupported Browser
The American College of Surgeons website is not compatible with Internet Explorer 11, IE 11. For the best experience please update your browser.
Menu
Become a member and receive career-enhancing benefits

Our top priority is providing value to members. Your Member Services team is here to ensure you maximize your ACS member benefits, participate in College activities, and engage with your ACS colleagues. It's all here.

Become a Member
Become a member and receive career-enhancing benefits

Our top priority is providing value to members. Your Member Services team is here to ensure you maximize your ACS member benefits, participate in College activities, and engage with your ACS colleagues. It's all here.

Membership Benefits
ACS
RISE

Learner Engagement when Utilizing the Virtual Platform for Education

Katharine E. Caldwell, MD, MSCI; Kevin Y. Pei, MD, MHSEd, FACS; Karen J. Dickinson MBBS, BSc, MD, MEd, FRCS

At the beginning of the COVID-19 pandemic, many people believed the transition to virtual learning would be temporary with post-pandemic return to full in-person education. However, with each additional variant and wave, demand persists for educational modalities that allow for asynchronous and remote options. Virtual learning enables training to continue during this time of educational disruption. Online platforms for surgical education have offered many benefits, including use of novel technologies, the ability for asynchronous or self-paced learning, and exploration of new assessment and teaching methods.1,2 Engagement has been identified as the most important variable in learning through online modalities, and appropriately, significant concerns have been raised about a learner’s ability to stay focused during virtual education.3,4 Online learning has been shown to lead to decreased levels of learner engagement.5 There have been many proposed contributors to this decline, including Zoom fatigue, increased demands at home, or distractions from the easy ability to click over to another window.6–10

Surgical educators have faced challenges with learner engagement during both the transition to virtual learning and continuation of this modality as the pandemic persists. Recent discourse within the online Association for Surgical Education (ASE) “DocMatters” community, a discussion forum for surgical educators to share experiences, has eludicated that many educators are discouraged by a wall of “black boxes” on zoom when delivering education online. Educators report struggling to engage learners, with examples including all learners participating with cameras off, learners not responding to direct questions, and the “awkward pause” after questions are asked. Further, many surgical educators have faced personal challenges due to the demands placed by virtual education during the pandemic. Examples include greater number of out of hours educational sessions, increased effort required to engage when teaching online, and the impact of the increased cognitive load of managing Q&A or chat functions while delivering education. Crowd sourcing educator opinion on platforms such as the ASE discussion forum are important as instituting virtual educational strategies can be variably successful, and it can be difficult for educators to determine which strategies are most effective for learner engagement and learning.

Educators have long understood that learner engagement and focus are the keys to information transfer, but work in this area has been challenged by the lack of a common definition of an engaged learner. A good working definition is “someone who has good learning strategies, a will to learn and who is involved behaviorally, emotionally and intellectually with their task.”11,12 The ability to effectively assess learner engagement is important to determine efficacy of educational strategies.  Initial tools to assess learner engagement were developed for elementary and secondary education learning environments.13,14 At the level of graduate medical education, the STROBE tool has been used to measure student engagement in the health professions classroom.15 Observational engagement tools can be valuable in determining the external behavior of a learner, but self-assessment is crucial to any effective engagement measure. It has previously been demonstrated that learners are able to “fake” engagement when necessary, and formal learner self-assessment must be incorporated in order to gauge a learner’s internal behavior.16

It is more challenging to assess learner engagement in the virtual environment than in-person and educators may struggle to utilize engagement tools designed for an in-person classroom in the virtual sphere due to differences in learner behavior. To combat these challenges, our team developed the Virtual In-class Engagement Measure (VIEM) to assess learner engagement in online surgical education events.17 This tool is an adaptation of the STROBE tool, consisting of two parts: (1) assessment of an online learning event by a trained observer and (2) learner self-assessment. Expectantly, engagement scores as measured by the VIEM were higher in activities that required learner participation (e.g., mock orals, journal clubs) and lower in activities in which learners were passive observers (e.g., lecture-based). Additionally, VIEM scores demonstrated a positive correlation with number of questions asked during sessions and participation in the chat. Interestingly, on learner self-assessment, we found that up to 20% of learners across multiple educational activities reported that they pretended to participate15. On the online platform, fake participation continues to be the most significant limitation in accurate assessment of learner engagement.16,17

While no single method to increase engagement will be successful with all learner groups, several strategies, anchored in active learning principles, have been utilized (see Table 1).  Some—such as the use of case-based learning or flipped classroom models—may be familiar to surgical educators due to their previous use in in-person learning spaces, but others may require more preparation on the part of the educator. Each strategy offers benefits and challenges, with some being more suitable for certain learner types or learning events, and we recommend experimentation combined with evaluation of success through learner engagement tools, with revision of tactics as required.

Table 1. Suggested Active Learning Strategies for Increasing Virtual Learner Engagement

Strategy

Examples

Benefits

Challenges

Gamification

Gamification software (TopHat, Kahoot)

Jeopardy 

  • Online gamification software allows learners to score points and naturally encourage competition.
  • Consider “upping the stakes” with prizes for winning individuals or teams. 
  • Potential cost depending on software chosen.
  • Additional preparation required to create games.
  • Learner buy-in.

Small Group Sessions

Breakout rooms

Pre-classroom small groups  

  • Learners can be easily moved into small groups using zoom breakout rooms to facilitate discussion.
  • Some learners may feel more comfortable speaking up in small groups vs. large groups.
  • Consider breaking up learners by stage (i.e. by PGY year) or integrating learners of different stages.
  • Consider use of a moderator (faculty or resident) within each room to keep groups on task and to facilitate stalled discussions.
  • Learners may meet in small groups first to discuss their ideas regarding a topic in a “low-stakes” way before a larger group session.
  • Educators should consider strategies to optimize psychological safety of learners within groups, especially if hierarchies within learner groups exist.
  • Increased requirement for personnel if facilitators within each room.
  • Learners may struggle to engage before receiving more formal instruction.
  • May require facilitators to monitor and promote discussion.

Flipped classroom 

PDF, video review prior to the learning event
  • Learners may familiarize themselves with the material before session allowing for more comfort answering/asking questions.
  • Richer discussion if learners are prepared.
  • Educators need to circulate material with enough time for learners to prepare.
  • May require protected learner time to prepare.
  • Requires some method to monitor learner completion of pre-course materials.
  • Learner buy-in.

Q&A Based Sessions

Case-based presentations
  • Encourages learner participation by providing dedicated time within the teaching for learners to answer questions.
  • Allows learners to test their knowledge and identify knowledge gaps.
  • Learner buy-in.
  • Requires adequate preparation by learners to enable them to answer questions in a way that facilitates group learning.

Polling Software

PollEverywhere
  • Learners may be more comfortable engaging anonymously.
  • Allows all learners in a group to get feedback on their own learning.
  • Learners must commit to an answer.
  • Assesses collective understanding of a topic so educator can spend more/less time explaining.
  • Allows word cloud options to enable visual representation of concepts.
  • May require learners to access website/download App.
  • Requires learners to participate using a smart device during the session.
  • Educator requires software and pre-event rehearsal to ensure no technical issues.
  • Educator must prepare the polling feature before the event.

Expectation Setting

“Cameras on”
  • Learners may be more likely to engage when engagement expectations are set e.g.,encouraging, but not requiring, learners to turn their cameras on.
  • Indicating at the outset that questions will be asked and who they will be asked of (random calling, structured set-up, open to all) so learners expect this and engage.
  • Learners know how to answer and therefore educational time is not wasted with logistical questions.
  • If educator encounters or is aware of more introverted learners within the group, more appropriate engagement strategies such as the chat can be employed to enrich discussion.
  • Learners may not turn cameras on despite expectations being discussed.
  • Learners may not engage with chat.
  • Certain types of learner may be intimidated by this unless a clear safe space is created for discussion.

Assistance

Moderators 
  • Moderators of small group breakout rooms can help facilitate discussion vs. a single educator who cannot be present in all rooms.
  • An assistant can monitor the chat and read out questions so the educator does not have to attempt to monitor the chat and lecture to prevent cognitive overload.
  • Education champions can be encouraged start the discussion by asking a question so others feel comfortable to ask.
  • Increased personnel requirements.
  • Need for all educators/ moderators/ assistants/champions to have a pre-event “huddle” to ensure all are adequately prepared and understand their roles. This may be challenging to coordinated based on schedules.

Unique Set Ups 

Role play 
  • Leverages active learning strategies.
  • Encourages healthy competition in some scenarios.
  • Allows peer-to-peer learning.
  • More preparation may be required by the educator—role play requires scenario writing and assessment rubric, mock orals require cases and examiners.
  • Journal clubs require learners to prepare to provide a considered critique.

Virtual learning is likely to be a permanent fixture in medical education. Soon, a generation of medical students whose majority of medical school training has been completed virtually will be entering the clinical environment. It is imperative that residency educators are ready to meet the needs of these learners. Understanding the engagement of learners is essential to understanding both whether information is transmitted and the efficacy of learning; there is much left to study in terms of the impact of virtual engagement on competency. We suggest that formal assessment of learner engagement is even more important in for virtual learning events, where it may be challenging for an educator to evaluate the engagement of their learners in real time. Formal assessment of learner engagement can be used to evaluate the effectiveness of certain educational strategies and to help determine which learning points are unclear, in order to optimize the experience for both students and educators. As we enter the post-pandemic educational landscape, evaluating the efficacy of virtual and hybrid educational strategies will be essential to determine the best pedagogical approach for surgical education in the future.

References

  1. Chick RC, Clifton GT, Peace KM, et al. Using Technology to Maintain the Education of Residents During the COVID-19 Pandemic. Journal of surgical education. 2020;77(4):729-732. doi:10.1016/J.JSURG.2020.03.018
  2. Dedeilia A, Sotiropoulos MG, Hanrahan JG, Janga D, Dedeilias P, Sideris M. Medical and Surgical Education Challenges and Innovations in the COVID-19 Era: A Systematic Review. In Vivo. 2020;34(3 Suppl):1603. doi:10.21873/INVIVO.11950
  3. Bientzle M, Hircin E, Kimmerle J, et al. Association of Online Learning Behavior and Learning Outcomes for Medical Students: Large-Scale Usage Data Analysis. JMIR medical education. 2019;5(2). doi:10.2196/13529
  4. Dickinson K, Caldwell K, Graviss E, et al. Perceptions and behaviors of learner engagement with virtual educational platforms. American Journal of Surgery. 2022:S0002-9610(22)00102-7. doi:10.1016/j.amjsurg.2022.02043
  5. Challenge Success. Kids Under Pressure: A Look at Student Well-Being and Engagement During the Pandemic. Published online February 2021. Accessed February 2, 2022. https://challengesuccess.org/wp-content/uploads/2021/02/CS-NBC-Study-Kids-Under-Pressure-PUBLISHED.pdf
  6. Wiederhold BK. Connecting Through Technology During the Coronavirus Disease 2019 Pandemic: Avoiding “Zoom Fatigue.” Cyberpsychology, behavior and social networking. 2020;23(7):437-438. doi:10.1089/CYBER.2020.29188.BKW
  7. Singh Chawla D. Zoom fatigue saps grant reviewers’ attention. Nature. 2021;590(7844):172. doi:10.1038/D41586-021-00161-5
  8. Muric G, Lerman K, Ferrara E. Gender Disparity in the Authorship of Biomedical Research Publications During the COVID-19 Pandemic: Retrospective Observational Study. Journal of medical Internet research. 2021;23(4). doi:10.2196/25379
  9. Ipe TS, Goel R, Howes L, Bakhtary S. The impact of COVID-19 on academic productivity by female physicians and researchers in transfusion medicine. Transfusion. 2021;61(6):1690-1693. doi:10.1111/TRF.16306
  10. Robinson LJ, Engelson BJ, Hayes SN. Who Is Caring for Health Care Workers’ Families Amid COVID-19? Academic medicine : journal of the Association of American Medical Colleges. 2021;96(9):1254-1258. doi:10.1097/ACM.0000000000004022
  11. O’Malley KJ, Moran BJ, Haidet P, et al. Validation of an observation instrument for measuring student engagement in health professions settings. Evaluation & the health professions. 2003;26(1):86-103. doi:10.1177/0163278702250093
  12. Järvelä S, Järvenoja H, Malmberg J, Isohätälä J, Sobocinski M. How do types of interaction and phases of self-regulated learning set a stage for collaborative engagement? Learning and Instruction. 2016;43:39-51. doi:10.1016/J.LEARNINSTRUC.2016.01.005
  13. Stallings J. Needles M. Stallings Observation Instrument Revised Ed.; 1985.
  14. Sattler JM. Assessment of Children. 3rd ed.; 1988.
  15. Alimoglu MK, Sarac DB, Alparslan D, Karakas AA, Altintas L. An observation tool for instructor and student behaviors to measure in-class learner engagement: a validation study. Medical education online. 2014;19(1). doi:10.3402/MEO.V19.24037
  16. Fuller KA, Karunaratne NS, Naidu S, et al. Development of a self-report instrument for measuring in-class student engagement reveals that pretending to engage is a significant unrecognized problem. PLOS ONE. 2018;13(10):e0205828. doi:10.1371/JOURNAL.PONE.0205828
  17. Dickinson KJ, Caldwell KE, Graviss EA, et al. Assessing learner engagement with virtual educational events: Development of the Virtual In-Class Engagement Measure (VIEM). American journal of surgery. 2021;222(6):1044-1049. doi:10.1016/J.AMJSURG.2021.09.021
  18. Alimoglu MK, Sarac DB, Alparslan D, Karakas AA, Altintas L. An observation tool for instructor and student behaviors to measure in-class learner engagement: a validation study. http://dx.doi.org.beckerproxy.wustl.edu/103402/meo.v1924037. 2014;19(1). doi:10.3402/MEO.V19.24037
katharinecaldwell.png

About the Author

Katharine E. Caldwell, MD, MSCI, Washington University in St. Louis, Department of General Surgery, St. Louis, MO; Kevin Y. Pei, MD, MHSEd, FACS, Department of Graduate Medical Education, Parkview Health, Fort Wayne, IN; Karen J. Dickinson, MBBS, BSc, MD, MEd, FRCS, University of Arkansas for Medical Sciences, Department of Surgery, Little Rock, AR