Artificial intelligence (AI) has rapidly swept into many aspects of everyday life, including the OR.
On Monday, the Panel Session “Surgical Application of Artificial Intelligence/Computer Vision in the OR” (PS215) presented AI technologies for surgical care, with all speakers emphasizing that the best options are highly multimodal.
Gabriel A. Brat, MD, MPH, FACS, a trauma surgeon at Beth Israel Deaconess Medical Center and associate professor of surgery and biomedical informatics at Harvard Medical School in Boston, Massachusetts, began his presentation by calling AI “near and dear to my heart.”
Describing webcam-based models for recording and analyzing videos of surgical procedures as relatively outmoded, Dr. Brat said his laboratory is “in the midst of an exciting revolution or evolution in our understanding,” which enables “much more granular representation of what is happening” in a surgical procedure.
A key transition is to smart glasses, which can use embedded cameras to track gaze more accurately than webcam videos. Because focusing the eyes while operating reduces the standard deviation in gaze tracking, the new method also more accurately determines when a surgeon starts operating. Such egocentric video collection can also help researchers track surgeons’ head position and surrounding environment.
Dr. Brat noted that progress is ongoing, but described meaningful data that his team has already generated. Using head position data, the researchers have shown that “the longer a surgeon operates, the more they move into nonergonomic positioning,” a finding that could help surgeons learn to maintain healthier positioning.
Multimodal Data Could Sharpen Surgical Insights
Filippo Filicori, MD, FACS, a minimally invasive surgeon and the system chief of surgical innovation at Northwell Health in New York, New York, described his efforts to bring AI into clinical practice. “I’m trying to fast-forward the implementation of AI in the operating room,” he said.
His laboratory has focused on kinematic data captured from humans, including surgical residents. “We have looked at the gesture of suturing and needle positioning to give feedback to the residents about whether they are doing a good job or what step they should be improving,” he explained.
His work has also demonstrated stapler shape and angle in sleeve gastrectomy are key to postoperative outcomes, while stapling force is closely associated with bleeding and leaks.
Dr. Filicori emphasized that multimodal data streams capturing video, language, sensor data, and other elements are necessary to understand the complexities of surgery. He also emphasized the importance of implementation research and testing AI in the OR under real-life conditions.
The next presenter, Daniel Hashimoto, MD, MTR, FACS, is an assistant professor of surgery at the University of Pennsylvania and director of the Penn Computer-Assisted Surgery and Outcomes Laboratory in Philadelphia. “I have taken a particular interest in how surgeons can drive innovation in the radiology space to meet our needs,” he said.
His work has included training an AI model on hundreds of computed tomography (CT) scans of normal human anatomy until it could identify structures with high accuracy. He described how some AI models can interpret CT scans by detecting subtle differences in resolution and uniformity that humans may be unable to perceive.
Dr. Hashimoto has also engaged in research aimed at personalizing predictions of complication rates based on patients’ specific clinical characteristics, allowing surgeons to move beyond vague predictions. “It’s all about trying to find that valuable data that is going to impact your clinical decision-making,” he said, “to augment our performance as human surgeons because it can see what our eyes cannot see.”
Despite fast-moving research, Dr. Hashimoto noted that most funding for clinical applications of AI is going toward ambient scribes and billing and coding: “It doesn’t even register yet to develop these things for surgeons.”
Nonetheless, he said, AI represents a significant opportunity for surgeons: “You have the opportunity to advocate for your needs as surgeons, to see a return on investment not just on the profit-and-loss sheet, but how you’re doing in terms of burnout and happiness and just the joy of being a surgeon.”
Moving AI Toward Autonomous Surgery
Finally, Justin Chen, a doctoral student in mechanical engineering at Johns Hopkins University in Baltimore, Maryland, focused his presentation on vision-language-action models, another multimodal approach to AI. His aim is to help address growing surgical needs and workforce constraints by training robots to operate autonomously.
To date, his laboratory has worked on a robotic system designed to perform an autonomous cholecystectomy in an experimental setting.
The work has limitations, including training a single robot to perform a single, ex vivo procedure while using curated laboratory data rather than real clinical data. “How can we make this a real product? That’s one thing we are looking for,” Chen said, before describing ongoing work on in vivo porcine models, as well as ambitions to use different robots and procedures in the future.
Both Dr. Filicori and Chen noted their involvement with Open-H, a large-scale dataset created with the participation of numerous institutions. All presenters encouraged audience members to engage with this and other AI-based tools and opportunities for surgeons.
