Become a member and receive career-enhancing benefits
Our top priority is providing value to members. Your Member Services team is here to ensure you maximize your ACS member benefits, participate in College activities, and engage with your ACS colleagues. It's all here.
Become a member and receive career-enhancing benefits
Our top priority is providing value to members. Your Member Services team is here to ensure you maximize your ACS member benefits, participate in College activities, and engage with your ACS colleagues. It's all here.
“Improving surgical care requires access to high-quality data from one of the most secretive environments in modern society—the operating room (OR),” said Teodor Grantcharov, MD, PhD, FACS, inventor of the OR Black Box.
Inspired by the aviation industry’s black box, which helps transportation investigators determine the cause of an adverse event, the OR Black Box is a sophisticated system of sensors and software—not a physical box—that captures data with the goal of minimizing risks and improving patient outcomes.1
The OR Black Box tracks all activity in the OR, including patient vital signs, equipment malfunctions, and surgical team performance—virtually any factor that might affect the outcome of a surgical procedure. The streams of data collected by this platform are then analyzed using a combination of artificial intelligence (AI) and specialty-trained analysts to produce feedback intended to improve efficiency in the OR, enhance surgical training, and optimize best surgical practices.
“The OR Black Box processes an enormous amount of data in excess of half a million data points per operating room per day,” explained Dr. Grantcharov, adding that the purpose of this technology is not to point fingers or assign blame but to reduce errors and improve systems of care.
Last September, Stanford Hospital in Palo Alto, California, installed this technology in four operating rooms—the first on the West Coast—joining a total of 24 hospitals in the US, Canada, and Western Europe that are using the OR Black Box system.1,2
“We expect this number will double in the next 12 months,” said Dr. Grantcharov, who is the associate chief quality officer for innovation and safety for Stanford Healthcare and professor of surgery at Stanford University.
Duke University in Durham, North Carolina, has featured the OR Black Box platform in two of its ORs, with plans to expand across several surgical specialties, including thoracic surgery, urology, gynecology, neurosurgery, and trauma.2,3
“The OR Black Boxes are quite good for minimally invasive surgery, whether its robotic or laparoscopic, because the filming of it can be taken straight off of the monitor,” explained Christopher R. Mantyh, MD, FACS, FASCRS, vice-chair of clinical operations at Duke University Medical Center, and a professor of surgery in the Division of Surgical Oncology.
“Historically, the OR has been a very closed shop,” added Dr. Mantyh. “We’ve never really conducted a close deep dive into what actually occurs there until now.”
A specialized camera behind Dr. Grantcharov captures video and audio data. The output has all faces blurred to de-identify patients and the surgical team. (Photo by Yuri Makarov)
Evidence Supporting OR Black Box Implementation
The idea to develop what eventually evolved into the OR Black Box came to Dr. Grantcharov as a surgical resident in Copenhagen, Denmark, where he observed sharp differences in skill levels of surgeons objectively measured by a virtual reality simulator.4
After moving to the University of Toronto in 2006, he worked with engineers and data specialists to create the first prototype of the OR Black Box, which was installed at St. Michael’s Hospital in Toronto.5
Advancing the OR Black Box from a prototype to a system adopted by hospitals around the globe started with a fundamental question: Does the platform deliver on its mission to make healthcare safer and more cost effective?
In a study published in a 2020 issue of the Annals of Surgery, Dr. Grantcharov and coauthors conducted a cohort study of patients undergoing laparoscopic surgery at an academic hospital during the first year of OR Black Box implementation.6
Analyzing the data from 132 consecutive patients, a median of 20 errors per case or 3,435 errors were identified. According to the study’s authors, “…errors most frequently occurred due to the application of insufficient force or underestimation of distance to target tissue when performing surgical tasks. Errors often took place during dissection (median of 18 errors per hour), resection (13 per hour), and reconstruction phases (18 per hour), and when performing a grasping or dissecting task (6 per hour).”6
The study also revealed that auditory distraction occurred a median of 138 times per case, including alarms from equipment, pagers, phones, and surgical devices.6
“The topic of distractions in the OR was something that really surprised us in the early days of the data analysis,” revealed Dr. Grantcharov. “In fact, the number of distractions we found was extraordinarily high. I think my colleagues will identify with the fact that somebody inevitably comes in and asks a question during the most physical step of the procedure or how often someone comes in and asks when we will be done so that the next case will be able to follow.”
According to the study, the OR door opened a median of 42 times per case or approximately once every 2 minutes. The authors assert that surgeons who encounter auditory distractions exhibit “lower surgical skill proficiency, speed, and accuracy” in a simulated environment.6
“Often, these distractions adversely impact our ability to execute critical steps and can lead to an increased number of errors,” Dr. Grantcharov said. Both Drs. Grantcharov and Mantyh said findings such as these present informed opportunities to construct better ORs and intentionally design devices with more effective alarms.
Further studies using the OR Black Box technology have demonstrated the impact of stress, leadership style, teamwork, and communication on safety outcomes. The results have highlighted the importance of modern leadership and safety culture in the journey toward high reliability in surgery. Ongoing studies are investigating the impact of data-driven improvement achieved through the OR Black Box on patient outcomes and are expected to be published later in 2023.
Notably, the OR Black Box design is intentionally nonintrusive in order to avoid interrupting normal behavior or processes in the OR. This approach is aimed at eliminating the Hawthorne effect, which occurs when individuals behave differently if they know they are being watched, according to Dr. Grantcharov.
The OR Black Box console is a touch panel screen that provides information to the surgical team.
OR Black Box and the M&M Conference
Mitigating errors and eliminating distractions in the OR are not the only ways the OR Black Box can drive a culture of safety—this technology also can augment surgical education and training with feedback that addresses specific procedures focused on the system rather than a particular individual.
After data from an OR are collected, they are analyzed via explainable AI algorithms to uncover variations in the procedure that are then tagged for human review.
Dr. Mantyh described how a tagged case provided a notable teaching moment during a Morbidity and Mortality (M&M) conference at Duke.
“It was a low interior resection for rectal cancer, and one of the large veins was entered, which is a known complication from these procedures,” he said. “The decision was made to open and anesthesia saw that there was bleeding, and they actually called for some blood and additional help. Within a few minutes, the patient had an open laparotomy, and the bleeding was repaired surgically with a couple of sutures.”
According to Dr. Mantyh, the case—which was presented at an M&M conference a week or so later—is used as an example of how well a surgical team can perform if there is an untoward event.
“I think what we try to do with this technology is look at something that occurred and figure out how we can make it better. Or determine what we did if something actually worked out quite well. The old mantra of blame and shame at M&M conferences is out—we really want to examine things to learn how we can improve,” said Dr. Mantyh.
Developing an objective process for evaluating surgeon proficiency is key to improving surgical outcomes. Rather than relying on human memory to reconstruct an event, the OR Black Box is an assessment tool that can be used, not only in the peer-review setting of the M&M conference, but also in a self-directed review to identify areas for improvement. For surgical trainees, this platform promotes effective coaching and opportunities for enhancing technical skills like closing an incision or inserting a breathing tube.
“We’ve known for a long time that high-quality, objective, meaningful measurement of skill and performance is important,” Dr. Grantcharov said. “Today, this is done through questionnaires that are subject to recall bias. Often, the way we evaluate our cases and provide feedback to our trainees is subjective and biased, and it’s often meaningless in terms of improvement.”
Replacing subjective and unreliable measurement and feedback with a machine-generated process that is objective and actionable could dramatically improve surgical education, as well as surgeon development at three critical steps: selection, promotion, and certification.
“A lot of work still has to be done in this area, especially for high-stakes or summative assessment, but we are well into that process. And, for the first time in many years, I feel optimistic about the introduction of competency-based education in surgery,” said Dr. Grantcharov.
Improving OR Efficiency
Implementing procedures that decrease inefficiencies in the OR, such as scheduling delays, canceled cases, and surgical wait times, also can improve the care of the surgical patient. “Everyone thinks that the Black Box is going to just be looking at what’s going on in the operating room, but there’s a lot of activity that goes on before the case even starts,” explained Dr. Mantyh.
In a 2015 article published in the Journal of the American College of Surgeons that examined factors contributing to OR delays, the authors noted that “delays in surgical start times can be attributed to both human errors and system deficiencies, with both occurring in the OR.”7 The article suggested that improvements in scheduling lead to more efficient allocation of staff and resources, enhanced patient flow, and higher patient satisfaction scores.
“More than half of procedures are under-scheduled, which means that if the surgeon believes that a procedure will take 1 hour, it often takes 2 to 3 hours,” said Dr. Grantcharov. “We’ve shown with the objective data that we generate through the OR Black Box that we can cap scheduling errors dramatically resulting in better OR utilization.”
For example, if three cases are booked in one OR and each of them is an hour longer than originally scheduled, this obviously can lead to significant delay-related inefficiency throughout the day. The OR Black Box uses machine learning, rather than subjective and retrospective information that is manually input into the system, to make scheduling predictions.
This technology also can help improve OR turnover rates by looking at how surgical equipment is arranged in the OR, including the positioning of trays or the placement of laparoscopic devices.
“If there’s a delay getting the patient into the room because a piece of equipment’s not there, we can look at the Black Box and figure that out,” added Dr. Mantyh. “Was it simply not in the room or did we have to call it up from our supply area? Turnover time has always been an issue at our hospital, and the question is, 'Why?' With data from the Black Box, there is a lot of opportunity to streamline the process and make it more efficient.”
Inefficiency has a tremendous effect on the quality of life of doctors, nurses, and staff in the OR, shared Dr. Grantcharov. “We found that one of the reasons operating room nurses were leaving their profession was unpredictable overtime. It’s hard to organize your life if you can’t predict when you’ll be home.”
Enhanced efficiency in the OR not only increases job satisfaction, it also reduces spending on overtime. “This reduction in cost leads to better use of one of the most valuable resources in the hospital—the operating room,” said Dr. Grantcharov.
The OR Black Box captures data analyzed by both artificial intelligence and human reviewers. (Photo by Yuri Makarov)
Positive not Punitive
A significant barrier to widespread adoption of the OR Black Box is the concern that the data could be used for punitive or legal purposes.
“This is a natural first reaction,” said Dr. Grantcharov. “Obviously, we practice medicine in a litigious society, and we’ve got enough stressors in our lives without adding another one. We’ve introduced a number of protective mechanisms to make the data de-identified in order to aggregate it and make it practically impossible to be used for any type of legal action."
Dr. Mantyh added, "Not only is the patient information de-identified, but the surgeon’s voice and his or her image are also de-identified as is anyone’s who is in the operating room."
To date, OR Black Box data remain untested in the legal arena, likely because the system is relatively new.2
Both Drs. Grantcharov and Mantyh urge hospitals to embrace the OR Black Box platform, as these data reinforce and promote surgical safety and best practices, leading to fewer adverse outcomes and, therefore, fewer liability and malpractice claims.
“I believe this type of technology will be the mirror that we need to look at as individuals, as teams, as organizations, and as a profession in general,” said Dr. Grantcharov. “I think that continuous access to objective data will allow us to critically reflect on our performance. Without objective and reliable data, we have absolutely no chance of improving.”
Tony Peregrin is the Managing Editor of Special Projects in the ACS Division of Integrated Communications in Chicago, IL.
Jung JJ, Jüni P, Lebovic G, Grantcharov T. First-year analysis of the operating room black box study. Ann Surg. 2020;271(1):122-127.
Kimbrough CW, McMasters KM, Canary J, et al. Improved operating room efficiency via constraint management: Experience of a tertiary-care academic medical center. J Am Coll Surg. 2015;221(1):154-162.
