Extended Reality Demos at Stanford
As more people look to apply extended reality (XR), a field that includes real-time data and augmented reality, to health and wellness, an award-winning lab at the UC San Diego Qualcomm Institute (QI) is creating innovative approaches to addressing predicted global shortages in surgeons.
On January 23, QI Senior Research Scientist Yang Cai’s Visual Intelligence Studio (VIS) was invited to demonstrate XR technology at Stanford University’s Goodman Surgical Education Center. Three live demos offered Stanford medical students and Doctor of Medicine (MD) residents the chance to immerse themselves in XR technology replicating laparoscopic or “keyhole” surgeries in the abdomen.
Pairing augmented reality and real-time feedback from artificial intelligence, the VIS’s system allowed medical students to operate on a hyper-realistic simulated abdominal cavity with perceptual and haptic (touch-based) perceptions. The XR technology enables laparoscopic instrument tracking, performance assessment and procedure knowledge navigation.
“We asked the medical students for their impression of the demos, and they said, ‘Cool!’” Cai recalled. “If it’s not cool, we won’t do it. That’s our mantra.”
The VIS lab’s $1.8 million project is funded by the Wellcome Leap Foundation’s “Surgery: Assess/Validate/Expand (SAVE)” program. The initiative tackles a projected global shortage of surgeons, particularly in low-resource communities.
With no shortage of need on the horizon, Project SAVE invites teams to dream up new tools and technologies that may streamline preparation of MD surgeons and non-MD practitioners with hands-on training and that reduce postoperative deaths.
At Stanford University, the VIS lab demonstrated the technology for locating Calot’s triangle, a triangular region inside the body that should be identified for a safe operation, tracking instruments inside the body during surgery, and visualizing surgical scenarios for cholecystectomy, or gallbladder removal.
Instead of using existing machine learning algorithms, Cai and his students, who hail from UC San Diego’s Jacobs School of Engineering, Rady School of Management and the School of Arts and Humanities, created novel algorithms for simpler, faster and more effective functions.
“I believe engineering students can be just as creative as art students,” said Cai. “Our lab allows people of various backgrounds to work together. The teamwork was what made this work.”
VIS will present its work at the 15th International Conference on Applied Human Factors and Ergonomics (AHFE) in Nice, France, in July 2024.
Cai also heads a $1.27 million project funded by the U.S. Department of Commerce’s National Institute of Standards and Technology (NIST). The project’s goal is to develop augmented reality technology to train first responders for or aid those in already dangerous situations. In 2019, Cai’s lab placed first in the NIST Haptics Interface Challenge for their invention of a haptic helmet that helped firefighters navigate smoke-filled buildings to save lives and prioritize safety.
A third, ongoing initiative, supported by the National Oceanic and Atmospheric Administration (NOAA), uses ocean samples collected by citizens to detect harmful algal blooms on the coast.
Within its first year at UC San Diego, the Visual Intelligence Studio has spearheaded three research projects with an innovative team, a cutting-edge laboratory and a budget totaling $2.8 million.
