The first metaverse experiences? See what’s going on in medicine


Surgeon Shafi Ahmed poses for a photo wearing a Microsoft HoloLens headset inside his operating room at the Royal London Hospital on Thursday, January 11, 2018.

Bloomberg | Bloomberg | Getty Images

The metaverse, the next big thing in the digital world, is touted as the internet realm where the animated avatars of our physical being will be able to do virtually all kinds of interactivity, from shopping to gambling to traveling – someday. Wonks says it could be a decade or more before the necessary technologies catch up with the hype.

Right now, however, the healthcare industry is using some of the core components that will ultimately include the metaverse – virtual reality (VR), augmented reality (AR), mixed reality (MR), and intelligence. artificial (AI) – as well as the software and hardware to power their applications. For example, medical device companies use MR to assemble surgical tools and design operating rooms, the World Health Organization (WHO) uses AR and smartphones to train Covid-19 responders, psychiatrists are using VR to treat post-traumatic stress (PTS) among combat soldiers, and medical schools are using virtual reality for surgical training.

Facebook, Oculus and Covid

From Facebook – now Meta Platforms – acquired Oculus and its VR headset technology in 2014 for $ 2 billion, many healthcare applications have been developed. One of the latest was a collaboration with Facebook Reality Labs and Nexus Studios and the WHO Academy. The organization’s R&D incubator has designed a mobile learning app for healthcare workers battling Covid-19 around the world. One of the training courses involves AR to simulate on a smartphone the proper techniques and sequence for putting on and taking off personal protective equipment. With content available in seven languages, the app is built around the needs expressed by 22,000 global health workers surveyed by WHO last year.

Oculus technology is used at UConn Health, the University of Connecticut Medical Center in Farmington, Connecticut, to train residents in orthopedic surgery. The educators have partnered with PrecisionOS, a Canadian medical software company that offers virtual reality training and educational modules in orthopedics. Wearing Oculus Quest headsets, residents can 3D visualize performing a range of surgeries, such as putting a pin in a fractured bone. Because the procedure is performed virtually, the system allows students to make mistakes and receive feedback from faculty to incorporate on their next try.

Meanwhile, as the Metaverse remains under construction, “we see a great opportunity to continue the work Meta is already doing to support health efforts,” a Meta spokesperson said. “As Meta’s experiences, applications, and services evolve, you can expect health strategy to play a role, but it’s far too early to say how this might intersect with technologies and third-party vendors. “

When Microsoft introduced its HoloLens AR smart glasses in 2016 for business development, early adopters included Stryker, the medical technology company from Kalamazoo, Michigan. In 2017, he started leveraging the AR device to improve the design processes of operating rooms in hospitals and surgery centers. Since operating theaters are shared by different surgical departments – from general surgery to orthopedics, cardiology and more – surgical lighting, equipment and tools vary depending on the procedure.

Recognizing the opportunity offered by HoloLens 2 in the evolution of operating room design from 2D to 3D, Stryker engineers are able to design shared operating rooms using holograms . The MR experience visualizes all people, equipment and configurations without requiring the presence of physical objects or people.

Zimmer Biomet, a medical device company based in Warsaw, Indiana, recently unveiled its OptiVu mixed reality solutions platform, which uses HoloLens devices and three applications – one using MRI in the manufacture of surgical tools, another that collects and stores data to track patient progress before and after surgery, and a third that allows clinicians to share an MRI experience with patients before a procedure.

“We are currently using HoloLens on a pilot basis with remote assistance in the United States, EMEA and Australia,” said a spokesperson for Zimmer Biomet. The technology has been used for remote case coverage and training programs, and the company is developing software applications on HoloLens as part of data solutions focused on pre- and post-procedure procedures, the spokesperson said. .

Microsoft’s holographic vision of the future

In March, Microsoft introduced Mesh, an MR platform powered by its Azure cloud service, which allows people in different physical locations to participate in 3D holographic experiences on a variety of devices, including HoloLens 2, a line of VR headsets. , smartphones, tablets and PCs. . In a blog post, the company imagined avatars of medical students, learning human anatomy, gathered around a holographic model and peeling back muscles to see what was underneath.

Microsoft sees many opportunities for its MR technology, and in secure March a $ 20 billion contract with the US military for use with soldiers.

In actual applications of AR medical technology, Johns Hopkins neurosurgeons performed the institution’s first AR surgeries on living patients in June. During the initial procedure, doctors placed six screws into a patient’s spine during a spinal fusion. Two days later, a separate team of surgeons removed a cancerous tumor from a patient’s spine. The two teams donned helmets made by Augmedics, an Israeli company, fitted with a transparent eye shield that projects images of a patient’s internal anatomy, such as bones and other tissues, based on CT scans. “It’s like having a GPS navigator in front of your eyes,” said Timothy Witham, MD, director of the Johns Hopkins Neurosurgery Spinal Fusion Laboratory.

At the Miller School of Medicine at the University of Miami, instructors from the Gordon Center for Simulation and Innovation in Medical Education use AR, VR, and MR to train emergency first responders to treat trauma patients, including those who have had a stroke, heart attack or gunshot wound. Students perform vital heart procedures on Harvey, a lifelike mannequin that realistically simulates nearly any heart disease. By wearing virtual reality headsets, students can “see” the underlying anatomy that is graphically displayed on Harvey.

“In the digital environment, we are not bound by physical objects,” said Barry Issenberg, MD, professor of medicine and director of the Gordon Center. Before developing the virtual technology program, he said, students had to be physically on site and training on real trauma patients. “We can now ensure that all learners have the same virtual experience, regardless of their location. “

Since its inception in 1999, the Institute for Creative Technologies (ICT) at the University of Southern California has developed VR, AI, and other technologies to treat a variety of medical and mental health issues. “When I first got involved, the technology was the Stone Age,” said Albert “Skip” Rizzo, psychologist and director of medical virtual reality at ICT, recalling his tinkering with an Apple IIe and a portable Game Boy console. Today, he uses VR and AR headsets from Oculus, HP, and Magic Leap.

Rizzo helped create a virtual reality exposure therapy called Bravemind aimed at relieving PTSD, especially among veterans of the wars in Iraq and Afghanistan. During exposure therapy, a patient, guided by a trained therapist, confronts their memories of trauma through simulations of their experiences. Wearing a helmet, the patient can be immersed in several different virtual scenarios including a Middle Eastern themed city and desert road environments.

“Patients use a keyboard to simulate people, insurgents, explosions, even smells and vibrations,” Rizzo said. And rather than relying exclusively on the imagination of a particular scenario, a patient can experience it in a secure virtual world as an alternative to traditional talk therapy. Evidence-based Bravemind therapy is now available in more than a dozen Veterans Administration hospitals, where it has been shown to produce a significant reduction in PTS symptoms. Additional randomized controlled studies are underway.

As Big Tech continues to develop the metaverse, alongside software and hardware companies, universities, and other R&D partners, the healthcare industry remains a real testing ground. “While the metaverse is only in its infancy, it holds enormous potential for transforming and improving healthcare,” wrote Paulo Pinheiro, head of software at Cambridge-based Sagentia Innovation, in the United Kingdom. UK, on ​​the consulting firm’s website. “It will be fascinating to see the situation evolve.”

Stay connected with Healthy Returns

For a front row seat at CNBC Events, you can hear firsthand from the visionary leaders, innovators, leaders and influencers taking the stage in “The main podcast. ” Listen now, however, you get your podcasts.

For more exclusive information from our journalists and speakers, subscribe to our Healthy Returns newsletter to receive the latest delivery straight to your inbox each week.


Margie D. Carlisle