Levelling up healthcare: Understanding games' life-changing medical potential | Playable Futures

This series of Playable Futures articles considers how the design, technology, people, and theory of video games are informing and influencing the wider world. You can find all previous Playable Futures articles here.

Video games' relationship with training and simulation goes back decades. In the healthcare sector in particular, the applied gaming movement has generated all manner of training tools to guide professionals through different procedures and equipment.

And yet, thanks in no small part to a new generation of adaptable, user-centric game development tools and technologies, the future holds much potential and promise for our medium to have a profound and remarkable impact in the healthcare realm.

Over at facial animation specialist Speech Graphics, the team has served healthcare providers for some time through the Rapport platform. But there is increasingly an unmissable sense that the opportunities for game tech and game makers in supporting patients, medical professionals, and broad healthcare systems are about to get considerably more varied, impactful, and innovative.

Speech Graphics, which was founded in 2010 after spinning out of the University of Edinburgh's famed School of Informatics, has some proven tech within the video game sector, most notably its facial animation tools. It is this tech that drives the Rapport platform; this enables companies to create and animate characters, using AI to map recorded dialogue and messages to facial animation that matches the emotion of the original speech. Applicable in a variety of industries, this particularly offers an impressive breadth of potential in healthcare. But to co-founder and CEO Gregor Hofer, there's a cultural element that underpins much of games' rise in the medical space.

"We're absolutely seeing more going on in terms of what games can contribute to healthcare, and the future there is fascinating," Hofer offers. "And I think a core part of where this comes from is how present games have become in daily life, culturally.

"The best films and TV shows are now based on games. Games are places people go for culture; for exposure to music and so on. People are becoming more familiar with concepts like avatars through games, and now we see all these beautiful game-like computer animated characters in so much we consume. Today, whoever you are, you can't really fail to notice the power and potential of games. So – in tandem with there being so much real-time content out there now – people across all kinds of industries are seeing what games can bring.

"And to a degree this is really nothing new. Game tech has been used in spaces like simulation and training for many years. But now game engines and tools are so much more powerful, flexible, and welcoming to users, we're seeing this explosion, from Unreal becoming a standard of film production, to the various places Rapport is being used, like in healthcare."

"Digital brings so much potential to the healthcare world, but physical presence will always be a requirement... That's why bringing those elements together through gaming technology is so powerful"

Speech Graphics' core proposition feels very much a technology particularly specialised for games, with the Rapport platform providing obvious applications in other forms of entertainment media and sectors like advertising. That begs a question; how might a facial animation tech with a reputation for delivering realistic emotional expressiveness be relevant to a sector focused on the wildly complex business of understanding and working on the human body? In exploring the answers to that question, we can understand how in the future a great many game technologies could have a role supporting medics and other professionals in the space.

First, though, it's worth briefly profiling what Speech Graphics does.

"We have a core real-time animation technology that analyses speech and then produces matching facial expressions on a digital character," Hofer explains. "It's all built on our model of what the muscles in the face need to do at different intensities, but also at different emotional valence. So we're not just looking at automating lip-syncing based on speech audio – our technology also looks at the full face and how all the muscles in the face need to be configured to express sound at a certain intensity, and at a certain emotional balance.

"How we communicate in reality is about more than just our voice and how we move our lips. Our technology replicates all the other facial expressions and movements that help with communication and showing emotion. So, we are really automating full facial performance from an audio file. And the real gain there is that it becomes super efficient to animate large amounts of dialogue."

It's a highly nuanced, complicated application of technology. After all, human emotion and software's internal machinations could be seen as diametric opposites. But Speech Graphics' technology can clearly deliver, having become popular across the triple-A space in particular. And it just so happens human emotion is a central part of medical practice.

"The first, most obvious use for our technology was in training," Hofer recalls. "Rapport, uses Speech Graphics' technology to bring the same efficiency to creating training content as it does to games, but it also makes the training more convincing and engaging, and so more impactful.

"We started doing medical training tools, where nurses, doctors, and medical professionals can actually interact with simulated patients or staff or others. And that communication and understanding element is so important to the provision of healthcare, whether you are trying to understand patients' needs or potential conditions, or be part of an efficient, effective team. And that training really excels when the communication it provides is convincing. And that might be delivered on a PC screen, or in VR. It's a flexible system and it's proved popular in healthcare. So we've been doing that, and we're continually developing the realism and potential there.

"But there are other ways we're doing work to explore how else we can support patients more directly. My business partner and co-founder Michael Berger worked with UC San Francisco and UC Berkeley to develop a system that lets people with severe paralysis speak and express emotion through a digital avatar. The university teams developed a device that worked as an interface between a patient's brain and our technology."

As published in the scientific journal Nature in August 2023, the electrode-based device can intercept brain signals that would normally be sent to the muscles and related systems that control our faces and voices. Instead, the signals are picked up and sent to a specially adapted form of Speech Graphic's technology which – after several weeks of machine learning training to better understand and translate brain signals – interprets and translates, offering the patient the power of vocal and facial communication through an expressive digital avatar capable of 80-words-per-minute.

"It's incredible to contribute to those kinds of projects – to be a part of improving patients' lives and helping advance healthcare," Hofer says, before touching on the practical gains for Speech Graphics and Rapport. "And there are practical gains too. Our training work absolutely brings in revenue, and with some of the research-based work like the medical avatar system, we can absolutely get to a point where that is commercial. But what is really cool is that these more ambitious research projects really validate our technology and its quality and potential. So that really helps Speech Graphics in the games space, and Rapport in all the other sectors we now serve.

"Other games companies should see if they can become involved with research and other forward-looking projects around healthcare"

"That's something others from games should see if they can become involved with; research and other forward-looking, exploratory projects around healthcare. And, of course, that really ambitious research teaches us things we can take back to our commercial technology. Rapport connects our core facial animation technology with character building, AI, text-to-speech and so on. And that can support what we offer to medical training, or other things like retail experiences. There really are gains in a lot of different directions."

There are challenges to taking all this on, of course. Tech will almost certainly need to be adapted to be understood by medical professionals, and to potentially couple with healthcare technologies and systems – including prototype and research-based devices. There is a very necessary mountain of privacy and data protection to consider, especially when working with teams and patients to establish research projects. Across the world heavy regulation defines the sector – with very different systems existing in distinct countries. Large organisations like the UK's NHS can be mechanically slow when it comes to innovation and adopting new systems, for example.

But all those challenges, Hofer asserts, are manageable and well worth facing.

"Training is a huge area of opportunity and business for us, but there are a lot of other opportunities in healthcare for game technology and game makers. We're seeing developments like game-like VR experiences that can help with pain management. That's a really interesting part of this future, and we might see other use cases where game technology is used to create experiences that apply auditory and visual approaches as a complement to other established options.

"Elsewhere, there's work going on to see if we can use VR experiences to help patients with comfort and movement and rehabilitation. And today we're seeing more instances where game tech is being used to create immersive experiences that can give a patient a chance to feel mentally prepared for more intimidating procedures."

Again, some of those things have existed to some degree in the past. Over a decade ago, former UK stalwart studio Blitz Games' TruSim division worked on medical training tools, and even Vitalize, a Kinect-based game intended to allow for custom physiotherapy courses to be delivered to recovering soldiers. But right now opportunities for an intersection of game industry experience and tools with healthcare are blossoming – and still have much further to go. The potential is dazzling.

Advances in scanning technology in combination with increasing game engine and middleware real-time capabilities may soon allow for detailed, even live 3D models to be displayed on medical devices or through the likes of VR headsets, greatly improving the understanding and practice of surgery and other procedures. There have been scans to inform medical procedures before, but in no way at the level of fidelity, data-informed detail, and interactivity that may soon be coming.

There is currently also encouraging work being done by specialists in breast cancer. In that specialty, lumpectomies currently fail to move an entire tumour some 16-to-25% of the time. Using scans and AR headsets powered by game tech and theory to get a more accurate perspective on an individual's case may greatly increase the chances of precise surgery and prompt recovery.

Others are enthusing about the increasing potential for real-time telepresence, where experts scattered across the world could use AR or VR to coexist in a single space – perhaps with a live patient scan realised via a game engine – sharing expertise and insight during some of the most complex procedures currently known (and potential increasing tremendous efficiency gains and budgetary savings for stretched health services). Game engine and back ends' knack for connecting players in live, asset-rich, real-time environments makes them a perfect foundation for such approaches.

"It's incredible to be a part of improving patients' lives and helping advance healthcare"

Meanwhile, transcranial magnetic stimulation (TMS) provides an imperfect treatment technique that uses a magnetic field to influence brain activity. Over at Stanford University's engineering department there is much interest in the notion that game tech could help bring more accuracy to TMS' delivery. The same team is also exploring how lightweight AR devices could be used by surgeons to provide live data, precise information, and scan overlays that might greatly improve their work. Advances in motion-capture and VR tracking systems, meanwhile, mean custom physiotherapy remotely prescribed in gaming form could now be much more meaningful and impactful than as seen in previous exploratory work in that space.

Much of the appeal of game tech in these instances is that it has been tested at scale, from the hundreds of thousands or even millions of game makers, to the billions of players on the consumer side. As such, new gaming technologies are eagerly eyed by innovative minds in healthcare.

"The word 'metaverse' now means a lot of different things – good and bad – to a lot of different people," Hofer states, considering an overall theme shaping games' relationship with healthcare. "But I do think that a blending of the digital and physical will define a lot of the opportunities in the future of games and game tech serving medical and healthcare purposes.

"I think we'll see that in training, diagnosis, physical and other therapy, in managing large healthcare organisations, and in the delivery of all kinds of care. That is because the healthcare world is one where digital brings so much potential, but physical presence will always be a requirement of so much provision of healthcare. That's why bringing those physical and digital elements together through gaming technology is so powerful."

Games' legacy absolutely extends far beyond their own medium. To improve patient lives, help advance healthcare, and to expand the capacity and impact of organisations like the NHS. That would be about as good as any legacy could get.

Playable Futures is a collection of insights, interviews and articles from global games leaders sharing their visions of where the industry will go next. This article series has been brought to you by GamesIndustry.biz, Ukie, and Diva. You can find previous Playable Futures articles and podcasts here.