François Patou discusses how the recent advances in virtual reality technologies can be used to support people living with hearing loss. He outlines some of the novel virtual reality tools that are emerging as resources to support delivery of audiology services in the future.
Virtual reality (VR) has finally made its way to our living rooms. With portable, affordable, and powerful systems capable of transporting us to enchanted worlds of synthetic beauty or to the remotest parts of the globe while sitting on our sofas, VR is delightedly immersive. But VR is certainly not only fun.
Many professionals and institutions in fields ranging from manufacturing to aviation, from tourism to architecture, have realised that this immersivity has great potential to transform and enrich the public’s experiences of their businesses. They are joined daily by health tech companies aiming to train the next generation of surgeons, aid with pain management, or rehabilitate movement disorders. For professionals working with clinical audiology or hearing research, these developments raise the question: is audiology ripe for VR? I believe that it is.
My conviction is mostly based on the undebated crying need for developing methods (assessments, rehabilitations regimes) that better translate the daily challenges hearing device users face in real life: we are in desperate need of ecologically valid clinical tools [1]. Most readers will concur that our assessments of the benefits of hearing interventions (the use of a specific assistive device, specific device feature, fitting, training regime, etc.) are still very widely limited to auditory thresholds and speech intelligibility testing in the simplest of configurations, e.g. single talker in quiet at zero degree azimuth or with collocated noise. Without much surprise, our hearing interventions therefore often fail to provide the real-life benefits intended. Our rehabilitation strategies for instance, e.g. for cochlear implant users, even when thoroughly probed for efficacy, are still too often designed to optimise performance on unrealistic assessment methods, which eventually prove ineffectual for transferring the benefit to real-world listening environments, to the detriment of cochlear implant users worldwide.
VR can bring a whole new level of contextual and sensory realism to audiological assessment. Just as equally important as the realism of the visual scenes is the realism of the audio scene in which the subject is immersed, whether over headphones or as here over loudspeakers. Realistic virtual acoustics contribute to making VR the perfect tool to ensure that our hearing interventions are as effective in real life as they are in the lab.
So, we need assessments and rehabilitation tools that reflect real-life listening situations; easier said than done. Indeed, realism is only part of the problem: designing interventions (choosing devices, validating rehabilitation protocols, etc.) optimised for real life requires us to hypothesise, test, re-test, and validate empirically that the benefits we hope to achieve are in fact observed. We do need realism, yet not at the expense of a loss of control over how we explore and validate our designs. It is therefore a combination of a high-level of precision with realistic sensory input and situational context - aka realism - that we need. VR allows us to do just that: combine a high level of control with a high degree of realism: VR is the ecological validity medium of excellence.
RELATED TOPIC - Tap the box below
From the control of sound playback timings and levels, to the synchronous (or asynchronous) controlled display of elements of visual scenery that may be relevant (e.g. lip reading, visual distractors, etc.) VR offers all the control one may hope from an experimental test-bench. VR can also support the realisation of better (more sensitive, specific, robust) assessments thanks to its capabilities for behavioural and biometrics tracking. Indeed, VR comes with headtracking for free. Some VR head-mounted displays also already include eye tracking, which can be used for pupillometry measurements to assess listening effort/cognitive load. From an engineering standpoint, the main VR development tools come with various software libraries and integration layers that make it fairly easy to acquire any biometric data in a synchronous manner (e.g. EEG, ECG). In short VR-based audiological assessments are attractive not only because they can be ecologically valid, but also because they can rely on a deeper objective assessment of an individual’s hearing and associated cognitive processes.
One thing that is worth mentioning is that by realism we do not only mean visual realism. Indeed, virtual acoustics have made tremendous progress in recent years and allow VR applications to render realistic audio stimuli, including spatial rendering, room acoustics, and/or the consideration of each user’s individual head and pinnae shape and how these affect the soundwave actually transmitted to the eardrum (or what is called head-related transfer function). Whether you are working for a high-end audiology research clinic with access to 72 loudspeakers on a sphere, or whether you can only afford a more reasonable five-loudspeaker configuration or can make do with a pair of headphones (binaural rendering [2], there probably is a virtual acoustics engine that fits your purpose.
Capability-wise, VR offers so much. So why hasn’t VR made its way to the audiology clinic yet? Well mostly because of the challenge to specify, test and explore VR-based assessment methods. A good deal of research still needs to be carried out to explore and refine VR-based assessments adapted to specific user groups. Pioneering work in Oldenburg [3], at Oticon Medical [4] and within the BEARS project [5] have already made steps in that direction. The VR paradigms that are and will be developed will, in turn, need to be carried and supported by industrials and professional organisations to generate the scales required for VR-based assessment to be worthwhile from a health economic standpoint (whether for the private clinic or public institution and generally for our healthcare systems). The path has been laid down – we just need to walk along it.
Can VR/XR technology benefit from advances in audiology and acoustics?
If hearing research and clinical audiology are ripe for VR, what about the other way around? The VR industry (and other immersive technologies such as augmented or mixed reality – XR for short) has certainly understood that immersivity requires much more than realistic visuals. They appreciate that sound and touch are part of what makes XR immersion or plausibility so compelling and are keen on building more realistic acoustics for the user. But beyond what is already proposed, e.g. binaural spatialisation using generic HRTF, the next steps for VR is certainly the individualisation of acoustic signals to a specific user. Recent initiatives at the Facebook Reality Lab have surely taken a big step in that direction with a large hearing research team looking for ways not only to provide plausible acoustic experiences, but augmented hearing experiences. Where this will take us, we can only imagine. But surely what the XR industry ought to realise is that their acoustic individualisation strategies should be directed to satisfy both normal hearing users and people living with hearing loss and assistive devices. Bringing audiology into VR development is therefore a matter of inclusiveness and necessity.
References
1. Keidser G, Naylor G, Brungart DS, et al. The Quest for Ecological Validity in Hearing Science: What It Is, Why It Matters, and How to Advance It. Ear and Hearing 2020;41:5S-19S.
2. Cuevas-Rodríguez M, Picinali L, González-Toledo D, et al. 3D Tune-In Toolkit: An open-source library for real-time binaural spatialisation. PloS one 2019;14(3): e0211899.
3. Fichna S, Biberger T, Seeber BU, Ewert SD. Effect of acoustic scene complexity and visual scene representation on auditory perception in virtual audio-visual environments. International Conference on Immersive and 3D Audio - IC3A 2021.
4. Lundby Pedersen R, Kajs N, Rico KMJ, et al. Virtual-Reality-based Research in Cochlear Implants: Oticon Medical’s Virtual Experiment Building Platform, 15th European Symposium on Paediatric Cochlear Implantation 2021.
5. Vickers D, Salorio-Corbetto M, Driver S, et al. Involving children and teenagers with bilateral cochlear implants in the design of the BEARS (Both EARS) virtual reality training suite improves personalization of the intervention. Frontiers in Digital Health 2021.
Declaration of Competing Interests: Francois Patou is employed as Senior Translational Research Manager at Oticon Medical.
