The development of effective hearing-assistive devices is essential as the prevalence of deafness grows with an ageing population. Where can AI support speech understanding? A team from the University of Glasgow discusses how lip‑reading hearing aids could be the future.
Nowadays, various wearable devices have been used to improve hearing loss, such as hearing aids, cochlear implants or middle ear implants [2]. They work by amplifying the input sound so that the vibrations can be more easily processed by the brain. However, accessibility in noise environments and affordability are still major barriers for many individuals with hearing impairment. In this context, using visual information, such as sign language or lip reading, to assist hearing can be a feasible alternative.
There have been many studies that propose camera-based systems for sign language detection and lip reading. However, the performance of vision-based systems is limited by lighting conditions (e.g. dark rooms), obstructions (e.g. face masks), cluttered backgrounds and many other environmental factors. In addition, capturing visual information through cameras may raise serious privacy concerns and is also illegal in many parts of the world. To overcome these drawbacks, researchers are exploring the use of radio frequency (RF) sensing technology to help people with hearing loss recognise spoken words, and have already demonstrated the great potential of RF-based hearing aid systems [3].
"The idea of RF-based sensing is to perceive environmental variations by analysing the difference between the received signal and the original signal"
RF signals travel through space in the form of electromagnetic waves. Their properties, such as signal energy and phase, are varied due to distance-related losses, reflection, absorption by objects, etc. Particularly, a moving object can cause dramatic changes in RF signals. Based on this fact, the idea of RF-based sensing is to perceive environmental variations by analysing the difference between the received signal and the original signal. So far, RF sensing technology has achieved great success in the fields of indoor positioning and localisation, activity classification, gesture recognition, occupancy detection and many others [4]. Furthermore, it is also capable of monitoring micro-movements, such as breathing, heart beating and lip movements. For example, Figure 1 shows a visual illustration of vowel detection using RF sensing technologies (Wi-Fi channel state information and ultra-wideband radar), where we can clearly see the difference in RF sensing results when different vowels are pronounced.
Figure 1. Visualisation of RF-based lip-reading results: row (a) shows lip status captured by a camera; row (b)
shows Wi-Fi channel state information response; row (c) shows ultra-wideband radar time-Doppler representations.
Compared to camera systems, RF sensing has many advantages. First, RF sensing systems do not have as many restrictions (e.g. lighting conditions) on the environment as camera-based systems, for example, they can also work well in a dark place. Furthermore, the RF signal is able to penetrate obstacles (e.g. walls, masks) so that it can enable detection through obstacles, which is not possible with camera sensors. Meanwhile, RF sensing systems do not directly capture sensitive information, such as video and sound. The received signal is initially a sequence of complex numbers and is then passed to Fast Fourier Transform or other signal processing algorithms to extract position and velocity information. The final RF output is normally in the form of range-Doppler or time-Doppler representation. Therefore, this abstract representation makes RF sensing more advantageous in terms of privacy preservation. However, this processing method also sacrifices the intuitiveness and ease of interpretation of the RF output. In this case, AI-assisted RF hearing aids can be a feasible solution and have attracted the attention of many researchers.
"The door to AI-assisted RF hearing aid technology has now been opened and we can see that the theoretical feasibility has been well established"
SEE ALSO
In 2022, Abbasi et al proposed the first work using RF signals for hearing aids [3]. They demonstrated the performance of two different RF sensing systems: a Wi-Fi channel state information system and an ultra-wideband radar system, on lip reading, and finally reported an accuracy of over 95% for the recognition of five vowels. One of the keys to the success of this work is the use of AI algorithms for feature extraction and classification, allowing for the effective and efficient interpretation of difficult-to-observe information. On the other hand, this work also compared the recognition accuracy across different types of AI algorithms, including support vector machine (SVM), Naive Bayes, Convolutional Neural Networks (CNNs), etc. Experimental results show that the range-Doppler or time-Doppler representation of the RF sensing output has strong spatial correlations, and the CNNs is the algorithm that best fits this characteristic, thus CNNs achieved the best performance among these AI algorithms. Finally, we can conclude a basic framework from this work for the development of future RF hearing aid systems where AI algorithms play a critical role as the core of data analysis and recognition, as shown in Figure 2.
Figure 2. A basic framework of AI-assisted RF hearing aids.
The door to AI-assisted RF hearing aid technology has now been opened and we can see that the theoretical feasibility has been well established. However, current advances cannot yet be applied to real-life scenarios, and researchers still need to continuously improve the capabilities of existing technologies. On the other hand, with the development of AI algorithms, we can expect more accurate, lower latency and smaller models to overcome the adaptive adjustment of gain and filtering based on the user’s environment and listening preferences. This allows for a more personalised and optimal user experience, such as recognising words previously unknown to listeners using traditional hearing aids.
References
1. Deafness and hearing loss. World Health Organization.
www.who.int/news-room/fact-sheets/
detail/deafness-and-hearing-loss
Last accessed March 2023.
2. Kim HH, Barrs DM. Hearing aids: a review of what’s new. Otolaryngology—Head and Neck Surgery 2006;134(6):1043-50.
3. Hameed H, Usman M, Tahir A, et al. Pushing the limits of remote RF sensing by reading lips under the face mask. Nature Communications 2022;13(1):5168.
4. Shah SA, Fioranelli F. RF sensing technologies for assisted daily living in healthcare: A comprehensive review. IEEE Aerospace and Electronic Systems Magazine 2019;34(11):26-44.
This work was supported in parts by Engineering and Physical Sciences Research Council (EPSRC) grants EP/T021020/1 and EP/T021063/1 and by the RSE SAPHIRE grant.
