Case study: Using human voice to uncover mental wellbeing insights in maritime
30/05/2023This Safetytech Accelerator pilot was a collaboration between Hilo, TORM and audEERING to uncover mental wellbeing insights for seafarers.
The Challenge
The shipping industry is increasingly recognising the importance of addressing mental health and wellbeing issues for seafarers as it has significant health, social and economic consequences. Even before the COVID-19 pandemic, seafarers reported high levels of stress, isolation, and fatigue, and the pandemic only compounded these issues. Ship owners are realising that good mental wellbeing among employees is not only a moral obligation but is also economically beneficial.
This open innovation challenge sought mental wellbeing measurement solutions that can interpret indicators of wellbeing and measure the impact of changes to the working or living environment and organizational drivers. Safetytech Accelerator searched for technology companies that provided mental wellbeing insights using a variety of digital technologies.
Following a thorough international search, four companies were shortlisted to pitch their solutions to the challenge partners. audEERING, a German start-up, was chosen as the technology partner. They were chosen for their audio analysis software, which is designed to provide emotional insights into employees using voice alone whilst still complying with privacy regulations such as GDPR.
The Pilot
The aim of the pilot was to use crew dialogue to understand their mental wellbeing, and was conducted in partnership with TORM who provided one of their vessels and its crew to participate in this trial for a period of 3 months. The seafarers onboard spoke in a mixture of native languages including Filipino, Thai, Croatian, Arabic and Tamil. Prior to making any recordings, the crew were provided with a brochure explaining the aims of the pilot, the mechanics around recordings and how privacy would be preserved. Each crew member also gave consent for recordings to be used in this pilot.
Two types of voice recordings were taken from the ship
- Passive recordings – Background conversations from the bridge obtained from the ship’s Voyage Data Recorder using 6 microphone channels.
- Individual recordings for each seafarer participating in the pilot
- Start: a baseline survey at the beginning of the pilot
- Daily: a short survey asking the individual to perform pre-set tasks such as describing work tasks and counting
- Weekly: a 15-minute survey to provide a view of the ship’s health status
- Final survey
The recordings were analysed post-voyage using the audEERING platform to identify aggregated emotions of sadness, pleasure, joy, anger along with subsets of these.
The voice recordings were correlated with actual ship activity data obtained from Hilo’s maritime decision support platform.
Results
31 crew members participated in this pilot over multiple voyages between December 2022 and February 2023.
Two types of voice recordings were taken from the ship
- Passive recordings (during work tasks) – 500k speech segments captured.
- Individual recordings – 7000 acoustic features extracted.
- A more controlled environment with less noise and focus on the individual.
- Allowed comparison of reported stress with stress determined from voice patterns.
Stress levels from individual recordings were found to be lower than those in passive recordings. The emotions uncovered in the passive recordings could be used as a proxy for stress in the ship’s bridge.
This pilot showed that the system could correctly identify more variance and higher intensity of emotions during the dynamic loading phases compared to calmer open water periods of the voyages. For instance the heatmaps below show stress detected from all passive radio channels during port and open-water activities. Brighter colours signify greater intensity of emotions.
The preliminary results obtained through the pilot showed that audEERING audio analytics has a strong potential to be able to detect automatically, and anonymously, the general wellbeing of the crew, or in particular areas of the ship such as the bridge.
It is worth noting the pilot did not aim to reduce mental wellbeing issues among seafarers per se but exploring if and how using voice as an anonymous indicator of crew wellbeing could allow fleet owners and operators to measure it in a tangible manner and support the development of more improvement strategies.
“Safetytech Accelerator provided us with an excellent opportunity to apply and evaluate our voice emotion recognition and vocal biomarker technology for the monitoring of mental wellbeing of seafarers with great support from HiLo and TORM. To build on the exciting findings from the data of the first vessel, we are now looking into collecting speech data from more vessels to deepen our understanding of how challenging situations for the crew are reflected in their vocal emotional expressions. We believe that vocal expression analysis can be a valuable indicator for upcoming critical situations, and that it can be used together with other indicators to prevent incidents.” – Dr Florian Eyben, CTO & Co-Founder at audEERING GmbH
Industry-Wide Impact
Beyond this pilot which was conducted on a single ship, the next step is to scale up the recordings across a fleet to compare aggregated mental wellbeing between vessels. The pilot partners have expressed an interest in exploring a follow-up with a larger pilot.
“Safetytech Accelerator was key in facilitating a partnership between HiLo Maritime Risk Management and AudEERING for a ground-breaking investigation into seafarer wellbeing. This project brought together HiLo’s unparalleled understanding of accidents and incidents and AudEERING’s voice recognition software with the intention of identifying early signs of seafarer stress. HiLo has developed the industry’s leading risk platform, with in depth knowledge of the warning signals of future incidents. This innovative project presents the opportunity to identify the links between seafarer stress and incidents at sea. The initial project gave some valuable insights, with the potential to grow into a powerful tool for protecting seafarers.” – Andy Cross, COO at Hilo
Beyond maritime, the ability to monitor mental wellbeing has the potential to help other forms of transportation and safety-critical sectors such as heavy industry.
“It is still early for the adoption of these audio analytics technologies in safety-critical industrial sectors, and especially maritime. These technologies have the potential to save lives but their adoption is not just about what the technology can do, it is also and especially about taking human factors into consideration. Real-world pilots such as this one between Hilo, Torm and audEERING are key stepping stones to gaining a better understanding of their potential.” – Dr. Maurizio Pilu, Managing Director at Safetytech Accelerator
>>>
About HiLo Maritime Risk Management
HiLo Maritime Risk Management is saving lives in the shipping industry by changing the way companies understand and address risk. HiLo collects and translates safety information from it’s clients into comprehensive risk profiles for each company and the whole HiLo fleet. Armed with this analysis, companies can stop catastrophes in their tracks. All reports are anonymised, meaning each customer gets the opportunity to learn from others’ experiences without putting their commercially sensitive information at risk.
About TORM
TORM is a world-leading specialist carrier of energy and clean petroleum products. Customers worldwide rely on TORM’s fleet of product tankers to move the gasoline, naphtha, diesel, and jet fuel that keep businesses running every day. Efficiently, safely, and reliably.
A pure play product tanker company, TORM’s unique business model and more than 3,500 dedicated employees leverage the strengths of owning and operating our own modern fleet by integrating all key shipping competences in-house. This provides our customers with the energy products they need efficiently, safely, and reliably – and investors with best-in-class return on investment.
About audEERING
audEERING is a world leading innovator in the application of voice- based artificial intelligence (AI) to various industries. Their AI technology can detect stress, emotion and other factors (vocal biomarkers) from human voice.
About Safetytech Accelerator
Safetytech Accelerator is a non-profit established by Lloyd’s Register. It is the first fully dedicated technology accelerator focused on safety-critical industrial sectors. Our mission is to make the world safer and more sustainable through wider adoption of Safetytech.