Understanding decision-making on a ship through sensing.

This pilot sought to explore how sensing technologies could improve the understanding of unsafe human behaviours and mindsets on commercial ships, leading to better knowledge of why safety incidents occur.

The Challenge

The growth of international sea trade in recent decades has drastically increased the commercial aspect of maritime activities and with it an increase to operational risk.  Although today’s ships are highly equipped with navigation aids, the number of shipping accidents is increasing, and human decisions are a contributing factor.

As the industry strives to reduce accidents and incidents caused by human error, Scorpio Group in conjunction with Safetytech Accelerator sought to explore AI-powered sensing capabilities to understand human decision making, reduce risk and improve safety on commercial ships.

AI has the potential to aid risk assessments by predicting and comparing risk exposures across vessels. Moreover, AI may assist at the operational level to warn of the consequences from crew behaviour patterns in real-time.

The Solution

UK based Sensing Feeling provides human behaviour and emotion-sensing capabilities, powered by Computer Vision and AI. Their core sensing system works without the need to collect personally identifiable information about the crew, complying fully with GDPR, PII and other privacy regulations.

Visual analysis of crew behaviour and responses at sea is a valuable source of new data for performance, safety, operations and training. By analysing behaviour patterns in the ship’s wheelhouse, Sensing Feeling’s solution aims to uncover insights into behaviour changes occurring at the helm of a vessel for the entire duration of a voyage.

The Pilot

Scorpio kindly provided access to one of their operational crude oil tankers for this pilot. Before installing the technology solution on the tanker, Sensing Feeling worked with Scorpio in a training facility to understand crew behaviour using realistic simulations.  Once confident that the solution could correctly capture and interpret crew movements, the solution was installed on the tanker solely by the crew due to the pandemic. The easy to install solution consisted of:

• A high resolution camera with a wide field of view and infrared night vision
• A Visual Processing Unit (‘VPU’, a compact computer unit providing real-time visual processing)
• A 4G cellular router for relaying data when GSM data networks were in range.

Correct positioning of the sensor was vital to enable analysis of face and body movements. The following figure shows the chosen location for the sensor.

 

 

Sensor data was collected by the solution on a 16 day voyage during the summer of 2020. Data captured by the camera was processed by Sensing Feeling’s VPU in real time. Data was stored and queued for transmission when available mobile data networks were detected. Once a mobile internet connection was established, non-sensitive data was relayed to the cloud for further analysis.

Sensing Feeling correlated their own behaviour data with the tanker’s E-Navigation Data Auditing (EDA) reports. The EDA uses voyage data to report on a vessel’s performance and safety incidents such as Close Proximity and Near Groundings.

The analysis by Sensing Feeling provided key data points to help understand crew behaviour

• Stress Index – Observations of body movements and alertness
• Fatigue Index – Observations associated with lethargy and tiredness
• Aggregate Emotional Response index – The AER index considers dwelling and occupancy of spaces in the wheelhouse and is like a pulse corresponding to the day and night cycles.

Results

One of the objectives of the pilot was to correlate voyage events with emotional and behavioural responses of crewmembers in the wheelhouse. Sensing Feeling’s sensor captured the relationship between operational situations and crew behaviour. The analysis offered a new perspective on the EDA reports.

This pilot found that events such as berthing and unmooring consistently followed sensor observations for increases in stress. The fatigue index also showed an accumulation of fatigue as the voyage progressed.

A summary of the Safety Accelerator pilot:

 

Task	Delivered	Remarks
Capture data required to understand human behaviour	✔	Camera sensor installed on the bridge house detected stress and fatigue.
Store data locally on the ship until it reaches shore.	✔	Data stored in a local computing unit and queued for transmission when mobile data networks were available
Monitoring an environment with multiple crew members	✔	Handled multiple people in the wheelhouse.
Analyse the data to create diagnostic reports that include situations which could have resulted in human-assisted accidents and near misses	✔	Data analysis detected instances of relevant stress factors and periods where fatigue accumulation could contribute to an inability to respond effectively.
Uncover the relationship between discovered patterns and undesired outcomes	✔	Stress spikes detected during berthing. Accumulation of fatigue detected over 13 days
Safely manage crew personal data and privacy	✔	No video or personal data was transmitted from any part of the system.
Optionally, gather and utilise data from existing ship equipment, crew reports and onshore staff to supplement your sensor data.	✔	Use of EDA reports correlating with indexes.

 

“In our ongoing efforts to keep improving safety at sea, Lloyd’s through its safety accelerator program supported Scorpio in trying to help address some complex safety-related challenges.  Using innovative digital tools and human analytics we approached our challenge: ‘Why incidents occur – understanding unsafe behaviours and mindsets’. We hope that going forward, our study and the understanding gained leads to the development of a ‘voyage risk index and alert system’ for vessels at sea.”  

Fletcher Martins, Director Quality Assurance & Safety, Scorpio Group.

Industry-Wide Impact

Currently, there is no standardised methodology for recording sleepiness or fatigue levels of crew onboard ships. The pilot’s findings present possibilities for developing adaptive shift patterns refined by real time monitoring and alerting.

Sensing Feeling’s ability to passively measure stress, fatigue and emotional responses by the crew is a demonstrable step towards predicting and preventing the incidents that may follow.

The insights uncovered during the Lloyds Register Acceleration program have since cross-pollinated into other sectors, a few key examples are safety on railways, office design, and enhanced learning in education.

“The growth of international sea trade has drastically increased maritime activities and their operational risk. At Sensing Feeling, we believe that smart sensing technologies can be entrusted to make the world a safer place. With the valuable support from Lloyds Register Foundation, we explored technological innovations that use AI-powered capabilities to understand human decision making, reduce risk, and improve safety on commercial ships. The Safety Accelerator program was an excellent opportunity to use our advanced human sensing to perform automatic visual detection of behavioural patterns onboard a maritime vessel to assess real-time risk. The work has resulted in new developments on automatic reporting, alerts and new measurement indices that enable our clients to analyse and predict patterns such as stress, fatigue, attention and collaboration. These have provided valuable commercial opportunities to use our sensing technology to bolster risk management and dramatically improve safe operations at sea.”

Jose RazGuzman, Head of Product, Sensing Feeling

About the Safetytech Accelerator

The Safetytech Accelerator is a dedicated open, not-profit initiative created by Lloyd’s Register Group and Lloyd’s Register Foundation, bringing together corporates, startups and the wider technology community to collaborate and accelerate the adoption of safetytech in safety-critical industries.

In early 2020, the Accelerator was awarded the Global Lloyd’s List Award for Excellence in Data and Technology Innovation, in recognition of its excellence and impact in innovation, setting it apart from the competition worldwide. The industry-leading panel felt that the programme, the first of its kind globally, embodied everything that the Lloyd’s List Awards were established to champion, displaying important and impressive global impact for safety and risk in critical industries, including the maritime sector.