Providing construction safety guidance using visual observations and historical reports

This pilot was a collaboration between Discovering Safety (a programme of work led by the Health and Safety Executive, HSE), Safetytech Accelerator, and Clarifai, to explore the use of historical contravention documents to assist inspectors and safety managers at new construction sites.

The Challenge

When preparing or inspecting building sites, workers and inspectors are susceptible to missing risks. HSE has amassed a sizable collection of historical Notice of Contravention (NOC) reports throughout the years containing photographs and text statements of safety breaches. In the NOCs, inspectors provide details of corrective actions and this advice is based on official HSE Guidance Resources.

This open innovation challenge aims to demonstrate the possibility of improving safety in the construction industry and elsewhere using image and text analytics. The hypothesis is that AI technology could help with automated identification of risk from new images taken at construction sites by understanding previous advice found in the historical NOCs. This would not only help inspectors to better identify risks but also put a wealth of safety knowledge into the hands of almost every construction worker if delivered as a mobile app.

Safetytech Accelerator began looking for technology companies that offer no-code AI platforms with text and image analytics capabilities for this pilot and future use-cases. Following a thorough international search, 17 companies were shortlisted, two were given the chance to present their products.

Clarifai, a US-based AI technology start-up, was chosen as the technology partner. Their platform has built-in capabilities to perform image recognition, unstructured text analytics, and ease of configuration and deployment so that clients can build their own solutions.

The Pilot

The pilot aimed to create a solution that would allow new images from construction sites to be uploaded and immediate safety guidance to be returned if risks were detected. HSE invited Clarifai to use their Artificial Intelligence (AI) platform to address some of the following scenarios:

• Excavation
• Lone working
• Working at height
• Work to roofs
• Demolition
• Work with machinery

After consultation with end-users, the working at height and work to roofs use-cases were deemed most appropriate for this pilot.

Clarifai began by cleaning and importing the HSE data into their platform and also supplemented the data set with publicly available images. Once the images were in the platform, ladders and roofs were annotated and checked for accuracy by the Clarifai technical team.

 

 

 

Existing general AI models in the platform were enhanced to generate construction specific models and further tuned to automatically detect the presence of roof and ladder types, eg. step ladders or extension ladders.

 

 

The final step was to incorporate the models into a workflow where the image output was sent to a language processor to find relationships with appropriate text in the NOCs.

 

 

 

Results

The pilot solution was able to detect a number of risk scenarios from construction images Involving roofs and ladders. Clarifai had performed the configuration and later made the platform available to HSE staff for hands-on evaluation without the need for extensive training. HSE were pleased to see a simple no-code interface for uploading images and making model predictions. The ability to link models in a graphical workflow was very useful and a well-documented API allowed connectivity to external systems such as mobile apps if required.

“With the support of Safetytech Accelerator, Discovering Safety were able to work with Clarifai to test the feasibility of automatically detecting, using computer algorithms, unsafe working practices in photographs of work being carried out on construction sites. The work demonstrated that, for certain scenarios of risk prevalent on construction sites, captured in photographs by HSE inspectors when routinely inspecting sites, that this was indeed possible. We are now working to see whether the computer vision technology trialled might be used in tandem with natural language processing techniques so as to be able to identify and serve up relevant content in published health and safety guidance, pertaining to the risks detected, to help those with responsibilities for managing such risks on construction sites.” 

Steven Naylor, Technical Research Lead, HSE Science Division

Further work 

One of the objectives of Discovering Safety for this pilot was to understand the potential of cutting- edge visual analytics to democratize access to safety advice. Discovering Safety also wanted to understand if and how HSE’s existing unique national-scale data sets on Notice of Contraventions could be used for training visual analytics algorithms as well future requirements for data collection.

In that respect the pilot provided invaluable information and areas for future improvement. For instance, it was discovered that due to the nature of the training data, it was difficult to determine finer details in the images such as guard rails. This was due to contravention reports mostly reporting poor practice and often missing the positive examples. A next step could be to add training sets with good practice images and more consistent viewing angles.

Using text statements in the contravention reports was particularly challenging because the reports varied considerably in format and the link to image examples was not easy to establish other than by an experienced human. This highlighted the need for report authors, or report format, to describe what each image was intended to illustrate in a more standardised format.

Using the learning from the pilot, Discovering Safety will continue to explore which data sets are most likely to unlock safety insights and how new data can be captured in a more usable format. Clarifai have deepened their understanding of the construction sector and will use this knowledge to further develop their platform for safety use-cases.

“Discovering Safety and Safetytech Accelerator collaborated with Clarifai to undergo a study utilizing computer vision to assist HSE inspectors with real-time automated risk identification from images taken at construction sites. During this process, it became clear that certain risk items could be identified using AI, aiding HSE in identifying and managing unsafe conditions quickly. Clarifai enjoyed the collaboration with Discovering Safety to explore this together, utilizing expert’s expertise in workplace safety initiatives and Clarifai’s AI expertise to discover solutions to make everyone safer.” 

Alfredo Ramos, Senior Vice President – Platform, Clarifai 

>>>  

About Discovering Safety 

Discovering Safety aspires to be a leader in innovative, data driven health and safety with the aim of improving operational performance through the use of data and analytical techniques. To date it has developed and is delivering a programme of work with partners based on extracting insights from HSE and other organisations data to feed into a range of industry endorsed proof of concept applications. For more information or to get in touch visit discoveringsafety.com 

 About the Health and Safety Executive  

HSE’s Science Division, is one of the world’s leading providers of health and safety solutions to industry, government and professional bodies. The main focus of our work is on understanding and reducing health and safety risks. We provide health and safety consultancy, research, specialist training and products to our customers worldwide. Our long history developing health and safety solutions means that we’re well placed to understand the changing industrial, regulatory and societal landscape, and to anticipate future issues. We employ scientific, medical and technical specialists, drawing on their wealth of knowledge and experience to deliver evidence-based solutions to our clients.

About Clarifai 

Clarifai offers a leading computer vision, NLP, and deep learning AI lifecycle platform for modelling unstructured image, video, text, and audio data. It helps both public sector and enterprise customers solve complex use cases through object classification, detection, tracking, geolocation, visual search, and natural language processing.

Founded in 2013, Clarifai has been a market leader in AI since winning the top five places in image classification at the 2013 ImageNet Challenge. In 2019, Clarifai was named a leader in Forrester’s New Wave Computer Vision Platforms report, the only start-up to receive a differentiated rating. Clarifai is headquartered in New York City with more than 100 employees and offices in New York City, San Francisco, Washington, D.C., and Tallinn, Estonia.

About Safetytech Accelerator

Safetytech Accelerator is a non-profit established by Lloyd’s Register. It is the first fully dedicated technology accelerator focused on safety and risk in industrial sectors. Our mission is to make the world safer and more sustainable through wider adoption of safetytech.