Want to know how safe your warehouse job is? There’s an app for that

In 2017 there were nearly 350,000 incidents of workers taking sick leave due to injuries affecting muscles, nerves, ligaments or tendons – like carpal tunnel syndrome – according to the US Bureau of Labor Statistics.

Among the workers with the highest number of incidents: people who work in factories and warehouses.

Musculoskeletal disorders happen at work when people use awkward postures or perform repeated tasks. These behaviors generate strain on the body over time. So it’s important to point out and minimize risky behaviors to keep workers healthy on the job.

Researchers at the University of Washington have used machine learning to develop a new system that can monitor factory and warehouse workers and tell them how risky their behaviors are in real-time.

The algorithm divides up a series of activities – such as lifting a box off a high shelf, carrying it to a table and setting it down – into individual actions and then calculates a risk score associated with each action.

The team published its results June 26 in IEEE Robotics and Automation Letters and will present the findings August 23 at the IEEE International Conference on Automation Science and Engineering in Vancouver, British Columbia.

Senior author Ashis Banerjee, an assistant professor in both the industrial and systems engineering and mechanical engineering departments at the UW, says: “Right now workers can do a self-assessment where they fill out their daily tasks on a table to estimate how risky their activities are.

“But that’s time-consuming, and it’s hard for people to see how it’s directly benefiting them. Now we have made this whole process fully automated. Our plan is to put it in a smartphone app so that workers can even monitor themselves and get immediate feedback.”

For these self-assessments, people currently use a snapshot of a task being performed. The position of each joint gets a score, and the sum of all the scores determines how risky that pose is.

But workers usually perform a series of motions for a specific task, and the researchers wanted their algorithm to be able to compute an overall score for the entire action.

Moving to video is more accurate, but it requires a new way to add up the scores. To train and test the algorithm, the team created a dataset containing 20 three-minute videos of people doing 17 activities that are common in warehouses or factories.

First author Behnoosh Parsa, a UW mechanical engineering doctoral student, says: “One of the tasks we had people do was pick up a box from a rack and place it on a table.

“We wanted to capture different scenarios, so sometimes they would have to stretch their arms, twist their bodies or bend to pick something up.”

The researchers captured their dataset using a Microsoft Kinect camera, which recorded 3D videos that allowed them to map out what was happening to the participants’ joints during each task.

Using the Kinect data, the algorithm first learned to compute risk scores for each video frame. Then it progressed to identifying when a task started and ended so that it could calculate a risk score for an entire action.

The algorithm labeled three actions in the dataset as risky behaviors: picking up a box from a high shelf, and placing either a box or a rod onto a high shelf.

Now the team is developing an app that factory workers and supervisors can use to monitor in real-time the risks of their daily actions. The app will provide warnings for moderately risky actions and alerts for high-risk actions.

Eventually, the researchers want robots in warehouses or factories to be able to use the algorithm to help keep workers healthy. To see how well the algorithm could work in a hypothetical warehouse, the researchers had a robot monitor two participants performing the same activities.

Within three seconds of the end of each activity, the robot showed a score on its display.

Banerjee says: “Factories and warehouses have used automation for several decades. Now that people are starting to work in settings where robots are used, we have a unique opportunity to split up the work so that the robots are doing the risky jobs.

“Robots and humans could have an active collaboration, where a robot can say, ‘I see that you are picking up these heavy objects from the top shelf and I think you may be doing that a lot of times. Let me help you’.”

Additional co-authors are Ekta Samani, Rose Hendrix and Cameron Devine, who are UW mechanical engineering doctoral students; Shashi Singh, who completed this research as a summer intern at the UW; and Santosh Devasia, a professor in the UW mechanical engineering department.

Funding and support for this project has been provided by the State of Washington, Department of Labor and Industries, Safety and Health Investment Projects. This research was also funded by a gift from Amazon Robotics.