Helping robots and humans work hand in hand

As the internet-connected world continues to transform the manufacturing and logistics industry, Technische Universität Dortmund is leading the way in research into human-robot interaction.

With the help of Vicon’s cutting-edge motion capture and VR solutions, the University is developing innovative ways to approach the coexistence of drones and robots with people in warehouse environments.

The manufacturing and logistics sector faces a major challenge: how best to serve the ever-growing e-commerce market – with increasingly small-scale shipments and shorter delivery deadlines – in a cost-effective and sustainable way.

This is where TU Dortmund, one of the leading technical universities in Europe, steps in. It’s helping to meet that challenge with research that’s focused on human activity recognition and ergonomics in logistics and manufacturing environments.

This research program is led by the Chair of Materials Handling and Warehousing at the Faculty of Mechanical Engineering and based in its InnovationLab, an interdisciplinary research center.

To deliver the visual and data evidence that is vital to guide the team’s insights into human activity and drone tracking, Vicon’s cutting-edge motion capture system was installed at the InnovationLab.

The lab focuses on the design and organization of algorithms and protocols to guide human-robot interaction in shared spaces.

The installation features 30 Vicon Vero and 10 Vicon Vantage motion capture cameras, combined with Vicon’s Nexus and Tracker software to analyze the data captured. At 22x10x6 meters, the volume is one of the largest installations of its kind in Europe.

The InnovationLab installation is used by up to 25 people and is proving to be a popular training and special projects facility for students, PhD thesis writers and computer science visitors on summer schools.

Human and machine interaction

A major project for the InnovationLab is to improve robot control by tracking drones to analyze the interaction between humans and machines.

Recognizing and avoiding obstacles, particularly people, who move quickly and erratically, is a key challenge in warehouses. In a stand-out demo, a person in a motion capture suit walks through a swarm of 12 drones flying autonomously.

This showcases their ability to react almost instantaneously, avoiding contact and keeping a minimum prescribed distance away.

The motion capture system is running at up to 300 fps, and simultaneously processes high volumes of data and fast response times with precision.

The information obtained is used as a reference system for safely using drones or robots in a warehouse environment, to check that the drone sensors are working correctly, to both improve positional placement and facilitate the decision-making process.

The lab also employs machine learning technology so that the drones can recognize activities such as carrying boxes, order picking, and conveying goods from one point to another.

This enables the development of safety protocols and creates flexibility, so in the future the drones won’t need a fixed infrastructure to navigate an environment.

Activity recognition to improve ergonomics

The InnovationLab is also being used to develop an algorithm that recognizes activities in IMU sensor data. To train the algorithm, Vicon’s optical motion capture system visualizes the human skeleton with Plug-in Gait (PiG) and synchronizes with video data from Vue cameras, to capture and perfectly label manual warehouse tasks.

The algorithm’s architecture can then be transferred to be deployed on IMU data, which is comparatively hard to annotate. Using Vicon’s Nexus and Tracker data capture software, the InnovationLab team can also see the angles of limbs.

Outputs are used to identify more efficient ways of working, improve workstation placement, and avoid repetitive injuries, to inform future research on ergonomics in the workplace.

Activities examined include everything from ways of lifting, to pushing a cart, and order picking.

Ultimately, the data will improve the ergonomics of these activities and, therefore, warehouse and production efficiency, with applications extending to cover more efficient tool usage, as well as human-robot interaction.

Virtual warehouse (VR to recreate warehouse environment in limited space)
Another project in development is to recreate a warehouse environment virtually.

Without having to duplicate the exact, and usually immense, volumes of a typical warehouse, the VR environment can deliver the same results as in the real world, in a much smaller footprint.

Using the Unity engine, huge spaces can be imagined in a restricted space such as the InnovationLab’s volume, and, covered by Vicon cameras, used to capture accurate data for use in real-world applications.

The system is also linked with a laser projection system to externally visualize everything that’s going on in the virtual environment.

Christopher Reining, MSc, research associate, says: “Our work has far-reaching implications and benefits for the manufacturing and logistics industry and we plan to curate the data to make sure other researchers can access it and use the findings.”

Moritz Roidl, Dipl-Inform, senior engineer, says: “Because of this, we chose to work with Vicon because we needed the most precise and high-quality scientific data available, to prove our research is of the highest standard.

“The Vicon team set up the system within an hour, demonstrating how it works and showing the data capture results.

“And the specialist support the Vicon team provided on-site and on the phone, gives us confidence that they have the technical expertise we need.”