HiSMoT

HIGH-SPEED MOTION TRACKING AND COUPLING FOR HUMAN-ROBOT COLLABORATIVE ASSEMBLY TASKS —

Funding: German Research Foundation (DFG)
Grant number: 500490184
Project duration: 01.09.2023 - 31.12.2025
Collaborating University: Ruhr Universität Bochum (RUB), Chair for Production System (LPS)

Background

Today, the capabilities of human-robot collaboration (HRC) are widely investigated for manufacturing processes, mainly in lab environments. However, they are not well applied in production systems. Collaborative handling of an object is one of the current challenges regarding HRC’s real collaboration capabilities. In this context, it is still unclear how robots and humans should behave to handle an object and anticipate for mutual care. The speed of the robot tool center point (TCP) that depends on the human motion might create high joints speeds, which can create potential risk to human body parts. In this regard, robots have been represented using kinematic and dynamic models of rigid bodies that yield a structured and controlled motion profile compared to data-driven human motion generation techniques.

Project objective

To investigate real-time HRC motion coupling and harmonization methods that highly anticipate human motion behavior with high mutual care.

Method

• Our approach is based on a Latent Space in which each vector represents a human action.
• The Latent Space is derived from Motion Capture (original idea see Min et al. 2012 (doi)).
• Motions are generalized using Gaussian Mixtures.

Design of experiment

• Two humans handling jointly a rigid object
• The test procedure includes: a. Layout planning using digital environment e.g., Unity3D, b. Physical and digital twin, c. Visual guide for randomly generated target positions, d. Motion capturing process using MVN analyze (Submitted for CIRP CMS 2025 Conference, Saeed et al. 2025).
• Overhead light projector is a visual spatial clue object positioning
• The human motions are captured using Inertial Measurement Units (IMUs) and eye tracker
  • Frame rate (speed) - 240 fps,
  • Gaze scene - 30 fps,
  • World scene from top, side and front view - 4K video with 50 fps

Figure:

Research question

In HiSMoT aspect, the following basic questions may lead to investigation;

• What is the best method to couple two latent space models?
• How do we map one latent space to the other latent space or from one latent space to another latent space? E.g. human reach and robot reach, grasp action after reach action.
• How do we control the robot dynamics (force/torque) and human behavior (trajectory junk) in latent space?

Preliminary results

• Digital twin scene and simulation setting setup for human motion simulation is successfully created using game engine tools such as Unity3D.

• Human motion model using latent space features for pick and place is successfully created using Morphoble Graph libraries, and the motions are analyzed in the digital twin.

• Latent space controller for position control is promising for coupling human and robot model.

Publications

Conference Articles

1. Raza, S.M., Tuli, T.B., and Manns, M., 2024. Human Action Sequence Prediction for (Re)Configuring Machine Tools. Procedia CIRP, 130, 1170–1175. (doi)
2. Saeed, R., Tuli, T.B., Habersang, T., Weikum, M., Kuhlenkötter, B., and Manns, M., 2024. Human motion generation using latent space constraint for manual assembly tasks. In: WGP Jahreskongress 2024 (under production).
3. Saeed, R., Tuli, T.B, Weikum, M., and Manns, M., 2025. Towards trajectory-based latent space control for human-robot collaboration. In: Proceedings of the 58th CIRP Conference on Manufacturing Systems. (Accepted).
4. Saeed, R., Tuli, T.B., Manns, M., 2025. Experimental setup for motion capturing two humans jointly handling an object. In: Proceedings of the 58th CIRP Conference on Manufacturing Systems. (Accepted).
5. Habersang, T., Saeed, R., Manns, M. and Kuhlenkötter, B., 2025. Motion harmonization for human-robot collaborative handling tasks. In: Procedia CIRP Design. (Accepted)

Book chapter

1. Manns, M., Tuli, T.B., 2025. Digital human modeling for human robot interaction (Accepted for publication). Springer Nature.

Miscellaneous

1. Tuli, T.B., Saeed, R., and Manns, M., 2024. Experimental design for human-to-human collaboration during collaborative object handling using digital twin (Poster presentation, ZESS PhD Forum).
2. Tuli, T.B., Saeed, R., and Manns, M., 2024. Target position-constrained ‘reach’ human motion exhibits either Gaussian or Gaussian Mixture (Poster presentation, ZESS Open day).

Project team

• Univ.-Prof. Dr.-Ing. Martin Manns
  Chair leader, FAMS - Manufacturing Automation and Assembly
  E-mail: martin.manns@uni-siegen.de

• Dr.-Ing. Tadele B. Tuli,
  Chief Engineer (Collaborative robotics), FAMS
  E-mail: tadele-belay.tuli@uni-siegen.de

• Dr. Raza Saeed,
  HiSMoT Project leader, FAMS
  E-mail: raza.saeed@uni-siegen.de