A multimodal visual guided robot collaborative system based on the classification of multi-class human motion

V. Voronin; M. Zhdanova; O. Tokareva; E. Semenishchev; A. Zelensky; N. Gapon

doi:10.1117/12.2691154

27 November 2023 A multimodal visual guided robot collaborative system based on the classification of multi-class human motion

V. Voronin, M. Zhdanova, O. Tokareva, E. Semenishchev, A. Zelensky, N. Gapon

Author Affiliations +

Proceedings Volume 12766, Advanced Optical Imaging Technologies VI; 1276614 (2023) https://doi.org/10.1117/12.2691154
Event: SPIE/COS Photonics Asia, 2023, Beijing, China

Conference Poster

Abstract

The key technology of modern intelligent production is the concept of human-robot interaction. Operators and robots collaborate to perform complex tasks in various scenarios in such a system. Understanding human behavior introduces challenges for collaborative robot systems to perform efficient tasks in unstructured and dynamic environments. The proposed algorithm is based on classifying multi-class human motion from multimodal visual data. For this purpose, we apply the Laplace pyramid for visible, depth, and thermal data from imaging sensors. To prevent the appearance of visual artifacts, a multiscale approach is proposed for combining images based on the Laplace pyramid and calculating the optimal weights. The next stage is fused data preprocessing by the two-sided Gabor quaternionic Fourier transform and calculating 3D local binary dense micro-block difference. Also, we obtain human skeleton data using a convolutional neural network. Based on the coordinates of the unique points of the human body skeleton and the distances between them, a descriptor is constructed that describes the person's posture on each frame and considers the time information between frames. At the next stage, the 3D local binary dense micro-block difference and skeletal descriptor are combined into a single feature vector. Moreover, finally, this descriptor is classified by a neural network. We present simulation modeling on the effectiveness of the proposed action recognition algorithm in the RoboGuid environment.

(2023) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

V. Voronin, M. Zhdanova, O. Tokareva, E. Semenishchev, A. Zelensky, and N. Gapon "A multimodal visual guided robot collaborative system based on the classification of multi-class human motion", Proc. SPIE 12766, Advanced Optical Imaging Technologies VI, 1276614 (27 November 2023); https://doi.org/10.1117/12.2691154

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available