Real-time human action recognition from aerial videos using autozoom and synthetic data

Ruiqi Xian; Bryan I. Vogel; Celso M De Melo; Andre V. Harrison; Dinesh Manocha

doi:10.1117/12.3013547

7 June 2024 Real-time human action recognition from aerial videos using autozoom and synthetic data

Ruiqi Xian, Bryan I. Vogel, Celso M De Melo, Andre V. Harrison, Dinesh Manocha

Proceedings Volume 13035, Synthetic Data for Artificial Intelligence and Machine Learning: Tools, Techniques, and Applications II; 130350I (2024) https://doi.org/10.1117/12.3013547
Event: SPIE Defense + Commercial Sensing, 2024, National Harbor, Maryland, United States

Abstract

In this paper, we propose a novel approach for real-time human action recognition (HAR) on resource-constrained UAVs. Our approach tackles the limited availability of labeled UAV video data (compared to ground-based datasets) by incorporating synthetic data augmentation to improve the performance of a lightweight action recognition model. This combined strategy offers a robust and efficient solution for UAV-based HAR.We evaluate our method on the RoCoG v2¹ and UAV-Human² datasets, showing a notable increase in top-1 accuracy across all scenarios on RoCoG: 9.1% improvement when training with synthetic data only, 6.9% with real data only, and the highest improvement of 11.8% with a combined approach. Additionally, using an X3D backbone further improves accuracy on the UAV-Human dataset by 5.5%. Our models deployed on a Qualcomm Robotics RB5 platform achieve real-time predictions at approximately 10 frames per second (fps) and demonstrate a superior trade-off between performance and inference rate on both low-power edge devices and high-end desktops.

Conference Presentation

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Ruiqi Xian, Bryan I. Vogel, Celso M De Melo, Andre V. Harrison, and Dinesh Manocha "Real-time human action recognition from aerial videos using autozoom and synthetic data", Proc. SPIE 13035, Synthetic Data for Artificial Intelligence and Machine Learning: Tools, Techniques, and Applications II, 130350I (7 June 2024); https://doi.org/10.1117/12.3013547

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available