LCIP: a retargetable framework for optimized CNN inference

Lei Pan; Yan Zhang; Shuvra S. Bhattacharyya

doi:10.1117/12.2662945

13 June 2023 LCIP: a retargetable framework for optimized CNN inference

Lei Pan, Yan Zhang, Shuvra S. Bhattacharyya

Proceedings Volume 12528, Real-Time Image Processing and Deep Learning 2023; 1252803 (2023) https://doi.org/10.1117/12.2662945
Event: SPIE Defense + Commercial Sensing, 2023, Orlando, Florida, United States

Abstract

Deep Neural Networks (DNNs) are adopted in numerous application areas of signal and information processing with Convolutional Neural Networks (CNNs) being a particularly popular class of DNNs. Many machine learning (ML) frameworks have evolved for design and training of CNN models, and similarly, a wide variety of target platforms, ranging from mobile and resource-constrained platforms to desktop and more powerful platforms, are used to deploy CNN-equipped applications. To help designers navigate the complex design spaces involved in deploying CNN models derived from ML frameworks on alternative processing platforms, retargetable methods for implementing CNN models are of increasing interest. In this paper, we present a novel software tool, called the Lightweight-dataflow-based CNN Inference Package (LCIP), for retargetable, optimized CNN inference on different hardware platforms (e.g., x86 and ARM CPUs, and GPUs). In LCIP, source code for CNN operators (convolution, pooling, etc.) derived from ML frameworks is wrapped within dataflow actors. The resulting coarse grain dataflow models are then optimized using the retargetable LCIP runtime engine, which employs higher- level dataflow analysis and orchestration that is complementary to the intra-operator performance optimizations provided by the ML framework and the back-end development tools of the target platform. Additionally, LCIP enables heterogeneous and distributed edge inference of CNNs by offloading part of the CNN to additional devices, such as onboard GPU or network devices. Our experimental results show that LCIP provides significant improvements in inference throughput on commonly-used CNN architectures, and the improvement is consistent across desktop and resource-constrained platforms.

Conference Presentation

Citation Download Citation

Lei Pan, Yan Zhang, and Shuvra S. Bhattacharyya "LCIP: a retargetable framework for optimized CNN inference", Proc. SPIE 12528, Real-Time Image Processing and Deep Learning 2023, 1252803 (13 June 2023); https://doi.org/10.1117/12.2662945

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available