Efficient Deep Neural Network Inference with Near-Edge Accelerator Offload

Overview

Deep neural networks enable a wide range of intelligent real world applications that require accurate and efficient inference. Many of these must be deployed in the real world, favoring the use of compute resources tailored to such scenarios. However, the increasing computational and memory demands of modern DNNs make inference on resource-constrained edge devices challenging, while offloading to cloud datacenters incurs substantial communication latency, bandwidth consumption, and energy costs. Recent advances in hardware accelerators provide enhanced computational capability and energy efficiency in smaller form factors, offering an intermediate computing tier between the edge and cloud, which can support new approaches to offloading. This thesis investigates how to effectively exploit this new class of near-edge accelerators to optimize distributed DNN inference.

This thesis first extends a traditional single split edge-cloud inference to a two-split approach across the edge, near-edge, and cloud tiers. Autoencoders at each split compress data to reduce communication overhead, while a profiling-based two-split search algorithm identifies latency and energy trade-offs under predefined accuracy thresholds without exhaustively fine-tuning all candidate configurations. Building on this framework, the thesis then considers a more general multi-split inference scheme across the edge and multiple near-edge accelerators. Within this setting, different compression methods are compared and a more precise performance model is developed to capture device heterogeneity, DNN runtime execution, intra-device data movement, and inter-device communication. This model is validated on a real edge and near-edge testbed, achieving mean absolute percentage errors in DNN latency prediction of 3.7% on VGG16 and 2.2% on ResNet50, compared with 38.9% and 15.7% for previous models. It can be applied to guide multi-split search, dynamic bandwidth adaptation, and shared near-edge offloading. Finally, the thesis demonstrates that collaborative inference for more advanced Vision Transformer architectures can also benefit from near-edge accelerators. The proposed framework orchestrates a lightweight generalist model on the edge and multiple specialist experts on a near-edge accelerator. A proposed routing mechanism repurposes the edge model's predictions as a router with negligible overhead, dynamically routing uncertain samples from the edge to the most relevant expert for refinement. Furthermore, a progressive specialist training strategy effectively enforces expert specialization on target subsets while preserving overall generalization capability, raising expert accuracy on target subsets by 4.12% and overall accuracy by 2.76%. Overall, our framework achieves accuracy gains of up to 1.77% over generalist co-inference, while reducing latency by up to 46% and energy consumption by 46% compared to standalone inference.