Accelerating Edge AI Models on Jacinto7 Processors with TIDL

Our team specializes in empowering clients to develop Edge AI solutions by leveraging Texas Instruments' Deep Learning (TIDL) capabilities on Jacinto7 processors. The Jacinto7 series, specifically designed for automotive and industrial applications, includes powerful components like the C7x DSP and matrix multiplication accelerator (mma), offering significant computational power for AI workloads.

We have successfully partnered with multiple clients, each aiming to improve the performance and energy efficiency of AI models on constrained edge hardware setups. Our expertise spans not only optimizing pre-existing models but also implementing custom operators specifically designed to run on the C7x DSP, allowing us to tailor solutions to the unique needs of our clients.

The Challenges in Executing AI Inference on Edge Devices

Clients initially implemented their AI models on the Jacinto7’s Cortex A72 cores. While the Cortex A72 provided a general-purpose processing platform, the complexity and size of AI models led to sub-optimal inference speeds and higher power consumption, limiting their real-time capabilities. Furthermore, some clients required highly specialized operators that weren’t part of standard deep learning libraries.

Our task was to optimize these models using the C7x DSP+mma accelerator and develop custom operators to handle specific computations efficiently, enabling real-time performance and reduced energy consumption.

The Solution

Using TIDL, we helped our clients migrate their AI workloads from the Cortex A72 to the C7x DSP+mma accelerator. TIDL provides an efficient framework for optimizing and deploying deep learning models on specialized hardware in Jacinto7, offering substantial benefits in terms of speed and power efficiency. We also implemented custom operators to handle specific operations that were not supported natively, further enhancing performance.

Optimization and Custom Operator Implementation

• Model Conversion and Quantization: We first converted clients' pre-trained models to a format compatible with the Jacinto7 TIDL framework. Optimizations, such as 8-bit quantization, were applied to reduce the memory footprint and computational demand without significantly affecting model accuracy.
• Custom Operators: For specialized tasks not covered by standard deep learning layers, we developed custom operators that were efficiently executed on the C7x DSP. These custom operators allowed our clients to extend the functionality of their models and achieve the performance they required for specific applications.
• Offloading Computation to C7x DSP + MMA: By leveraging the C7x DSP and mma accelerator, computationally intensive tasks such as convolutions, matrix multiplications, and custom operations were handled by dedicated hardware, ensuring maximum throughput and minimal latency.

Expected Results for inference Acceleration on C7x DSP + MMA

By migrating AI models from the Cortex A72 to the C7x DSP+mma, our clients can expect substantial improvements in inference speed, energy efficiency, and memory usage, particularly when executing well-known AI models. Below are some expected performance improvements based on industry benchmarks and our expertise in optimizing models.

C7x*MMA Performance Examples on Well-Known Models:

TIDL Key Expected Optimizations

• Inference Time: By offloading model execution to the C7x DSP+mma, we estimate a 15x reduction in inference time compared to the Cortex A72, as seen in models like ResNet-50 and YOLOv3-Tiny.
• Energy Efficiency: With the C7x DSP+mma optimized for AI workloads, we expect a significant reduction in power consumption, typically 30-40% lower than running on the Cortex A72.
• Memory Efficiency: Quantized models will reduce memory usage by around 50%, enabling the models to fit into constrained memory environments common in edge devices.

Conclusion

By leveraging the TIDL framework and the C7x DSP+mma on the Jacinto7 platform, our team delivers highly optimized AI solutions that offer significant improvements in inference speed, energy efficiency, and scalability. Clients can expect dramatic performance boosts for popular AI models, and our capability to implement custom operators ensures that specialized AI workloads can also benefit from these optimizations.

The C7x DSP and mma accelerators offer vastly superior performance compared to the Cortex A72, making them the go-to solution for real-time Edge AI applications. This case study reflects our deep expertise in accelerating AI models and implementing custom solutions tailored to the specific needs of our clients.