CascadeLUT: Info.-Ordered Streaming Inference for Bandwidth-Constrained FPGAs (arxiv.org)

Researchers have introduced CascadeLUT, an innovative framework aimed at improving streaming inference on bandwidth-constrained FPGAs (Field-Programmable Gate Arrays). Unlike traditional methods that rely on complete data availability, CascadeLUT restructures the input data into ordered subsets, enabling progressive predictions as information is received. This approach mitigates bottlenecks caused by large input transfers, optimizing data movement while ensuring deterministic inference without runtime branching. The significance of CascadeLUT lies in its remarkable efficiency improvements, achieving latency reductions of 4.0 to 12.5 times and throughput increases of 3.0 to 5.0 times compared to prior LUT-based models. Furthermore, it offers up to 13.8 times lower energy consumption per sample, making it a highly attractive solution for real-time applications. The co-design of feature scheduling with hardware dataflow not only enhances accuracy but also integrates on-device input quantization, leading to a five-fold decrease in quantization overhead. This advancement represents a crucial step forward for the AI/ML community, particularly for deployments where bandwidth limitations are a critical concern.