As deep learning models grow in size to achieve state-of-the-art accuracy, there is a pressing need for compact models. To address this challenge, we introduce a novel operation called Personal Self-Attention (PSA). It is specifically designed to learn non-linear 1-D functions, enhancing existing spline-based methods while remaining compatible with gradient backpropagation. By integrating these non-linear functions with linear transformations, we can achieve the accuracy of larger models but with significantly smaller hidden dimensions, which is crucial for FPGA implementations. We evaluate PSA by implementing it in a Multi-Layer Perceptron (MLP)-based vision model, ResMLP, and testing it on the CIFAR-10 classification task. MLP is gaining increasing popularity due to its widespread use in large-language models. Our results confirm that PSA achieves equivalent accuracy with a 2x smaller hidden size compared to conventional MLPs. Furthermore, by quantizing our non-linear function into a simple lookup table, we reduce the number of operations required by 45%-28%, which offers significant benefits for hardware accelerators. To showcase this, we design an end-to-end unrolled streaming accelerator for ResMLP, demonstrating that our compressed model maintains an 88% accuracy while reducing LUT + DSP resource requirements by 25%, and doubling throughput to 32kFPS. Additionally, we implement a fixed-size SIMD accelerator for the same compressed model that achieves a 62.1% improvement in throughput while only consuming 3.5% extra LUTs.

Gaurav Singh

Kia Bazargan

ACM Transactions on Reconfigurable Technology and Systems

TRETS 是专注于可重构系统及其基础技术研究的顶级期刊。其他期刊的范围、基本原理和报道通常仅限于可重构技术或可重构系统的特定方面。 TRETS 是一本涵盖可重构性的期刊。适合 TRETS 的主题将包括所有级别的可重构系统抽象和可重构技术的所有方面，包括平台、编程环境和支持这些系统用于计算或其他应用程序的应用程序成功.-可重构平台的板和系统架构。-可重构系统的编程环境，尤其是那些设计用于可重构系统的环境，这将提高程序员的生产力。-可重构系统的语言和编译器。-逻辑综合和相关工具，因为它们与可重构系统有关。-可以证明成功的应用程序。开发可重构系统的基础技术。 （目前这项技术是 FPGA 的技术，但对后续技术的性质和使用的研究适合 TRETS。）在考虑一篇论文是否适合 TRETS 时，最重要的问题应该是可重构性是否对成功至关重要。如果上下文合适，架构、编程语言、编译器和环境、逻辑综合和高性能应用程序等主题都是合适的。例如，碰巧使用 FPGA 的嵌入式应用架构不一定适合 TRETS，但使用 FPGA 的架构，其 FPGA 的可重构性是规范的固有部分（可能是由于需要重用在多个应用程序上）将适用于 TRETS。

ACM Transactions on Reconfigurable Technology and Systems (TRETS) is a premier journal dedicated to research on reconfigurable systems and their underlying technologies. Unlike other journals that may focus on specific aspects of reconfigurable technology or systems, TRETS covers a broader range of topics. Suitable subjects for TRETS include all levels of abstraction for reconfigurable systems and all aspects of reconfigurable technology, including platforms, programming environments, and applications that support the successful use of these systems in computing or other applications. Specifically, TRETS focuses on the following areas: hardware and system architecture of reconfigurable platforms; programming environments for reconfigurable systems, especially those designed to enhance programmer productivity; programming languages and compilers for reconfigurable systems; logic synthesis and related tools as they pertain to reconfigurable systems; and foundational technologies for developing successful applications (currently primarily FPGA technology, but research on future technologies is also appropriate for TRETS). When considering whether a paper is suitable for TRETS, the most important question is whether reconfigurability is critical to its success. If the context is appropriate, topics such as architecture, programming languages, compilers and environments, logic synthesis, and high-performance applications are all suitable. For example, while an embedded application architecture that happens to use FPGA may not be suitable for TRETS, an architecture where the reconfigurability of the FPGA is an inherent part of the specification (possibly due to the need for reuse across multiple applications) would be appropriate for TRETS.

ACM Transactions on Reconfigurable Technology and Systems（TRETS）是一个专注于可重构系统及其基础技术研究的顶级期刊。与其他期刊相比，TRETS 的范围更广，涵盖了可重构技术和系统的各个方面。适合发表在 TRETS 上的主题包括：可重构系统的各级抽象和技术，包括平台、编程环境，以及支持这些系统在计算或其他应用中成功运行的应用程序。具体来说，TRETS 关注以下几个方面：可重构平台的硬件和系统架构；可重构系统的编程环境，尤其是那些旨在提高程序员生产力的环境；可重构系统的编程语言和编译器；与可重构系统相关的逻辑综合和工具；以及成功应用程序的开发基础技术（目前主要是 FPGA 技术，但未来技术的研究也适合 TRETS）。在考虑论文是否适合发表在 TRETS 时，最重要的标准是可重构性是否对成功至关重要。如果上下文合适，架构、编程语言、编译器和环境、逻辑综合以及高性能应用程序等主题都是合适的。例如，虽然使用 FPGA 的嵌入式应用架构不一定适合 TRETS，但如果 FPGA 的可重构性是设计的固有部分（例如由于需要在多个应用程序中重用），那么这样的架构就适合发表在 TRETS 上。

Category	Quartile
计算机科学	4
计算机科学, 计算机硬件	4

Compressing Neural Networks Using Learnable 1-D Non-Linear Functions

ivySCI AI Smartly Parses PDF, Answers Researchers' Questions, and Helps You Understand Papers in Seconds

Journal Info

Category	Quartile
COMPUTER SCIENCE, HARDWARE & ARCHITECTURE	2