Large language models (LLMs) like BERT and GPT have revolutionized artificial intelligence, enabling machines to perform complex tasks such as translation, summarization, and dialogue with human-like proficiency. Despite their prowess, these models often rely on formidable computational power provided by hefty servers or cloud infrastructure, posing hurdles for real-time, private, and low-latency applications specifically on mobile devices.
To address these challenges, researchers at Sejong University have pioneered the Scalable Transformer Accelerator Unit (STAU), a cutting-edge hardware innovation designed to facilitate efficient on-device execution of transformer-based language models. Their work, prominently featured in the journal Electronics, details an adaptable solution that scales to different input sizes and model architectures, making it particularly amenable to on-device artificial intelligence requirements.
The innovation at STAU’s core is the Variable Systolic Array (VSA) architecture. This design is crucial for scaling matrix operations—the heart of transformer models—by easily adjusting to varying input sequence lengths. This adaptability is indispensable for large language models that process different sentence lengths and token sequences. Within this architecture, input data streams row by row while weights are paralleled, cutting down memory lags and boosting throughput.
Benchmark assessments have revealed that STAU delivers an impressive 3.45× performance boost compared to traditional CPU-only executions, maintaining over 97% numerical accuracy, even when managing extended input sequences. With further optimization, this increase extends to a 5.18× speedup, underscoring the architecture’s robust scalability.
An intriguing enhancement in STAU’s design is its revamped softmax function. Given its historical role as a computational bottleneck due to its complexity, the researchers streamlined it using a Radix-2 approach, which reduces complexity without sacrificing performance. Moreover, the developers introduced a proprietary 16-bit floating-point format, tailored for the intricate demands of transformer tasks, circumventing the need for exhaustive layer normalization stages.
STAU was brought to life on a Xilinx FPGA and managed in conjunction with an Arm Cortex-R5 processor. This blend of hardware and software equips developers with the flexibility to support an array of transformer models, including LLMs. As a result, functionality can be enhanced with straightforward software updates rather than costly and complicated hardware overhauls.
This development marks a crucial advancement in the domain of AI technology, propelling sophisticated language models to be more accessible and widely applicable, particularly on mobile platforms, wearables, and edge systems. By facilitating on-device AI execution, it lays the groundwork for integrating transformative AI applications into our everyday gadgets.
Key Takeaways:
- Sejong University’s Scalable Transformer Accelerator (STAU) advances the deployment of LLMs on embedded systems, connecting the gap for on-device AI.
- With a Variable Systolic Array architecture, STAU achieves substantial speed enhancements while maintaining precision.
- Refined processing techniques, including a streamlined softmax function and custom floating-point format, augment model efficiency and adaptability.
- STAU empowers small devices to run advanced AI models effectively, broadening AI’s applications beyond data servers and cloud environments.