Architecture and System Co-Design for High-performance LLM Inference and Training

Overview

Artificial intelligence breakthroughs like LLMs have led to a sharp increase in computing power demand. Computing power data centers extensively use GPU processors and NPU processors to meet their computing needs, becoming the main engines of computing infrastructure. On the other hand, AI models can continuously learn and evolve during interaction with users, such that they will frequently switch between training and inference tasks. Thus, the integrated architecture of training and inference will become increasingly popular. Compared to the separate training and inference architecture, the integrated training and inference architecture has advantages in response speed, operational cost, and resource utilization, such as reducing data transmission and storage redundancy. However, the integrated training and inference architecture also brings new full-stack system challenges, including numerical data types, multi-operator fusion, memory management, and co-located deployment. In this talk, I will share our group’s ideas, preliminary explorations, and practices in the integrated training and inference system architecture, while proposing future prospects to fully utilize the potential of the integrated training and inference architecture.