Large language models (LLMs) have become increasingly popular due to their exceptional performance in various artificial intelligence applications. However, their development often suffers from the scarcity of high-quality data and the extensive requirements for computing resources. These obstacles are even more severe for enterprises in vertical industries, which have limited computer resources but urgently require large-scale models for specific activities. To address these issues, LLMs call for the integration of federated learning (FL), which enables the collaborative learning of a powerful LLM using private data and computing resources from multiple entities. In this article, we present a systematic introduction to the federated large language model (Fed-LLM), a distributed learning of LLM in the FL manner. We first introduce the learning paradigm of Fed-LLM, which is called federated parameter-efficient fine-tuning (Fed-PEFT). Fed-PEFT empowers the collaborative fine-tuning of pre-trained LLMs by only involving a small subset of parameters in local LLMs. Specifically, we detail the workflow of Fed-PE-FT, and summarize the state-of-the-art solutions in this area. Additionally, we discuss the challenges faced in Fed-LLMs, including efficiency, privacy, and security. Finally, we introduce future directions to facilitate the research of Fed-LLMs and guide coming explorations in this nascent field.

Jiahui Hu

Dan Wang

Zhibo Wang

Xiaoyi Pang

Huiyu Xu

Ju Ren

Kui Ren

IEEE Wireless Communications

IEEE 无线通信杂志处理所有媒体中与个性化、与位置无关的通信相关的所有技术和政策问题。论文重点介绍了诸如便携式电话、通信掌上电脑、协议、消息传递、通信和个性化流量过滤等主题。它还涵盖了频谱分配、产业结构和技术演进等政策问题。

The IEEE Wireless Communications magazine addresses all technical and policy issues related to personalized and location-independent communication across all media. The papers focus on topics such as portable phones, communication PDAs, protocols, messaging, communication, and personalized traffic filtering. It also covers policy issues related to spectrum allocation, industry structure, and technological evolution.

《IEEE Wireless Communications》杂志涵盖了与个性化和位置无关的通信技术及政策问题。文章重点讨论便携式电话、通信掌上电脑、协议、消息传递、通信和个性化流量过滤等主题。此外，杂志还涉及频谱分配、产业结构和技术演进等政策问题。

Category	Quartile
计算机科学	1
计算机科学, 计算机硬件	1
计算机科学, 计算机信息系统	1
计算机科学, 工程电子与电气	1
计算机科学, 电信学	1

Federated Large Language Model: Solutions, Challenges and Future Directions

ivySCI AI Smartly Parses PDF, Answers Researchers' Questions, and Helps You Understand Papers in Seconds

Journal Info