NEWS · RESEARCH · #141
FLoKD: Adaptive knowledge distillation for federated LoRA fine-tuning of LLMs over wireless networks
This arXiv preprint proposes FLoKD, an adaptive knowledge-distillation framework for federated LoRA fine-tuning of large language models over bandwidth‑constrained wireless networks. Instead of sharing full parameters or token logits, FLoKD transmits intermediate LoRA activations and selectively sends only important transformer blocks and public samples aligned with local distributions; experiments on WikiText-103, PTB and a dialog dataset report 50–65% communication reduction while converging to competitive perplexity compared to baselines.
KEY POINTS
- This arXiv preprint proposes FLoKD, an adaptive knowledge-distillation framework for federated LoRA fine-tuning of large language models over bandwidth‑constrained wireless networks.
- Instead of sharing full parameters or token logits, FLoKD transmits intermediate LoRA activations and selectively sends only important transformer blocks and public samples aligned with local distributions; experiments on WikiText-103, PTB and a dialog dataset report 50–65% communication reduction while converging to competitive perplexity compared to baselines.
- If validated widely, FLoKD could materially reduce communication costs for privacy-preserving federated fine-tuning of LLMs on bandwidth‑limited wireless devices by replacing logits/parameter transfers with selective activation-based distillation.
WHY IT MATTERS
If validated widely, FLoKD could materially reduce communication costs for privacy-preserving federated fine-tuning of LLMs on bandwidth‑limited wireless devices by replacing logits/parameter transfers with selective activation-based distillation.