About Me
I am a system researcher at Huawei (HKRC). I obtained my PhD degree in Computer Science and Engineering from The Hong Kong University of Science and Technology, advised by Prof. Bo Li. Prior to that, I received my Bachelor’s Degree from Zhejiang University.
I am broadly interested in machine learning systems and algorithms, with a special focus on: 1) distributed DNN training, 2) second-order optimization, and 3) learning on graphs. I am currently working on LLMs training and inference.
News
- 08/2023: I passed my PhD defense.
- 04/2023: Two papers are accepted to ICDCS 2023.
- 01/2023: One paper is accepted to ICLR 2023.
- 12/2022: One paper is accepted to INFOCOM 2023.
- 09/2022: One paper is accepted to NeurIPS 2022.
- 09/2022: One paper is accepted to TCC 2022.
- 06/2022: One paper is accepted to TKDD 2022.
Publications
(* indicates equal contributions, + indicates the corresponding author)
Preprints
- Longteng Zhang*, Lin Zhang*, Shaohuai Shi, Xiaowen Chu, and Bo Li, “LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning”, arXiv preprint, Aug 2023. [PDF]
- Lin Zhang, Shaohuai Shi, and Bo Li, “Eva: A General Vectorized Approximation Framework for Second-order Optimization”, arXiv preprint, Aug 2023. [PDF]
Conference Publications
- Lin Zhang, Shaohuai Shi, Xiaowen Chu, Wei Wang, Bo Li, and Chengjian Liu, “DeAR: Accelerating Distributed Deep Learning with Fine-Grained All-Reduce Pipelining”, in the IEEE International Conference on Distributed Computing Systems (ICDCS), 2023. [PDF, Code]
- Lin Zhang, Longteng Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li, “Evaluation and Optimization of Gradient Compression for Distributed Deep Learning”, in the IEEE International Conference on Distributed Computing Systems (ICDCS), 2023. [PDF, Code]
- Lin Zhang, Shaohuai Shi, and Bo Li, “Eva: Practical Second-order Optimization with Kronecker-vectorized Approximation”, in the International Conference on Learning Representations (ICLR), 2023. [PDF, Code]
- Lin Zhang, Shaohuai Shi, and Bo Li, “Accelerating Distributed K-FAC with Efficient Collective Communication and Scheduling”, in the IEEE International Conference on Computer Communications (INFOCOM), 2023.
- Barakeel Fanseu Kamhoua, Lin Zhang+, Yongqiang Chen, Han Yang, Kaili Ma, Bo Han, Bo Li, and James Cheng, “Exact Shape Correspondence via 2D graph convolution”, in the Conference on Neural Information Processing Systems (NeurIPS), 2022. [PDF, Code] [Spotlight]
- Barakeel Fanseu Kamhoua*, Lin Zhang*, Kaili Ma, James Cheng, Bo Li and Bo Han, “HyperGraph Convolution Based Attributed HyperGraph Clustering”, in the ACM Conference on Information and Knowledge Management (CIKM), 2021. [PDF, Code]
- Shaohuai Shi, Lin Zhang, and Bo Li, “Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks”, in the IEEE International Conference on Distributed Computing Systems (ICDCS), 2021. [PDF, Code]
- Yuqing Li, Hok Chun Ng, Lin Zhang, and Bo Li, “Online Cooperative Resource Allocation at the Edge: A Privacy-Preserving Approach”, in the IEEE International Conference on Network Protocols (ICNP), 2020. [PDF]
Journal Publications
- Lin Zhang, Shaohuai Shi, Wei Wang, and Bo Li, “Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning”, in the IEEE Transactions on Cloud Computing (TCC), 2022. [PDF, Code]
- Barakeel Fanseu Kamhoua*, Lin Zhang*, Kaili Ma, James Cheng, Bo Li and Bo Han, “GRACE: A General Graph Convolution Framework for Attributed Graph Clustering”, in the ACM Transactions on Knowledge Discovery in Data (TKDD), 2022. [PDF, Code]
Work Experience
- 09/2021-08/2022, Huawei HKRC Intern
Teaching Assistant
- Spring 2019, Introduction to Computing with Excel VBA
- Fall 2019, Fall 2022, Operating Systems
Awards and Honours
- Zhejiang Provincial Government Scholarship, 2016
- Excellent Graduate Student of Zhejiang University, 2018
- Huawei PhD Fellowship, 2019
Technical Skills
- General: Python, C/C++, Linux Shell
- Deep Learning Framework: PyTorch, Horovod, Megatron-LM, DeepSpeed