About Guangyu Chen
Guangyu Chen (Nathan) is a machine learning researcher at Kimi, Moonshot AI, specializing in model architecture, efficient attention mechanisms, and continual learning. Despite being a high school student from Shenzhen, China, he has made significant contributions to AI research, most notably as the co-first author of the "Attention Residuals" paper, which proposes a novel approach to replacing fixed residual connections in Transformers with learned, input-dependent attention over depth. The paper garnered widespread attention and praise from Elon Musk and Andrej Karpathy. Chen's journey into ML began through contributing to the Flash Linear Attention open-source project, which led to his invitation to join the Kimi team.
Education
Senior Year
Currently a high school senior at an international school in Shenzhen, Guangdong, China. His interest in machine learning began through self-directed learning, competitive programming on Codeforces, and contributing to open-source projects.
Skills
Core technical competencies in machine learning research, model architecture, and systems optimization.
Efficient Attention
Model Architecture
Linear Attention
Residual Connections
Scaling Laws
Continual Learning
Triton Kernels
CUDA Optimization
GPU Compute
PyTorch
Distributed Training
Codeforces
Algorithms
Flash Linear Attention
Open Source Development
Python
Domains
Efficient Attention Mechanisms, Transformer Architecture, Model Scaling
Continual Learning, Hardware-Aligned ML Algorithms, Interpretability
Tags
Self-taught, Open Source Advocate, Prodigious, Driven, Collaborative
Novel Architectures, Scaling Efficiency, Depth-wise Aggregation, Linear Attention
Experience
ML Researcher
Working on ML research at Kimi, focusing on model architecture and efficient attention mechanisms. Co-first author of the Attention Residuals paper, which proposes replacing fixed residual connections with softmax attention over preceding layer outputs, achieving approximately 25% compute efficiency improvement on the Kimi Linear architecture (48B total / 3B activated parameters).
Research Intern
Completed a seven-week internship at a Silicon Valley AI startup. Managed a project involving 144 H100 GPUs and engaged directly with leadership on operational matters including recruitment and financing discussions.
Research Contributor
Worked on applied interpretability research, exploring how neural networks process and represent information internally.
Notable Work
Co-first author. Proposes AttnRes, replacing fixed residual accumulation with softmax attention over preceding layer outputs. Validated on Kimi Linear 48B architecture with 1.4T tokens, achieving ~25% compute efficiency improvement with <2% inference latency overhead. Praised by Elon Musk and Andrej Karpathy.
Core contributor to the FLA open-source project (4.7K+ GitHub stars), providing efficient implementations of state-of-the-art linear attention models in PyTorch and Triton.
Profile Summary
Efficient Attention, Model Architecture, Linear Attention, Continual Learning, Open Source
Twitter, LinkedIn, GitHub
Related Profiles
Alfred Paul "Al" Seckel was an American cognitive neuroscientist, author, and prominent skeptic known for his work in popularizing optical illusions and perceptual phenomena. He was an instructor at the California Institute of Technology and co-founded the Southern California Skeptics, serving as its executive director. Seckel lectured extensively on illusions at prestigious institutions like Harvard, MIT, and Caltech, using them as a window into the hidden rules of the human perceptual system.
Alan Dershowitz is an American lawyer and legal scholar, renowned for his expertise in U.S. constitutional and criminal law. He served as a professor at Harvard Law School for nearly five decades, where he was the Felix Frankfurter Professor of Law. Dershowitz is also a prominent defender of civil liberties and human rights, known for his extensive writings and high-profile litigation.
Alexey Kurakin is a Senior Staff Research Engineer at Google DeepMind, specializing in the security and privacy of machine learning, large language models (LLMs), and differential privacy. He has a long tenure at Google, previously working as a Research Software Engineer at Google Brain, where he focused on fundamental research and applications in machine learning. Kurakin is a prolific researcher, publishing papers at top machine learning conferences like NeurIPS, ICLR, and ICML, and is highly cited in the field.
Anca Dragan is a prominent figure in AI and robotics, currently serving as the Director of AI Safety and Alignment at Google DeepMind while on leave from her role as an Associate Professor at UC Berkeley's EECS Department. Her research, conducted through the InterACT Lab, focuses on algorithms for human-AI and human-robot interaction, particularly on AI alignment to ensure agents act in accordance with human goals and values. She is recognized for her work in reward engineering and enabling AI agents to work collaboratively with people.
Andrew Ng is a globally recognized leader in Artificial Intelligence, known for co-founding Coursera and founding DeepLearning.AI, AI Fund, and LandingAI. He is a former head of Baidu AI Group and the founder of the Google Brain team. He currently serves as an Adjunct Professor of Computer Science at Stanford University.
Been Kim is a Senior Staff Research Scientist at Google DeepMind, specializing in interpretable machine learning (Explainable AI). Her research focuses on developing human-centered tools and concepts to help people communicate with and understand complex ML models. She is known for her work on TCAV (Testing with Concept Activation Vectors), which received the UNESCO Netexplo award, and has given keynotes at major conferences like ICLR 2022 and ECML 2020.
Chelsea Finn is an Assistant Professor of Computer Science and Electrical Engineering at Stanford University, where she directs the IRIS (Intelligence, Robotics, and Interactive Systems) Lab. She is a leading American computer scientist specializing in machine learning, robotics, and reinforcement learning. Finn is also the co-founder of Physical Intelligence, a company focused on developing intelligent physical systems. Her work has been highly influential, evidenced by over 100,000 citations.
Dawn Song is a highly distinguished Professor of Computer Science at UC Berkeley, where she also co-directs the Berkeley RDI Center. Her research focuses on critical areas including AI safety and security, Agentic AI, deep learning, and decentralization technology. A serial entrepreneur, she is the founder of Oasis Labs and is recognized for her significant contributions to computer security and privacy, including being a recipient of the prestigious MacArthur Fellowship.
Fei-Fei Li is a Chinese-born American computer scientist, best known as the inventor of ImageNet and the ImageNet Challenge, a pivotal dataset that fueled the rapid advancement of modern computer vision and deep learning. She is the Sequoia Professor of Computer Science at Stanford University and the Co-Director of the Stanford Institute for Human-Centered Artificial Intelligence (HAI). She also co-founded the non-profit AI4ALL and is the CEO and Cofounder of World Labs.