Awesome Transformers LM Analytics

This paper list focuses on the theoretical and empirical analysis of language models, especially large language models (LLMs). The papers in this list investigate the learning behavior, generalization ability, and other properties of language models through theoretical analysis, empirical analysis, or a combination of both.

Scope of this list:

• Currently, this list focuses on transformer-based models.
• We hope to collect papers that only focus on the theoretical and empirical analysis of language models, instead of papers that aim to improve the performance of language models.

Limitations of this list:

• This list is not exhaustive, and we may miss some very important papers.
• This list is not well-organized yet, and we may need to reorganize the list in the future.
• Some popular topics are not well-covered yet, such as mechanistic engineering, probing, and interpretability.

Statistics of This paper list:

• Total number of different papers: 279
• For more detailed statistics, please refer to the end of this page.

If you have any suggestions or want to contribute, please feel free to open an issue or a pull request.

For details on how to contribute, please refer to the contribution guidelines.

You can also share your thoughts and discuss with others in the Discussions.

Table of Content

• Awesome Transformers LM Analytics
• Table of Content
  • Phenomena of Interest
    • In-Context Learning
    • Chain-of-Thought
    • Hallucination
    • Reversal Curse
    • Scaling Laws / Emergent Abilities / Grokking / etc.
    • Knowledge / Memory mechanisms
    • Training Dynamics / Landscape / Optimization / Fine-tuning / etc.
    • Learning / Generalization / Reasoning / Weak to Strong Generalization
    • Other Phenomena
  • Representational Capacity
    • What Can Transformer Do? / Properties of Transformer
    • What Can Transformer Not Do? / Limitation of Transformer
  • Architectural Effectivity
    • Layer-normalization
    • Tokenization
    • Linear Attention / State Space Models / etc.
  • Training Paradigms
  • Mechanistic Engineering / Probing / Interpretability
  • Miscellanea

Phenomena of Interest

Here are some phenomena that are interesting to investigate in language models.

In-Context Learning

paper list (click to fold / unfold)

• State Soup: In-Context Skill Learning, Retrieval and Mixing [paper link] 2024-06-12
  Maciej Pióro; Maciej Wołczyk; Razvan Pascanu; Johannes von Oswald; João Sacramento

• Estimating the Hallucination Rate of Generative AI [paper link] 2024-06-11
  Andrew Jesson; Nicolas Beltran-Velez; Quentin Chu; Sweta Karlekar; Jannik Kossen; Yarin Gal; John P. Cunningham; David Blei

• BERTs are Generative In-Context Learners [paper link] 2024-06-07
  David Samuel

• Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective [paper link] 2024-06-06
  Xinhao Yao; Xiaolin Hu; Shenzhi Yang; Yong Liu

• What Do Language Models Learn in Context? The Structured Task Hypothesis [paper link] 2024-06-06
  Jiaoda Li; Yifan Hou; Mrinmaya Sachan; Ryan Cotterell

• Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers [paper link] 2024-06-05
  Brian K Chen; Tianyang Hu; Hui Jin; Hwee Kuan Lee; Kenji Kawaguchi

• Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks [paper link] 2024-06-04
  Tianyu He; Darshil Doshi; Aritra Das; Andrey Gromov

• Why Larger Language Models Do In-context Learning Differently? [paper link] 2024-05-30
  Zhenmei Shi; Junyi Wei; Zhuoyan Xu; Yingyu Liang

• Is In-Context Learning Sufficient for Instruction Following in LLMs? [paper link] 2024-05-30
  Hao Zhao; Maksym Andriushchenko; Francesco Croce; Nicolas Flammarion

• Does learning the right latent variables necessarily improve in-context learning? [paper link] 2024-05-29
  Sarthak Mittal; Eric Elmoznino; Leo Gagnon; Sangnie Bhardwaj; Dhanya Sridhar; Guillaume Lajoie

• A Theory of In-Context Learning in Transformers [paper link] 2024-05-29
  Yifei Wang; Yuyang Wu; Zeming Wei; Stefanie Jegelka; Yisen Wang

• On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability [paper link] 2024-05-27
  Chenyu Zheng; Wei Huang; Rongzhen Wang; Guoqiang Wu; Jun Zhu; Chongxuan Li

• Transformer In-Context Learning for Categorical Data [paper link] 2024-05-27
  Aaron T. Wang; Ricardo Henao; Lawrence Carin

• Automatic Domain Adaptation by Transformers in In-Context Learning [paper link] 2024-05-27
  Ryuichiro Hataya; Kota Matsui; Masaaki Imaizumi

• Unifying Demonstration Selection and Compression for In-Context Learning [paper link] 2024-05-27
  Jun Gao

• On the Noise Robustness of In-Context Learning for Text Generation [paper link] 2024-05-27
  Hongfu Gao; Feipeng Zhang; Wenyu Jiang; Jun Shu; Feng Zheng; Hongxin Wei

• MLPs Learn In-Context [paper link] 2024-05-24
  William L. Tong; Cengiz Pehlevan

• Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification [paper link] 2024-05-24
  Shang Liu; Zhongze Cai; Guanting Chen; Xiaocheng Li

• In-Context Learning with Long-Context Models: An In-Depth Exploration [paper link] 2024-04-30
  Amanda Bertsch; Maor Ivgi; Uri Alon; Jonathan Berant; Matthew R. Gormley; Graham Neubig

• What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation [paper link] 2024-04-10
  Aaditya K. Singh; Ted Moskovitz; Felix Hill; Stephanie C. Y. Chan; Andrew M. Saxe

• Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability [paper link] 2024-04-01
  Ivan Lee; Nan Jiang; Taylor Berg-Kirkpatrick

• Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality [paper link] 2024-02-29
  Siyu Chen; Heejune Sheen; Tianhao Wang; Zhuoran Yang

• How Transformers Learn Causal Structure with Gradient Descent [paper link] 2024-02-22
  Eshaan Nichani; Alex Damian; Jason D. Lee

• In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initialization [paper link] 2024-02-22
  Ruiqi Zhang; Jingfeng Wu; Peter L. Bartlett

• Identifying Semantic Induction Heads to Understand In-Context Learning [paper link] 2024-02-20
  Jie Ren; Qipeng Guo; Hang Yan; Dongrui Liu; Xipeng Qiu; Dahua Lin

• An Information-Theoretic Analysis of In-Context Learning [paper link] 2024-01-28
  Hong Jun Jeon; Jason D. Lee; Qi Lei; Benjamin Van Roy

• The Transient Nature of Emergent In-Context Learning in Transformers [paper link] 2023-12-11
  Aaditya K. Singh; Stephanie C. Y. Chan; Ted Moskovitz; Erin Grant; Andrew M. Saxe; Felix Hill

• In-Context Learning Functions with Varying Number of Minima [paper link] 2023-11-21
  David Oniani; Yanshan Wang

• Exploring the Relationship between In-Context Learning and Instruction Tuning [paper link] 2023-11-17
  Hanyu Duan; Yixuan Tang; Yi Yang; Ahmed Abbasi; Kar Yan Tam

• When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks [paper link] 2023-11-15
  Hao Peng; Xiaozhi Wang; Jianhui Chen; Weikai Li; Yunjia Qi; Zimu Wang; Zhili Wu; Kaisheng Zeng; Bin Xu; Lei Hou; Juanzi Li

• In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax [paper link] 2023-11-13
  Aaron Mueller; Albert Webson; Jackson Petty; Tal Linzen

• Transformers learn to implement preconditioned gradient descent for in-context learning [paper link] 2023-11-09
  Kwangjun Ahn; Xiang Cheng; Hadi Daneshmand; Suvrit Sra

• Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study with Linear Models [paper link] 2023-10-26
  Deqing Fu; Tian-Qi Chen; Robin Jia; Vatsal Sharan

• In-Context Learning Creates Task Vectors [paper link] 2023-10-24
  Roee Hendel; Mor Geva; Amir Globerson

• Function Vectors in Large Language Models [paper link] 2023-10-23
  Eric Todd; Millicent L. Li; Arnab Sen Sharma; Aaron Mueller; Byron C. Wallace; David Bau

• In-context Learning with Transformer Is Really Equivalent to a Contrastive Learning Pattern [paper link] 2023-10-19
  Ruifeng Ren; Yong Liu

• Trained Transformers Learn Linear Models In-Context [paper link] 2023-10-19
  Ruiqi Zhang; Spencer Frei; Peter L. Bartlett

• How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations [paper link] 2023-10-16
  Tianyu Guo; Wei Hu; Song Mei; Huan Wang; Caiming Xiong; Silvio Savarese; Yu Bai

• Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions [paper link] 2023-10-13
  Satwik Bhattamishra; Arkil Patel; Phil Blunsom; Varun Kanade

• How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? [paper link] 2023-10-13
  Jingfeng Wu; Difan Zou; Zixiang Chen; Vladimir Braverman; Quanquan Gu; Peter Bartlett

• In-Context Learning Learns Label Relationships but Is Not Conventional Learning [paper link] 2023-10-13
  Jannik Kossen; Yarin Gal; Tom Rainforth

• In-context Convergence of Transformers [paper link] 2023-10-13
  Yu Huang; Yuan Cheng; Yingbin Liang

• In-Context Learning through the Bayesian Prism [paper link] 2023-10-13
  Madhur Panwar; Kabir Ahuja; Navin Goyal

• Do pretrained Transformers Really Learn In-context by Gradient Descent? [paper link] 2023-10-12
  Lingfeng Shen; Aayush Mishra; Daniel Khashabi

• What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalization [paper link] 2023-10-10
  Yufeng Zhang; Fengzhuo Zhang; Zhuoran Yang; Zhaoran Wang

• Explaining Emergent In-Context Learning as Kernel Regression [paper link] 2023-10-05
  Chi Han; Ziqi Wang; Han Zhao; Heng Ji

• CausalLM is not optimal for in-context learning [paper link] 2023-09-02
  Nan Ding; Tomer Levinboim; Jialin Wu; Sebastian Goodman; Radu Soricut

• One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention [paper link] 2023-07-07
  Arvind Mahankali; Tatsunori B. Hashimoto; Tengyu Ma

• Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection [paper link] 2023-07-06
  Yu Bai; Fan Chen; Huan Wang; Caiming Xiong; Song Mei

• Transformers Learn In-Context by Gradient Descent [paper link] 2023-06-15
  Johannes Von Oswald; Eyvind Niklasson; Ettore Randazzo; Joao Sacramento; Alexander Mordvintsev; Andrey Zhmoginov; Max Vladymyrov

• The Closeness of In-Context Learning and Weight Shifting for Softmax Regression [paper link] 2023-04-26
  Shuai Li; Zhao Song; Yu Xia; Tong Yu; Tianyi Zhou

• A Theory of Emergent In-Context Learning as Implicit Structure Induction [paper link] 2023-03-14
  Michael Hahn; Navin Goyal

• The Learnability of In-Context Learning [paper link] 2023-03-14
  Noam Wies; Yoav Levine; Amnon Shashua

• What Can Transformers Learn In-Context? A Case Study of Simple Function Classes [paper link] 2023-01-14
  Shivam Garg; Dimitris Tsipras; Percy Liang; Gregory Valiant

• Transformers generalize differently from information stored in context vs in weights [paper link] 2022-10-13
  Stephanie C. Y. Chan; Ishita Dasgupta; Junkyung Kim; Dharshan Kumaran; Andrew K. Lampinen; Felix Hill

• In-Context Learning and Induction Heads [paper link] 2022-09-24
  Catherine Olsson; Nelson Elhage; Neel Nanda; Nicholas Joseph; Nova DasSarma; Tom Henighan; Ben Mann; Amanda Askell; Yuntao Bai; Anna Chen; Tom Conerly; Dawn Drain; Deep Ganguli; Zac Hatfield-Dodds; Danny Hernandez; Scott Johnston; Andy Jones; Jackson Kernion; Liane Lovitt; Kamal Ndousse; Dario Amodei; Tom Brown; Jack Clark; Jared Kaplan; Sam McCandlish; Chris Olah

Chain-of-Thought

paper list (click to fold / unfold)

• Iteration Head: A Mechanistic Study of Chain-of-Thought [paper link] 2024-06-04
  Vivien Cabannes; Charles Arnal; Wassim Bouaziz; Alice Yang; Francois Charton; Julia Kempe

• Let's Think Dot by Dot: Hidden Computation in Transformer Language Models [paper link] 2024-04-24
  Jacob Pfau; William Merrill; Samuel R. Bowman

• Chain of Thought Empowers Transformers to Solve Inherently Serial Problems [paper link] 2024-02-20
  Zhiyuan Li; Hong Liu; Denny Zhou; Tengyu Ma

• Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective [paper link] 2023-12-22
  Guhao Feng; Bohang Zhang; Yuntian Gu; Haotian Ye; Di He; Liwei Wang

• Why Can Large Language Models Generate Correct Chain-of-Thoughts? [paper link] 2023-10-20
  Rasul Tutunov; Antoine Grosnit; Juliusz Ziomek; Jun Wang; Haitham Bou-Ammar

• How Large Language Models Implement Chain-of-Thought? [paper link] 2023-10-13
  Yiqun Wang; Sile Hu; Yonggang Zhang; Xiang Tian; Xuesong Liu; Yaowu Chen; Xu Shen; Jieping Ye

• The Expressive Power of Transformers with Chain of Thought [paper link] 2023-10-13
  William Merrill; Ashish Sabharwal

Hallucination

paper list (click to fold / unfold)

• Estimating the Hallucination Rate of Generative AI [paper link] 2024-06-11
  Andrew Jesson; Nicolas Beltran-Velez; Quentin Chu; Sweta Karlekar; Jannik Kossen; Yarin Gal; John P. Cunningham; David Blei

• Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? [paper link] 2024-05-09
  Zorik Gekhman; Gal Yona; Roee Aharoni; Matan Eyal; Amir Feder; Roi Reichart; Jonathan Herzig

• Mechanisms of non-factual hallucinations in language models [paper link] 2024-03-26
  Lei Yu; Meng Cao; Jackie Chi Kit Cheung; Yue Dong

• Unfamiliar Finetuning Examples Control How Language Models Hallucinate [paper link] 2024-03-08
  Katie Kang; Eric Wallace; Claire Tomlin; Aviral Kumar; Sergey Levine

• In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation [paper link] 2024-03-05
  Shiqi Chen; Miao Xiong; Junteng Liu; Zhengxuan Wu; Teng Xiao; Siyang Gao; Junxian He

• Calibrated Language Models Must Hallucinate [paper link] 2023-11-24
  Adam Tauman Kalai; Santosh S. Vempala

• The Curious Case of Hallucinatory Unanswerablity: Finding Truths in the Hidden States of Over-Confident Large Language Models [paper link] 2023-10-18
  Aviv Slobodkin; Omer Goldman; Avi Caciularu; Ido Dagan; Shauli Ravfogel

Reversal Curse

paper list (click to fold / unfold)

• Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics [paper link] 2024-05-07
  Hanlin Zhu; Baihe Huang; Shaolun Zhang; Michael Jordan; Jiantao Jiao; Yuandong Tian; Stuart Russell

• The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" [paper link] 2024-04-04
  Lukas Berglund; Meg Tong; Max Kaufmann; Mikita Balesni; Asa Cooper Stickland; Tomasz Korbak; Owain Evans

• An Investigation of LLMs' Inefficacy in Understanding Converse Relations [paper link] 2023-12-01
  Chengwen Qi; Bowen Li; Binyuan Hui; Bailin Wang; Jinyang Li; Jinwang Wu; Yuanjun Laili

• Physics of Language Models: Part 3.2, Knowledge Manipulation [paper link] 2023-09-25
  Zeyuan Allen-Zhu; Yuanzhi Li

• The Reversal Curse: Which Tokens You Predict Underlie the Factorization Curse and More [paper link] 2023-06-07
  Ouail Kitouni; Niklas Nolte; Diane Bouchacourt; Adina Williams; Mike Rabbat; Mark Ibrahim

Scaling Laws / Emergent Abilities / Grokking / etc.

This section includes papers that investigate how the performance of language models scales with model size, data size, or compute, and how emergent abilities arise in language models.

paper list (click to fold / unfold)

• Deep Grokking: Would Deep Neural Networks Generalize Better? [paper link] 2024-05-29
  Simin Fan; Razvan Pascanu; Martin Jaggi

• Linguistic Collapse: Neural Collapse in (Large) Language Models [paper link] 2024-05-28
  Robert Wu; Vardan Papyan

• Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations [paper link] 2024-05-28
  Alexander Hägele; Elie Bakouch; Atli Kosson; Loubna Ben Allal; Leandro Von Werra; Martin Jaggi

• gzip Predicts Data-dependent Scaling Laws [paper link] 2024-05-26
  Rohan Pandey

• Emergence of a High-Dimensional Abstraction Phase in Language Transformers [paper link] 2024-05-24
  Emily Cheng; Diego Doimo; Corentin Kervadec; Iuri Macocco; Jade Yu; Alessandro Laio; Marco Baroni

• A rationale from frequency perspective for grokking in training neural network [paper link] 2024-05-24
  Zhangchen Zhou; Yaoyu Zhang; Zhi-Qin John Xu

• Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization [paper link] 2024-05-23
  Boshi Wang; Xiang Yue; Yu Su; Huan Sun

• Data Mixing Made Efficient: A Bivariate Scaling Law for Language Model Pretraining [paper link] 2024-05-23
  Ce Ge; Zhijian Ma; Daoyuan Chen; Yaliang Li; Bolin Ding

• 4+3 Phases of Compute-Optimal Neural Scaling Laws [paper link] 2024-05-23
  Elliot Paquette; Courtney Paquette; Lechao Xiao; Jeffrey Pennington

• Slaves to the Law of Large Numbers: An Asymptotic Equipartition Property for Perplexity in Generative Language Models [paper link] 2024-05-22
  Raghu Mudumbai; Tyler Bell

• Quantifying Emergence in Large Language Models [paper link] 2024-05-21
  Hang Chen; Xinyu Yang; Jiaying Zhu; Wenya Wang

• Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory [paper link] 2024-05-14
  Xueyan Niu; Bo Bai; Lei Deng; Wei Han

• More Compute Is What You Need [paper link] 2024-04-30
  Zhen Guo

• An exactly solvable model for emergence and scaling laws [paper link] 2024-04-26
  Yoonsoo Nam; Nayara Fonseca; Seok Hyeong Lee; Ard Louis

• Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck [paper link] 2024-04-11
  Nathan Godey; Éric de la Clergerie; Benoît Sagot

• A Large-Scale Exploration of $\mu$-Transfer [paper link] 2024-04-08
  Lucas Lingle

• Emergent Abilities in Reduced-Scale Generative Language Models [paper link] 2024-04-02
  Sherin Muckatira; Vijeta Deshpande; Vladislav Lialin; Anna Rumshisky

• Understanding Emergent Abilities of Language Models from the Loss Perspective [paper link] 2024-03-23
  Zhengxiao Du; Aohan Zeng; Yuxiao Dong; Jie Tang

• Unraveling the Mystery of Scaling Laws: Part I [paper link] 2024-03-21
  Hui Su; Zhi Tian; Xiaoyu Shen; Xunliang Cai

• Language models scale reliably with over-training and on downstream tasks [paper link] 2024-03-13
  Samir Yitzhak Gadre; Georgios Smyrnis; Vaishaal Shankar; Suchin Gururangan; Mitchell Wortsman; Rulin Shao; Jean Mercat; Alex Fang; Jeffrey Li; Sedrick Keh; Rui Xin; Marianna Nezhurina; Igor Vasiljevic; Jenia Jitsev; Alexandros G. Dimakis; Gabriel Ilharco; Shuran Song; Thomas Kollar; Yair Carmon; Achal Dave; Reinhard Heckel; Niklas Muennighoff; Ludwig Schmidt

• When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method [paper link] 2024-02-26
  Biao Zhang; Zhongtao Liu; Colin Cherry; Orhan Firat

• Interpreting Grokked Transformers in Complex Modular Arithmetic [paper link] 2024-02-26
  Hiroki Furuta; Gouki Minegishi; Yusuke Iwasawa; Yutaka Matsuo

• A Tale of Tails: Model Collapse as a Change of Scaling Laws [paper link] 2024-02-10
  Elvis Dohmatob; Yunzhen Feng; Pu Yang; Francois Charton; Julia Kempe

• Scaling Data-Constrained Language Models [paper link] 2023-10-25
  Niklas Muennighoff; Alexander M. Rush; Boaz Barak; Teven Le Scao; Aleksandra Piktus; Nouamane Tazi; Sampo Pyysalo; Thomas Wolf; Colin Raffel

• The Cost of Down-Scaling Language Models: Fact Recall Deteriorates before In-Context Learning [paper link] 2023-10-06
  Tian Jin; Nolan Clement; Xin Dong; Vaishnavh Nagarajan; Michael Carbin; Jonathan Ragan-Kelley; Gintare Karolina Dziugaite

• Are Emergent Abilities of Large Language Models a Mirage? [paper link] 2023-04-28
  Rylan Schaeffer; Brando Miranda; Sanmi Koyejo

• Training Compute-Optimal Large Language Models [paper link] 2022-03-29
  Jordan Hoffmann; Sebastian Borgeaud; Arthur Mensch; Elena Buchatskaya; Trevor Cai; Eliza Rutherford; Diego de Las Casas; Lisa Anne Hendricks; Johannes Welbl; Aidan Clark; Tom Hennigan; Eric Noland; Katie Millican; George van den Driessche; Bogdan Damoc; Aurelia Guy; Simon Osindero; Karen Simonyan; Erich Elsen; Jack W. Rae; Oriol Vinyals; Laurent Sifre

• Scaling Laws for Neural Language Models [paper link] 2020-01-22
  Jared Kaplan; Sam McCandlish; Tom Henighan; Tom B. Brown; Benjamin Chess; Rewon Child; Scott Gray; Alec Radford; Jeffrey Wu; Dario Amodei

Knowledge / Memory mechanisms

paper list (click to fold / unfold)

• Knowledge Circuits in Pretrained Transformers [paper link] 2024-05-28
  Yunzhi Yao; Ningyu Zhang; Zekun Xi; Mengru Wang; Ziwen Xu; Shumin Deng; Huajun Chen

• Upper and lower memory capacity bounds of transformers for next-token prediction [paper link] 2024-05-22
  Liam Madden; Curtis Fox; Christos Thrampoulidis

• A Multi-Perspective Analysis of Memorization in Large Language Models [paper link] 2024-05-19
  Bowen Chen; Namgi Han; Yusuke Miyao

• Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws [paper link] 2024-04-08
  Zeyuan Allen-Zhu; Yuanzhi Li

• Memorization Capacity of Multi-Head Attention in Transformers [paper link] 2024-03-02
  Sadegh Mahdavi; Renjie Liao; Christos Thrampoulidis

• Birth of a Transformer: A Memory Viewpoint [paper link] 2023-11-06
  Alberto Bietti; Vivien Cabannes; Diane Bouchacourt; Herve Jegou; Leon Bottou

• Physics of Language Models: Part 3.2, Knowledge Manipulation [paper link] 2023-09-25
  Zeyuan Allen-Zhu; Yuanzhi Li

• Can Neural Network Memorization Be Localized? [paper link] 2023-07-18
  Pratyush Maini; Michael C. Mozer; Hanie Sedghi; Zachary C. Lipton; J. Zico Kolter; Chiyuan Zhang

• Quantifying Memorization Across Neural Language Models [paper link] 2022-02-15
  Nicholas Carlini; Daphne Ippolito; Matthew Jagielski; Katherine Lee; Florian Tramer; Chiyuan Zhang

Training Dynamics / Landscape / Optimization / Fine-tuning / etc.

This section focuses on the training dynamics of language models, including the optimization landscape, fine-tuning, and transfer learning.

paper list (click to fold / unfold)

• Understanding Linear Probing then Fine-tuning Language Models from NTK Perspective [paper link] 2024-05-27
  Akiyoshi Tomihari; Issei Sato

• Infinite Limits of Multi-head Transformer Dynamics [paper link] 2024-05-24
  Blake Bordelon; Hamza Tahir Chaudhry; Cengiz Pehlevan

• Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics [paper link] 2024-05-07
  Hanlin Zhu; Baihe Huang; Shaolun Zhang; Michael Jordan; Jiantao Jiao; Yuandong Tian; Stuart Russell

• Control Theoretic Approach to Fine-Tuning and Transfer Learning [paper link] 2024-04-16
  Erkan Bayram; Shenyu Liu; Mohamed-Ali Belabbas; Tamer Başar

• Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Think [paper link] 2024-04-12
  Xinpeng Wang; Chengzhi Hu; Bolei Ma; Paul Röttger; Barbara Plank

• On Training Data Influence of GPT Models [paper link] 2024-04-11
  Qingyi Liu; Yekun Chai; Shuohuan Wang; Yu Sun; Keze Wang; Hua Wu

• Best Practices and Lessons Learned on Synthetic Data for Language Models [paper link] 2024-04-11
  Ruibo Liu; Jerry Wei; Fangyu Liu; Chenglei Si; Yanzhe Zhang; Jinmeng Rao; Steven Zheng; Daiyi Peng; Diyi Yang; Denny Zhou; Andrew M. Dai

• How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse [paper link] 2024-04-07
  Mohamed El Amine Seddik; Suei-Wen Chen; Soufiane Hayou; Pierre Youssef; Merouane Debbah

• Unveiling the Generalization Power of Fine-Tuned Large Language Models [paper link] 2024-03-14
  Haoran Yang; Yumeng Zhang; Jiaqi Xu; Hongyuan Lu; Pheng Ann Heng; Wai Lam

• Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models [paper link] 2024-03-14
  Akhil Kedia; Mohd Abbas Zaidi; Sushil Khyalia; Jungho Jung; Harshith Goka; Haejun Lee

• Linear Attention is (Maybe) All You Need (to Understand Transformer Optimization) [paper link] 2024-03-13
  Kwangjun Ahn; Xiang Cheng; Minhak Song; Chulhee Yun; Ali Jadbabaie; Suvrit Sra

• Hallmarks of Optimization Trajectories in Neural Networks and LLMs: The Lengths, Bends, and Dead Ends [paper link] 2024-03-12
  Sidak Pal Singh; Bobby He; Thomas Hofmann; Bernhard Schölkopf

• The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models [paper link] 2024-03-06
  Adithya Bhaskar; Dan Friedman; Danqi Chen

• Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality [paper link] 2024-02-29
  Siyu Chen; Heejune Sheen; Tianhao Wang; Zhuoran Yang

• How Transformers Learn Causal Structure with Gradient Descent [paper link] 2024-02-22
  Eshaan Nichani; Alex Damian; Jason D. Lee

• LoRA Training in the NTK Regime has No Spurious Local Minima [paper link] 2024-02-19
  Uijeong Jang; Jason D. Lee; Ernest K. Ryu

• Transformers learn through gradual rank increase [paper link] 2023-12-10
  Enric Boix-Adsera; Etai Littwin; Emmanuel Abbe; Samy Bengio; Joshua Susskind

• Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks [paper link] 2023-11-21
  Samyak Jain; Robert Kirk; Ekdeep Singh Lubana; Robert P. Dick; Hidenori Tanaka; Edward Grefenstette; Tim Rocktäschel; David Scott Krueger

• Connecting Pre-trained Language Model and Downstream Task via Properties of Representation [paper link] 2023-11-02
  Chenwei Wu; Holden Lee; Rong Ge

• Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer [paper link] 2023-07-02
  Yuandong Tian; Yiping Wang; Beidi Chen; Simon Du

• A Kernel-Based View of Language Model Fine-Tuning [paper link] 2023-06-15
  Sadhika Malladi; Alexander Wettig; Dingli Yu; Danqi Chen; Sanjeev Arora

• A Stability Analysis of Fine-Tuning a Pre-Trained Model [paper link] 2023-01-24
  Zihao Fu; Anthony Man-Cho So; Nigel Collier

Learning / Generalization / Reasoning / Weak to Strong Generalization

This section includes papers that investigate the generalization ability of language models, and the general learning behavior of language models.

paper list (click to fold / unfold)

• How Truncating Weights Improves Reasoning in Language Models [paper link] 2024-06-05
  Lei Chen; Joan Bruna; Alberto Bietti

• Understanding Transformer Reasoning Capabilities via Graph Algorithms [paper link] 2024-05-28
  Clayton Sanford; Bahare Fatemi; Ethan Hall; Anton Tsitsulin; Mehran Kazemi; Jonathan Halcrow; Bryan Perozzi; Vahab Mirrokni

• Linguistic Collapse: Neural Collapse in (Large) Language Models [paper link] 2024-05-28
  Robert Wu; Vardan Papyan

• Reality Only Happens Once: Single-Path Generalization Bounds for Transformers [paper link] 2024-05-26
  Yannick Limmer; Anastasis Kratsios; Xuwei Yang; Raeid Saqur; Blanka Horvath

• A statistical framework for weak-to-strong generalization [paper link] 2024-05-25
  Seamus Somerstep; Felipe Maia Polo; Moulinath Banerjee; Ya'acov Ritov; Mikhail Yurochkin; Yuekai Sun

• Theoretical Analysis of Weak-to-Strong Generalization [paper link] 2024-05-25
  Hunter Lang; David Sontag; Aravindan Vijayaraghavan

• Quantifying the Gain in Weak-to-Strong Generalization [paper link] 2024-05-24
  Moses Charikar; Chirag Pabbaraju; Kirankumar Shiragur

• Towards Understanding How Transformer Perform Multi-step Reasoning with Matching Operation [paper link] 2024-05-24
  Zhiwei Wang; Yunji Wang; Zhongwang Zhang; Zhangchen Zhou; Hui Jin; Tianyang Hu; Jiacheng Sun; Zhenguo Li; Yaoyu Zhang; Zhi-Qin John Xu

• Initialization is Critical to Whether Transformers Fit Composite Functions by Inference or Memorizing [paper link] 2024-05-08
  Zhongwang Zhang; Pengxiao Lin; Zhiwei Wang; Yaoyu Zhang; Zhi-Qin John Xu

• When can transformers reason with abstract symbols? [paper link] 2024-04-16
  Enric Boix-Adsera; Omid Saremi; Emmanuel Abbe; Samy Bengio; Etai Littwin; Joshua Susskind

• A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Task [paper link] 2024-02-19
  Jannik Brinkmann; Abhay Sheshadri; Victor Levoso; Paul Swoboda; Christian Bartelt

• Provably learning a multi-head attention layer [paper link] 2024-02-06
  Sitan Chen; Yuanzhi Li

• Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks [paper link] 2023-11-21
  Samyak Jain; Robert Kirk; Ekdeep Singh Lubana; Robert P. Dick; Hidenori Tanaka; Edward Grefenstette; Tim Rocktäschel; David Scott Krueger

• The Impact of Depth and Width on Transformer Language Model Generalization [paper link] 2023-10-30
  Jackson Petty; Sjoerd van Steenkiste; Ishita Dasgupta; Fei Sha; Dan Garrette; Tal Linzen

• On the Optimization and Generalization of Multi-head Attention [paper link] 2023-10-19
  Puneesh Deora; Rouzbeh Ghaderi; Hossein Taheri; Christos Thrampoulidis

• Large Language Models Cannot Self-Correct Reasoning Yet [paper link] 2023-10-13
  Jie Huang; Xinyun Chen; Swaroop Mishra; Huaixiu Steven Zheng; Adams Wei Yu; Xinying Song; Denny Zhou

• How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition [paper link] 2023-10-09
  Guanting Dong; Hongyi Yuan; Keming Lu; Chengpeng Li; Mingfeng Xue; Dayiheng Liu; Wei Wang; Zheng Yuan; Chang Zhou; Jingren Zhou

• A Theory for Emergence of Complex Skills in Language Models [paper link] 2023-07-29
  Sanjeev Arora; Anirudh Goyal

• On the Power of Foundation Models [paper link] 2023-07-03
  Yang Yuan

• Task-Specific Skill Localization in Fine-tuned Language Models [paper link] 2023-06-15
  Abhishek Panigrahi; Nikunj Saunshi; Haoyu Zhao; Sanjeev Arora

• Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Tasks [paper link] 2023-02-11
  Jiachun Pan; Pan Zhou; Shuicheng Yan

• Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models [paper link] 2022-10-25
  Hong Liu; Sang Michael Xie; Zhiyuan Li; Tengyu Ma

• Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning [paper link] 2022-04-20
  Colin Wei; Sang Michael Xie; Tengyu Ma

• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks [paper link] 2021-04-14
  Nikunj Saunshi; Sadhika Malladi; Sanjeev Arora

• Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning [paper link] 2020-12-22
  Armen Aghajanyan; Luke Zettlemoyer; Sonal Gupta

• How fine can fine-tuning be? Learning efficient language models [paper link] 2020-06-03
  Evani Radiya-Dixit; Xin Wang

Other Phenomena

paper list (click to fold / unfold)

• Linguistic Collapse: Neural Collapse in (Large) Language Models [paper link] 2024-05-28
  Robert Wu; Vardan Papyan

• Exploring Activation Patterns of Parameters in Language Models [paper link] 2024-05-28
  Yudong Wang; Damai Dai; Zhifang Sui

• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs [paper link] 2024-05-26
  Mustafa Shukor; Matthieu Cord

• Your Transformer is Secretly Linear [paper link] 2024-05-19
  Anton Razzhigaev; Matvey Mikhalchuk; Elizaveta Goncharova; Nikolai Gerasimenko; Ivan Oseledets; Denis Dimitrov; Andrey Kuznetsov

• The Platonic Representation Hypothesis [paper link] 2024-05-13
  Minyoung Huh; Brian Cheung; Tongzhou Wang; Phillip Isola

• Algorithmic progress in language models [paper link] 2024-03-09
  Anson Ho; Tamay Besiroglu; Ege Erdil; David Owen; Robi Rahman; Zifan Carl Guo; David Atkinson; Neil Thompson; Jaime Sevilla

• Massive Activations in Large Language Models [paper link] 2024-02-27
  Mingjie Sun; Xinlei Chen; J. Zico Kolter; Zhuang Liu

• The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers [paper link] 2023-02-01
  Zonglin Li; Chong You; Srinadh Bhojanapalli; Daliang Li; Ankit Singh Rawat; Sashank J. Reddi; Ke Ye; Felix Chern; Felix Yu; Ruiqi Guo; Sanjiv Kumar

Representational Capacity

Investigate the expressiveness of transformer-based models about what they can do and what they can't do.

What Can Transformer Do? / Properties of Transformer

This section includes positive results on the representational capacity and properties of transformer-based models.

paper list (click to fold / unfold)

• Seperations in the Representational Capabilities of Transformers and Recurrent Architectures [paper link] 2024-06-13
  Satwik Bhattamishra; Michael Hahn; Phil Blunsom; Varun Kanade

• Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot [paper link] 2024-06-11
  Zixuan Wang; Stanley Wei; Daniel Hsu; Jason D. Lee

• What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages [paper link] 2024-06-07
  Nadav Borenstein; Anej Svete; Robin Chan; Josef Valvoda; Franz Nowak; Isabelle Augenstein; Eleanor Chodroff; Ryan Cotterell

• Transformers Can Do Arithmetic with the Right Embeddings [paper link] 2024-05-27
  Sean McLeish; Arpit Bansal; Alex Stein; Neel Jain; John Kirchenbauer; Brian R. Bartoldson; Bhavya Kailkhura; Abhinav Bhatele; Jonas Geiping; Avi Schwarzschild; Tom Goldstein

• A One-Layer Decoder-Only Transformer is a Two-Layer RNN: With an Application to Certified Robustness [paper link] 2024-05-27
  Yuhao Zhang; Aws Albarghouthi; Loris D'Antoni

• The Power of Hard Attention Transformers on Data Sequences: A Formal Language Theoretic Perspective [paper link] 2024-05-25
  Pascal Bergsträßer; Chris Köcher; Anthony Widjaja Lin; Georg Zetzsche

• Transformers represent belief state geometry in their residual stream [paper link] 2024-05-24
  Adam S. Shai; Sarah E. Marzen; Lucas Teixeira; Alexander Gietelink Oldenziel; Paul M. Riechers

• ALPINE: Unveiling the Planning Capability of Autoregressive Learning in Language Models [paper link] 2024-05-15
  Siwei Wang; Yifei Shen; Shi Feng; Haoran Sun; Shang-Hua Teng; Wei Chen

• What Formal Languages Can Transformers Express? A Survey [paper link] 2024-05-06
  Lena Strobl; William Merrill; Gail Weiss; David Chiang; Dana Angluin

• Transformers Can Represent $n$-gram Language Models [paper link] 2024-04-23
  Anej Svete; Ryan Cotterell

• Mechanics of Next Token Prediction with Self-Attention [paper link] 2024-04-18
  Yingcong Li; Yixiao Huang; Muhammed E. Ildiz; Ankit Singh Rawat; Samet Oymak

• When can transformers reason with abstract symbols? [paper link] 2024-04-16
  Enric Boix-Adsera; Omid Saremi; Emmanuel Abbe; Samy Bengio; Etai Littwin; Joshua Susskind

• The Illusion of State in State-Space Models [paper link] 2024-04-12
  William Merrill; Jackson Petty; Ashish Sabharwal

• Attention is Naturally Sparse with Gaussian Distributed Input [paper link] 2024-04-03
  Yichuan Deng; Zhao Song; Chiwun Yang

• What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks [paper link] 2024-04-01
  Xingwu Chen; Difan Zou

• The Topos of Transformer Networks [paper link] 2024-03-27
  Mattia Jacopo Villani; Peter McBurney

• Simulating Weighted Automata over Sequences and Trees with Transformers [paper link] 2024-03-12
  Michael Rizvi; Maude Lizaire; Clara Lacroce; Guillaume Rabusseau

• Simplicity Bias of Transformers to Learn Low Sensitivity Functions [paper link] 2024-03-11
  Bhavya Vasudeva; Deqing Fu; Tianyi Zhou; Elliott Kau; Youqi Huang; Vatsal Sharan

• On the Origins of Linear Representations in Large Language Models [paper link] 2024-03-06
  Yibo Jiang; Goutham Rajendran; Pradeep Ravikumar; Bryon Aragam; Victor Veitch

• How Well Can Transformers Emulate In-context Newton's Method? [paper link] 2024-03-05
  Angeliki Giannou; Liu Yang; Tianhao Wang; Dimitris Papailiopoulos; Jason D. Lee

• RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval [paper link] 2024-02-29
  Kaiyue Wen; Xingyu Dang; Kaifeng Lyu

• Implicit Bias of Next-Token Prediction [paper link] 2024-02-28
  Christos Thrampoulidis

• On the Expressive Power of a Variant of the Looped Transformer [paper link] 2024-02-21
  Yihang Gao; Chuanyang Zheng; Enze Xie; Han Shi; Tianyang Hu; Yu Li; Michael K. Ng; Zhenguo Li; Zhaoqiang Liu

• From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers [paper link] 2024-02-20
  M. Emrullah Ildiz; Yixiao Huang; Yingcong Li; Ankit Singh Rawat; Samet Oymak

• Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context [paper link] 2024-02-15
  Xiang Cheng; Yuxin Chen; Suvrit Sra

• Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks [paper link] 2024-02-05
  Rahul Ramesh; Ekdeep Singh Lubana; Mikail Khona; Robert P. Dick; Hidenori Tanaka

• Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? [paper link] 2024-01-29
  Tokio Kajitsuka; Issei Sato

• Transformers are Multi-State RNNs [paper link] 2024-01-11
  Matanel Oren; Michael Hassid; Yossi Adi; Roy Schwartz

• How Capable Can a Transformer Become? A Study on Synthetic, Interpretable Tasks [paper link] 2023-12-12
  Rahul Ramesh; Mikail Khona; Robert P. Dick; Hidenori Tanaka; Ekdeep Singh Lubana

• Transformers can optimally learn regression mixture models [paper link] 2023-11-14
  Reese Pathak; Rajat Sen; Weihao Kong; Abhimanyu Das

• The Expressive Power of Low-Rank Adaptation [paper link] 2023-10-26
  Yuchen Zeng; Kangwook Lee

• What Algorithms can Transformers Learn? A Study in Length Generalization [paper link] 2023-10-24
  Hattie Zhou; Arwen Bradley; Etai Littwin; Noam Razin; Omid Saremi; Josh Susskind; Samy Bengio; Preetum Nakkiran

• Transformers as Support Vector Machines [paper link] 2023-09-07
  Davoud Ataee Tarzanagh; Yingcong Li; Christos Thrampoulidis; Samet Oymak

• How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding [paper link] 2023-06-15
  Yuchen Li; Yuanzhi Li; Andrej Risteski

• Tighter Bounds on the Expressivity of Transformer Encoders [paper link] 2023-06-15
  David Chiang; Peter Cholak; Anand Pillay

• Fast Attention Requires Bounded Entries [paper link] 2023-02-26
  Josh Alman; Zhao Song

• Transformers Learn Shortcuts to Automata [paper link] 2023-02-01
  Bingbin Liu; Jordan T. Ash; Surbhi Goel; Akshay Krishnamurthy; Cyril Zhang

• Transformer Vs. MLP-Mixer: Exponential Expressive Gap For NLP Problems [paper link] 2022-11-17
  Dan Navon; Alex M. Bronstein

• Small Transformers Compute Universal Metric Embeddings [paper link] 2022-10-18
  Anastasis Kratsios; Valentin Debarnot; Ivan Dokmanić

• The Lipschitz Constant of Self-Attention [paper link] 2021-06-09
  Hyunjik Kim; George Papamakarios; Andriy Mnih

• On Identifiability in Transformers [paper link] 2020-02-07
  Gino Brunner; Yang Liu; Damián Pascual; Oliver Richter; Massimiliano Ciaramita; Roger Wattenhofer

What Can Transformer Not Do? / Limitation of Transformer

The papers in this section investigate the limitations of transformer-based models, including the limitations of their expressiveness and learning abilities.

paper list (click to fold / unfold)

• How Far Can Transformers Reason? The Locality Barrier and Inductive Scratchpad [paper link] 2024-06-10
  Emmanuel Abbe; Samy Bengio; Aryo Lotfi; Colin Sandon; Omid Saremi

• Transformers Need Glasses! Information Over-squashing in Language Tasks [paper link] 2024-06-06
  Federico Barbero; Andrea Banino; Steven Kapturowski; Dharshan Kumaran; João G.M. Araújo; Alex Vitvitskyi; Razvan Pascanu; Petar Veličković

• On Limitation of Transformer for Learning HMMs [paper link] 2024-06-06
  Jiachen Hu; Qinghua Liu; Chi Jin

• Language Models Need Inductive Biases to Count Inductively [paper link] 2024-05-30
  Yingshan Chang; Yonatan Bisk

• Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theory [paper link] 2024-05-26
  Nikola Zubić; Federico Soldá; Aurelio Sulser; Davide Scaramuzza

• Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers [paper link] 2024-05-22
  Tobias Leemann; Alina Fastowski; Felix Pfeiffer; Gjergji Kasneci

• Collapse of Self-trained Language Models [paper link] 2024-04-02
  David Herel; Tomas Mikolov

• The pitfalls of next-token prediction [paper link] 2024-03-11
  Gregor Bachmann; Vaishnavh Nagarajan

• Why are Sensitive Functions Hard for Transformers? [paper link] 2024-03-03
  Michael Hahn; Mark Rofin

• Transformers are Expressive, But Are They Expressive Enough for Regression? [paper link] 2024-02-23
  Swaroop Nath; Harshad Khadilkar; Pushpak Bhattacharyya

• Limits of Transformer Language Models on Learning Algorithmic Compositions [paper link] 2024-02-13
  Jonathan Thomm; Aleksandar Terzic; Geethan Karunaratne; Giacomo Camposampiero; Bernhard Schölkopf; Abbas Rahimi

• Representational Strengths and Limitations of Transformers [paper link] 2023-11-16
  Clayton Sanford; Daniel Hsu; Matus Telgarsky

• Large Language Models Cannot Self-Correct Reasoning Yet [paper link] 2023-10-13
  Jie Huang; Xinyun Chen; Swaroop Mishra; Huaixiu Steven Zheng; Adams Wei Yu; Xinying Song; Denny Zhou

• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth [paper link] 2023-08-01
  Yihe Dong; Jean-Baptiste Cordonnier; Andreas Loukas

• Limits for Learning with Language Models [paper link] 2023-06-21
  Nicholas Asher; Swarnadeep Bhar; Akshay Chaturvedi; Julie Hunter; Soumya Paul

• Your Transformer May Not be as Powerful as You Expect [paper link] 2022-10-31
  Shengjie Luo; Shanda Li; Shuxin Zheng; Tie-Yan Liu; Liwei Wang; Di He

• The Devil in Linear Transformer [paper link] 2022-10-19
  Zhen Qin; XiaoDong Han; Weixuan Sun; Dongxu Li; Lingpeng Kong; Nick Barnes; Yiran Zhong

Architectural Effectivity

discussion of the effectiveness of different architectures in terms of learning and generalization

Layer-normalization

paper list (click to fold / unfold)

• On the Role of Attention Masks and LayerNorm in Transformers [paper link] 2024-05-29
  Xinyi Wu; Amir Ajorlou; Yifei Wang; Stefanie Jegelka; Ali Jadbabaie

• The Expressive Power of Tuning Only the Normalization Layers [paper link] 2023-07-12
  Angeliki Giannou; Shashank Rajput; Dimitris Papailiopoulos

• ResiDual: Transformer with Dual Residual Connections [paper link] 2023-04-28
  Shufang Xie; Huishuai Zhang; Junliang Guo; Xu Tan; Jiang Bian; Hany Hassan Awadalla; Arul Menezes; Tao Qin; Rui Yan

• DeepNet: Scaling Transformers to 1,000 Layers [paper link] 2022-03-01
  Hongyu Wang; Shuming Ma; Li Dong; Shaohan Huang; Dongdong Zhang; Furu Wei

• On Layer Normalization in the Transformer Architecture [paper link] 2020-06-29
  Ruibin Xiong; Yunchang Yang; Di He; Kai Zheng; Shuxin Zheng; Chen Xing; Huishuai Zhang; Yanyan Lan; Liwei Wang; Tie-Yan Liu

Tokenization

paper list (click to fold / unfold)

• Toward a Theory of Tokenization in LLMs [paper link] 2024-04-12
  Nived Rajaraman; Jiantao Jiao; Kannan Ramchandran

• On the Effect of (Near) Duplicate Subwords in Language Modelling [paper link] 2024-04-09
  Anton Schäfer; Thomas Hofmann; Imanol Schlag; Tiago Pimentel

• Tokenization Is More Than Compression [paper link] 2024-02-28
  Craig W. Schmidt; Varshini Reddy; Haoran Zhang; Alec Alameddine; Omri Uzan; Yuval Pinter; Chris Tanner

Linear Attention / State Space Models / etc.

The section includes papers that investigate the effectiveness of linear attention, state space models, and other prevelant architectures in language models.

paper list (click to fold / unfold)

• Parallelizing Linear Transformers with the Delta Rule over Sequence Length [paper link] 2024-06-10
  Songlin Yang; Bailin Wang; Yu Zhang; Yikang Shen; Yoon Kim

• Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality [paper link] 2024-05-31
  Tri Dao; Albert Gu

Training Paradigms

paper list (click to fold / unfold)

• Knowledge Distillation vs. Pretraining from Scratch under a Fixed (Computation) Budget [paper link] 2024-04-30
  Minh Duc Bui; Fabian David Schmidt; Goran Glavaš; Katharina von der Wense

• Why are Adaptive Methods Good for Attention Models? [paper link] 2020-10-23
  Jingzhao Zhang; Sai Praneeth Karimireddy; Andreas Veit; Seungyeon Kim; Sashank J. Reddi; Sanjiv Kumar; Suvrit Sra

Mechanistic Engineering / Probing / Interpretability

This section includes papers that mainly investigate the mechanisms of language models through probing, mechanistic engineering, and other papers generally related to interpretability.

paper list (click to fold / unfold)

• Scaling and evaluating sparse autoencoders [paper link] 2024-06-06
  Leo Gao; Tom Dupré la Tour; Henk Tillman; Gabriel Goh; Rajan Troll; Alec Radford; Ilya Sutskever; Jan Leike; Jeffrey Wu

• From Neurons to Neutrons: A Case Study in Interpretability [paper link] 2024-05-27
  Ouail Kitouni; Niklas Nolte; Víctor Samuel Pérez-Díaz; Sokratis Trifinopoulos; Mike Williams

• Mechanistic Interpretability of Binary and Ternary Transformers [paper link] 2024-05-27
  Jason Li

• InversionView: A General-Purpose Method for Reading Information from Neural Activations [paper link] 2024-05-27
  Xinting Huang; Madhur Panwar; Navin Goyal; Michael Hahn

• Not All Language Model Features Are Linear [paper link] 2024-05-23
  Joshua Engels; Isaac Liao; Eric J. Michaud; Wes Gurnee; Max Tegmark

• Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers [paper link] 2024-05-22
  Tobias Leemann; Alina Fastowski; Felix Pfeiffer; Gjergji Kasneci

• Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Models [paper link] 2024-05-21
  Charles O'Neill; Thang Bui

• Anchored Answers: Unravelling Positional Bias in GPT-2's Multiple-Choice Questions [paper link] 2024-05-06
  Ruizhe Li; Yanjun Gao

• A Primer on the Inner Workings of Transformer-based Language Models [paper link] 2024-05-02
  Javier Ferrando; Gabriele Sarti; Arianna Bisazza; Marta R. Costa-jussà

• Talking Nonsense: Probing Large Language Models' Understanding of Adversarial Gibberish Inputs [paper link] 2024-04-25
  Valeriia Cherepanova; James Zou

• Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions [paper link] 2024-02-22
  Clement Neo; Shay B. Cohen; Fazl Barez

• Universal Neurons in GPT2 Language Models [paper link] 2024-01-22
  Wes Gurnee; Theo Horsley; Zifan Carl Guo; Tara Rezaei Kheirkhah; Qinyi Sun; Will Hathaway; Neel Nanda; Dimitris Bertsimas

• Interpretability Illusions in the Generalization of Simplified Models [paper link] 2023-12-06
  Dan Friedman; Andrew Lampinen; Lucas Dixon; Danqi Chen; Asma Ghandeharioun

• Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars [paper link] 2023-12-03
  Kaiyue Wen; Yuchen Li; Bingbin Liu; Andrej Risteski

• White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is? [paper link] 2023-11-22
  Yaodong Yu; Sam Buchanan; Druv Pai; Tianzhe Chu; Ziyang Wu; Shengbang Tong; Hao Bai; Yuexiang Zhai; Benjamin D. Haeffele; Yi Ma

• Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks [paper link] 2023-11-21
  Samyak Jain; Robert Kirk; Ekdeep Singh Lubana; Robert P. Dick; Hidenori Tanaka; Edward Grefenstette; Tim Rocktäschel; David Scott Krueger

• Understanding the Mechanics and Dynamics of Memorisation in Large Language Models: A Case Study with Random Strings [paper link] 2023-10-13
  Till Speicher; Aflah Mohammad Khan; Qinyuan Wu; Vedant Nanda; Soumi Das; Bishwamittra Ghosh; Krishna P. Gummadi; Evimaria Terzi

Miscellanea

paper list (click to fold / unfold)

• Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Models [paper link] 2024-06-13
  Sarah Ball; Frauke Kreuter; Nina Rimsky

• Interpretability of Language Models via Task Spaces [paper link] 2024-06-10
  Lucas Weber; Jaap Jumelet; Elia Bruni; Dieuwke Hupkes

• How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States [paper link] 2024-06-09
  Zhenhong Zhou; Haiyang Yu; Xinghua Zhang; Rongwu Xu; Fei Huang; Yongbin Li

• Attention as a Hypernetwork [paper link] 2024-06-09
  Simon Schug; Seijin Kobayashi; Yassir Akram; João Sacramento; Razvan Pascanu

• Verbalized Machine Learning: Revisiting Machine Learning with Language Models [paper link] 2024-06-06
  Tim Z. Xiao; Robert Bamler; Bernhard Schölkopf; Weiyang Liu

• Local to Global: Learning Dynamics and Effect of Initialization for Transformers [paper link] 2024-06-05
  Ashok Vardhan Makkuva; Marco Bondaschi; Chanakya Ekbote; Adway Girish; Alliot Nagle; Hyeji Kim; Michael Gastpar

• Pre-trained Large Language Models Use Fourier Features to Compute Addition [paper link] 2024-06-05
  Tianyi Zhou; Deqing Fu; Vatsal Sharan; Robin Jia

• Computational Limits of Low-Rank Adaptation (LoRA) for Transformer-Based Models [paper link] 2024-06-05
  Jerry Yao-Chieh Hu; Maojiang Su; En-Jui Kuo; Zhao Song; Han Liu

• Rethinking Spiking Neural Networks as State Space Models [paper link] 2024-06-05
  Malyaban Bal; Abhronil Sengupta

• LongSSM: On the Length Extension of State-space Models in Language Modelling [paper link] 2024-06-04
  Shida Wang

• On Affine Homotopy between Language Encoders [paper link] 2024-06-04
  Robin SM Chan; Reda Boumasmoud; Anej Svete; Yuxin Ren; Qipeng Guo; Zhijing Jin; Shauli Ravfogel; Mrinmaya Sachan; Bernhard Schölkopf; Mennatallah El-Assady; Ryan Cotterell

• A Theory of In-Context Learning in Transformers [paper link] 2024-05-29
  Yifei Wang; Yuyang Wu; Zeming Wei; Stefanie Jegelka; Yisen Wang

• Lower Bounds on the Expressivity of Recurrent Neural Language Models [paper link] 2024-05-29
  Anej Svete; Franz Nowak; Anisha Mohamed Sahabdeen; Ryan Cotterell

• Demystifying amortized causal discovery with transformers [paper link] 2024-05-27
  Francesco Montagna; Max Cairney-Leeming; Dhanya Sridhar; Francesco Locatello

• Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective [paper link] 2024-05-27
  Zhen Qin; Xuyang Shen; Dong Li; Weigao Sun; Stan Birchfield; Richard Hartley; Yiran Zhong

• Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words? [paper link] 2024-05-27
  Gal Yona; Roee Aharoni; Mor Geva

• Towards Understanding How Transformer Perform Multi-step Reasoning with Matching Operation [paper link] 2024-05-24
  Zhiwei Wang; Yunji Wang; Zhongwang Zhang; Zhangchen Zhou; Hui Jin; Tianyang Hu; Jiacheng Sun; Zhenguo Li; Yaoyu Zhang; Zhi-Qin John Xu

• Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers [paper link] 2024-05-24
  Lorenzo Tiberi; Francesca Mignacco; Kazuki Irie; Haim Sompolinsky

• Attention as an RNN [paper link] 2024-05-22
  Leo Feng; Frederick Tung; Hossein Hajimirsadeghi; Mohamed Osama Ahmed; Yoshua Bengio; Greg Mori

• Surgical Feature-Space Decomposition of LLMs: Why, When and How? [paper link] 2024-05-17
  Arnav Chavan; Nahush Lele; Deepak Gupta

• Dynamic Activation Pitfalls in LLaMA Models: An Empirical Study [paper link] 2024-05-15
  Chi Ma; Mincong Huang; Chao Wang; Yujie Wang; Lei Yu

• Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis [paper link] 2024-05-14
  Yao Fu

• Understand LLMs Requires More Than Statistical Generalization [paper link] 2024-05-03
  Patrik Reizinger; Szilvia Ujváry; Anna Mészáros; Anna Kerekes; Wieland Brendel; Ferenc Huszár

• Compression Represents Intelligence Linearly [paper link] 2024-04-15
  Yuzhen Huang; Jinghan Zhang; Zifei Shan; Junxian He

• Language Generation in the Limit [paper link] 2024-04-10
  Jon Kleinberg; Sendhil Mullainathan

• Do language models plan ahead for future tokens? [paper link] 2024-03-31
  Wilson Wu; John X. Morris; Lionel Levine

• What's In My Big Data? [paper link] 2024-03-05
  Yanai Elazar; Akshita Bhagia; Ian Magnusson; Abhilasha Ravichander; Dustin Schwenk; Alane Suhr; Pete Walsh; Dirk Groeneveld; Luca Soldaini; Sameer Singh; Hanna Hajishirzi; Noah A. Smith; Jesse Dodge

• Do Efficient Transformers Really Save Computation? [paper link] 2024-02-21
  Kai Yang; Jan Ackermann; Zhenyu He; Guhao Feng; Bohang Zhang; Yunzhen Feng; Qiwei Ye; Di He; Liwei Wang

• Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning [paper link] 2024-02-07
  Hao Zhao; Maksym Andriushchenko; Francesco Croce; Nicolas Flammarion

• Provably learning a multi-head attention layer [paper link] 2024-02-06
  Sitan Chen; Yuanzhi Li

• Universality and Limitations of Prompt Tuning [paper link] 2023-11-16
  Yihan Wang; Jatin Chauhan; Wei Wang; Cho-Jui Hsieh

• Data Similarity is Not Enough to Explain Language Model Performance [paper link] 2023-11-15
  Gregory Yauney; Emily Reif; David Mimno

• Simplifying Transformer Blocks [paper link] 2023-11-03
  Bobby He; Thomas Hofmann

• Causal Interpretation of Self-Attention in Pre-Trained Transformers [paper link] 2023-10-31
  Raanan Y. Rohekar; Yaniv Gurwicz; Shami Nisimov

• How do Language Models Bind Entities in Context? [paper link] 2023-10-26
  Jiahai Feng; Jacob Steinhardt

• Understanding prompt engineering may not require rethinking generalization [paper link] 2023-10-13
  Victor Akinwande; Yiding Jiang; Dylan Sam; J. Zico Kolter

• Understanding Catastrophic Forgetting in Language Models via Implicit Inference [paper link] 2023-09-18
  Suhas Kotha; Jacob Mitchell Springer; Aditi Raghunathan

• Attention-Only Transformers and Implementing MLPs with Attention Heads [paper link] 2023-09-15
  Robert Huben; Valerie Morris

• On the Role of Attention in Prompt-tuning [paper link] 2023-06-15
  Samet Oymak; Ankit Singh Rawat; Mahdi Soltanolkotabi; Christos Thrampoulidis

---

Detailed Statistics

• Phenomena of Interest:

  • In-Context Learning: 56

  • Chain-of-Thought: 7

  • Hallucination: 7

  • Reversal Curse: 5

  • Scaling Laws / Emergent Abilities / Grokking / etc.: 28

  • Knowledge / Memory mechanisms: 9

  • Training Dynamics / Landscape / Optimization / Fine-tuning / etc.: 22

  • Learning / Generalization / Reasoning / Weak to Strong Generalization: 26

  • Other Phenomena: 8

• Representational Capacity:

  • What Can Transformer Do? / Properties of Transformer: 41

  • What Can Transformer Not Do? / Limitation of Transformer: 17

• Architectural Effectivity:

  • Layer-normalization: 5

  • Tokenization: 3

  • Linear Attention / State Space Models / etc.: 2

• Training Paradigms: 2

• Mechanistic Engineering / Probing / Interpretability: 17

• Miscellanea: 39

---

Contact

• Shiguang Wu, furyton AT outlook.com / shiguang.wu AT mail.sdu.edu.cn