ICML 2026
Interesting papers, ideas, and personal notes from ICML 2026.
ICML 2026 was my first one of the “big three” ML conferences (NeurIPS, ICML, and ICLR). It was also my first time in Seoul (or anywhere in East Asia actually). Of course, it was exciting, huge, and much more. I had a great time visiting it with my advisor Jae Hee Lee and another former student of his, Jeremy Herbst. Jeremy and Jae had a paper in the main track, while Jae and I had a workshop paper in the 2nd Compositional Learning workshop.
Attending the conference inspired me a lot, and I came away with many new ideas and perspectives on my research. I think a large part of that was also due to me taking some time off work to fully immerse myself in the conference experience, which reminded me how much I enjoy research and the process of learning from others in the field. Usually due to full-time work, my time would be fragmented and I wouldn’t have the opportunity to think and engage with research for days at a time. Also, It was very rewarding to meet and chat with many researchers whose work I have been following for a few years, it really felt like the world is small after all.
The conference received 24,661 submissions, up from 12,468 in 2025, which is a 98% increase, or basically 2x in one year. The number sounds absurd, and given that it’s only going to keep growing, I’m interested to see how the community will manage the review process and maintain quality. I’d guess we will see much more automatic reviewing to help with the flood of AI-powered submissions, which reminds me of a russian saying: “против лома нет приёма, если нет другого лома” (roughly: “there’s no way to deal with a crowbar if you don’t have another crowbar”). This already was the case at ICML this year, where they used a two-policy review process regarding AI usage, and also ran a randomized experiment that showed no statistical difference between the acceptance rates. Also I think implementing a submission fee might be necessary at this point (though yes, it is also unfair to some researchers).
I spent a lot of time walking through the poster sessions. There were a lot of posters, with pretty wide variation in quality and in how relevant they were to what I work on. The list below is the subset I found most interesting, roughly ordered by relevance to my research.
Looped transformers
My advisor and I were already interested in depth-recurrent (a.k.a. looped) models for a few years now, especially given their potential for learning soft algorithms, reasoning and generalization. Our workshop paper also explored adaptive recurrence in transformers on the toy task of addition and showed some promising OOD length generalization results. I saw multiple interesting papers in depth-recurrent models and talked to several researchers about their approaches and findings.
There seems to be a lot of interest in looped transformers, and I expect it to grow even more given the rumors that GPT-6 Astra uses some form of depth-recurrence.
One way to think about looped transformers is they are more computationally efficient, by reusing parameters and allowing “early exit” at lower depths. But recurrence can also improve generalization and reasoning, which I personally find more interesting. Another question is: with the repeated application of the same layers, do we converge to a fixed point, or do the representations keep evolving in a meaningful way? It seems this is an open question, and some models have been explicitly designed to encourage fixed-point convergence, while others rely on the dynamics to keep changing meaningfully (like ours). It’s a nice clean perspective that models keep iteratively refining the answer representation over multiple recurrent steps, and at some point that “converges” to a stable solution and we can stop iterating. However, I think that’s not the full story, since you can imagine a case where the recurrent block has learned an underlying algorithm, say something like a finite state machine, where the representations cycle through a set of meaningful states rather than settling to a single fixed point. In that case it’s also important to decide when to stop iterating. This is just a simple example though, the latent space dynamics could be much more complex.
Papers/posters I found interesting
-
Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models (Yang et al., 2026).
-
Step-Resolved Data Attribution for Looped Transformers (Kaissis et al., 2026).
-
From Growing to Looping: A Unified View of Iterative Computation in LLMs (Kapl et al., 2026).
-
Length Generalization Bounds for Transformers (A. Yang et al., 2026).
-
Symbol-Equivariant Recurrent Reasoning Models (Freinschlag et al., 2026).
-
A Constrained Optimization Perspective of Unrolled Transformers (Porras-Valenzuela et al., 2026).
-
Skip a Layer or Loop It? Learning Program-of-Layers in LLMs (Li et al., 2026).
-
An Algebraic View of the Expressivity of Recurrent Language Models (Nowak et al., 2026).
-
Causally Evaluating the Learnability of Formal Language Tasks (Snæbjarnarson et al., 2026).
-
Procedural Pretraining: Warming Up Language Models with Abstract Data (Jiang et al., 2026).
-
Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models (Uselis et al., 2026).
-
Query Circuits: Explaining How Language Models Answer User Prompts (Wu & Barez, 2026).
-
Understanding the Parameter Space Geometry of Transformers Encoding Boolean Functions (Kövér et al., 2026).
-
Equivalence of Context and Parameter Updates in Modern Transformer Blocks (Goldwaser et al., 2026).
-
A model of errors in transformers (Raju & Netrapalli, 2026).
-
Unveiling the Coexistence of Generalization and Memorization: A Case Study on Arithmetic Tasks with Label Noise (Liu & Lu, 2026).
-
Efficient numeracy in language models through single-token number embeddings (Kreitner et al., 2026).
-
Reasoning Structure of Large Language Models (Berdoz et al., 2026).
-
Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMs (Kim et al., 2026).
-
On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (Zhang et al., 2026).
-
Learning Coherent Representations: A Topological Approach to Interpretability (Gaukstad et al., 2026).
-
Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge (Ma et al., 2026).
-
Learning syntax without semantics: Disentangled tiny language models (Winston & Kolter, 2026).
-
Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Models (C. Liu et al., 2026).
-
MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers (Jaiswal et al., 2026).
-
On Minimum Depth and Width of Floating-Point Neural Networks for Representing Floating-Point Functions (Park et al., 2026).
-
How much can language models memorize? (Morris et al., 2026).
-
Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights (Gan & Isola, 2026).
-
Deriving Neural Scaling Laws from the Statistics of Natural Language (Cagnetta et al., 2026).
-
Width Independent Bounds for the Local Lipschitz Constant of Deep Neural Networks at Random Initialization and after Lazy Training (Evangelidis & Krahmer, 2026).
-
Flatland: The Adventures of Gradient Descent with Large Step Sizes (Galli et al., 2026).
-
On the Epistemic Uncertainty of Overparametrized Neural Networks (Rügamer, 2026).
-
Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts (Ye et al., 2026).
-
Hedging on the Frontier: Learning New Tasks with Few Samples (Wegel et al., 2026).
-
A Penalty Approach For Differentiation Through Black-box Quadratic Programming Solvers (Linghu et al., 2026).