ICML 2026

Interesting papers, ideas, and personal notes from ICML 2026.

ICML 2026 was my first one of the “big three” ML conferences (NeurIPS, ICML, and ICLR). It was also my first time in Seoul (or anywhere in East Asia actually). Of course, it was exciting, huge, and much more. I had a great time visiting it with my advisor Jae Hee Lee and another former student of his, Jeremy Herbst. Jeremy and Jae had a paper in the main track, while Jae and I had a workshop paper in the 2nd Compositional Learning workshop.

The entrance hall of the ICML 2026 venue in Seoul
The entrance hall of the ICML 2026 venue COEX
Jeremy, Jae, and I standing in front of the ICML 2026 conference sign
Jeremy, Jae, and I standing in front of the ICML 2026 conference sign

Attending the conference inspired me a lot, and I came away with many new ideas and perspectives on my research. I think a large part of that was also due to me taking some time off work to fully immerse myself in the conference experience, which reminded me how much I enjoy research and the process of learning from others in the field. Usually due to full-time work, my time would be fragmented and I wouldn’t have the opportunity to think and engage with research for days at a time. Also, It was very rewarding to meet and chat with many researchers whose work I have been following for a few years, it really felt like the world is small after all.

The conference received 24,661 submissions, up from 12,468 in 2025, which is a 98% increase, or basically 2x in one year. The number sounds absurd, and given that it’s only going to keep growing, I’m interested to see how the community will manage the review process and maintain quality. I’d guess we will see much more automatic reviewing to help with the flood of AI-powered submissions, which reminds me of a russian saying: “против лома нет приёма, если нет другого лома” (roughly: “there’s no way to deal with a crowbar if you don’t have another crowbar”). This already was the case at ICML this year, where they used a two-policy review process regarding AI usage, and also ran a randomized experiment that showed no statistical difference between the acceptance rates. Also I think implementing a submission fee might be necessary at this point (though yes, it is also unfair to some researchers).

A presentation slide showing the growth of ICML submissions from 2017 to 2026
Growth of ICML submissions from 2017 to 2026
The large main hall at ICML 2026
The large main hall
A crowded upper level inside the ICML 2026 venue
One of the floors at the venue, with a super nice view of the Bongeunsa Temple

I spent a lot of time walking through the poster sessions. There were a lot of posters, with pretty wide variation in quality and in how relevant they were to what I work on. The list below is the subset I found most interesting, roughly ordered by relevance to my research.

Presenting the poster at the CompLearn workshop
Presenting our poster at the CompLearn workshop
My advisor Jae and I standing next to our workshop poster
My advisor Jae and I standing next to our workshop poster

Looped transformers

My advisor and I were already interested in depth-recurrent (a.k.a. looped) models for a few years now, especially given their potential for learning soft algorithms, reasoning and generalization. Our workshop paper also explored adaptive recurrence in transformers on the toy task of addition and showed some promising OOD length generalization results. I saw multiple interesting papers in depth-recurrent models and talked to several researchers about their approaches and findings.

There seems to be a lot of interest in looped transformers, and I expect it to grow even more given the rumors that GPT-6 Astra uses some form of depth-recurrence.

One way to think about looped transformers is they are more computationally efficient, by reusing parameters and allowing “early exit” at lower depths. But recurrence can also improve generalization and reasoning, which I personally find more interesting. Another question is: with the repeated application of the same layers, do we converge to a fixed point, or do the representations keep evolving in a meaningful way? It seems this is an open question, and some models have been explicitly designed to encourage fixed-point convergence, while others rely on the dynamics to keep changing meaningfully (like ours). It’s a nice clean perspective that models keep iteratively refining the answer representation over multiple recurrent steps, and at some point that “converges” to a stable solution and we can stop iterating. However, I think that’s not the full story, since you can imagine a case where the recurrent block has learned an underlying algorithm, say something like a finite state machine, where the representations cycle through a set of meaningful states rather than settling to a single fixed point. In that case it’s also important to decide when to stop iterating. This is just a simple example though, the latent space dynamics could be much more complex.

Papers/posters I found interesting

  • Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models (Yang et al., 2026).

  • Step-Resolved Data Attribution for Looped Transformers (Kaissis et al., 2026).

  • From Growing to Looping: A Unified View of Iterative Computation in LLMs (Kapl et al., 2026).

  • Length Generalization Bounds for Transformers (A. Yang et al., 2026).

  • Symbol-Equivariant Recurrent Reasoning Models (Freinschlag et al., 2026).

  • A Constrained Optimization Perspective of Unrolled Transformers (Porras-Valenzuela et al., 2026).

  • Skip a Layer or Loop It? Learning Program-of-Layers in LLMs (Li et al., 2026).

  • An Algebraic View of the Expressivity of Recurrent Language Models (Nowak et al., 2026).

  • Causally Evaluating the Learnability of Formal Language Tasks (Snæbjarnarson et al., 2026).

  • Procedural Pretraining: Warming Up Language Models with Abstract Data (Jiang et al., 2026).

  • Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models (Uselis et al., 2026).

  • Query Circuits: Explaining How Language Models Answer User Prompts (Wu & Barez, 2026).

  • Understanding the Parameter Space Geometry of Transformers Encoding Boolean Functions (Kövér et al., 2026).

  • Equivalence of Context and Parameter Updates in Modern Transformer Blocks (Goldwaser et al., 2026).

  • A model of errors in transformers (Raju & Netrapalli, 2026).

  • Unveiling the Coexistence of Generalization and Memorization: A Case Study on Arithmetic Tasks with Label Noise (Liu & Lu, 2026).

  • Efficient numeracy in language models through single-token number embeddings (Kreitner et al., 2026).

  • Reasoning Structure of Large Language Models (Berdoz et al., 2026).

  • Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMs (Kim et al., 2026).

  • On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (Zhang et al., 2026).

  • Learning Coherent Representations: A Topological Approach to Interpretability (Gaukstad et al., 2026).

  • Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge (Ma et al., 2026).

  • Learning syntax without semantics: Disentangled tiny language models (Winston & Kolter, 2026).

  • Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Models (C. Liu et al., 2026).

  • MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers (Jaiswal et al., 2026).

  • On Minimum Depth and Width of Floating-Point Neural Networks for Representing Floating-Point Functions (Park et al., 2026).

  • How much can language models memorize? (Morris et al., 2026).

  • Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights (Gan & Isola, 2026).

  • Deriving Neural Scaling Laws from the Statistics of Natural Language (Cagnetta et al., 2026).

  • Width Independent Bounds for the Local Lipschitz Constant of Deep Neural Networks at Random Initialization and after Lazy Training (Evangelidis & Krahmer, 2026).

  • Flatland: The Adventures of Gradient Descent with Large Step Sizes (Galli et al., 2026).

  • On the Epistemic Uncertainty of Overparametrized Neural Networks (Rügamer, 2026).

  • Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts (Ye et al., 2026).

  • Hedging on the Frontier: Learning New Tasks with Few Samples (Wegel et al., 2026).

  • A Penalty Approach For Differentiation Through Black-box Quadratic Programming Solvers (Linghu et al., 2026).

References

Berdoz, F., Lanzendörfer, L. A., Farestam, F., & Wattenhofer, R. (2026, June 24). Reasoning Structure of Large Language Models. https://openreview.net/forum?id=SX7LSFG4IH
Cagnetta, F., Raventos, A., Ganguli, S., & Wyart, M. (2026, June 24). Deriving Neural Scaling Laws from the Statistics of Natural Language. https://openreview.net/forum?id=VXlniX0C5y
Evangelidis, A., & Krahmer, F. (2026, June 24). Width Independent Bounds for the Local Lipschitz Constant of Deep Neural Networks at Random Initialization and after Lazy Training. https://openreview.net/forum?id=83l762JFa9
Freinschlag, R., Bertram, T., Kobler, E., Mayr, A., & Klambauer, G. (2026, June 24). Symbol-Equivariant Recurrent Reasoning Models. https://openreview.net/forum?id=1YCpxT3t5V
Galli, L., Fox, C., Bartolomaeus, W., Schmidt, M., & Rauhut, H. (2026, June 24). Flatland: The Adventures of Gradient Descent with Large Step Sizes. https://openreview.net/forum?id=NpSI4x2vBS
Gan, Y., & Isola, P. (2026, June 24). Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights. https://openreview.net/forum?id=92oF5bU4cU
Gaukstad, S., Vaupel, M., Olsen, V. K., Hermansen, E., & Dunn, B. A. (2026, June 24). Learning Coherent Representations: A Topological Approach to Interpretability. https://openreview.net/forum?id=VnlpOuA3Bs
Goldwaser, A., Munn, M., Gonzalvo, J., & Dherin, B. (2026, June 24). Equivalence of Context and Parameter Updates in Modern Transformer Blocks. https://openreview.net/forum?id=aZ5Di8Ii4X
Jaiswal, A. K., Hannah, L., Kim, H.-B., Hoang, D. N. M., Kundu, A., Farajtabar, M., & Cho, M. (2026, June 24). MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers. https://openreview.net/forum?id=1vshJGUVBS
Jiang, L., Shinnick, Z., Hengel, A. van den, Saratchandran, H., & Teney, D. (2026, June 24). Procedural Pretraining: Warming Up Language Models with Abstract Data. https://openreview.net/forum?id=XFTTezxLdU
Kaissis, G., Mildenberger, D., Gomez, J. F., Menten, M. J., & Triantafillou, E. (2026, June 24). Step-Resolved Data Attribution for Looped Transformers. https://openreview.net/forum?id=R3GDb4y629
Kapl, F., Angelis, E., Maile, K., Oswald, J. von, & Bauer, S. (2026, June 24). From Growing to Looping: A Unified View of Iterative Computation in LLMs. https://openreview.net/forum?id=FrXKVuK0yS
Kim, J., Chang, H., Hwang, H., Kim, C., & Ye, J. C. (2026, June 24). Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMs. https://openreview.net/forum?id=JheeAwquCb
Kövér, B., Butoi, A., Svete, A., Hahn, M., & Cotterell, R. (2026, June 24). Understanding the Parameter Space Geometry of Transformers Encoding Boolean Functions. https://openreview.net/forum?id=rUZVcnQjYW
Kreitner, L., Hager, P., Mengedoht, J., Kaissis, G., Rueckert, D., & Menten, M. J. (2026, June 24). Efficient numeracy in language models through single-token number embeddings. https://openreview.net/forum?id=Bh4Ubk80M8
Li, Z., Li, Y., & Zhou, T. (2026, June 24). Skip a Layer or Loop It? Learning Program-of-Layers in LLMs. https://openreview.net/forum?id=pl10b6EQAN
Linghu, Y., Liu, Z., & Deng, Q. (2026, June 24). A Penalty Approach For Differentiation Through Black-box Quadratic Programming Solvers. https://openreview.net/forum?id=2jpMiRwsrL
Liu, C., Sun, X., Xiao, X., Tassel, A. V., Xu, K., Reimann, K., Liao, D., Gerstein, M., Wang, T., Wang, X., & Krishnaswamy, S. (2026, June 24). Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Models. https://openreview.net/forum?id=pd6A7jB5D6
Liu, L., & Lu, P. (2026). Unveiling the Coexistence of Generalization and Memorization: A Case Study on Arithmetic Tasks with Label Noise. https://openreview.net/forum?id=19ScezMmQr&referrer=%5Bthe%20profile%20of%20Linyu%20Liu%5D(%2Fprofile%3Fid%3D~Linyu_Liu1)
Ma, X., Huang, Y., Zhu, H., & Sojoudi, S. (2026, June 24). Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge. https://openreview.net/forum?id=kKUrbW8xJ3
Morris, J. X., Sitawarin, C., Guo, C., Kokhlikyan, N., Suh, G. E., Rush, A. M., Chaudhuri, K., & Mahloujifar, S. (2026, June 24). How much can language models memorize? https://openreview.net/forum?id=bA6BgSbaUi
Nowak, F., Cotterell, R., & Boumasmoud, R. (2026, June 24). An Algebraic View of the Expressivity of Recurrent Language Models. https://openreview.net/forum?id=7pbmZatDuD
Park, S., Park, Y., & Hwang, G. (2026, June 24). On Minimum Depth and Width of Floating-Point Neural Networks for Representing Floating-Point Functions. https://openreview.net/forum?id=8IxY2DUiBi
Porras-Valenzuela, J., Hadou, S., & Ribeiro, A. (2026, June 24). A Constrained Optimization Perspective of Unrolled Transformers. https://openreview.net/forum?id=aYe8j2jOmK
Raju, S., & Netrapalli, P. (2026, June 24). A model of errors in transformers. https://openreview.net/forum?id=fbabBXNeqw
Rügamer, D. (2026, June 24). On the Epistemic Uncertainty of Overparametrized Neural Networks. https://openreview.net/forum?id=oiMjaUbSWp
Snæbjarnarson, V., Svete, A., Valvoda, J., Boumasmoud, R., DuSell, B., & Cotterell, R. (2026, June 24). Causally Evaluating the Learnability of Formal Language Tasks. https://openreview.net/forum?id=sYUCXUeApn
Uselis, A., Dittadi, A., & Oh, S. J. (2026, June 24). Compositional Generalization Requires Linear, Orthogonal Representations in Vision Embedding Models. https://openreview.net/forum?id=AQZZWVp6XA
Wegel, T., Gennaro, F. D., So, G., & Yang, F. (2026, June 24). Hedging on the Frontier: Learning New Tasks with Few Samples. https://openreview.net/forum?id=J4wRLmh29t
Winston, E., & Kolter, J. Z. (2026, June 24). Learning syntax without semantics: Disentangled tiny language models. https://openreview.net/forum?id=p7HVrmZwWB
Wu, T.-Y., & Barez, F. (2026, June 24). Query Circuits: Explaining How Language Models Answer User Prompts. https://openreview.net/forum?id=7F0sragazb
Yang, A., Bergsträßer, P., Zetzsche, G., Chiang, D., & Lin, A. W. (2026, June 24). Length Generalization Bounds for Transformers. https://openreview.net/forum?id=XNXU8BJRT7
Yang, X.-W., Han, Z., Zhang, X.-H., Wei, W.-D., Shao, J.-J., Guo, L.-Z., & Li, Y.-F. (2026, June 24). Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models. https://openreview.net/forum?id=tbdSfBMlbq
Ye, J., Feldman, V., & Talwar, K. (2026, June 24). Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts. https://openreview.net/forum?id=YL9lAEeoXw
Zhang, C., Neubig, G., & Yue, X. (2026, June 24). On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models. https://openreview.net/forum?id=TBaUfO9znF