LooM, a Method for Making Looped Transformers More Flexible
LooM is an arXiv method for Looped Transformers: the cache does not grow with loops and compute can be set from 3 to 21 executions per token.
A paper published on arXiv on 4 November 2026 presents LooM (Loop-invariant Memory), a method designed for Looped Transformers, language models that repeat the same computation layers several times.1 The authors say LooM addresses three limits that hold back the adoption of these models: memory that grows with the number of loops, computation that can be adjusted only in whole blocks, and a number of loops that is hard to change after training.1
What Looped Transformers are and where they get stuck
Looped Transformers increase the model’s effective depth by rerunning shared layers instead of adding new ones.1 The first limit the authors point to is that memory grows with the number of loops. The second is that computation can only be adjusted in steps corresponding to whole blocks of layers. The third is that the number of loops is hard to change once training is over.1
How LooM works
LooM works on an architecture called causal encoder-decoder (CED). Here it repeats only the decoder layers after the first one, which read a global memory written just once for each token, that is, for each unit of text processed.1 Each of these layers publishes a token’s key-value (KV) entry only on the first visit. The cache, that is, the model’s working memory, therefore does not grow with the number of loops.1 The repeated layers loop independently and offer every compute budget from 3 to 21 layer executions per token. A single model, trained on randomly drawn budgets, covers them all without needing a router, the component that would otherwise decide how much computation to use.1
What the authors report about the results
With a sufficient number of loops, according to the authors LooM outperforms a single-pass model trained on the same data. The comparison covers language modeling, zero-shot accuracy (without examples specific to the task) and perplexity on long contexts, at equal cache size.1 Their analysis also shows that the distribution of budgets used in training determines what the additional depth delivers.1 The authors conclude that a CED is a natural place to repeat loops: memory is written only once and depth becomes a budget chosen at the time of use.1
This article was written by artificial intelligence from the sources cited, without human review. Automatic newsroom by Redazia.
