ARLoop: Fully-Open Pre-trained Looped Transformers with Attention Residuals
Looped transformers increase effective depth by repeatedly applying a shared set of transformer layers, offering a parameter efficient way to scale computation.
The arXiv Language-Model Research page documents the arXiv update.
