Retrofitting Recurrent Vision–Language Models from Pretrained Backbones

Overview

Depth recurrence allocates test-time compute to iterating a block of layers rather than generating tokens, which keeps the context size and the memory overhead independent of the reasoning length. Recurrent VLMs have so far been built by adding a vision backbone to a recurrent language model or by looping an existing VLM without removing layers. We pursue a third option: we prune a block of intermediate layers from the language backbone of a pretrained VLM and loop over a subset of the remaining layers, converting the pretrained non-recurrent VLM into a recurrent one while reusing its weights. We refer to the result as retrofitted recurrent VLM (RR-VLM).

Method

We prune six of the 22 layers of the language backbone and apply an eight-layer recurrent block six times. An adapter concatenates the block's output with the prelude's output, and a linear layer compresses it back to the original dimensionality. Since the surgery degrades the backbone and misaligns it with the vision-language projector, we retrain in three stages: text-only repair of the backbone (S₀), projector-only re-alignment (S₁) and fine-tuning on visual instruction data (S₂), with the pretrained model as distillation teacher in S₀ and S₂. No new vision-language data is introduced.

Results

On the TinyLLaVA-1.4B backbone, RR-VLM reaches 58.13 on average over nine benchmarks against 55.80 for the unpruned baseline trained on the same data, while storing 23.3% fewer language-model parameters. The pruned model without the loop reaches 52.41, so the gain is due to the recurrence itself. Retrofitting a pretrained VLM also outperforms adding a vision encoder to a pretrained recurrent language model (55.59). The gains concentrate on the reasoning-oriented benchmarks MMBench, RealWorldQA, SEED-Bench, AI2D and CV-Bench-2D, whereas basic visual understanding benchmarks stay on par.

Table 1: Comparison to baselines. The retrofit outperforms the baselines on average over nine benchmarks with 23.3% fewer stored language-model parameters. Sec. 5.1 of the paper defines the models. RR-VLM denotes our retrofitted recurrent VLM F⁶_4/8/4 of Sec. 3 of the paper and F¹ the pruned single-pass model.

More qualitative examples (Figures 3 and 4)

Ablations

• Training stages: removing the text-only stage S₀ costs 3.68 pp, removing the alignment stage S₁ costs 1.28 pp (Table 2).
• Layer allocation, adapter and looping: at equal compute a larger recurrent block is better, 4/8/4 beats 6/4/6 by 3.19 pp; the linear adapter is the best adapter; looping the whole block beats per-layer looping by 2.83 pp at r=2 and 5.40 pp at r=6 (Table 3).
• Distillation: distilling from the pretrained model in S₂ improves the retrofitted model by 1.48 pp and the unpruned baseline by only 0.25 pp (Table 4).
• Training depth: a model trained at a fixed depth only works at that depth; a model trained with sampled recurrence works from r=2 to r=16 but does not improve beyond the trained depth (Table 5).
• KV-cache sharing: with n=3 cache copies the cache shrinks to 1.45× the base model's at a cost of 0.75 pp, with one copy to 0.727× at a cost of 5.43 pp (Table 6).

BibTeX

@article{iyengar2026retrofitting,
  title   = {Retrofitting Recurrent Vision--Language Models from Pretrained Backbones},
  author  = {Iyengar, Madhav and Geiger, Andreas and K{\"u}hne, Hilde},
  journal = {arXiv preprint arXiv:2610.22075},
  year    = {2026}
}

Acknowledgments

This work was funded by the Federal Ministry of Research, Technology and Space (BMFTR) under the project XEI, FKZ: 16IS24079A. The authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for providing computing time on the JUPITER Booster at Jülich Supercomputing Centre (JSC) under the project token-compression, and the computing resources provided by the ML Cloud of the Tübingen AI Center. We are grateful to Jonas Geiping and Takeru Miyato for insightful discussions on looped models.