← sources

[Meta FAIR + NYU] Unbundling Multimodal Vision Encoding: training 200+ MLLMs to find encoder pretraining is optional but vision-only parameters are not [R]

source: reddit (reddit:MachineLearning)published 2026-11-15 17:38ingested 2026-11-16 02:36

retained text

X threads (with videos!): 

* [https://x.com/choang333/status/2121335484236386574](https://x.com/choang333/status/2121335484236386574)
* [https://x.com/\_ellisbrown/status/2121335477244428419](https://x.com/_ellisbrown/status/2121335477244428419)

Paper: [https://www.alphaxiv.org/abs/2611.17334](https://www.alphaxiv.org/abs/2611.17334)

Blog and FAQ: [https://unbundling.github.io](https://unbundling.github.io/)

**Labs disagree on vision encoders right now.** Gemma 4 12B dropped the encoder, Kimi K3 trains its encoder from scratch, and Inkling 2 Small brought an encoder back. Published models can't settle this, because they differ in data and compute and rarely account for encoder pretraining. A standard encoder also bundles three choices: separate pretraining, parameters that only vision tokens use, and placement before the LLM. We tested each choice separately.

Setup:

* 200+ models, all trained from scratch (LLM included) on the same data.
* Lifetime compute: encoder pretraining counts against the same budget as multimodal training.
* Main comparisons use Qwen3-1.7B and 4B configurations with 0.4B parameters of vision capacity. A scaling ladder covers 0.6B to 6.4B backbone sizes.
* Evaluations are on 16 VQA benchmarks from Cambrian at 448 px. and 12 language benchmarks testing commonsense, QA, and broad knowledge.

Findings:

1. **Separate pretraining is optional.** Encoders pretrained with four objectives (contrastive, self-supervised, two captioning) at best tie an encoder trained from scratch with the LLM at matched compute. Even free pretraining doesn't help. Pretraining MetaCLIP 6.5x longer raised ImageNet linear-probe accuracy (80.4% → 84.4%) but not VQA.
2. **Vision-only parameters are required.** Encoder-free models trail at every size from 0.6B to 6.4B and need about 3x the compute to catch up. Giving the LLM the encoder's parameters instead improves text, not VQA.
3. **Placement before the LLM is optional.** "In-fusion" adds zero-initialized vision-only weights (attention projections and FFNs) to the LLM's first 8 layers, and text tokens never use them. It's on par with the encoder from 1.7B to 6.4B, with about 13% lower time to first token. At 1.7B, vision-only branches in layers 1–8 reach 45.0 VQA (encoder 44.4), versus 42.3 when text shares them and 35.1 in the last 8 layers.
4. **The same holds with a pretrained LLM** (vision added to a pretrained 1.7B LLM in midtraining).

So vision needs its own parameters, but not its own pretraining or its own encoder.

Disclosure: I'm one of the authors. Happy to answer questions!

document fetched 2026-11-16 00:26 · canonical https://reddit.com/r/MachineLearning/comments/1xx1k2o/meta_fair_nyu_unbundling_multimodal_vision