# [Meta, NYU] Unbundling Multimodal Vision Encoding: 200+ MLLMs trained from scratch show you don't need a pretrained vision encoder, just vision-only parameters — retained record

> Retained reddit record from reddit:deeplearning.

Canonical: https://highsignal.app/data/records/e60b3c3c59419a0a

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:deeplearning)  published 2026-11-14 18:16 ingested 2026-11-15 02:34

[original source ↗](https://www.reddit.com/r/deeplearning/comments/1xw98vi/meta_nyu_unbundling_multimodal_vision_encoding/)

## retained text



```
Paper: [https://www.alphaxiv.org/abs/2611.17334](https://www.alphaxiv.org/abs/2611.17334)

X threads (with videos): 

* [https://x.com/choang333/status/2121335484236386574](https://x.com/choang333/status/2121335484236386574)
*  [https://x.com/\_ellisbrown/status/2121335477244428419](https://x.com/_ellisbrown/status/2121335477244428419)

Blog and FAQ: [https://unbundling.github.io](https://unbundling.github.io/) 

New paper from Meta FAIR and NYU! Labs disagree on vision encoders right now: Gemma 4 12B scrapped the encoder, Kimi K3 dropped encoder pretraining, and Inkling 2 Small brought the encoder back. Published models differ in data, compute and whether encoder pretraining is counted at all, so they can't settle it.

This paper runs a large controlled study to help shed light on the debate, with 200+ models trained from scratch, LLM included, on the same data, with encoder pretraining charged to the same lifetime compute budget, changing one design choice at a time. Findings (VQA after instruction tuning, 1B to 6.4B models):

1. Encoder pretraining is optional. Four pretrained encoders (contrastive, self-supervised, captioning) at best tie an encoder trained from scratch with the LLM at matched compute. Even with pretraining compute not counted, longer MetaCLIP pretraining raised ImageNet linear-probe accuracy from 80.4% to 84.4% and didn't improve VQA.
2. Vision-only parameters are not optional. Encoder-free models trail at every size and need about 3x the compute to match the encoder. Giving the encoder's parameters to the LLM instead improves text benchmarks but not VQA.
3. Those parameters don't have to be an encoder. Vision-only weights added to the LLM's early layers ("in-fusion") are on par with a separate encoder from 1.7B to 6.4B, with about 13% lower time to first token.
4. The conclusions hold when adding vision to a pretrained 1.7B LLM.

Disclosure: I'm one of the authors. Happy to answer questions!
```



document fetched 2026-11-15 00:23 · canonical https://reddit.com/r/deeplearning/comments/1xw98vi/meta_nyu_unbundling_multimodal_vision_encoding
