GenLIP
Generative Language-Image Pre-training

Let ViT Speak

One Transformer. One language modeling objective. A vision encoder that learns by predicting words.

Yan Fang1,2,* Mengcheng Lan2,3,* Zilong Huang2,† Weixian Lei2 Yunqing Zhao2 Yujie Zhong2 Yingchen Yu2 Qi She2 Yao Zhao1 Yunchao Wei1,4,†
1 Beijing Jiaotong University 2 ByteDance 3 Nanyang Technological University 4 Beijing Academy of Artificial Intelligence

*Equal contribution   †Corresponding authors

Key results

73.6Overall score with Qwen2.5-7B
GenLIP-g/16, frozen encoder
+4.7Points over SigLIP2-g/16
under the same evaluation
8BStage 1 seen samples
versus 40B for SigLIP2
39MAdditional adaptation samples
at native aspect ratios

A simpler way to learn vision.

Generative Language-Image Pre-training (GenLIP) trains a Vision Transformer to predict language tokens directly from visual tokens. One Transformer jointly models images and text, using a standard language modeling objective without contrastive batch construction or an additional text decoder.

With 8B seen samples in its first pretraining stage, GenLIP matches or surpasses strong vision-encoder baselines trained on up to 40B samples. A second stage of native-aspect-ratio adaptation further improves detail-sensitive tasks, including OCR and chart understanding.

Pretrain by speaking. Deploy as a vision encoder.

GenLIP replaces separate image and text Transformers with a single model trained by next-token prediction; a perplexity diagnostic compares compatibility with an LLM.
Figure 1. A direct generative objective for the vision backbone. The perplexity diagnostic suggests better compatibility with language modeling.In the paper ↗

The method

Language supervision. Visual representations.

Learn from image-caption pairs, then use the same backbone to supply visual features to an MLLM.

One shared Transformer

Image patches precede text tokens in one sequence. Multimodal rotary position embeddings (MRoPE) encode position.

A visual prefix, a text objective

Image tokens attend bidirectionally; text tokens attend causally. Next-token prediction loss applies only to text.

Gated attention

Input-dependent gates reduce first-token attention sinks, improve training stability, and preserve more distributed visual features.

GenLIP architecture showing a shared Transformer, gated attention, MRoPE indices, and the Prefix-LM attention mask.
Figure 2. At deployment, final-layer normalized visual features feed a two-layer MLP projector. The tokenizer and LM head are discarded; attention becomes fully bidirectional.In the paper ↗
Stage 1

Learn at a fixed resolution

8B seen samples over 1B unique Recap-DataComp-1B pairs. Eight epochs at 224 × 224 resolution.

Stage 2

Adapt to native aspect ratios

39M samples, one epoch. BLIP3o-Long-Caption, Infinity-MM (stage 1), and CapRL. Between 16 and 1,024 visual tokens per image.

Why gated attention matters

The controlled So/16 study shows less attention concentrating on the first sequence token, with stronger text-to-vision attention in deeper layers. This supports the gate's role in stabilizing visual representation learning.

Three layer-wise attention plots compare vision-to-first-token, text-to-first-token, and text-to-vision attention with and without gating.
Figure 3. Attention allocation across all 27 layers of controlled So/16 models.In the paper ↗

Stronger features. Fewer seen samples.

Consistent overall gains across model scales, with the clearest improvements on document understanding and OCR.

A shared evaluation setup

Frozen vision encoders · 576 visual tokens · Qwen2.5-Instruct · more than 3M LLaVA-OneVision instruction-tuning samples. All encoders use the same token budget and a two-layer MLP projector.

Qwen2.5-7B · frozen vision encoder

Table 4 in paper ↗
Qwen2.5-7B frozen visual representation results. Doc/OCR is the mean of seven tasks. Overall averages use all 14 benchmarks, not just the selected columns shown here.
ModelArchSamplesDoc/OCRMME-PTextCapsAll avg.
CLIPL/1412.8B48.21316127.958.8
AIMv2L/1412.0B48.31240130.558.6
OpenVision2L/1612.8B52.41325133.864.9
SigLIPL/1640.0B52.01275131.164.5
GenLIPL/168.0B59.21320139.469.0
SigLIP2So/1640.0B56.71422139.369.4
GenLIPSo/168.0B62.01424142.171.8
SigLIP2g/1640.0B56.61422142.768.9
GenLIPg/168.0B63.51483144.873.6
Compare with the smaller Qwen2.5-1.5B backbone

Qwen2.5-1.5B · frozen vision encoder

Table 3 in paper ↗
Qwen2.5-1.5B frozen visual representation results. Doc/OCR is the mean of seven tasks. Overall averages use all 14 benchmarks, not just the selected columns shown here.
ModelArchSamplesDoc/OCRMME-PNoCapsAll avg.
CLIPL/1412.8B39.1121855.553.1
AIMv2L/1412.0B39.6115780.155.7
OpenVision2L/1612.8B44.3123084.358.7
SigLIPL/1640.0B42.4120384.056.9
SigLIP2L/1640.0B45.0116582.958.7
GenLIPL/168.0B49.3125882.661.5
SigLIP2So/1640.0B46.8122084.360.6
GenLIPSo/168.0B50.1121587.562.6
SigLIP2g/1640.0B47.3128484.461.5
GenLIPg/168.0B53.2125688.365.2

Reading the tables. Doc/OCR averages seven tasks. All avg. is the unweighted mean of 14 benchmarks, with MME-P divided by 20 and captioning CIDEr scores unchanged. The selected columns above do not reproduce that full mean. GenLIP's 8B denotes Stage 1 seen samples; Stage 2 adds 39M samples.

Joint fine-tuning

With the vision encoder unfrozen, GenLIP-So/16 reaches 68.5 at 576 patches and 70.3 at 729 patches in the standard LLaVA-NeXT setting.

Table 6 ↗

Broader evaluation

The advantage also holds in the Cambrian-1-style suite. With Qwen2.5-7B, GenLIP-g/16 scores 66.0 versus 63.0 for SigLIP2-g/16.

Table 13 ↗
Data scaling curves for OCR, VQA, and Caption tasks from 1B to 8B pretraining samples.
Figure 4. Data scaling from 1B to 8B first-stage seen samples, with and without gated attention.

Scaling signal. Performance improves with more pretraining, but gains flatten between 4B and 8B samples, especially for VQA and captioning. Plot averages follow the task groups labeled in the figure.

Stage 1 versus Stage 2 validation curves across evaluation resolutions.
Figure 5. Stage 1 and Stage 2 GenLIP-So/16 across evaluation resolutions.

Resolution signal. Native-aspect adaptation improves detail-sensitive multimodal understanding. These curves use the task groups labeled in the figure, distinct from the seven-task Doc/OCR table average.

Inside the representation

What does "speak" reveal?

Caption generation and patch readout offer a closer look at the visual-language alignment learned during pretraining.

Select a figure to read it at full size.

GenLIP caption generation examples comparing model scales and pretraining stages.
Figure 6. Direct image descriptions across three model scales and two pretraining stages.

Generation probe. Larger models and native-aspect adaptation produce more detailed descriptions. These qualitative probes illustrate learned alignment; GenLIP is deployed as a vision encoder.

Patch semantic readout showing local visual regions mapped to language tokens.
Figure 7. Top-five language tokens read directly from selected image patch features.

Patch semantics. Selected regions in So/16 and g/16 align with meaningful language concepts, with more stable readouts in g/16. This behavior is not established for L/16.

Inspect the OCR examples and failure cases
OCR-heavy GenLIP generation probes on receipt, geometry, and tiny text examples.
Figure 8. Qualitative OCR examples with both correct details and generation errors.

Doc/OCR behavior. Long number strings, precise spatial layouts, and tiny text remain challenging, even when descriptions are fluent.

What the evidence tells us.

Strong results as an MLLM vision encoder, with tradeoffs that remain visible.

Controlled pretraining and language initialization

In Table 8, GenLIP reaches 57.2 versus 55.9 for OpenVision2 and 54.4 for SigLIP after 2B first-stage samples. The baselines additionally receive 0.2B high-resolution adaptation samples, while GenLIP is evaluated directly after Stage 1.

In the separate 1B-sample study (Table 9), training the gated SAIL-style architecture from scratch scores 56.0, compared with 54.8 using Qwen3 initialization. The paper interprets this as possible language-prior bias in visual grounding, not a universal claim about language initialization.

Read the ablations ↗
Discriminative features and retrieval

Native-aspect adaptation improves multimodal understanding but lowers ImageNet-1K accuracy at every model scale. GenLIP-g/16 changes from 85.2 to 83.0 top-1 accuracy, and reaches 44.5 mIoU on ADE20K. GenLIP remains behind SigLIP2 on these discriminative evaluations and on zero-shot retrieval.

Table 11 ↗ Table 12 ↗
Scope and remaining limitations

The experiments use the academic-scale LLaVA-NeXT framework; transfer to frontier MLLMs remains to be established. Scaling beyond 1B unique pretraining pairs is unverified, and high-quality captions carry acquisition costs. Qualitative generations can still misread text or spatial relations.

Read the limitations ↗

Explore the models.

Three model scales, with checkpoints after native-aspect-ratio adaptation. See the code repository for training and configuration details.

Page content follows the paper dated .

BibTeX

@article{fang2026letvitspeakgenerative,
  title={Let ViT Speak: Generative Language-Image Pre-training},
  author={Yan Fang and Mengcheng Lan and Zilong Huang and Weixian Lei and Yunqing Zhao and Yujie Zhong and Yingchen Yu and Qi She and Yao Zhao and Yunchao Wei},
  journal={arXiv preprint arXiv:2605.00809},
  year={2026}
}