Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?: plain AI paper summary
AI-generated summary. Read the original paper before relying on it.
Hugging Face paper page · Original paper on arXiv
The paper finds that quantization error does not reliably predict semantic loss in discrete visual tokens. It introduces Factorized Vector Quantization to match continuous visual representations in multimodal understanding.
Key findings
- Quantization error is unreliable as a predictor of semantic loss; discrete representations with substantially larger errors can achieve better multimodal understanding.
- The effective bit budget of discrete indices, not quantization error, determines downstream performance.
- Factorized Vector Quantization quantizes complementary fixed-dimensional views with symmetric parallel codebooks, continuing to improve at larger bit budgets where existing quantizers saturate.
- The model outperforms UniAR and Emu3-Chat by 22.8% and 59.2% on a nine-metric average, and is the first discrete-representation model competitive with strong continuous models like Qwen3-VL.
How you might use it
- Replace the current visual tokenizer with Factorized Vector Quantization and measure multimodal understanding scores against the previous discrete baseline.
- Plot downstream accuracy against effective bit budget to confirm the model does not saturate at higher budgets.
Limits
The abstract does not report inference speed, memory cost, or training compute overhead. It does not specify the exact bit budgets tested or the size of the industrial-scale training data.
Related research
- What Generation Knows: Unlocking Fine-Grained Understanding in Unified Multimodal Models Related paper on arXiv: Both papers address fine-grained visual understanding in multimodal models, with the main paper focusing on discrete visual token quantization and the related paper on leveraging generation-side representations for fine-grained understanding.