---
title: Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models
url: https://www.ml-quant.com/papers/arxiv/2501.01423/
site: ML-Quant (https://www.ml-quant.com)
updated: 2026-09-26
license: Summaries CC BY 4.0; links go to the original sources
index: https://www.ml-quant.com/llms.txt
identifier: arXiv:2501.01423
source_url: https://arxiv.org/abs/2501.01423
featured: 2025-01-08
citations: 395
topic: ML & AI Methods
---


# Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models

The paper proposes a new model, VA-VAE, that aligns the latent space with pre-trained vision foundation models, enabling faster convergence of Diffusion Transformers in high-dimensional latent spaces and achieving top performance on ImageNet 256x256 generation.

- Source: https://arxiv.org/abs/2501.01423
- Identifier: arXiv:2501.01423
- Released: 2025-01-02
- First featured: Quant Letter No. 81 (2025-01-08): https://www.ml-quant.com/issues/2025-01-08/
- Citations (Semantic Scholar): 395
- Published in: 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Topic: ML & AI Methods

## Related

- [Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models](https://www.ml-quant.com/papers/arxiv/2411.04996/): The Mixture-of-Transformers (MoT) is a sparse multi-modal transformer architecture that reduces pretraining costs and allows modality-specific processing with global self-attention.
- [ECG-FM: An Open Electrocardiogram Foundation Model](https://www.ml-quant.com/papers/arxiv/2408.05178/): ECG-FM, a transformer-based model for ECG analysis, shows strong performance in predicting cardiac conditions, having been pretrained on 2.5 million samples.
- [FiT: Flexible Vision Transformer for Diffusion Model](https://www.ml-quant.com/papers/arxiv/2402.12376/): Improved Vision Transformer: The Flexible Vision Transformer (FiTv2) is presented, a design that generates images with unrestricted resolutions and aspect ratios, showing excellent performance across various resolutions.
- [Geometry-Aware Generative Autoencoders for Warped Riemannian Metric Learning and Generative Modeling on Data Manifolds](https://www.ml-quant.com/papers/arxiv/2410.12779/): The Geometry-Aware Generative Autoencoder (GAGA) addresses challenges of high-dimensional datasets by combining manifold learning with generative modeling.
- [xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations](https://www.ml-quant.com/papers/arxiv/2408.12590/): Video Synthesis: xGen-VideoSyn-1 is a text-to-video generation model that creates realistic scenes from text descriptions using a video variational autoencoder and a Diffusion Transformer model.
- [Banach-Tarski Embeddings and Transformers](https://www.ml-quant.com/papers/arxiv/2311.09387/): Interpretable Transformers: A novel method for embedding recursive data structures into high-dimensional vectors has been developed, offering an interpretable model for transformer's latent state vectors and enabling computations without decoding.