Machine learningML & AI Methods

Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models

The Mixture-of-Transformers (MoT) is a sparse multi-modal transformer architecture that reduces pretraining costs and allows modality-specific processing with global self-attention.