Machine learningLLMs & Text

Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free

The research shows that Mixture-of-Experts Large Language Models can be effective embedding models without finetuning, and suggests a combination of routing weights and hidden state for better performance.

Read the paper

View as Markdown Open in Claude Open in ChatGPT Open in PerplexitySite index for AI (llms.txt)

Featured in No. 70 on 17 Oct 2024 · 3 days after release · 42 citations today · published in International Conference on Learning Representations

Released: 14 Oct 2024
First featured: No. 70 · 17 Oct 2024
Citations (Semantic Scholar): 42
Influential citations: 2
Published in: International Conference on Learning Representations
Shares when featured: 47
Identifier: arXiv:2410.10814

Citations and venue from Semantic Scholar (ODC-BY), refreshed weekly. Summary: Quant Letter (CC BY 4.0).

Related papers

SliceGPT: Compress Large Language Models by Deleting Rows and Columns

Efficient Adversarial Training in LLMs with Continuous Attacks

Gecko: Versatile Text Embeddings Distilled from Large Language Models

CoLLEGe: Concept Embedding Generation for Large Language Models

Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling