Machine learningLLMs & Text
Scalable-Softmax Is Superior for Attention
The study proposes Scalable-Softmax (SSMax), a replacement for Softmax in language models, which improves performance in long contexts and key information retrieval, and allows better focus on key information.
Featured in No. 84 on 5 Feb 2025 · 5 days after release · 47 citations today
- Released
- 31 Jan 2025
- First featured
- No. 84 · 5 Feb 2025
- Citations (Semantic Scholar)
- 47
- Influential citations
- 8
- Published in
- Not yet, as far as Semantic Scholar knows
- Shares when featured
- 108
- Identifier
- arXiv:2501.19399
Citations and venue from Semantic Scholar (ODC-BY), refreshed weekly. Summary: Quant Letter (CC BY 4.0).