Machine learningLLMs & Text
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
KVQuant, a new method for quantizing cached KV activations in large language models, has been developed, allowing the LLaMA-7B model to be served on an 8-GPU system with minimal degradation.
Featured in No. 44 on 10 Apr 2024 · · 701 citations today · published in Neural Information Processing Systems
- Released
- 31 Jan 2024
- First featured
- No. 44 · 10 Apr 2024
- Citations (Semantic Scholar)
- 701
- Influential citations
- 69
- Published in
- Neural Information Processing Systems
- Shares when featured
- 64
- Identifier
- arXiv:2401.18079
Citations and venue from Semantic Scholar (ODC-BY), refreshed weekly. Summary: Quant Letter (CC BY 4.0).