NEWS · RESEARCH · #356
BITCOS layout pushes ternary LLM storage below 1.58 bits and speeds inference
The arXiv paper (2609.16338v1) introduces BITCOS, a distribution-adaptive layout for ternary LLM weights that stores a dense presence bitmap plus a compacted sign vector and costs 2 - z bits per weight given zero density z. Evaluated on 29 ternary models, BITCOS beats conventional five-trit packing in 26 cases (reaching 1.485 bits/weight on the sparsest model), provides optimized unpacking for AVX-512/AVX2 and Intel Xe2 GPUs, and yields up to 1.28× kernel speedups and up to 1.18× CPU / 1.27× GPU end-to-end decode throughput improvements.
KEY POINTS
- The arXiv paper (2609.16338v1) introduces BITCOS, a distribution-adaptive layout for ternary LLM weights that stores a dense presence bitmap plus a compacted sign vector and costs 2 - z bits per weight given zero density z.
- Evaluated on 29 ternary models, BITCOS beats conventional five-trit packing in 26 cases (reaching 1.485 bits/weight on the sparsest model), provides optimized unpacking for AVX-512/AVX2 and Intel Xe2 GPUs, and yields up to 1.28× kernel speedups and up to 1.18× CPU / 1.27× GPU end-to-end decode throughput improvements.
- BITCOS materially reduces storage per ternary weight and translates that compression into measurable inference throughput gains on CPUs and GPUs, improving deployment efficiency for ternary LLMs.
WHY IT MATTERS
BITCOS materially reduces storage per ternary weight and translates that compression into measurable inference throughput gains on CPUs and GPUs, improving deployment efficiency for ternary LLMs.