Intel's BITCOS Squeezes Ternary Model Weights Below the Theoretical Limit
By rethinking how weights are stored rather than retraining models, Intel's new compression format pushes ternary language models below 1.58 bits per weight and accelerates decoding by up to 27% on GPUs.
The 1.58-bit threshold in ternary language models has long seemed immovable, but Intel researchers have found a way around it. Rather than retraining or modifying the model itself, they developed BITCOS, a storage format that compresses ternary weights to 1.485 bits per weight while maintaining the model's original accuracy. On CPUs, the approach yielded decoding speedups of up to 18%, while GPU performance improved by as much as 27%.
The breakthrough hinges on a simple observation: the theoretical 1.58-bit figure assumes ternary models distribute their three possible weight values—negative one, zero, and positive one—equally. In practice, real models contain far more zeros than this calculation presumes. BITCOS exploits this sparsity by storing the location and sign of nonzero weights separately, allowing zeros to occupy less space. The technique requires no model retraining and produces no change in output, functioning much like fitting identical contents into a more compact container.
The origin of the 1.58-bit figure
Ternary quantization restricts weights to just three values: −1, 0, and +1. The information-theoretic minimum to represent three equally probable options is 1.58 bits. However, practical storage differs from this ideal. The standard method packs five ternary values into an eight-bit byte, yielding an average of 1.6 bits per weight. Since models typically organize weights into blocks of 128, the final byte in each block remains partially empty, pushing the actual rate to 1.625 bits per weight.
Intel researchers analyzed weight distributions across 29 checkpoints from seven ternary model families and discovered that zeros ranged from 29.7% to 51.5% of all weights. In 26 of these checkpoints, the zero density was high enough for BITCOS to outperform five-trit packing. The sparsest model tested was a ternary variant of Qwen3-1.7B, created using CAT-Q post-training quantization, where zeros comprised 51.48% of weights. BITCOS reduced storage to 1.485 bits per weight in this case.
Models commonly store weights in blocks of 128; however, this leaves the final byte partly unused and pushes the actual rate to 1.625 bits per weight.
How zero density enables compression
BITCOS stands for "BITmap and COmpacted Signs." The format divides model weights into two separate streams. The first assigns a single bit to every weight to indicate whether it is zero or nonzero. The second assigns a sign bit exclusively to nonzero weights.
A weight with a sign—positive or negative—requires two bits, but a zero needs only the presence bit since it carries no sign information. When z represents the proportion of zero weights, BITCOS consumes 2 − z bits per weight. At 40% zeros, this yields 1.6 bits; at 51.5% zeros, it drops to 1.485 bits. Since the format alters only how weights are stored, unpacking restores the original −1, 0, and +1 values without sacrificing accuracy.
BITCOS becomes more efficient than five-trit packing when a model's zero proportion exceeds 37.5%. Of the 29 checkpoints Intel examined, 26 crossed this threshold.
BITCOS becomes smaller than five-trit packing once more than 37.5% of a model's weights are zero, a threshold reached by 26 of the 29 checkpoints Intel examined.
Translating compression into faster inference
Designed for token-by-token decoding with small batch sizes, BITCOS reduces the volume of weight data traversing memory. Intel created dedicated unpacking kernels for AVX-512 and AVX2 CPUs, as well as for Xe2 GPUs, joining a broader push to integrate compressed models into faster inference pipelines for AI agents.
On AVX-512 systems, the kernel leverages the presence bitmap as a mask and employs pdep to distribute the compacted sign bits across nonzero weight positions. Xe2 GPUs lack an equivalent instruction, so Intel substituted a 2KB lookup table to perform the same operation.
Performance across five hardware platforms
When measured against 2-bit kernels, BITCOS delivered 10% to 18% faster decoding on a 64-core Xeon server and 2% to 15% faster performance on a 24-core Core Ultra 9. The integrated Arc 140V GPU showed 9% to 22% improvement, while the discrete Arc Pro B70 achieved 2% to 27% gains. These measurements capture decoding performance after model loading, independent of efforts to reduce GPU inference cold starts from minutes to seconds.
Where the format fell short
The eight-core Lunar Lake CPU presented a different picture. Intel's fixed 2-bit kernel outpaced BITCOS on every model tested because the system possessed sufficient bandwidth to make unpacking itself the limiting factor. BITCOS retained its speed advantage on GPUs, though decoding overhead constrained the magnitude of improvements. This pattern aligns with observations from computer scientist and AI infrastructure author Chip Huyen, who has emphasized that optimal inference strategies depend on whether compute, memory bandwidth, or memory capacity represents the actual bottleneck.
On the eight-core Lunar Lake CPU, Intel's fixed 2-bit kernel beat BITCOS on every model because the system had enough bandwidth to make unpacking the bottleneck.
Remaining limitations and unanswered questions
The research has not undergone peer review. All five test systems ran Intel hardware, and end-to-end benchmarks covered seven models at batch size one. Intel has not yet evaluated the format on Nvidia, AMD, or Arm processors.