Which 1-Bit Weights Can You Trust?

Sep 17, 2026

1-bit language models compress every weight to a ternary value +1, 0, or −1 during the forward pass, while keeping the weights for gradient updates. This does make inference fast and cheap, but the information about each weight flips is gone.

During training, each weight crosses zero whenever its sign flips. A weight that crossed zero twenty times in the first hundred steps and then did not change its sign for nine hundred more is different from one that crossed zero at step nine hundred and ninety. The first weight has committed. The second has not. But both look the same in the saved model.

I instrumented a BitNet training run with a tracker that records, for every BitLinear weight: (a) how many times it flipped, and (b) at which step it last flipped. From these two numbers we get a reliability score and classify every weight into one of four groups.

Methodology

with adding a tracker to the training loop that runs once after every step. For each weight matrix I maintained two counters: flip count (INT16, how many times the sign changed) and last flip (INT16, the step at which it last changed). Both start at zero. After each step we compare the sign of each weight to its sign at the previous step. Wherever the sign changed, flip count increases and writes the current step number to the last flip.

this is skipped during warmup, weights change and flip often. so recording these early flips would fill noise. For a 16.4M parameter model, the tracker covers 3.5M weights across 42 tensors. Memory cost: 2 bytes per weight (INT16), about 7 MB, which is freed after training. using flip count and last flip we calculate two values for each weight after T training steps:

Sign age:

Sign age is 1 for a weight that never flipped.

Flip rate:

Flip rate is 0 for weights that never moved.

Reliability:

Experimental Setup

I then trained a 1-bit language model on WikiText-2 using a BitNet b1.58 architecture. The model has 16.4M parameters: 6 layers, hidden size 256, 4 attention heads, 2 KV heads, FFN size 512, maximum sequence length 256, used the GPT-2 tokenizer. Weights are quantized to {−1, 0, +1} using the absmean scaling; activations are quantized to INT8 per token. Training runs for 1,000 optimizer steps with batch size of 32 sequences (8 micro-batch, 4 gradient accumulation), used AdamW with learning rate 3×10⁻⁴, cosine decay to 3×10⁻⁵, weight decay 0.1, and gradient clip 1.0. warmup is the first 100 steps and training runs in BF16 autocast.

The tracker recorded 11,165,218 total flips over 900 steps across 3.5M weights.

Results

Table below shows the weight breakdown. 68.8% of tracked weights never flipped after warmup, i.e. they are locked. 14.0% are reliable: they flipped but stopped early with low flip rate. 8.9% are still settling. 8.3% are volatile, i.e. unstable.

Sign flip tracking adds nothing to the deployed model. It only requires that during training you keep two INT16 counters per weight, costing about 7 MB for a model of this size, freed when training ends. From these counters you get a reliability score for every weight.