SJ-KVaRN

1. What a KV cache is, and what KL in nats means

A language model writes one token at a time. To write the next token, each attention layer looks back at every earlier token. It does not recompute them. It keeps two vectors per earlier token per layer, a key and a value, in a store called the KV cache.

The cache grows with every token. Model weights are a fixed size. So at long context the cache becomes the part of memory that decides whether a job fits on the card.

A concrete case. Qwen3.8 27B is a hybrid model: only 16 of its layers use full attention and keep a KV cache. Each of those layers stores 1,024 numbers for the key and 1,024 for the value, per token. Stored as 16-bit floats, that is 65,536 bytes per token. At a 110,592-token context the cache alone is 6.75 GB. A common 8-bit format roughly halves this, to 3.59 GB. These two figures are computed from the format sizes. SJ-KVaRN at 4/4 bits allocates 1.99 GB at the same context, measured on the card (section 8.4).

Throughout, GB means 1,024 MiB, the convention GPU tools use.

The cost of compression: KL divergence in nats

Fewer bits per value means each key and value is stored a little wrong. The question is how much that changes what the model says. We measure it directly:

1. Run the model once with a high-precision cache. Record its full probability distribution over the vocabulary at each generated token. This is the reference, p.
2. Replay the exact same token stream with the compressed cache. Record its distribution, q, at the same positions.
3. Compute the KL divergence KL(p ‖ q) at each position, in nats, and average.

Zero means identical predictions. Bigger numbers mean the compressed model drifts from the reference more often or by more.

We score generated tokens only. Prompt-text positions produce rare, very large spikes under any numeric change, and those spikes dominate a mean. Where we have them, we also report medians and high percentiles, because KL distributions are heavy-tailed.

This note is about quality per bit. Speed depends on the engine and its kernels and is not covered here.

2. How the codec works

SJ-KVaRN is built on KVarN, a KV-cache quantization method from Huawei's CSL group (credits). KVarN compresses keys and values in groups of 128 tokens. Before quantizing, it rotates and rebalances each group so that plain rounding works well. SJ-KVaRN keeps that core and changes what happens around it: where new tokens live before compression, at what precision, and when they get compressed.

2.1 The life of a token

Every token passes through up to four regions. The attention kernel reads all of them in one pass.

1. Sink. The first 128 tokens stay at 16 bits for the whole run. Attention puts outsized weight on the first few tokens of a context, so compressing them is costly. The KVarN reference keeps the same 128-token sink.
2. Staging. A new token's key and value are rotated (see 2.2) and written to a ring buffer in a 6.125-bit format (tq6_0, a rotated 6-bit format: 98 bytes per 128 values; tq5_0 and turbo4, used as comparators below, are 5-bit and 4-bit relatives). This is the "Staged" in the name.
3. Exact tail. The newest tokens stay in that ring, uncompressed by KVarN, so recent context is read at near-full precision.
4. Seal. When the tail reaches its maximum, every full 128-token group older than the newest 4,096 tokens is compressed into body records in one batch. This is the "Journaled" in the name: like a write-ahead log, recent writes sit in a precise journal and are committed to compact storage in batches.

2.2 Rotation and balancing

Plain rounding fails when a few channels carry values much larger than the rest. One outlier sets the scale, and every small value rounds to zero. KVarN fixes this in two steps before rounding.

• Rotation. Each 256-value head vector is multiplied by a 256-point Hadamard matrix, an orthonormal matrix of +1 and −1 entries. Queries are rotated the same way, so query·key dot products do not change. Values are rotated too, and the attention output is rotated back. The effect is to spread an outlier's energy across all channels. On Gemma 4 31B, whose full-attention heads are 512 values wide, each half of the head gets its own 256-point rotation.
• Balancing. A 128-token by 256-channel tile is then rescaled by alternating row and column passes until every row and column has a similar spread. This is Sinkhorn-style balancing, done in the log domain. We run 16 passes and keep the final scales. The KVarN reference configuration runs 8 passes and keeps the best scales seen.
• Rounding. Each row is rounded to the target bit width with its own scale and zero point. All scale factors are stored as fp16 next to the payload.

2.3 Two body types: scalar and trellis

Scalar body. Plain rounding, as described above.

Trellis body. Trellis-coded quantization treats a run of values as one path through a small state machine. Each step emits a few bits; the state picks which part of a trained codebook those bits index. A Viterbi search with 64 states finds the path with the lowest squared error. The 3-bit codebook has 512 trained entries and the 2-bit codebook has 256, both trained offline for mean squared error. After coding, each row's scale and offset are refit so the decoded values match the original mean and spread. The default in our current development build (not yet released) runs the trellis along the token axis of each tile. Earlier versions ran it along the channel axis; section 4 shows the difference.

Trellis and scalar records have exactly the same size. The trellis places the same bits better. By default the trellis is used for 3/3, 3/2 and 2/2, and 4/4 stays scalar.

3. Bits per value

"4/4" means 4-bit keys and 4-bit values; "3/2" means 3-bit keys and 2-bit values. A "t" marks a trellis body. Each 128-token record also stores fp16 scales, 2,304 bytes per record per KV head, which adds 0.28125 bits per value.

Effective bits average over the whole occupied cache: the body, the 6.125-bit tail (4,096 tokens just after a seal, up to 8,192 just before one), and the 128-token sink. Formula: [(N − 128 − t) × body + t × 6.125 + 128 × (6.125 + 16)] / N, for N cached tokens and tail length t. The range shows t = 4,096 to 8,192. The staging ring is allocated at full size up front, so actual allocated memory is a little higher than the occupied figure (section 8.4).

4. What SJ-KVaRN changes over KVarN

Each change below lists what it buys and what it costs, with the measured cells it rests on. Where the effect has not been measured in isolation, we say so.

4.1 Staging precision: a 6-bit tail instead of 16-bit

What it is. Tokens waiting to be sealed are stored at 6.125 bits (tq6_0) instead of 16-bit floats. The sink keeps a 16-bit copy.

What it buys. Memory. Each 4,096 tokens of tail costs 98 MiB across Qwen's 16 attention layers in this format, against 256 MiB in 16-bit floats (computed from the formats). That is what makes a 4,096 to 8,192-token exact tail affordable.

What it costs. A little accuracy in the tail. Two measured comparisons:

• 16-bit vs 6-bit staging, nine short histories (no confidence intervals): KL 16% lower at 4/4 with 16-bit staging (0.00083 vs 0.00099) and 11% higher at 3/3 (0.00253 vs 0.00227). 16-bit staging used 0.33 to 0.39 GB more at 100K.
• 8-bit vs 6-bit staging, 100K trajectories: 8-bit staging gave lower KL at 4/4 (ratio 0.870, 95% CI 0.811–0.924, 3 histories) and at 3/3t (0.900, CI 0.821–0.975, 3 histories). At 3/2t there was one history and no detected difference (1.053, CI 0.948–1.212).

So 6-bit staging has about 11–15% higher KL at 4/4 and 3/3t than 8-bit staging, in exchange for the smaller ring.

4.2 Journaled exact tail

What it is. The newest tokens are not compressed by KVarN at all. The reference compresses each 128-token group as soon as it fills, so its exact suffix is 0 to 127 tokens.

What it buys. The most recent context is usually what the model attends to most. On 3/2t at a 102K depth (vs an 8-bit reference, one corpus, 6,141 positions, RTX 3090):

What it costs. Memory for the ring (section 8.4), and the tail is part of every attention read.

4.3 Adaptive tail

What it is. The tail is not a fixed length. It grows from 4,096 to 8,192 tokens with nothing sealed, then everything older than the newest 4,096 is sealed in one batch. An idle server also seals back to 4,096 between requests. This is the default at every bit width.

How it behaves. Sealing happens once every ~4,096 tokens in one batch instead of once every 128 tokens. On average the tail holds more exact tokens than a fixed 4,096 tail, with the ring sized for 8,192 tokens plus margin.

Measured: on the nine short histories, the 4,096/8,192 adaptive tail cut 4/4 KL by 49% against a 512/1,024 adaptive tail (0.001045 vs 0.002049), and 3/3t KL by 43% (0.001878 vs 0.003314). At 9.4K–15K tokens a larger tail means a much larger share of the cache is exact, so this mostly measures "more exact tokens", not the adaptive policy itself.

4.4 Trellis bodies

What it buys. Lower KL at the same bytes for 3-bit and 2-bit bodies. Measured cells, all paired on the same positions and reference:

Not the default at 4/4. A 4-bit trellis exists. On the token axis it lowered 4/4 KL (ratio 0.844, CI 0.804–0.889, one run), but 4/4 stays on the scalar body by default for reasons outside the scope of this note.

4.5 Tiered edge layers (measured, not enabled)

We tested keeping the first and last layers of the model at 4/4 while the middle runs at 3/3t or 3/2t. At 32K, the ratios against the uniform config were 0.928 (CI 0.77–1.10) for 3/3t and 0.850 (CI 0.64–1.09) for 3/2t. Both intervals include 1, so tiering is available as an option and is not the default.

5. Headline: versus paper-faithful KVarN

What was compared.

• Model: Qwen3.8 27B, 4-bit weights (14.5 GB file), the same file on both sides.
• Data: nine frozen real histories (3 agentic, 3 retrieval, 3 coding), 9.4K to 15K tokens each. 964 generated-token positions that already have a sealed body behind them, plus 140 early-position controls.
• Reference: each engine against its own 16-bit-cache run. Two engines' 16-bit runs differ from each other by 0.000847 nats on average, about the size of our 4/4 error, so cross-engine references would swamp the result.
• Paper-faithful KVarN, as its port recommends: a public third-party llama.cpp port of KVarN (BeeLlama v0.4.6) with the 1,024-token 16-bit exact tail its README recommends.
• Paper-faithful KVarN, tail 0: the same port with no extra exact tail. It keeps a 128-token exact sink and a 128-token exact suffix. This is the closest available match to the KVarN reference configuration, whose exact suffix is the 0 to 127-token unfilled group.
• SJ-KVaRN: 16-bit 128-token sink, 6-bit staging, adaptive 4,096/8,192 tail. Scalar body at both 4/4 and 3/3, so the body format matches the stock side.

Brackets are paired-position bootstrap 95% CIs. A task-level bootstrap over the nine histories gives 30.5–46.0% (4/4) and 41.2–56.9% (3/3) against the 1,024-tail configuration, and 41.4–54.3% and 45.6–66.3% against tail 0. Medians at 4/4: SJ-KVaRN 9.57e-6 vs 1.99e-5 for tail 0. Against tail 0, SJ-KVaRN is lower at 687 of 964 positions (4/4) and 732 of 964 (3/3).

Counts are summed over the nine histories. They are small, so read them as an illustration of the KL result, not a separate test.

5.1 At 3/2 bits, with the current default

The same nine histories and 964 positions, rerun on our current default build. Our 3/3 and 3/2 sides use the default trellis body on the token axis; the stock port has no trellis, so its 3/3 and 3/2 bodies are scalar. This is a comparison of default configurations, body coding included. Our 16-bit references are fresh on this build. The stock side uses the same port, flags and 16-bit references as section 5; four of those references were regenerated and are byte-identical to the originals.

"Lower by" = 1 − mean(ours) ÷ mean(stock). Brackets are stratified paired-position bootstrap 95% CIs; "Task CI" resamples the nine histories. Bits per value are averaged over the scored positions and computed from each engine's allocation rules. At 3/2 ours has the lower KL at 822 of 964 positions against the 1,024-tail configuration and 866 against tail 0.

5.2 Closer to equal bytes

Two ways to bring the bytes closer on the same nine histories: give the stock side a longer 16-bit tail, or compare our lower-bit configuration with stock at 4/4. Same positions, references and statistics as 5.1.

Reading it.

• At about the same memory (4.98 vs 4.65 bits per value, ours 7% more), SJ-KVaRN 3/3t has 20.7% lower KL (CI 3.3–35.1%, 9 of 9) than stock 4/4 with no tail. This compares our 3/3 trellis default with stock's 4/4 scalar, not like-for-like settings.
• SJ-KVaRN 3/2t against stock 4/4 with no tail (4.76 vs 4.65 bits) shows no significant difference either way.
• Stock with a 4,096-token 16-bit tail spends 1.67 times our 4/4 bits and still has higher KL: ours is 20.2% lower.
• Stock with an 8,192-token 16-bit tail is not significantly different from our 4/4 (−3.9%, CI −23.8 to 12.8, 5 of 9) at 2.5 times the bits.
• These are short histories, where any exact tail is a large share. An equal-bytes comparison at 100K tokens, where the tails are a small share, has not been run.

5.3 At long range: the 200K needle study

A second study placed 8 facts ("needles") in each of 6 long documents and asked for them back at look-back distances of 5K, 10K, 20K, 50K, 75K, 100K, 150K and 200K tokens. We scored KL on the answer tokens (4,806 positions per arm), with each engine measured against its own 8-bit-cache run. The model was a fine-tuned variant of the same Qwen3.8 27B. Our trellis arms in this study used the earlier channel-axis trellis.

Results are pooled across all 8 look-back distances: the ratio is mean answer-token KL over every answer position, ours ÷ stock. The 95% intervals come from a nested cluster bootstrap: the 6 haystacks are resampled, then the needles within each one (10,000 resamples). The same method was applied to every pair, and nothing was excluded.

Bits per value for the stock side at these contexts are not reported here, so this compares default configurations, not equal bytes. The 3/3t and 3/2t rows also compare our trellis body with the stock scalar body. Weighting each distance equally instead of each position gives the same significance calls (for example 4/4 vs tail 0: 0.20, CI 0.10–0.73).

Reading it. Over 5K–200K look-back, SJ-KVaRN 4/4 had 79.7% lower answer-token KL than paper-faithful KVarN 4/4 (CI 27–90%) and 82.1% lower than the 1,024-tail configuration (CI 8–91%). 3/2t was lower than stock 3/2 with its 1,024-token tail (70.7%, CI 9–91%). Against stock 3/2 at tail 0 the difference was not significant. 3/3t was not significantly different from either stock 3/3 config, but its point estimates favour stock. Answer-token KL is heavy-tailed: in every arm, the 10 largest of 4,806 positions carry 60–90% of the total, so these ratios mostly compare rare large errors, and with 6 haystacks the intervals are wide. All arms, ours and stock, returned the gold answer as the top-1 token at 99.6% or more of answer positions at every distance.

6. Versus other cache formats: quality per bit on ~100K-token trajectories

How does SJ-KVaRN compare with the other KV-cache formats someone could pick today? We ran every format on the same data in one matched test.

• Model: Qwen3.8 27B, the same 14.5 GB 4-bit weight file as the headline.
• Data: six real trajectories of about 100K tokens (coding, agentic and retrieval, two each). Every format replays the same generated token streams. 37,502 scored generated-token positions per format, spread from about 3K to 107K tokens into each trajectory.
• Reference: one 8-bit cache run (q8_0 keys and values) on the same engine build, so all rows share one reference and one build.
• Formats: q8_0 is an 8-bit block format; q5_1 a 5-bit block format with a scale and an offset; tq5_0 a rotated 5-bit format; turbo4 a rotated 4-bit format with one 16-bit norm per 128 values. SJ-KVaRN used the settings of section 5, with the trellis on the channel axis, which was the default when this test ran (section 8.1 has the current token-axis default).
• Bits per value: to compare like with like, every format is charged the KV memory it allocates for a 110,592-token context, divided by the number of key and value numbers that context holds. For the plain formats this equals the format size. For SJ-KVaRN it includes the 16-bit sink and the full-size staging ring, so it is higher than the body size.

Percentages use unrounded means; 41% is 1 − 0.000665/0.001128. CIs are paired bootstrap 95% intervals over the 37,502 positions; neighbouring positions are correlated, so these intervals are somewhat optimistic.

Format bits come from the block layouts: q8_0 stores 34 bytes per 32 values (8.5 bits), q5_1 24 bytes per 32 (6.0), tq5_0 82 bytes per 128 (5.125), turbo4 66 bytes per 128 (4.125). Measured memory is the allocated KV cache at a 110,592-token context (section 8.4); computed rows are format size × 110,592 tokens × 16 layers × 2,048 values; the measured tq5_0 / turbo4 allocation matches its computed size. Allocated bits = memory ÷ (110,592 × 16 × 2,048) values. The SJ-KVaRN memory figures come from the current build; the build that produced these KL rows allocated slightly more (0.04 to 0.07 GB) because of an over-sized record pool that has since been fixed; the extra records were never written, so KL is unaffected.

Reading it.

• Next closest format. Of the other formats tested, tq5_0 / turbo4 has the lowest KL after q8_0 / q5_1 (section 6.1). SJ-KVaRN 4/4 has 41% lower KL than tq5_0 / turbo4, with 2% more memory (4.72 vs 4.63 bits per value).
• Same 4-bit class. Against turbo4 / turbo4, SJ-KVaRN 4/4 has 68% lower KL with 14% more memory (4.72 vs 4.125 bits per value).
• Less memory, still better. SJ-KVaRN 3/3t allocates 9% less memory than turbo4 / turbo4 (3.76 vs 4.125 bits per value) and has 18% lower KL.
• Where it does not win. SJ-KVaRN 3/2t, at 3.27 bits per value, has higher KL than the 4.125-bit turbo4 / turbo4 (0.003325 vs 0.002071). It is the smallest cache in the table. The current token-axis trellis lowers 3/2t KL by about 23% (section 8.1), which this test predates.

6.1 The format with lower KL in this test: q8_0 / q5_1

In this test one format beats SJ-KVaRN 4/4 on KL: 8-bit keys with 5-bit q5_1 values. (Higher-bit formats such as q8_0 / q8_0 and tq6_0 / tq6_0 also have lower KL than 4/4 in the other tests below.) Its mean KL is about half of ours, 0.000329 against 0.000665 nats. The absolute gap is 0.00034 nats. In top-token terms it agrees with the reference at 99.59% of positions against 99.39%, about 2 more agreements per 1,000 positions.

It pays for that with memory: 3.06 GB against 1.99 GB at a 110,592-token context, 7.25 against 4.72 bits per value. Put the other way: SJ-KVaRN 4/4 gets within 0.00034 nats of an 8-bit-key, 5-bit-value cache in 65% of its memory; q8_0 / q5_1 needs 1.54 times the memory for that last 0.00034 nats.

7. 12 GB cards: SJ-KVaRN 3/2t

On a 12 GB card the KV cache decides how much context a 27B model can hold. This section asks two questions: how much context each cache option leaves room for, and what each option costs in quality. The short answer: SJ-KVaRN 3/2t ran a full 204,800-token context for a 27B model quantized to fit a 12 GB card, with about 0.74 GB of the 11 GB budget to spare and about 0.48 GB under the stricter 11,000 MiB (calibrated to an RTX 3060). It buys that context with higher KL than the 4-bit formats measured so far.

7.1 The measured fit

• Model: Qwen3.8 27B, quantized to fit a 12 GB card (6.81 GB of weights), with its multi-token-prediction draft head loaded.
• Cache: SJ-KVaRN 3/2 with a trellis body, the 6.125-bit staging tail (adaptive, 4,096 to 8,192 tokens) and the 16-bit sink. The model's recurrent layers keep their state at 16 bits; the draft head keeps an 8-bit cache of its last 8,192 tokens.
• Run: a 199,424-token prompt plus 5,120 generated tokens, so the cache was full to 204,800 tokens at the peak.
• Card: measured on a 24 GB RTX 3090 Ti as the process's own memory, then calibrated to a 12 GB RTX 3060. The calibration was set from one run on a real RTX 3060. The full 204,800-token run itself was not repeated on a 12 GB card.
• Budget: about 11 GB usable (11,264 MiB) of the card's 12 GB. We also report a stricter 11,000 MiB budget, which keeps a full 1 GB free for the desktop after the driver's own reservation.

"Everything else" is the measured total minus the listed items. Ledger items are measured allocations on the 24 GB card. The RTX 3060 calibration is applied to the card-level peak minus idle (10.33 GB), not to the process figure. 1 GB = 1,024 MiB.

7.2 How much context each cache option allows

Only the SJ-KVaRN 3/2t row was run at 204,800 tokens. The other rows are computed: we take the measured memory of the run above without its cache and without its attention mask (7.77 GB), and add each option's cache size and the mask (1 KiB per token of context, which matches the measured compute buffers at 65,536, 131,072 and 204,800 tokens). That assumes the rest of the memory is the same for every cache format; we have not measured that for the other formats on this setup.

K = 1,000 tokens, rounded down. "Bits per value" for SJ-KVaRN is the body size; counting the sink and staging ring, the allocated figure at 204,800 tokens is about 4.52 (4/4), 3.54 (3/3t) and 3.02 (3/2t, measured). Format sizes come from the block layouts: q4_0 stores 18 bytes per 32 values (4.5 bits) and turbo3 50 bytes per 128 values (3.125 bits); the others are listed in section 6. Computed cache size = bits per value × 4,096 ÷ 8 bytes per token (16 layers × 2,048 values). SJ-KVaRN rows use the rule that reproduces the measured allocations in section 8.4: body bytes per token × (context − 4,224) + 259 MiB for the sink, staging rows and ring. For 3/2t at 204,800 tokens that rule gives 2.38 GB, against 2.36 GB measured, so the computed SJ-KVaRN rows are slightly conservative. The largest-context columns for SJ-KVaRN use the same rule. The KL and top-token columns are from the matched 100K test of section 6, which used Qwen3.8 27B with 4-bit weights, not the 12 GB weights, and the earlier channel-axis trellis.

Reading it.

• Context. Of the options in the table, SJ-KVaRN 3/2t leaves room for the most context in 11 GB, about 261K tokens, and it is the only row run at 204,800 tokens on this setup. turbo3 / turbo3 comes next at about 250K. An 8-bit cache stops at about 96K.
• Quality. 3/2t pays for that in KL. In the 100K test its mean KL was 1.6 times that of turbo4 / turbo4 (0.003325 vs 0.002071); it kept the same top token at 98.78% of positions against 98.97%, about 2 more changes per 1,000 positions. The current token-axis trellis lowered 3/2t KL by about 23% against the channel axis used here (section 8.1, a separate run with its own reference).
• Task accuracy. On a fine-tuned variant of the 4-bit-weight model, 3/2t showed no detected loss against the 8-bit cache, at shallow depth or after a 100K-token prefix (section 8.5). That test is small and is not a proof of equivalence.
• The middle option. SJ-KVaRN 3/3t reaches about 224K tokens in 11 GB with 18% lower KL than turbo4 / turbo4, which reaches about 193K. Computed, 3/3t also fits 204,800 tokens, with about half the KL of 3/2t in the same test (0.001694 vs 0.003325). Its computed margin is about 0.27 GB under 11 GB but only about 0.01 GB under 11,000 MiB, too thin to claim without a run; 3/2t is the option measured at 204,800 with margin under both budgets.
• The same pattern on a 24 GB card. On Gemma 4 31B under a 23 GB cap, 3/2t and 3/3t were the measured options that reached the model's full 262,144-token window (section 9.2).

Against paper-faithful KVarN at 3/2. On the nine short histories of section 5, 3/2t had 76.3% lower KL than paper-faithful KVarN 3/2 with its 1,024-token tail, and 81.4% lower at tail 0 (section 5.1). Those histories are short, so our side used more bits per value than the stock side there (4.76 vs 4.43 and 3.19); stock has not been scored at 100K or on the 12 GB weights. Our side used the trellis body, which the stock port lacks. At about equal bytes on the same histories (3/2t at 4.76 vs stock 4/4 tail 0 at 4.65 bits), there was no significant difference (−15.3%, CI −42.4 to 8.5, 4 of 9; section 5.2).

8. More results on Qwen3.8 27B

Model: Qwen3.8 27B, hybrid architecture with 16 full-attention layers, 4 KV heads of 256 dims each, 4-bit weights. GPU: RTX 3090 Ti, 24 GB, except where a subsection says otherwise. All KL is on generated tokens.

8.1 The current default (token-axis trellis) at 100K

A later run on a newer build scored SJ-KVaRN on the same six ~100K trajectories with both trellis axes, against its own regenerated 8-bit reference, with a block bootstrap. Its reference differs from section 6's, which moves arm KL by up to about 10% per history, so compare rows within this table, not with section 6.

The token axis lowered KL by 23% at both 3/3t and 3/2t (paired ratios 0.769 and 0.772, section 4.4). The 4/4 cell, 0.000664, is close to the section 6 run's 0.00066.

8.2 At 32K, against a 16-bit reference

Three trajectories to 32K generated tokens, 3,074 positions, reference = 16-bit cache. Channel-axis trellis.

8.3 Same harness as Gemma

To compare the two models directly, we ran the Gemma harness (section 9) on a fine-tuned variant of Qwen3.8 27B: three prompts generating up to 25K tokens each, 74,873 generated tokens, reference = 16-bit cache, channel-axis trellis, one seed.

8.4 Memory

KV cache memory measured on the card at a 110,592-token context, Qwen3.8 27B (16 attention layers). The 16-bit and 8-bit rows are computed from the format sizes for comparison.

The SJ-KVaRN rows are body plus a fixed 259 MiB (0.25 GB) for the sink, the staging rows and a ring sized for 8,192 tokens plus margin, which is allocated in full up front. At this context, 4/4 uses about the same memory as tq5_0/turbo4. The fixed ring cost shrinks as a share of the cache as context grows.

8.5 Task accuracy

GPQA Diamond and LiveCodeBench v6 questions, a fine-tuned variant of Qwen3.8 27B with 4-bit weights, temperature 0.6, 4 repetitions. "Shallow" questions start from an empty context (100 questions × 4 = 400 answers). "Deep" questions are asked after a 100K-token real prefix (20 × 4 = 80 answers). Channel-axis trellis. Run on RTX 3090 (24 GB) cards; the 8-bit reference answers were reused from an earlier run with identical model, prompts and sampler. Deltas are paired against the 8-bit cache, with 95% paired-bootstrap intervals.

No arm shows a detected loss: every interval includes zero. That is not a proof of equivalence; the deep cohort is small. 3/3t on GPQA deep had a point estimate of −12.5 points (CI −27.5 to 0.0), which deserves a larger follow-up. One 3/3t deep answer failed with a server text-encoding error and is scored as wrong.

9. Results on Gemma 4 31B

Gemma 4 31B (4-bit weights) has 60 layers. Every sixth layer, 10 in all, uses full attention with 4 KV heads of 512 dims. The other 50 use sliding-window attention over 1,024 tokens. SJ-KVaRN compresses only the 10 full-attention layers. The sliding-window cache is 8-bit (q8_0) in every arm, including the reference arms. Each 512-dim head is rotated as two 256-point halves.

9.1 KL divergence

Three prompts (agentic, coding, retrieval) generating up to 25K tokens each, 70,912 generated tokens, one seed. Reference: 16-bit cache. Channel-axis trellis. ± is the token-level standard error, which understates the true error because neighbouring tokens are correlated.

Reading it.

• On Gemma, SJ-KVaRN 4/4 ties the 6-bit tq6_0 cache on mean KL (0.00127 vs 0.00135, overlapping errors) and has a lower p90 (0.00197 vs 0.00251), with a 4.28-bit body instead of 6.125 bits.
• Gemma's 8-bit floor is 3.4× Qwen's (0.00078 vs 0.00023 on the same harness, section 8.3). The absolute SJ-KVaRN cost is in the same range on both models: 4/4 0.00127 vs 0.00082, 3/3t 0.00215 vs 0.00180, 3/2t 0.00366 vs 0.00317.
• Retrieval is the hardest prompt for every SJ-KVaRN arm on both models. Medians on Gemma are at or below the printed resolution (1e-6) for every arm, so we report p90 instead.
• In the first 5K tokens every SJ-KVaRN arm sits close to the 8-bit cache, because nearly everything is still in the exact tail. KL rises from 5K onward on both models.

9.2 Memory and maximum context

Gemma 4 31B with a multi-token-prediction draft model loaded, on a 24 GB RTX 3090 Ti, under a 23 GB whole-card cap. About 1.05 GB of the card was used by other processes during these runs; the process peaks below exclude it.

Under this cap, SJ-KVaRN 4/4 fits about 1.9× the largest tested 6-bit context (216,064 vs 112,640), and 3/3t and 3/2t reach the model's full 262,144-token window.

9.3 What is not yet measured on Gemma

10. Limitations

• The headline is not an equal-bytes comparison. At 9.4K–15K tokens our larger exact tail stores more bits per value than tail-0 stock (5.32 vs 4.65 at 4/4 on the headline build) and than 1,024-tail stock at 3/2 (4.76 vs 4.43); against the 1,024-tail configuration at 4/4 and 3/3, ours allocates slightly fewer (5.32 vs 5.76, 4.82 vs 4.87). The closest equal-bytes readings (section 5.2) are on the same short histories: 3/3t is 20.7% lower than stock 4/4 at 4.98 vs 4.65 bits, and 3/2t vs stock 4/4 is not significant either way. An equal-bytes comparison at 100K has not been run.
• The 3/2 and 3/3t comparisons against stock mix body types. Our side uses the trellis body, which the stock port does not have, so they compare default configurations, not the same body.
• The headline is short-context. Paper-faithful KVarN has not been scored on the 100K trajectories of section 6. Beyond the short histories, our only comparison against stock is the needle study (section 5.3). Pooled over its 8 look-back distances (5K–200K), 4/4 had lower KL than both stock 4/4 configurations and 3/2t lower than stock 3/2 with its 1,024-token tail. 3/2t against tail-0 stock and 3/3t against either stock config were not significant, and 3/3t's point estimates favour stock. With 6 haystacks the intervals are wide, and the pooled ratio does not show how the gap changes with distance.
• The 4/4 and 3/3 headline used an earlier build. The current default build gives 37.5% and 49.7% lower at 4/4 against the 1,024-tail and tail-0 configurations, paired, on the same positions.
• The trellis default changed after most tests. The token-axis trellis is now the default. The Gemma results, the needle study, task accuracy and the 100K comparison against other formats (section 6) used the channel axis. On Qwen at 100K, token axis lowered KL by about 23% against channel axis.
• Higher-bit formats have lower KL than SJ-KVaRN 4/4. In the 100K test, q8_0 / q5_1 has about half its KL at 1.54 times the memory (section 6.1); 8-bit and 6-bit caches are also lower in sections 8 and 9. The section 6 comparison is one run with one reference, and its CIs resample positions, which understates uncertainty because neighbouring positions are correlated.
• The 12 GB context figures are mostly computed. Only SJ-KVaRN 3/2t was run at 204,800 tokens on the 12 GB setup, and that run was measured on a 24 GB card and calibrated to a 12 GB one. The other options' context limits are computed from format sizes, assuming the rest of the memory does not change with the cache format. KL for the 12 GB section comes from the 4-bit-weight model; KL on the 12 GB weights is not measured.
• Speed is not covered. This note measures quality per bit and memory only.
• Gemma coverage is partial. One seed, three prompts, KL and memory only. No stock comparison and no task accuracy.
• References differ between studies. Some use a 16-bit cache reference and some an 8-bit one. Compare numbers within one table, not across tables.
• Tail positions. Early in a context almost everything is in the exact tail, which flatters short-context KL for any long-tail config.
• Task accuracy shows no detected loss, not equivalence. The deep cohort is 80 answers per arm.
• Context editing. Sealed records cannot be partly rewritten. Removing tokens older than the seal boundary requires reprocessing the prompt from that point.

11. Credits and references

• KVarN, the method SJ-KVaRN builds on: Huawei CSL, github.com/huawei-csl/KVarN. The rotation, Sinkhorn variance balancing, 128-token grouping, scale absorption and 128-token sink all come from KVarN. SJ-KVaRN adds the staging precision, the journaled adaptive tail, trellis bodies and the GPU sealing path.
• Comparison runs used the public BeeLlama port of KVarN, v0.4.6.
• Trellis-coded quantization: M. W. Marcellin and T. R. Fischer, "Trellis coded quantization of memoryless and Gauss-Markov sources", IEEE Transactions on Communications, 1990.
• Matrix balancing: R. Sinkhorn and P. Knopp, "Concerning nonnegative matrices and doubly stochastic matrices", Pacific Journal of Mathematics, 1967.