Long-horizon video generation

dKV ForcingSeparating Generation and Memory Representations through a Differentiable KV Cache

Georgia Institute of Technology

Training KV cache writing takes no second pass. It takes a KV cache built for gradients: persistent pages, local rematerialization, and a writer that learns what to store from its future reads.

Trained on 5 s clips, generating 60 s and 240 s.

Abstract

A causal video generator produces two key–value (KV) states for every segment: the KV it uses while denoising that segment, and the KV it writes into the cache for later segments to read. In a pretrained generator the two already differ substantially, later segments attend to them differently, and replacing the written KV with the generation KV degrades the continuation.

We propose dKV Forcing (dKV denotes differentiable KV) to learn a dedicated memory representation from its future use: a writer-specific low-rank residual on the shared backbone separates memory writing from generation and is trained through the future reads of what it writes. Learning a write from its future use needs gradients through the cache, which the mutable cache in common use cannot provide. dKV Forcing replaces the training-time cache by persistent differentiable pages with local rematerialization and, in one temporal traversal, computes exactly the gradient that a two-pass reconstruction of the same objective computes.

One traversal computes the full memory gradient at 7% generator-update overhead over detached-write training, with 28% less resident memory and 20.8% shorter training cycles than the two-pass reconstruction. Over Self Gradient Forcing (SGF), the same objective trained with the same traversal, writer adaptation raises VBench-Long aesthetic quality by 6.6 points at 60 s and 8.5 points at 240 s with the highest subject and background consistency, and rollout dropout of the residual keeps most of this gain while restoring the motion of training without write gradients.

Side-by-side comparisons

Three training recipes, one prompt. Drag the timeline to compare.

Prompt

0:00 / 0:00

Same initialization, data, schedule and training seed for all three. Ticks mark the frames shown in the paper.

How the gradient reaches a write

One training step on a 5 s clip: 21 latent frames forward, then backward. Click any element for details and code.

Measured memory of the same step (rank 0, 4 H100, batch 2), aligned frame by frame. Click to jump.

Forward

Click any element in the animation for its explanation and code.


    

Method

Learn each write from its future reads, in one pass.

dKV Forcing: generation, memory writing and the memory gradient in one temporal traversal.
One temporal traversal. Each segment generates (Gθ) and writes its memory (𝒦θ,φ, writer residual φ). Writes are committed once to the page table; later segments read them through handles, and the gradient of every read returns to the write that produced it.
01 · Pages

Persistent differentiable pages

Each write commits an immutable page outside any replay region. Readers receive the captured prefix through data-free handles, so the graph holds handles while the page data stays outside it.

02 · Rematerialization

One temporal traversal

Every graph-carrying forward is rematerialized locally in groups of blocks. The retained activation footprint is bounded independently of the persistent state, and no second pass over the clip is needed.

03 · Writer residual

A writer-specific representation

A rank-64 residual on the K and V projections is active only when a segment writes its memory. It learns what to remember from the gradients of future reads while the generation backbone stays shared.

Autograd graphs and backward replay under three cache execution strategies.
Three ways to differentiate through the cache. (a) In-place writes are replayed in backward and run out of memory. (b) The two-pass reconstruction recomputes the whole sequence. (c) dKV Forcing commits each write once to an immutable page and replays one block group at a time.

Results

1.3B causal generator, 5 s training clips, three training seeds.

7%overhead for the full memory gradient
−28%resident memory vs. two-pass
−20.8%training time vs. two-pass
+8.5aesthetic quality over SGF at 240 s
Long-horizon qualityVBench-Long, frame-wise. Bold: best, underline: second.
MethodAesthetic ↑Imaging ↑Subject ↑Background ↑Dynamic ↑
60 s
CF++52.7±1.463.3±1.694.7±1.394.5±0.776.5±19.0
SGF56.2±1.467.4±2.296.5±0.495.7±0.538.5±6.9
Writer LoRA62.8±0.469.7±0.698.2±0.296.9±0.221.8±17.5
dKV Forcing (ours)59.8±1.368.8±1.396.6±1.294.8±1.171.4±34.5
240 s
CF++49.0±2.561.7±2.793.8±1.094.4±0.781.7±14.9
SGF51.3±1.764.6±3.595.9±0.395.3±0.438.9±7.4
Writer LoRA59.8±1.467.7±1.397.7±0.396.4±0.320.5±19.2
dKV Forcing (ours)57.4±0.468.2±0.596.2±1.094.8±0.772.0±33.8

Writer LoRA = SGF + writer residual. Full tables in the paper.

Training cost4 H100, batch 2 per GPU. Bold: best, underline: second, among methods with the memory gradient.
MethodCycle (s)vs. two-passGenerator update (s)Generator peak (GiB)Resident (GiB)
Two-pass reconstruction122.8—35.769.940.7
CF++ (no memory gradient)104.7−14.8%31.855.940.7
Mutable cache, write differentiatedout of memory in the first generator backward
dKV Forcing (default)97.3−20.8%33.962.129.2
dKV Forcing (fused)95.9−21.9%32.465.629.2
dKV Forcing (hidden)105.9−13.7%43.156.129.3
dKV Forcing (fused+hidden)100.9−17.8%37.058.929.3

All executions match the two-pass gradient to 4.1×10⁻¹² in fp64.

Fidelity versus motion at 60 s and 240 s.

Fidelity and motion

The writer residual raises fidelity. Rollout dropout of the residual brings the motion back: at 240 s, 57.4 aesthetic and 72 dynamic against 51.3 and 39 for SGF.

BibTeX

@article{li2026dkvforcing,
  title   = {dKV Forcing: Separating Generation and Memory Representations
             through a Differentiable KV Cache},
  author  = {Li, Zhiqi and Zhu, Bo},
  journal = {arXiv preprint},
  year    = {2026}
}