Teams from ByteDance, the University of Hong Kong (HKU), and the University of Science and Technology of China (USTC) have released Veda, a sparse attention method for video diffusion transformers; the paper was accepted to ICML 2026 (arXiv:2605.30325). Instead of full attention, whose cost grows quadratically with sequence length, a lightweight predictor determines which key blocks are actually needed for each query, and only the selected portion of attention is computed. On the Waver-T2V-12B model, generating a 720P clip of 241 frames was accelerated from 19.4 to 3.8 minutes — a 5.1x speedup at 95% sparsity. Quality remained at the level of full attention, the code was published under the MIT license, the predictor weights in fp8 were uploaded to Hugging Face, and the claimed numbers are reproducible on a single RTX 4090.

image
image
image

What happened

The Miowtion repository and predictor weights were published on September 23-26, 2026, simultaneously with the acceptance of the Veda: Scalable Video Diffusion via Distilled Sparse Attention paper to ICML 2026. The method is structured as follows: a 275-million-parameter predictor in fp8 format works separately for each layer and each head and determines which 128-token key tiles are actually needed for a query, while a block-sparse kernel based on FlashAttention-4 computes attention only in the selected blocks instead of full attention with O(N²) complexity. On the Waver-T2V-12B model, layer-wise attention was 10.5x faster than FlashAttention-3: 309 milliseconds versus 1576 on a sequence of 245,000 tokens. In a human comparison on the WaverBench1.0 benchmark at 90% sparsity, viewers preferred Veda to full attention in 50% of cases versus 49%.

Context

Quadratic attention was considered the main bottleneck in video diffusion for long clips: as the number of frames and tokens grows, the cost of full attention grows quadratically, so long high-resolution generation required tens of minutes and serious GPUs. Early training-free approaches to sparse attention were criticized for losing the local structure of the scene, because the decision on which tokens to discard was made heuristically. Veda addresses this criticism by changing the problem formulation: what matters is not the proportion of discarded tokens, but the accuracy of reproducing the structure of dense attention. The sparsity mask is distilled here via KL-divergence from max-pooled dense attention before binarization, and the tile shape is chosen separately for each head and each layer — heads and layers receive their own tile shapes. It is precisely this design that explains why quality does not drop at 90-95% sparsity.

Why this matters for the industry

Attention stops being a bottleneck in two ways. By share of computations: in video diffusion transformers, it accounted for 92% of calculations, and with Veda this share drops to 50%. By hardware: the vendor patch fa4_sm8x includes a block-sparse kernel for FlashAttention-4 on consumer SM8x cards, including RTX 30 and RTX 40, although upstream FlashAttention-4 on such cards silently reverts block sparsity to dense mode. A 3-5x speedup on hardware at the level of a single RTX 4090 moves video generation into a new price category: a startup no longer needs a GPU fleet for a product based on the 33-billion MiniMax-H3 to have normal unit economics. Finally, the code under MIT and the open predictor weights make fast inference reproducible from the outside — in such competition, one will have to defend with data, UX, and distribution, not generation speed.

Why this matters for users

Everything described can be assembled on your own hardware today. It is enough to clone the Miowtion repository and download the fp8 predictor from Hugging Face: the file takes up 263 MiB versus 525 MiB for the bf16 version, and the recall differs by only 2e-5. Deployment instructions in AGENTS.md cover RTX 4090 (SM89), Hopper (SM90), and Blackwell (SM120), and the scripts/generate.py script renders dense and Veda versions of the same prompt side by side with timings, so the difference is immediately visible. On a single RTX 4090 with 24 GB of memory, the 33-billion MiniMax-H3 generates a clip with a speedup of up to 2.76x end-to-end, and on clips 14.4 seconds long at 90% sparsity — about 3.1x; the team can verify these numbers in a working day.

What is still unknown / limitations

A preview checkpoint of the predictor is currently being distributed: it was trained for only 600 updates and only on clips 5.17 seconds long — the method generalizes to 10.1 and 14.4-second clips, but was not trained on them. The configuration is designed for 8 denoising steps, and the predictor is tied to its 12 tile plans (4 aspect ratios for 3 durations) and is useless without a paired plan. The result of 50% versus 49% on WaverBench1.0 is a statistical tie: this is the absence of measurable degradation, not the superiority of Veda over dense attention. The transfer of the method to other video diffusion models has not been verified, so it will only be possible to say that fast inference has become a commodity after independent reproductions outside the original teams. Expectations of full checkpoints instead of preview and an expansion of the tile plan grid are interpretations, not confirmed facts.

Sources

Author

Look at AI, editorial team