SLA2 cuts video diffusion attention costs 97% while actually improving output quality.
Sparse-Linear Attention (SLA) combines sparse and linear attention to accelerate diffusion models and has shown strong performance in video generation. However, (i) SLA relies on a heuristic split that assigns computations to the sparse or linear branch based on attention-weight magnitude, which can be suboptimal. Additionally, (ii) after formally analyzing the attention error in SLA, we identify a mismatch between SLA and a direct decomposition into sparse and linear attention. We propose SLA2, which introduces (I) a learnable router that dynamically selects whether each attention computation should use sparse or linear attention, (II) a more faithful and direct sparse-linear attention formulation that uses a learnable ratio to combine the sparse and linear attention branches, and (III) a sparse + low-bit attention design, where low-bit attention is introduced via quantization-aware fine-tuning to reduce quantization error. Experiments show that on video diffusion models, SLA2 can achieve 97% attention sparsity and deliver an 18.6x attention speedup while preserving generation quality.
SLA2 reduces video diffusion model attention computation by 97% while improving quality over the original.
The method combines sparse and linear attention mechanisms to handle both high-attention and low-rank components efficiently.
SLA2 fixes mathematical mismatches in previous SLA approach through learned routing instead of heuristic token assignment.
SLA relied on heuristics for token routing that weren't necessarily optimal for different situations.
Linear attention introduces approximation error compared to true softmax attention despite computational efficiency gains.
Jintao Zhang, Kai Jiang +6
Many training-free sparse attention methods are effective for accelerating diffusion models. Recently, several works suggest that making sparse attention trainable can further increase sparsity while ...
Yue Ding, Yiyan Ji +13
Omni-modal Large Language Models (Omni-LLMs) have demonstrated strong capabilities in audio-video understanding tasks. However, their reliance on long multimodal token sequences leads to substantial c...
Yongtong Wu, Shaoyuan Chen +11
The performance of multi-turn, agentic LLM inference is increasingly dominated by KV-Cache storage I/O rather than computation. In prevalent disaggregated architectures, loading the massive KV-Cache f...
Haocheng Xi, Shuo Yang +14
Despite rapid progress in autoregressive video diffusion, an emerging system algorithm bottleneck limits both deployability and generation capability: KV cache memory. In autoregressive video generati...
Zecheng Tang, Quantong Qiu +7
The quadratic complexity of standard attention mechanisms poses a significant scalability bottleneck for large language models (LLMs) in long-context scenarios. While hybrid attention strategies that ...
