A new attention method learns what to ignore, and reads 64x more text
Tencent's Hunyuan team introduced HiLS, a sparse-attention method that learns end-to-end which parts of a long document to focus on, matching full attention while handling context 64 times longer than it was trained on.

