[D] SparseFormer and the future of efficient Al vision models

Hi everyone,

I’ve been diving deep into sparse architectures for vision transformers, and I’m incredibly impressed with the potential of SparseFormer to solve the O(n²) compute bottleneck, especially for commercial applications like data labeling and industrial inspection.

It feels like this is where the industry is heading for efficiency, and it seems to have more commercial potential than it’s currently given credit for, especially with the push towards multimodal models.

Is anyone here working with or researching SparseFormer? Curious to hear thoughts on its commercial viability versus other sparse MoE approaches for vision tasks.