Apart from scale, what do transformer models get us?

Transformers showed up at the same time as a jump in data and compute, so the two are hard to pull apart.

Hold scale fixed and what’s left? Long-range dependence, flexible tokenization, one interface for every modality. Those are real. I’m not convinced they’re the binding constraint on most of what I work on. For contractility — local, motion-dominated, densely sampled — I’d want the architecture to earn its place against a decent optical flow baseline.