Effective inductive biases are the path to efficient models
Published:
Scale works and it costs more every year.
The priors didn’t disappear when we stopped hand-coding them, they moved into the dataset, the objective, the augmentation stack. Convolution bought translation equivariance for almost nothing. Attention bought flexible routing. Both were cheap in a way that more parameters isn’t. I think that’s where the gains are.