Flow Equivariant Transformers
Self-attention combines features from different times. For a moving peak, those features arrive at different positions, so changing the velocity changes the output shape.
Many sequences exhibit a geometric ‘flow’ structure that both relates time steps within the sequence via a time-parameterized Lie-group action, and relates sequences to one another via the Lie algebra of the underlying flow. In this work, we illustrate how Transformers do not inherently respect this type of flow structure, causing otherwise related sequences to be processed as unrelated inputs, i.e., the same sequence viewed from different moving reference frames may be seen entirely differently. We then introduce a provably Flow Equivariant Transformer and demonstrate empirically that it achieves the generalization benefits expected of geometric inductive biases.
Ibrahim Khaliliya and T. Anderson Keller
Flow Equivariant Transformers · Accepted at the ICML 2026 Workshop on Weight-Space Symmetries
Flow Equivariant Vision Transformer · Under review at ICLR 2027
Download the code from the ICLR submission’s supplementary material (ZIP)