Spontaneous symmetry breaking and Goldstone modes for deep information propagation

Fashion-MNIST test accuracy versus depth for generic, U(1)-equivariant, and O(4)-equivariant networks. On Fashion-MNIST, the O(4)-equivariant network retains about 86% test accuracy from 20 to 100 hidden layers. The network without this symmetry reaches chance near 50 layers.

In physical systems, whenever a continuous symmetry is spontaneously broken, the system possesses excitations called Goldstone modes, which allow coherent information propagation over long distances and times. In this work, we study deep neural networks whose internal layers are equivariant under a continuous symmetry and may therefore support analogous Goldstone-like degrees of freedom. We demonstrate, both analytically and empirically, that these degrees of freedom enable coherent signal propagation across depth and recurrent iterations, providing a mechanism for stable information flow without relying on architectural stabilizers such as residual connections or normalization. In feedforward networks, this results in improved trainability and representational diversity across layers. In recurrent settings, we demonstrate the same mechanism is valuable for long-term memory by propagating information over recurrent iterations, thereby improving performance of RNNs and GRUs on long-sequence modeling tasks.

Nabil Iqbal*, T. Anderson Keller*, Yue Song, Takeru Miyato, and Max Welling. *Equal contribution.

Read the manuscript on arXiv · Under review at ICLR 2027

Code on GitHub