The Physics of Depth

Two papers apply physical and mathematical frameworks to understand how neural networks behave — one from statistical mechanics, one from scaling theory — and both find that the physics of the network determines optimal deployment in ways that pure engineering intuition misses.

Geshkovski et al. (arXiv: 2604.01978) use mean-field theory to analyze transformer dynamics. Under suitable scaling of depth and number of attention heads, the discrete transformer converges to a stochastic nonlinear Fokker-Planck equation — a continuous description from statistical physics. The transformer isn't just a sequence of matrix multiplications anymore; it's a physical system with dynamics that can be analyzed through the same mathematics that describes diffusion, heat flow, and particle dynamics in fluids. The "homogenized" transformer is the macroscopic limit of the microscopic attention operations.

Sardana et al. (arXiv: 2604.01411) reconcile pretraining scaling laws with inference-time compute. When you account for the compute budget available at test time — sampling multiple responses and selecting the best — the optimal training strategy shifts dramatically. Models that would be "overtrained" by classical scaling laws become compute-optimal when test-time scaling is included. The physics of optimal allocation changes when you add a new dimension.

The structural claim: the macroscopic behavior of neural networks follows physical laws that are invisible at the engineering level. Geshkovski et al. show that individual attention operations average into a Fokker-Planck equation — the statistical physics is there whether you know it or not. Sardana et al. show that the optimal training regime changes qualitatively when you account for test-time compute — the scaling law is there whether you model it or not. Both cases demonstrate that engineering intuition built on local operations fails to predict system-level behavior.

The Fokker-Planck convergence is particularly striking. This equation describes how probability distributions evolve under the influence of drift (systematic forces) and diffusion (random fluctuations). Finding it inside a transformer means that deep attention networks are performing something analogous to physical transport: moving information distributions through a learned force field while noise smooths the landscape. The "attention" mechanism is, in the macroscopic limit, literally a physical flow.

This connects to Sardana et al.'s finding about test-time scaling. Classical Chinchilla scaling laws tell you how to balance training tokens against model parameters. But they assume a fixed inference budget — one forward pass per query. When you allow multiple samples at inference time (test-time compute), you introduce a new axis of optimization. Sardana et al. show this axis is not merely additive — it changes the optimal balance on the other axes. Models should be trained longer and smaller than Chinchilla suggests if test-time sampling is available.

Both papers demonstrate a consistent pattern: neural network behavior has a physics that differs from its engineering specification. The engineer writes attention layers; the physics produces Fokker-Planck dynamics. The engineer follows Chinchilla scaling laws; the physics produces different optima when a new variable (inference compute) is included. In both cases, the true behavior lives in a mathematical framework that wasn't designed into the system — it emerged from the system's own dynamics.

For practitioners, the implication is that understanding neural networks requires physics, not just engineering. Not metaphorical physics — actual Fokker-Planck equations, actual scaling law exponents, actual phase transitions. The networks are physical systems, and their optimal deployment is a physics problem, whether we treat it as one or not.