Simulator 2 of a series · interactive
If two solutions are secretly the same, why is there a hill between them?
Imagine two copies of a trained model that behave identically but sit at different spots in “weight space”. Walk straight from one to the other and the model gets worse in the middle. This page lets you rotate that landscape and find a way around.
TOY MODELThe surface below is a made-up loss, L = (x² + y² − 1)², not the paper’s measured landscape. It illustrates the idea that an extra dimension can open a flat path.
Step 1 · Predict
Two twins sit at x = −1 and x = +1. Which route keeps the loss at zero?
Step 2 · Rotate and walk
Loss landscape (drag to rotate)
Height and colour show loss L from 0 (dark blue) to 2.5 (yellow). Twins: x = ±1, y = 0. Plain-text version: the loss is zero on a ring of radius 1 and 1 at the centre, so a straight line crosses a hill of height 1, while the ring route stays at 0.
Expert notes
PAPERLakhanpal (2026), v2, Sections 2, 4.2 and Appendix B (Lemma 1, Proposition 1).
- What the toy borrows
- At width r, the 2rr! permutation and sign-flip copies of an irreducible network are isolated points of the set of equivalent networks (Zhao et al.). With one spare unit, Proposition 1 joins them by a piecewise-linear path along which the function is exactly constant. Here the ring plays that role.
- What it does not show
- The path’s existence says nothing about the straight-line barrier, which is what the experiments measure. The true loss surface is high-dimensional; this 2-D height field is only a picture.
- Barrier
- B = supα[L(θα) − ((1−α)L(θA)+αL(θB))]. In the toy the straight line gives B = 1 at the midpoint.
Full interactive paper page · Three.js version of this landscape · All simulators · github.com/MrigeshLakhanpal/sign-flips-lmc