Weights ≠ Function · 2 of a series
What does the loss landscape really look like?
The usual picture is one valley with one bottom. If many different weight settings compute the same function, there are many identical valleys, and the question is whether anything joins them. Smallest case solved exactly: one hidden layer, sigmoid units.
Drag to rotate. Off: you only see the line between the twins: two isolated wells and a hill. On: an extra dimension appears and the wells turn out to sit on one ring of equal-loss points (cyan). Cyan is low loss, orange is high loss.
Technical statement
At width r an irreducible one-hidden-layer sigmoid network has 2rr! permutation and sign-flip copies, which are isolated points of its set of equivalent networks. Proposition 1: adding one unit with output weight 0 joins every copy to the original by a piecewise-linear path (at most 10r−6 segments) along which the function is exactly constant. That path is an existence result; it does not show that the straight-line barrier is zero, which is what the experiments measure.
Evidence label: schematic. The surface is the invented loss L=(x²+y²−1)², with x as the position between the twins and y standing for the spare-unit direction. It illustrates the idea; it is not the measured landscape. Source: Lakhanpal (2026), preprint v2, App. B (Prop. 1) and Sec. 4.2; Zhao et al. (2025) for the isolated-points framing.