Weights ≠ Function · 1 of a series
Why is merging models so hard?
Two networks can compute exactly the same function and still sit far apart in weight space. Average their weights and you can erase what they know. Smallest case solved exactly: one hidden layer, sigmoid units.
Drag to rotate. Axes: the unit’s input weight w, its output weight s, and the spare unit’s output weight. Every faint line joins a unit to its mirror twin; every straight line passes through the same point, where the unit’s weights are zero. The strip below the view plots how much the function changes along each path (orange: straight, cyan: detour; scale 0 to |s|/2).
Technical statement
For f(x)=Σ sᵢσ(wᵢᵀx+bᵢ)+c, σ(−z)=1−σ(z) gives the exact symmetry (w,b,s,c) ↦ (−w,−b,−s,c+s). On the straight blend to a unit’s flipped copy, (w,b,s) all vanish at the midpoint and its logit contribution changes by −s(σ(z)−½), up to |s|/2. Permutation matching cannot undo a flip. Lemma 1: a spare unit with input (−w,−b) and output weight 0 lets output weight move across with the function exactly constant ((s−t)σ(z) − tσ(−z) + (c+t) = sσ(z)+c). The detour drawn here is that construction, projected onto (wᵢ, sᵢ, s_spare); its input-copy steps do not move the point in this projection.
Evidence label: geometry and numbers are computed exactly from Eq. 3 and Lemma 1 for one toy unit (w=2, b=0.5, s=−11.14); this is not an experiment. A flat path existing does not show a zero straight-line barrier. Scope: one hidden layer, sigmoid; ReLU has no sign flip. Source: Lakhanpal (2026), preprint v2, Sec. 4.2, App. B.