Simulator 1 of a series · interactive

If two networks compute exactly the same function, why can’t you simply average them?

Two neural networks can look different inside yet behave identically. Averaging their weights, which is the idea behind merging models, can still produce a broken network. This page lets you see why, with one neuron.

TOY MODELOne sigmoid unit with made-up weights (w=2, b=0.5), run on the paper’s identity. It illustrates the mechanism; it is not an experimental result.

Step 1 · Predict first

Take a neuron and its “mirror twin”. Blend them 50/50. What happens to its output?

Pick one, then play with the simulator below.

Step 2 · Play

Drag along the path from the neuron to its twin

Position α: 0.50 Output weight s: −11.14

Step 3 · See the whole path at once

Where along the path does the output change?

Heat map: horizontal axis is position α along the path, vertical axis is the input x from −4 to 4. Colour is |output − original output|, scaled from 0 to the bound |s|/2 (colour bar below). Follows the path chosen above. Tap or hover for exact values.

Tap the map to read a value.

Step 4 · Rotate the surface

The same heat map as a 3D surface

Axes: α runs left to right, input x runs front to back, height is |output change| from 0 to |s|/2. Drag the surface or use the sliders. Straight blend shows a ridge at α = 0.5; the detour shows a flat sheet.

Rotate (yaw °)Tilt (pitch °)

Expert notes

PAPERLakhanpal (2026), preprint v2, Sections 2 and 4.2, Appendix B.

Sign-flip symmetry: For one-hidden-layer sigmoid networks, σ(−z)=1−σ(z) gives an exact symmetry of the parameters; with permutations, the group has 2^hh! elements at width h.
Completeness: For irreducible networks (Sussmann), permutations and flips are the only function-preserving transformations.
Midpoint erasure: On the straight line to the flipped copy, the unit’s (w, b, s) all vanish at α=½; its contribution to the logit changes by −s(σ(z)−½), up to |s|/2.
Linear mode connectivity barrier: B(θ_A,θ_B) = sup_α[L(θ_α) − ((1−α)L(θ_A)+αL(θ_B))]. Permutation matching cannot undo a flip.
Spare-unit path: Lemma 1: (s−t)σ(z) − tσ(−z) + (c+t) = sσ(z)+c, so the function is constant along a piecewise-linear path (existence for tanh: Farrugia-Roberts). It shows a flat path exists, not that the straight-line barrier is zero.
Scope: One hidden layer, sigmoid, 2-D tasks. ReLU has no sign flip and is open.

(w, b, s, c) ↦ (−w, −b, −s, c+s)

Full explainer: interactive paper page · all simulators · Code and results: github.com/MrigeshLakhanpal/sign-flips-lmc (doi:10.5281/zenodo.23314245)