Interactive explainer · Lakhanpal (2026), preprint v2
When permutation matching fails, is the search broken or is a symmetry missing?
Entezari et al. and Ainsworth et al. found that aligning hidden units by permutation does not remove the straight-line loss barrier in thin networks. Ainsworth et al. left open whether matching fails or “some form of invariance other than permutation symmetries” is at play. This paper tests that for one-hidden-layer sigmoid networks, where the complete exact symmetry group (permutations and sign flips, 2hh! elements at width h) is small enough to search exhaustively.
PAPERvalues come from the paper’s tables. TOYa simplified simulation for intuition, not a result. EXPLORATORYdecided after seeing results. Scope: sigmoid, one hidden layer, 2-D convex-polygon tasks. ReLU is not addressed.
Mechanism · Eq. 3, Lemma 1
What does a sign flip do to a straight-line blend?
ILLUSTRATIVE TOY MODEL One sigmoid unit, 1-D input, with toy weights w=2, b=0.5, c=0. Because σ(−z)=1−σ(z), flipping a unit leaves the function unchanged:
Choose a path from the unit to its flipped copy and compare its function with the original. The default s=−11.14 is the output weight of the paper’s seed-1 unit; everything else here is toy.
Blend at α: (w,b,s) scaled by (1−2α) and c+αs, so at α=½ the unit’s weights, bias and output weight are all zero. The change in its contribution is −s(σ(z)−½), at most |s|/2. Lemma 1’s spare path instead moves output weight to a unit with parameters (−w,−b), keeping the function exactly constant. Source: Section 4.2, Appendix B (Figure 4). The paper reports 5.57 and 20.95% training error for its trained network; this toy does not reproduce those.
Evidence · Study 2, Table 4 (2% inclusion)
How big is the barrier with and without sign flips?
PAPER Orange-red: median barrier left by the best permutation alone, in pairs that needed a flip. Teal: upper quartile of the barrier left by the exhaustive search over permutations and flips, over all pairs. Training-error points, axis from 0 to 25, ε = 0.5.
Different statistics (median in flip pairs vs upper quartile in all pairs), as pre-registered in H4a/H4b. Coffee includes Study 1’s 11 pairs, so it is a re-test, not an independent replication. Source: Table 4, Section 5.1.
Sensitivity · Tables 5 and 6
Where does the claim fail?
PAPER Minimal width h* depends on the inclusion tolerance. Pick one.
Failing rows are marked ✗ (upper quartile above ε = 0.5). The same-line split was decided after seeing results. Source: Section 5.1–5.2, Tables 5–6.
Negative result · H7
Do the weights predict how large the barrier is?
PAPER Spearman ρ between M = Σ|sᵢ|/2 over flipped units and the barrier left without flips (129 pairs), with 95% bootstrap intervals.
Circle: pre-registered test, error barrier, ρ = 0.007 [−0.18, 0.19], not supported; the error barrier saturates near each task’s positive rate. Diamonds: EXPLORATORY loss re-tests (raw ρ = 0.86 [0.79, 0.90]; calibrated ρ = 0.65 [0.54, 0.75]). Raw loss grows with weight scale, so part of that correlation may be scale. Source: Section 5.4.
What this supports, and what it does not establish
- Supports: for these sigmoid tasks at minimal width, a missing sign symmetry, not a failed search, accounts for the barrier left by permutations (all pre-registered Study 2 tests passed on five tasks at 2%).
- Supports (exploratory): all 157 same-line pairs at 2% kept at most 0.21 points; the 8 different-line pairs kept 2.38–9.58.
- Does not establish: anything about ReLU (no sign flip; open), depth, real data, or tanh (not tested).
- Fails: the claim fails for the pentagon at 1% and the hexagon at 5%. Minimal width is operational, tied to error tolerance. The hexagon’s h* = 4 at 2% is an upper bound.
- Not shown here: pair-level curves; model files are not released. Barriers are on a 25-point grid, so they are lower bounds.
Claim ledger
| ID | Claim | Class | Source |
|---|---|---|---|
| C1 | Flip identity σ(−z)=1−σ(z) gives Eq. 3 | PAPER_FACT | Sec. 2, Eq. 3 |
| C2 | Blend midpoint shifts the logit by up to |s|/2 | DERIVED_RESULT | Sec. 4.2 |
| C3 | H4a, H4b, H4c, H5 hold on all five tasks (2%) | PAPER_FACT | Sec. 5.1, Table 4 |
| C4 | Same-line pairs close under the exact group | PAPER_FACT (exploratory) | Sec. 5.2, Table 6 |
| C5 | Error-barrier size not predicted by weights | PAPER_FACT | Sec. 5.4 |
| C6 | Simulator above illustrates C2 only | ILLUSTRATIVE_TOY_MODEL | this page |
| C7 | Does the effect occur in ReLU nets? | OPEN_QUESTION | Sec. 9 |
Inspect, don’t just trust: code and results at github.com/MrigeshLakhanpal/sign-flips-lmc (Zenodo doi:10.5281/zenodo.23314245). Pre-registration commits 349aa3a, 4c07b33, b7b5669, a09afcf.