Simulator 4 of a series · interactive

Does just making the network wider make the merging problem go away?

Prior work (Entezari et al.; Ainsworth et al.) reports that permutation matching works well on wide networks but leaves a barrier on thin ones. This page shows what happens to the barrier between two independently trained networks as width grows, on one task (coffee roasting), with and without the sign-flip fix.

PAPERMedians over independent pairs on the coffee-roasting task (Study 1, Appendix A Table 8). Barrier = training-error points above the straight line between the two networks. Axis starts at zero.

Step 1 · Predict

With sign-flip-aware matching, is the barrier removed at every width?

Pick one, then explore the graph.

Step 2 · Explore

Barrier vs hidden width

Bars: 95% bootstrap interval over pairs (11 pairs at width 3, 20 otherwise). Dashed line: ε = 0.5 points, the threshold for “removed”. Width axis is log-spaced. Brute-force (exhaustive) series exist only where the search was run: widths 3–8 (permutations), 3–6 (with flips).

Expert notes

Modes: none: no alignment. wm: weight matching over permutations (Hungarian). wm_sign: matching with sign flips, objective Σ|u_i^A⊤u_π(i)^B|. bf_perm / bf_sign: brute force over permutations / permutations and flips, chosen by midpoint loss.
Reading the result: With flips, the barrier is removed at widths 3 and 4 (upper quartiles 0.04 and 0.11) but not from width 6 to 64 (upper quartiles 0.77–1.30). Flips still lower the barrier at every width.
Caveat from the paper: At widths 4, 6 and 8 nearly every network has two units on almost the same line, so near-symmetries outside the finite group may remain; the exhaustive search does not cover them.
Scope: One task, one hidden layer, sigmoid. Study 1 rules were recorded in a private notebook (Section 3).

Full interactive paper page · All simulators · github.com/MrigeshLakhanpal/sign-flips-lmc