Our latest paper, LLM Compression by Block Removal with Constrained Binary Optimization, takes that correspondence literally. We reformulate block selection as a constrained binary optimization (CBO) problem that maps directly onto an Ising glass, a disordered spin system with all-to-all interactions and a fixed number of "up" spins. The energy of that spin system turns out to be a strong, cheap proxy for how well the pruned model will actually score on benchmarks, which means we can rank a huge number of candidate configurations without benchmarking any of them, and hand the hard instances to the same classical and quantum-inspired solvers we use elsewhere at Multiverse. The payoff in the deep-compression regime is large: at 50% compression of Llama-3.3-70B-Instruct, we gain almost 23 percentage points on MMLU over the best competing block-removal method.

Most existing block-removal methods score each block on its own, then remove the ones that look least important, using magnitude, sensitivity, or "block influence" heuristics. In physics terms these are mean-field methods: they treat each block as if its contribution were independent of the others, the way mean-field theory replaces a spin's neighbors with a single averaged field. A related shortcut is to only ever remove a single consecutive run of blocks, which keeps the problem small but throws away most of the search space.

The trouble is that blocks are not independent, any more than spins in a real magnet are. Whether removing block 20 hurts the model depends on whether you also removed block 19 or block 24, an interaction, or coupling , between the two decisions.

Source link