Multiverse maps block-removal LLM pruning to an Ising optimization and reports large MMLU gains
Multiverse's new paper, “LLM Compression by Block Removal with Constrained Binary Optimization,” formulates block selection for depth pruning as a constrained binary optimization problem that maps onto an Ising glass; it computes a Hessian of block couplings once on a small calibration set so candidate prunings can be scored cheaply by energy. The authors report that this search over interacting block combinations yields large gains in the deep-compression regime, e.g., nearly 23 percentage points on MMLU at 50% compression of Llama-3.3-70B-Instruct versus the best competing block-removal method.