Weight-Space Programming: AI Rewrites Neural Networks With a 0.016% Edit

Weight-space programming lets a large language model directly read and edit another neural network’s weights-without running a conventional training loop. In this Neural Intel deep dive, we unpack the MIT CSAIL research behind a 0.016% edit to Qwen3.5-0.8B that raises reported MATH-500 accuracy from 7.9% to 48.1% by reducing repetitive reasoning loops.Can an AI debug a neural network like an engineer debugs source code? “Sparks of Weight-Space Programming” investigates that question through eight experiments: recognizing games from raw weights, identifying transformer copying heads, repairing a failing control policy, writing game-playing networks from scratch, detecting sycophancy with attention tools, and editing a language model’s MLP projections.We explain how GPT-6 Astra inspects weight geometry, constructs neural control paths, and uses bounded forward-pass feedback to test tensor-level changes. The key distinction: removing a bottleneck that hides an existing capability is not the same as adding a capability the model never learned.KEY RESULTS & MECHANISMS• Qwen3.5-0.8B: 131,072 weights edited across MLP output projections in layers 15 and 19.• MATH-500: 7.9% → 48.1%; GSM8K: 13.8% → 67.1%; IFEval: 21.0% → 62.3%, as reported in the original research.• Kinetix control policy: a targeted weight patch changes a failing strategy into a successful reverse-and-launch maneuver.• MinAtar and 9×9 Go: task knowledge is translated into policy-network weights, with different tool and feedback budgets.• Gemma 3 270M: the math-editing attempt does not establish a meaningful improvement, revealing an important limit.For ML engineers, mechanistic interpretability researchers, MLOps teams, and AI-agent builders, the practical question is what a reproducible model-patch pipeline should look like: explicit tensor diffs, controlled tool access, held-out evaluation, regression testing, and rollback.More Neural Intel technical deep dives: https://neuralintel.orgCHAPTERS00:00 A new interface to learned software02:42 The research behind this episode04:33 Neural Signal Check: a patch is a hypothesis06:00 Read a game from 17,253 weights09:21 The paddle leaves a spatial footprint12:42 Read a generator, not a controller17:14 Remove the obvious input and output clues20:04 Induction: A B ... A -> B23:18 Could this become static model auditing?24:53 Fix the car that never takes a run-up26:44 Build a controlled override path29:16 Now start from zero weights30:34 Breakout: the striking one-shot result33:07 The longer-trained comparison is mixed33:39 Scale to a 1.48M-weight Go policy35:26 Forward-pass feedback becomes a debugging budget37:27 Which model agrees with a wrong suggestion?39:25 A narrow tool reveals where attention goes41:27 Can the same interface repair an LLM?43:49 Edit just 0.016% of the model45:24 The repair transfers beyond math46:38 Neural Signal Check: unblock versus inject47:31 Future possibilities need new experiments49:51 A new debugging interface—not the end of training50:58 Where does the evidence stop?PRIMARY RESEARCH & REPRODUCTION MATERIALSRikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab — MIT CSAIL.“Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights” — November 2, 2026.https://rikiyat.github.io/blog/weight-space-programming/Experiment logs, evaluations, networks, and weight-edit reconstruction materials:https://github.com/RikiyaT/lm_understand-logsEdited Qwen3.5-0.8B checkpoint, tensor hashes, and separately credited benchmark re-run:https://huggingface.co/RikiyaT/Qwen3.5-0.8B-weight-space-programmingWhat would you target first with a weight-patching agent: reasoning loops, control-policy failures, or instruction-following regressions? Share your take in the comments.#WeightSpaceProgramming #MechanisticInterpretability #NeuralIntel