Weight-Space Programming for Mechanistic Interpretability

Can we ask an AI to perform brain surgery on another AI to understand its behavior?

Two days ago, my roommate brought to my attention this recent discovery by MIT CSAIL on weight-space programming. In essence, it seemed to me like they’ve figured out that frontier models are pretty good at directly editing and interpreting the weights of smaller models. So, at least some of the time, we can have them perform a sort of “brain surgery” on these smaller models.

I was blown away! I jumped out of my seat, read their report, and had Claude get to work on building this for my thesis.

My sense is that if this is possible, then a more capable model could inspect a smaller one, figure out what might be going wrong (like our abstention case, wink wink), and edit the parts responsible. It might even explain why those parts of the model are causing a particular output. I think this is a potentially interesting tool for the field of mechanistic interpretability.

A quick recap of where I got stuck

In my previous article, I wrote about how a memory-management project became an investigation into models that fail despite having sufficient evidence.

One behavior kept showing up: the model would write the supported answer into its own note, then finish with “unknown.”

I tested Qwen2.5-3B-Instruct on short, controlled memory tasks: retrieving a stated fact, connecting two facts, or identifying which fact was most recent. The prompt asked the model to write a memory note before giving its final answer.

On the held-out set, there were 165 responses where the supplied evidence supported an answer, but the model answered “unknown.” In 152 of those, about 92%, its own preceding note already named the correct city.

Note: The 165 responses span 90 underlying base cases and represent the false-abstention subset, not the entire evaluation set.

So this wasn’t a one-off. It still doesn’t prove the model finished the reasoning and then suppressed the answer, but it does show that the mismatch from my last article is common enough to be worth chasing.

I started looking inside the model using vocabulary lenses, which turn internal activations into word scores, and activation patching, which lets us replace an internal state and observe what changes.

These experiments taught me to be careful with both. A lens could miss information at a position where a patch still changed the answer. And transferring an entire activation didn’t tell me exactly which feature caused that change. This replicated a known measurement limitation. It wasn’t a newly discovered mechanism.

Meanwhile, instructions and activation steering could recover correct answers. They could also make the model answer questions whose evidence was insufficient.

That left me with a frustrating problem. I could sometimes make the model answer, but I couldn’t reliably control when it should.

Making steering conditional

Before getting to weight edits, there’s another direction I want to explore: conditional steering.

An activation is a collection of numbers the model computes while processing an input. Activation steering adds a chosen direction to those numbers, nudging the model’s behavior during a run. I’ll call that addition a “push.”

Our earlier steering direction encouraged answering. Applying that push indiscriminately was where the trouble started.

Imagine two memories:

Now ask where Maya lives.

The first memory supports an answer. The second doesn’t. Both contain the same city, so simply detecting “Leeds” somewhere inside the model wouldn’t tell us when answering is justified.

What I’d like is a separate signal that determines whether to apply the push. It would need to distinguish those two cases using the relationship in the evidence.

The general idea already exists. Lee and colleagues’ conditional activation steering method uses activation patterns to decide when to apply a steering intervention. Their demonstrations include selectively refusing particular categories of requests. Applying that approach to evidence sufficiency is a hypothesis I want to test.

The hard part is learning a trustworthy condition. An earlier attempt to gate our intervention wasn’t selective enough. A new detector could also learn shortcuts, like associating a particular template with answerability.

So I’d test it on matched examples where we change who a fact belongs to, swap the answer, or remove the relevant relationship. Names and sentence patterns would need to vary across both answerable and unanswerable cases.

And I’d score the actual generated answers. A gate could recognize sufficient evidence correctly while the model still gives the wrong answer.

What weight-space programming adds

Conditional steering changes activations during a run. Weight-space programming lets an agent propose changes to the parameters that produce those activations.

The MIT CSAIL work gives a frontier model access to weights and tools for inspecting and editing them, without gradient-based training. In their Qwen experiment, the agent identified and edited a direction associated with repetitive reasoning loops. The authors reported improved task performance after the intervention. This is an early research report, with limits as well as successes. Source

The part that excited me was the possibility of turning an edit into a lead for investigation.

Suppose Claude proposes changing a small part of Qwen and says:

That would give me something concrete to test: a component, a proposed function, and predictions.

Of course, Claude could produce a very convincing explanation that happens to be wrong. We’d have another language model’s story about a language model. I’d want more than that before calling it an explanation.

Making the “surgeon” defend its diagnosis

The infrastructure I built is for exploring edits while keeping an independent evaluation separate. The proposed experiment would give Claude practice examples, access to a restricted family of small edits, and a limited evaluation budget. Other examples would remain hidden from it.

I want to evaluate two things separately.

First, does the edit help? Does it recover correct answers, preserve appropriate abstention, and avoid damaging answers that were already correct? It should also be compared with simpler interventions, including directly discouraging “unknown.”

Second, does its explanation hold up? If Claude identifies an activation through which the edit works, we can transfer that activation into the original model and see whether it reproduces the effect. We can also hold it at its original value in the edited model and check whether the effect disappears. Those comparisons need the same preceding tokens so we don’t accidentally change the question being tested.

Even then, we may only have explained how the edit works. It could have introduced a shortcut.

To explain the original failure, the account also needs to predict something about the unedited model on examples the agent hasn’t seen. For instance, how should the named activity change when the relevant evidence becomes insufficient? What result would contradict the explanation?

That’s the part I’m most interested in. A useful edit could help us find where to look. The follow-up experiments would determine what we actually learned.

While I still don’t have a generalizable mechanism for why the model is able to write an answer and then say “unknown,” it seems like there are more things to try. And so long as there are avenues unexplored, you bet your ass I’m gonna keep banging my head against this wall.

Have thoughts on the experiment, or an alternative explanation I should test? I’d love to hear them: qamilmirza@berkeley.edu.