Goal Encoding in LLMs

Paper
coming soon
Source
Code
GoalScope
Demo
Talk
Slides

Do LLMs have goals?

Language models now carry out tasks that unfold over thousands of tokens, and they behave as if they were pursuing a goal. We ask whether that behavior is backed by an explicit goal encoding inside the model’s computations. We fix three criteria that functionally define a goal: (i) pre-commitment, (ii) plasticity, and (iii) persistence. We find an encoding that meets these criteria in the outputs of a sparse set of attention heads. We further ask how a depth-bounded transformer manages to keep such a goal alive across a long generation without running out of layers.

What is a goal?

Inspired by Bratman’s planning theory, we treat a goal as a functional state with the three properties below.

Pre-commitment

The target state is settled before the action that realizes it begins.

Plasticity

The same goal can be realized by different means, as the environment demands.

Persistence

It is sustained while the action is carried out, and released once it is done.

In the LM’s computations, we look for these criteria in a controlled setup with multi-step coding tasks: the prompt lists several sub-tasks, and we prefill the code up to the point where a good model should start the next one (the frontier). In this setup, we find an encoding that meets all three criteria in the outputs of a sparse set of attention heads.
When we transplant the collective encoding of these heads into a neutral probe, the LM can verbalize the upcoming action in natural language and implement it in a different programming language. See the figure below, and browse the demo readouts →

GoalScope schema and readout results — **GoalScope.** (a) The goal heads’ outputs at one token form the encoding *v_g*. (b) We plant this encoding into a neutral probe: (c.1) asked to describe what it is doing, the model verbalizes the next action; and (c.2) in an information-less C++ stub, the LM implements the same sub-goal with a different sequence of tokens.
**Right**: faithfulness (c.1) and causality (c.2) at four pre-implementation token positions.

pre-commitment The LM’s next action becomes decodable well before the LM starts implementing it. At the indent token, three tokens before the LM writes the corresponding code, GoalScope can read out the next action in natural language with 94% accuracy.
plasticity When this encoding, cached from a Python context, is transplanted into an information-less C++ code stub, the LM implements the same action in C++, with different libraries, coding syntax, and variables.
persistence The code implementation of a sub-goal can take many tokens. Throughout this implementation, the LM keeps the sub-goal encoding alive in the same attention heads that first formed it.

Temporal persistence of the sub-goal encoding. Cosine similarity between a sub-goal’s goal-head encoding at the = token and the encoding sampled x% of the way through its implementation, averaged over data types and languages. Encodings of the same sub-goal stay well above those of a different sub-goal all the way through, whether the implementation is in the same programming language (solid) or a different one (dashed). The encoding is sustained throughout the implementation, and fades away at the end, when the LM is ready to move on to the next one.

How does a depth-bounded transformer LM achieve persistence?

The only way a transformer LM can pool information from a previous token is through the attention mechanism, which costs a layer. If the LM were to carry a running encoding of the goal from one token to the next, it would have only a limited number of layers to do so. As an example, an encoding formed at layer 40 of a 64-layer model has only 24 layers left to pass it on, yet we see goals sustained well beyond that limit.
We find that instead of carrying a goal-encoding forward, the LM frequently re-reads the goal from the task specification. Then the LM decides the next action by comparing the re-fetched goal with the progress made so far. We call this the non-recurrent goal-seeking account (NRGS) of how the transformer LM achieves persistence.

**Non-recurrent implementation of goal-pursuit in a transformer LM.** A goal such as implementing merge sort decomposes into sub-goals, each realized as a sequence of tokens. (i) An idealized agent forms an intention, acts on it until it is done, and moves on. (ii) A transformer cannot carry that commitment forward through a fixed number of layers, so at each step it re-reads the goal from the task span, computes its progress so far, and compares the two to pick the next token.

The goal is frequently re-read from the task specification. When we swap the key-value states that the goal heads read from at a token, while the LM is in the middle of implementing a specific goal, the LM is redirected to implement the next action of the transplanted goal. §5.1.1 · App. I.1

The same computational motif, implemented by the goal heads, works at different levels of the hierarchy. The same KV swap redirects a sub-sub-goal, a sub-goal, and the broader goal itself. §5.1.1 · App. I.1

KV transfer of a sub-goal from a C++ prompt into a Python prompt, and the swapped rows of one goal head's KV cache — **Sub-goal** (Fig. 5). At the `" ="` pivot the goal heads read the source’s task span (red) instead of the destination’s (green). The model implements the source next action, in the destination’s language. The GoalScope readout also flips with it.

The task span, not the code progress, dictates the LM’s next operation. The next action is pre-committed in the earlier generated code. The goal can be read out via GoalScope, and transplanting it into a different environment (C++) can trigger its execution. However, we find that the LM largely ignores this pre-commitment and instead re-reads the goal from the task span to decide the next operation. App. J · Fig. 27

Transplanting the task span vs. the code-progress span from a source run into a destination run — **Task span vs. code progress.** The two prompts share the first two sub-tasks, whose code is prefilled in the prompt (underlined), and differ only in the third. In the patched run we transplant the residual states of the source’s task span, its code-progress span, or both, and keep the destination’s states for everything else before the `" ="` pivot. Right: how often the model then implements the source’s next sub-task.

The next operation is chosen with a mix of different computations. The goal heads’ queries carry part of the “what is already done” signal. Transferring them can redirect the model to implement the next sub-goal in the destination list that hasn't been completed in the source prompt yet. However, this explains only 57% of the cases in our evaluation set. Through targeted interventions, we find non-trivial evidence for other potential computations, such as the order of the tasks and the number of completed tasks, that also participate in selecting the next action. §5.1.2 · App. I.2

Query transfer: swapping only the goal heads' query at the pivot, and the effect of the source frontier's position — **Query transfer.** Left: the source has done a and b, the destination only a. At the `" ="` pivot we swap in only the goal heads’ *query* from the source run, the side that should carry “what is already done”. The patched model skips ahead to the destination’s third task. Right: the swap works much better when the source frontier sits at the same position in its task list as the patched target (order-matched) than when it does not (order-diff), so progress is not the only cue: position matters too.

Can GoalScope read out the LM’s next action beyond code?

The goal heads were found on code, but GoalScope reads intentions elsewhere too, with no re-training.

GoalScope readouts in few-shot ICL and in an emergently misaligned model — **GoalScope beyond code.** Left: GoalScope can read out the implicit rule in few-shot ICL prompts. Right: the emergently misaligned model’s readouts at the start of its response.

Explore more examples on the ICL and chat (EM) pages of the demo.

Is "goal" the right term for this encoding?

By our criteria, the encoding is pre-committed (decodable before the action) and plastic (portable to another environment). It is also persistent, but through frequent re-derivation rather than a single running state. Whether that counts as a "goal" depends on whether there is a requirement on the mechanical implementation of persistence: if a goal must be a single state that survives the whole action, then this encoding does not qualify. But if we don’t care about the implementation, and only care whether the goal is present during the action, then this encoding qualifies. We leave the semantics of whether this is a “goal” or not to the reader.

How to cite

The paper is under review; a preprint and the final citation will be posted here.

bibliography

Arnab Sen Sharma, Eric Todd, Sheridan Feucht, Andy Arditi, Rohit Gandikota, Natalie Shapira, Gabriele Sarti, Constanza Fierro, Lisa Schut, David Bau. “Goal Encoding in LLMs.” Preprint, 2026.

bibtex

@misc{sensharma2026goals,
  title  = {Goal Encoding in {LLMs}},
  author = {Sen Sharma, Arnab and Todd, Eric and Feucht, Sheridan and Arditi, Andy and
            Gandikota, Rohit and Shapira, Natalie and Sarti, Gabriele and Fierro, Constanza and
            Schut, Lisa and Bau, David},
  year   = {2026},
  note   = {Preprint},
  url    = {https://goals.baulab.info/}
}